Teaching Sidekick to say no: automated data curation with LLM judge consensus

Shopify Engineering2026-06-16 03:300 views

Production training data only captures successful queries; it can't teach a model when to say no. We built an automated curation pipeline using LLM judge consensus to close that gap.

Source：https://shopify.engineering/sidekick-curation

My favorites

PreviousClustering billions of products for agentic commerce with Catalog API NextQuick: An internal hosting platform for the AI era

We deleted our old reminder CronJob from Git and Argo CD kept running it for six weeks

In September we moved invoice reminders from an old CronJob in our billing namespace to the new billing service, which s

0dev.to

Port-forwarding to a model on a private GKE GPU cluster: DNS endpoint or Connect gateway

6000 tokens from a vLLM server on a private GKE GPU node reached my Mac in 76.7 seconds through a plain kubectl port-for

0dev.to

You Were Sure You Knew How the Release Pipeline Worked Until You Wrote It Down

The new engineer asked how a release gets to production. You said you would write it up. You knew it well. You had used

0dev.to

Paradox of choice in design systems.

A good design system shouldn't maximize choices. It should minimize unnecessary decisions. In his famous TED Talk and bo

0dev.to

Global coordination requirements for 2038-class rollover events

Technical Paper ITU-T XSTP.epoch consolidates global understanding of risks associated with a cluster of structurally di

0Lobsters

Gentler Streak 6.0 adds a new wellbeing timeline and customizable dashboard

In an update rolling out today, Gentler Streak is introducing new ways for users to track their wellbeing and personaliz

09to5Mac

Teaching Sidekick to say no: automated data curation with LLM judge consensus

Related posts

We deleted our old reminder CronJob from Git and Argo CD kept running it for six weeks

Port-forwarding to a model on a private GKE GPU cluster: DNS endpoint or Connect gateway

You Were Sure You Knew How the Release Pipeline Worked Until You Wrote It Down

Paradox of choice in design systems.

Global coordination requirements for 2038-class rollover events

Gentler Streak 6.0 adds a new wellbeing timeline and customizable dashboard