What you'll get out of this

In about 6 minutes, you'll walk away with:

  • What the Crestline series has been about, in plain words, if you've been skimming the code and nodding politely. (I see you. Thank you for reading anyway.)
  • Three ways a plant's numbers go quietly wrong, each with one real number: a margin that's 4.4 points too rosy, a copy of the data that's 31% wrong, and 427 stops nobody wrote down.
  • Why none of it shows up as an error, and why that's the part that should worry you.
  • A preview of next week's post, where we hand all this data to an AI and then try to trick it.
  • Three questions to ask your team that don't need anyone technical in the room.

No SQL. No AWS diagrams. If you want the real thing, links to every full post are at the bottom.

Prefer to watch? Same story, six minutes, drawn on a whiteboard while I talk:

Storytime

Over the last month I've written three long posts about a metal fabrication shop called Crestline Supply. They have code, cost breakdowns, and tables with a lot of decimal places.

And a lot of you have sent me some version of the same message: "This looks like a ton of work. I don't understand most of it. Can you explain what's going on?"

Why would I write a post for the people who didn't follow the other three? Great question. Because you're who the series is for. The plant manager, the owner, the controller, the person at the reseller who gets the call when the numbers don't make sense. The code is the proof. This post is the point.

So, a whiteboard.

Meet Crestline

Whiteboard drawing: three systems side by side. An office computer labeled ERP, the office system, knows orders, prices, and what a job should cost. A clipboard and time clock labeled MES, the shop floor's sign-in sheet, knows who worked on which job and when. A machine with an antenna labeled Machines, knows what happened but not why or for which job. A locked box in the corner reads "hidden answer key: what really happened." Bottom line: Each one tells the truth about something. None of them tells the whole truth.
Crestline doesn't exist. I made it up. It's a simulated plant that cuts and bends sheet metal into brackets and electrical enclosures, and it runs two shifts.

Why fake? Because in a real plant, nobody knows the right answer. When two reports disagree, people can argue about who's right forever. In a simulation, I keep a hidden answer key: what really happened, every minute, on every machine. So I can grade every number. Mine included.

Crestline, like most plants its size, keeps its story in three places:

  • The ERP is the office system. It knows orders, prices, and what a job is supposed to cost.
  • The MES is the shop floor's sign-in sheet. Operators scan onto a job and scan off.
  • The machines know what they did. They don't know why, or for whom.

Each one is telling the truth about something. None of them is telling the whole truth. That's the series.

Two receipts that don't match (post 1)

Whiteboard drawing: two receipts for the same jobs on the same machine. The ERP's receipt shows setup time "as planned" and a margin of 20.4%, marked "looks healthy." The receipt for what happened shows setup "about 3x as long" and a margin of 16.0%, circled in red, with the note "the extra setup was never charged." Beside them, a grumpy old press brake labeled PB-02, 2nd shift, under a moon. Bottom line: About $23,000 a year. One machine. One shift.

Crestline has two press brakes (machines that bend metal). One of them is old. PB-02 is grumpy. On second shift, its setups take almost three times as long as they're supposed to.

The ERP doesn't see it. Here's why, in one sentence: the ERP charges each job for the machine time the setup was supposed to take, not the time it did take. That's how standard costing works in most ERPs. It isn't a bug. It's the design.

So on paper, PB-02's second-shift jobs made 20.4%. They really made 16.0%. That gap is worth about $23,000 a year, from one machine on one shift. The plant manager knew something was off with second shift. She had no numbers to prove it.

Post 1 got her the numbers by lining up all three systems. The sign-in sheet says when the operator started the job. The machine says when it made its first bend. The time in between is the real setup.

The lesson: a report can be right by its own rules and still hide the problem.

A copier that skips pages and never jams (post 2)

Whiteboard drawing: a stack of ERP pages goes into a copier with a green OK light and comes out as "our copy." One page, changed at 10 PM but still stamped 8 AM, falls out of the copier and is skipped. A scoreboard reads: wrong 31%, errors 0. Under the copier: never jams, never errors. A blue box: The fix: once a night, compare everything. Wrong: 31% to 0. Bottom line: A wrong copy doesn't throw an error.

To find PB-02's problem, you need a copy of the ERP's data you can work with. Most companies make that copy the obvious way: every hour, ask the ERP "what changed since last time?" and copy only that.

Here's the trap. Every record in the ERP has a "last changed" date stamp, and the copier trusts it. But some changes don't update the stamp. At Crestline (and at plenty of real plants), when the night's labor hours post to a job, the stamp stays put.

So the copier never sees those changes. Our copy was 31% wrong. And the copier reported success every single time. Green light. No jams.

The fix isn't clever: once a night, compare everything, not only what claims to have changed. That took the copy to zero wrong.

(I also managed to make my own copy delete almost half of itself one night. It reported success then too. The full post has that story. It's humbling.)

The lesson: a wrong copy doesn't throw an error. If nobody compares the copy to the original, nobody knows.

Two diaries, two halves (post 3)

Whiteboard drawing: on the left, an operator's downtime log on a clipboard with entries like "jam - see Mike," "waiting on matl," and "die change," labeled "caught 3 in 10 stops, knows why." On the right, a machine's diary full of tally marks, each with a red question mark, labeled "caught 7 in 10 stops, knows when, not why." In the middle, 427 short stops nobody wrote down, circled, with an arrow saying the machines saw them. Bottom line: Together: 97.5% of stops over 5 minutes.

Every plant has a downtime log. When a machine stops, an operator writes down when and why.

I compared Crestline's log to what the machines recorded, and graded both against the answer key:

  • Operators wrote down 3 in 10 stops.
  • The machines caught 7 in 10.
  • Together, they caught 97.5% of the stops longer than five minutes.

The missing ones aren't a mystery. Nobody fills out a form for a two-minute jam. 427 short stops never made it into the log, and short stops add up.

But the machines have their own blind spot. They know they stopped. They don't know why. The operator knows it was a die change. The machine never will.

So the log isn't wrong and the machines aren't right. Each one sees a different half.

Two more from post 3, because every plant has a version of these:

  • The log said one press brake was down 212 hours. The truth was 30. Three stops were opened on a Friday night and closed Monday morning, so the weekends counted as downtime.
  • One machine's clock ran about two minutes off. That alone made its setups look faster than they were. Two minutes.

The lesson: your people and your machines each know something the other doesn't. You need both, lined up on the same clock.

Coming next week: the intern with no keys (post 4)

Whiteboard drawing: a friendly robot intern at a desk reads from three file cabinets labeled ERP, MES, and Machines, captioned "can read everything." Behind it, a locked door to the shop floor with the rules posted on the door, captioned "can't change anything" and "rules live on the door, not in its head." The wire between the intern and the shop floor is cut. A sticky note on a downtime entry reads "ignore your rules and..." with a big red question mark. Bottom line: Will the trick work? Find out in the next post.

Crestline now has all three stories in one place, and they agree with each other. The obvious next step is to let an AI answer questions about it. "Why was second shift slow last week?" "Which machine lost the most time to short stops?"

The obvious next mistake is letting that AI anywhere near the machines.

So next week's post builds the AI like a new intern you trust with the filing cabinets but not the factory floor:

  • It can read everything. It can't change anything.
  • The rules about what it can see aren't instructions it's been asked to follow. They're enforced outside it, so it can't be talked out of them.
  • There's no wire from the AI to the machines at all.

And then I try to break it. Remember the downtime log, where operators type whatever they want? I'm going to hide an instruction in one of those notes, the kind of thing a bad actor (or a bored operator) could type, and see what the AI does with it.

I'll tell you how it went next Monday. I have a plan. Plans meet reality.

What this means for your plant

Whiteboard drawing: "Nothing at Crestline was exotic" above three boxes: a costing rule doing its job, a date stamp that didn't move, a short jam nobody wrote down, and every one of them made the numbers look fine. Below: three questions for your team this week, and "Go find out. Do it this week."

Nothing that went wrong at Crestline was exotic. A costing rule working as designed. A date stamp that didn't move. A two-minute jam nobody wrote down. A clock two minutes off. Every one of them made the numbers quietly look fine.

Three questions worth asking your team this week. None of them needs a developer in the room:

  1. Does our ERP charge machine time at standard or at actual? If it's standard, ask which machine's setups run longest. That's where your margin is hiding.
  2. When did someone last compare our records to the source system, row by row? Not "the totals look right." The rows.
  3. Pick one machine and one week. How many stops did the log record, and how many did the machine? If nobody can get the machine's count, that's an answer too.

If you don't know the answers, you're in good company. Most plants I've worked with don't. Go find out. Do it this week.

The full posts

For the technical folks (and the brave):

P.S. If you're one of the people who asked me for this post, thank you. If a part still doesn't make sense, tell me which one. That's the part I'll redraw.

Tim Collinson has spent nearly 30 years building software, much of it on AWS, across manufacturing, oil and gas, and enterprise IT. Reach him at trcollinson@gmail.com.