How to Deploy an LLM in Production: A Gated Checklist

Everyone wants to talk about the model. Which one, how big, what it costs per token.

The model is the least interesting decision in a production LLM deployment. It's also the most reversible one; you can swap it in an afternoon. What you can't swap in an afternoon is trust, and trust is built or destroyed by everything around the model.

So here's how I actually deploy, as a gated checklist. The rule that makes it work: each gate blocks the next. You don't proceed on a promise to circle back. A skipped gate doesn't disappear; it just moves downstream, gets a budget attached, and waits for you in month four.

Gate 0: The one sentence

Before anything: what does this system do, in one plain sentence containing four parts? What it reads. What it does, in a verb a dispatcher would use. What catches a wrong answer. Who acts on the output.

If you can't write the sentence, one of two things is true: you don't understand the system well enough to compress it, or it doesn't do anything specific enough to compress. Both are findings, and both are cheaper here than anywhere else on this list. The sentence isn't the pitch. The sentence is the spec.

Gate 1: Ground truth exists

There must be a known number, a validated document, or an answer set that the system's output can be checked against. A control total. Last quarter's reconciled figures. A hundred historical cases with confirmed outcomes.

No checkable truth means no way to ever know whether the system works, which means every downstream gate is theater. This is where most LLM ideas should die, and I mean that as praise for the gate.

A project that can't be verified isn't early. It's unfinishable.

If the ground truth doesn't exist yet, the honest move is to pause the AI project and go build the ground truth. That work is unglamorous and it is the actual foundation; everything after it is furniture.

Gate 2: The batting average

Run the real task, on real inputs, twenty times. Score every output against the ground truth from Gate 1: survived unchanged, needed correction, or failed. Write down the number.

Now you have a batting average instead of a vibe, and a decision instead of a debate. Set your threshold before you run, based on what a miss costs. A drafting task where a human reviews everything can deploy at a lower average. A task whose output touches money or safety needs a higher one, plus the review anyway.

Below threshold, you don't deploy. You diagnose: better inputs, tighter task definition, a validation layer, a different model. Then you run twenty more. The scorecard, not the demo, decides.

<aside> The pushback I get: "twenty runs isn't statistically rigorous." Correct. It's also twenty runs more than most production LLM deployments were tested with, and it catches the failure patterns that matter at a cost of one afternoon. Rigor can scale up later; the gate's job is to make "we never measured it" impossible. </aside>

Gate 3: The failure demo

Now you demo, and you demo the failure on purpose. In front of the actual stakeholders: feed it the malformed file, show the wrong answer happening, then show the check from Gate 1 catching it before anyone would have seen it.

People do not trust systems that claim to be right. They trust systems that get caught when they're wrong. The happy-path demo sells the model; the failure demo sells the system, and only systems get adopted. If you're not willing to run this demo, that reluctance is information about your gates, and everyone in the room already suspects it.

Gate 4: The 2 AM user

Put the tool in the hands of the person who didn't ask for it, under real conditions, without you in the room. The field tech mid-shift. The analyst at close. The dispatcher on a Saturday.

Their verdict outranks the steering committee's, because they're the ones who'll quietly route around the tool if it fails them, and adoption dies politely, in the hallway, as a shadow spreadsheet. What this gate usually surfaces: too many steps, unclear failure states, words nobody on the floor uses. All fixable, all invisible from the boardroom.

Gate 5: Boring engineering

The model is maybe 20% of a production system. This gate is the other 80:

Timeouts and retries, because the API will have a bad day. A fallback path, because the model will have a worse one. Monitoring that pages a human when output volume, latency, or failure rates drift. A kill switch someone non-technical can pull. And a documented human process for when the AI is simply off, because "the AI is down" cannot mean "the business is down."

None of this is AI work. All of it is the difference between a deployment and a liability with good uptime so far.

Gate 6: The receipt

Go live, but run parallel: for the first 30 days, every output reconciles against the old process. The spreadsheet, the manual check, the veteran's number. Publicly, visibly, on a scoreboard the team can see.

Two things happen. Divergences get caught while both systems still exist, which is the cheapest they will ever be to catch. And trust accumulates the only way it actually does: as a stack of receipts, not a promise. You'll know Gate 6 has passed by an unmistakable sign: the shadow spreadsheet retires itself, because its owner stopped updating it, because the tool kept being right in front of them.

The checklist

▌LLM PRODUCTION GATES — each blocks the next
▌0  One sentence     input, verb, check, audience
▌1  Ground truth     a known number to check against
▌2  Batting average  20 scored runs, threshold set first
▌3  Failure demo     wrong answer shown, catch shown
▌4  2 AM user        survives the person who didn't ask
▌5  Boring 80%       timeouts, fallback, monitor, kill switch
▌6  The receipt      30 days reconciled in parallel
▌
▌Skipped gates don't disappear.
▌They move downstream and acquire a budget.

Notice what's absent from the list: the model choice. It shows up implicitly at Gate 2, where the batting average tells you whether the one you picked is good enough, and it's swappable at every point after. Everyone wants the model.

The gates are the product.

---

Verified in this piece: no production figures used; the gate checklist is the artifact. Previous in this series: One Company, Fifty AIs, Zero Shared Truth. Related: Give It a Batting Average, Demo the Failure, The 2 AM Test.