Factory Floor · 01
Evals before prompt code.
Every time.
Including the time it hurts.
The class most clients arrive asking for: a retrieval, reasoning or generation pipeline that works in production, not in a demo. Which means the harness that decides whether it works exists before the code that has to pass it.
You hold: the production system, monitoring, alerting and drift detection tied to business impact and architecture decision records.
Why the order matters
A system whose success criteria were invented afterwards cannot be defended.
The usual sequence is: build the pipeline, demo it, then work out how to measure it. By then the measurement is chosen — consciously or not — to reflect what the system already does well. Nobody is lying; the incentive is simply pointing the wrong way, and the result is a system that passes its own exam.
We invert it. The evaluation harness is the first artefact: golden sets drawn from real cases with the right answers agreed by someone who owns the process, adversarial cases designed to break the retrieval or the reasoning, a latency budget, and a cost ceiling per transaction. Then the pipeline gets built against it, and it either passes or it does not.
This is occasionally uncomfortable. Agreeing golden answers surfaces disagreements inside the client organisation about what the correct output actually is — disagreements that were previously being absorbed by humans exercising judgement. Better to find that in week one than to discover it in production, when a model has been resolving it silently and consistently in one direction.
What runs in CI
Evals on every commit, not as a release gate. Regression on a golden set fails the build the same way a broken unit test does — which is the only way a harness stays honest once a delivery date is close.
How it runs
Observable by default, from the first commit.
A team that was not in the room has to be able to operate it. That is the gate, and it shapes every decision before it.
- EngagementProduction Build — 6–12 weeks
- Preceded byA Feasibility Sprint where build/no-build is still open
- MethodEvals-first · observable by default
- GateA team that was not in the room can operate it
- HandoverADRs, runbooks, and an eval maintenance guide
The harness, first
Golden sets from real cases, adversarial cases built to break it, a latency budget, a cost ceiling. Agreed with whoever owns the process, before there is a pipeline to flatter.
Architecture with failure modes named
Not just the happy path. What happens when retrieval returns nothing, when the model is confidently wrong, when the provider is slow, when the cost spikes. Each with a designed response.
Build against the harness, in CI
Evals on every commit. A regression fails the build. Guardrails and the empty, loading and error states most builds leave out are part of the work, not a later pass.
Monitoring tied to business impact
Drift detection and alerting that fire on what the business cares about rather than on model metrics nobody owns. A wrong number should be caught by the system, not by the client.
Handover that survives us leaving
Architecture decision records, runbooks, and a guide to maintaining the evals as the world moves. Plus the rollback, which was written before the first commit.
The two lists that matter
What you hold, and what we will not do.
Both are in the engagement letter before you sign it. The second list is the one worth reading twice — it is where most disappointment in this market actually comes from.
What you hold at the end
- The production system, and its evaluation harness
- Monitoring, alerting and drift detection tied to business impact
- Architecture decision records — what was decided, and why
- Runbooks, and a guide to maintaining the evals over time
- The rollback plan, written before the work started
What we won’t do
- Production systems without evals
- Chatbots without guardrails or monitoring
- Black-box models you cannot audit
- Builds without a rollback plan
- Handoffs to teams who were not in the room
- Fixed price without scope clarity
Before you fund it
A written “don’t build this” is a completed engagement.
If the build/no-build question is still genuinely open, a Feasibility Sprint answers it in two to three weeks with a running proof of concept and a harness you keep either way. We would rather sell you that than a production build that was never going to work.
See the Proof of Concept →Evidence on every claim
Every hand-back carries an evidence state: Verified, Built, Designed or Intended. “Should work” is not a state — see our delivery model.
Tell us what you are trying to build.
Thirty minutes with a founder. We will tell you how we would decompose it, and what the harness would have to contain, before you commit to anything.
Fixed fee · Quoted in writing before we start · NDA available