Guardrails and human-in-the-loop

A control plane is the set of constraints around the model, not the model being polite. Prompt policy: refusals, instruction hierarchy, 'sources are data.' Schema: cannot emit a field you did not allow. Classifiers and filters: PII, jailbreak, topic allow-list. Authz: user identity on retrieve and on tools. Runtime caps: steps, spend, time. HITL: a human on the expensive action. Monitoring: traces and evals after the fact. You want all of them in some form. You want none of them to be the only one.

A prompt-only 'be safe' is not a control plan. You would not accept a payments app whose only control was a comment in the code. You should not accept a copilot whose only control is a system message. Injection, long context, and sycophancy all degrade prompt policy. The prompt is the cheapest layer and the most brittle. Keep it. Do not worship it.

Schema is an underrated control. If the model must emit a JSON object with an enum for action type, it cannot emit a free-text 'go wire the money' as an action. If the UI only offers 'insert draft' and 'discard,' the model cannot send. Constrain the output shape and you constrain the blast radius without a committee. This is the same instinct as not giving a batch job a production write credential it does not need.

Authz belongs on retrieve and on tools, not only on the login to the app. A user who cannot see the HR space in Confluence must not retrieve it through the copilot. A user who cannot open a Jira ticket in that project must not have the open-ticket tool. The model is not a permission boundary. Identity is. If you skip this, you have built a search engine that launders access.

Runtime caps stop the quiet disasters. Max steps on an agent so it cannot loop until the bill is a news item. Max spend per user per day. Timeouts that fail to a fallback instead of spinning. You will write SLOs tomorrow. Today, put the caps on the control-plane diagram so they exist before the first enthusiastic loop.

Monitoring is a layer, not a dashboard hobby. Traces, eval drops, user reports, and a kill switch you have tested. A control you cannot observe is a control you cannot operate. Steal your existing SEV process. Add prompt version, model pin, and trace id to the ticket. That is the whole invention.

You would not accept a payments app whose only control was a comment in the code. Do not accept a copilot whose only control is a system message that says be careful. Say that sentence in design reviews. It lands. Then show the stack: schema, authz, caps, HITL, monitoring. People who wanted a prompt tweak will argue with a stack less easily than with your taste. The stack is the plan you can defend to Risk.

Runtime caps belong on the diagram before the first enthusiastic loop. Max steps, max spend per user per day, timeouts that fail to fallback. Agent-shaped features will eat the bill and the latency SLO in one afternoon if nothing stops the loop. Put the numbers in config, with an alarm, not in a standup comment. A cap nobody can see is not a cap. Day 18 will put spend on the dashboard. Today, make sure there is a cap to put there.

Observe, or shadow: the model drafts, the human does the job as today, you score in the background. Assist: the model drafts, the human sends. Confirm: the model wants to act, the human clicks. Bound-auto: the model acts inside a narrow envelope — small refund, known template — else confirm. Never-auto: legal, safety, large money, customer-facing irreversible send. Those are the notches. Name them. Do not invent twelve.

Start one notch more human than the sponsor wants. Loosen with evals, not with optimism. Publish the dial so nobody thinks v1 is the forever state. A sponsor who hears 'confirm on every send' without a date to revisit will either kill adoption or go around you. A sponsor who hears 'confirm for four weeks, then we look at the bar' has a deal they can live with.

Intensity follows blast radius, not 'AI maturity.' Wrong SOP citation to an internal PM: low, assist is fine. Wrong clinical or financial advice to a customer: never-auto. Wrong ticket filed internally: confirm. Wrong email to a customer about a missed SLA: confirm at least, often never-auto until the eval bar is boring. Audience, money, safety, irreversibility. Tune with those four, not with a vendor slide about autonomy.

You can run different notches on different actions in the same product. That is the point of a matrix. The wiki Q&A can be assist while the 'email the customer' action is confirm, and the 'close the incident' action is never-auto. One global setting is how you either block the useful thing or under-control the dangerous thing. Four actions, four rows. You will write them in practice today.

Bound-auto is earned, not claimed. The envelope has to be numeric and small: refunds under fifty dollars, templates from an allow-list, tickets in one project, no new recipients. Outside the envelope, confirm. If you cannot write the envelope, you do not have bound-auto. You have auto with a story. Auto with a story is the incident you will explain to a steering committee.

Shadow is not wasted time. It is how you collect the traces and the eval that justify the next notch. Two weeks of shadow on a real caseload will tell you more than a month of lab demos. Put shadow on the plan as a named phase with an eval bar to exit, not as 'we'll try it in the background if we have time.'

Publish the dial with a date to reconsider, or the sponsor will hear 'confirm forever' and either kill adoption or go around you. 'Confirm for four weeks, then we look at the bar' is a deal. 'We have to be safe' is a mood. Moods do not survive steering. Dates and numbers do. Write the exit from each notch the same way you write a phase gate: eval evidence, roster health, then the next intensity or a stay.

Different actions in the same product should sit on different notches. Wiki Q&A can be assist while customer email is confirm or never-auto. One global setting either blocks the useful thing or under-controls the dangerous thing. The matrix exists so you can show that thought per action. If someone asks 'are we human in the loop,' the honest answer is 'on which action.' Train the room to ask that.

Staffing HITL is a real roster. A line that says 'the team will review' with no hours in the plan is a lie. You know how to put FTE on a RACI. Do it. Name the humans, the hours per week, the backup, the hours of coverage, and what happens at 17:01 and on Tuesday when two people are out. If the action can fire outside those hours, either the intensity is wrong or you need an on-call. Pick one.

Capacity is math. If you expect eighty users to generate forty confirm-actions a day, and a review takes three minutes, that is two hours of review. Plus interruptions. Plus the people who will batch it at 16:30 and rubber-stamp. Either you staff it, you lower volume (smaller user set, tighter allow-list), or you lower intensity (stay in assist). Hoping the volume is lower than the model is a plan to be surprised.

The human must have a usable job. If the review UI is a wall of text with no source highlight, humans rubber-stamp. Give them the draft, the citations with the supporting span highlighted, the diff versus policy or versus the last sent version, and a one-click reject with a reason code. Track the reason codes. That list is your next eval slice and your next prompt change. A reject with no reason is a wasted click.

Reject reasons are gold. 'Wrong source,' 'tone,' 'out of policy,' 'missing the actual decision,' 'would not send this to a customer.' Tag them. Review them weekly with the same pulse as the golden set. If sixty percent of rejects are 'wrong source,' you have a retrieve problem, not a writing problem, and you should not be arguing about temperature. The roster is an instrument, not a moral comfort.

Do not use HITL as a place to hide missing authz. If the reviewer is expected to catch ACL leaks, you have already leaked by the time they see it. Reviewers catch content and judgment. Systems catch permissions. Mixing those jobs is how you burn the roster on work they cannot actually do, and then blame them in the incident.

Write the roster into the charter and the ops note. Names, hours, UI, reason codes, escalation (who they call if the draft wants to do something off-envelope), and the backup. When a hiring manager asks how you staff HITL, this page is the answer. 'We had humans in the loop' is not the answer.

Capacity is math you should do before you promise confirm. Forty confirm-actions a day at three minutes each is two hours, plus interruptions, plus the 16:30 batch that rubber-stamps. Either you staff it, you shrink the user set, or you stay in assist. Hoping the volume is lower than the model is a plan to be surprised. Put the math in the charter next to the roster names so a sponsor who wants more users can see they are also asking for more hours.

Reject reason codes are how HITL pays for itself. Wrong source, tone, out of policy, missing the decision, would not send to a customer. Tag them. Review weekly. If sixty percent are wrong source, you have a retrieve problem and you should not be arguing about temperature. If they are all blank, you have theatre. The roster is an instrument. Treat the codes like defect codes on a program. You already know what to do with a Pareto of defects.

Theatre is a human who cannot reasonably say no. Volume too high, UI too poor, reasons not tracked, no time in the plan, reviewer who does not know the domain, reviewer who is the intern because nobody else wanted it. The stamp is green. The control is fake. In the incident report this looks worse than no HITL, because you claimed a control you did not have.

A second form of theatre is HITL on the wrong action. Humans confirming every internal FAQ answer, while the customer email path is unbound auto, is a cargo cult. Move the human to where the blast radius is. If you can only afford four hours of review a week, those hours go to the high-risk action, not to the action that makes a steering slide look cautious.

A third form is HITL without a kill switch. The human is slow, the queue piles up, someone 'temporarily' lets the model send to clear the backlog. That temporary is the production path now. If the queue can overflow into auto, you did not have confirm. You had assist with extra latency. Design the overflow: the action waits, or the action is blocked, or you staff up. Overflow into auto is not a design. It is a failure mode.

Sycophancy toward the reviewer is a subtler theatre. The model learns, or appears to learn, the reviewer's taste and writes what will be approved, including painting status green. Track disagreement between model draft and what the human actually sends. If the human always accepts, either the system is excellent or the human is not looking. Spot-check accepts, not only rejects. Maker-checker that never checks is SOD in name only.

You already know this pattern from change-advisory boards that meet to bless what already shipped, and from quality gates with no authority to stop the train. Apply the same cynicism here, on purpose. If you cannot describe how a no happens — who says it, in what UI, with what consequence — you do not have HITL. You have a witness.

Fix theatre before you add more autonomy. Adding bound-auto on top of a rubber-stamp confirm is how you get auto with a story. Repair the roster, the UI, and the reason codes. Re-run eval. Then talk about the next notch. Delivery leads who skip the repair because the sponsor is impatient will own both the theatre and the incident.

Overflow into auto is the failure mode you must design against. The queue piles up, someone 'temporarily' lets the model send, and that temporary is now production. Confirm that can overflow into auto was never confirm. It was assist with extra latency. Design the overflow: the action waits, the action is blocked, or you staff up. Write it on the incident page and on the roster note. If you cannot, you are not ready for that notch.

Spot-check accepts, not only rejects. Maker-checker that never checks is SOD in name only. If the human always accepts, either the system is excellent or the human is not looking. A ten-percent sample of accepts, scored against the rubric, tells you which. Put it on the weekly pulse. You would not let a change board that blesses everything call itself a control. Do not let HITL do that either.

When it fails in public, you need a page you can open at speed. Detect: user report, trace anomaly, eval drop, social, a reviewer who caught it late. Contain: kill switch for the write path, maybe for the whole UI; disable tools; pin back the prompt if the pin is the cause. Communicate: internally first, then the people who saw the output, then any obligated notice. Learn: the row goes into the golden set the same week. This is incident management. Steal your existing SEV process.

The ticket template needs fields your current SEV template does not have. Prompt version. Model pin. Trace id. Retrieved chunk ids. User id. HITL reviewer if any, and whether they accepted. Kill-switch state. These are how you reconstruct. Without them you will have a fluent memory of what happened and no way to prove it. Add the fields before the first SEV, not during it.

Kill switch is named, tested, and reachable by more than one person. 'Restart the service' is not a kill switch if only one engineer knows how. A feature flag that disables generation, a flag that disables tools, and a flag that disables the whole entry point. Test it in a drill. Put the steps on the incident page in language a delivery lead can follow if the engineer is on a plane.

Communication order matters. Internally first so you do not learn the facts from a customer thread. Then the people who received the bad output, with a correction if you can make one, without promising the model is now 'fixed.' Then any legal or privacy notice the class of data requires. You already know not to brief Twitter before you brief the accountable exec. Same rule. Add 'do not ask the model to draft the apology until a human has approved the facts.'

Learning is a golden-set row and a control change, not a lessons-learned slide. The input that caused the fail, the expected behavior, the tag (injection, sycophancy, ACL, stale). A change: prompt, filter, HITL notch, or a removed tool. Re-run the set. If you cannot point to the row and the change, you had a meeting. You did not learn.

Write the page for the Atlas customer-email case even if v1 will not send customer email. The act of writing it will change what you allow in v1. If you cannot name the kill switch and the communicator, you are not ready for that action. That is a useful result. Put the action in non-goals until the page is real.

Communication order is internally first, then the people who saw the output, then any obligated notice. Do not learn the facts from a customer thread. Do not ask the model to draft the apology until a human has approved the facts. You already know not to brief the internet before you brief the accountable exec. Add prompt version and trace id to the comms pack so you are not reconstructing from memory while you write the correction.

Learning is a golden-set row and a control change, not a lessons-learned slide. The input, the expected behavior, the tag, the change (prompt, filter, HITL notch, removed tool), then re-run the set. If you cannot point to the row and the change a week later, you had a meeting. Schedule the learning in the SEV timeline the same way you schedule the customer correction. Same-week is the standard. Next-quarter is folklore.

HITL confirm is maker-checker. The model is the maker. The human is the checker. Separation of duties still applies: the checker should not be the person whose bonus depends on volume of sends. The checker should have the authority to stop the send. If that sentence is uncomfortable, your governance is uncomfortable, and the model did not cause it. The model exposed it.

You already have analogues. Change advisory. Dual control on payments. Four-eyes on production deploys. UAT sign-off. Use those words in the design so Risk and Internal Audit can see themselves in the diagram. 'Human in the loop' sounds like a research paper. 'Maker-checker on customer send, SOD vs the workstream owner, audit log of accept/reject' sounds like a control. Talk like the second.

Audit log of the gate is part of the control. Who saw the draft, what they clicked, when, the reason code, the trace id. Retention on that log follows the class of the action, not the class of the wiki. A customer send log may need to live longer than a wiki Q&A log. Write it down. This is ordinary. Do not let the AI wrapper make it mysterious.

Do not put the model on both sides of SOD. A model that drafts and a second model that 'approves' is not dual control. It is one vendor with extra latency. If you use a model to triage or to pre-check, a human still owns the high-risk action. Day 16 already told you a judge is a junior reviewer. Junior reviewers do not solely approve customer money.

Proportionality is the whole craft. Too little control is an incident. Too much is a PDF with extra steps, and users go back to personal ChatGPT, which has none of your controls. Your job is the same job you have on every program: enough control that a reasonable auditor nods, not so much that the process dies. The matrix is how you show you thought about it per action, not as a mood.

When Audit asks 'how do you know the human looked,' you want the UI timings, the reason codes, and the spot-check of accepts. When they ask 'how do you disable it,' you want the kill-switch drill date. When they ask 'how do you learn,' you want the golden-set changelog. Those three answers are a delivery lead doing the job. A slide that says 'responsible AI' is not.

Do not put a second model on the checker side of SOD and call it dual control. A generator that drafts and a judge that 'approves' is one vendor with extra latency. Junior reviewers, including model judges, do not solely approve customer money, legal send, or safety-affecting acts. If you use a model to pre-check, a human still owns the high-risk action. Write that sentence where Audit can find it. It will be asked.

Proportionality is the whole craft. Too little control is an incident. Too much is a PDF with extra steps, and users go back to personal ChatGPT, which has none of your controls. The matrix is how you show you thought per action, not as a mood. Enough control that a reasonable auditor nods, not so much that the process dies. That is the same judgment you already sell as a delivery lead. Apply it here and you are doing the job.