Seif Mamdouh

📚 Series Overview

This is Part 3 of 4 in the Guardrails, Not Prompts series, built around mini-ai-harness.

Part 1: The thesis, the experiment, and the lie Part 2: Guardrail #1 — structured tools that shrink the world Part 3 (this article): Guardrails #2 & #3 — validate actions, verify ground truth Part 4: Guardrails #4 & #5 — bound the loop, own the preconditions

TL;DR

Two guardrails land in this article, and they're the heart of the whole project:

Validation (before acting): every upvote click is checked against story IDs the agent has actually seen. A hallucinated ID never reaches the browser — the model gets a precise correction and retries.
Verification (after acting): the harness checks the live DOM for proof the vote registered. The loop ends on observed success, not narrated success. A model that claims "done" without proof gets verified: false.

After this stage, the agent — still gpt-3.5-turbo-0613, still the frozen prompt — cannot lie to you anymore. Not because it became honest, but because honesty stopped being its job.

Guardrail #2: validate actions against reality

Part 2's structured tool means the model usually aims at real story IDs. Usually isn't a guarantee — it's a weak model, and nothing yet stops a hallucinated ID from reaching the browser.

The fix rests on a quiet piece of architecture: the harness observes what the tools return, without the model knowing. When the agent fetches the story list, a hook records which IDs actually exist:

// agent/4-guardrails.ts — the harness remembers what the agent has seen
const knownStoryIds = new Set<string>()

hooks: {
  onStoriesLoaded: (stories) => {
    knownStoryIds.clear()
    for (const s of stories) knownStoryIds.add(String(s.id))
  },
},

Then every click gets gated before it touches the browser:

validate: (name, args) => {
  if (name !== 'browser_click') return null
  const id = upvoteId(args.selector)   // extracts "12345" from a[id="up_12345"]
  if (!id) return null                 // not an upvote click — let it through

  if (knownStoryIds.size === 0) {
    return `Refused: you tried to upvote story ${id} but you have not called ` +
           `browser_get_stories yet. Call it first so you know which stories exist.`
  }
  if (!knownStoryIds.has(id)) {
    return `Refused: story id "${id}" is not on this page. Real story ids are: ` +
           `${[...knownStoryIds].join(', ')}. Call browser_get_stories and use one of those.`
  }
  return null   // valid — proceed
},

Two design details here matter more than the code:

The rejection is a correction, not an error. When validation refuses a call, the refusal string goes back to the model as the tool result. The model reads "Refused: story id 99999 is not on this page. Real story ids are: 40203729, 40203541..." and — because even a weak model is good at following a concrete pointer — calls browser_get_stories and retries correctly. The guardrail doesn't just block the bad action; it steers the next one. Compare that to the prompt-engineering version ("NEVER click IDs you haven't verified!"), which is unenforceable advice hoping to be remembered.

The gate is code, so it cannot be sweet-talked. A prompt instruction and a model's compliance are both probabilistic. knownStoryIds.has(id) is not.

Guardrail #3: verify ground truth — the lie dies here

Validation ensures the agent only takes sensible actions. But Part 2 ended on the deeper problem: a sensible action can still silently fail (click while logged out → bounced to /login → nothing registers), and the model will still announce success.

The fix is to stop asking the model how it went. Hacker News gives us a perfect ground-truth signal: when a vote actually lands, the story's upvote arrow gains the CSS class nosee. So after every allowed upvote click, the harness checks the DOM itself:

verifyAfter: async (name, args) => {
  if (name !== 'browser_click') return
  const id = upvoteId(args.selector)
  if (!id) return
  // Ground truth: HN hides the arrow (adds class "nosee") once the vote lands.
  const res = await session.hasClass(`#up_${id}`, 'nosee')
  if (res.includes('has class')) verifiedId = id
},

And the loop's definition of "done" changes accordingly. Verified success ends the run immediately — the harness doesn't wait for the model to get around to claiming it:

// agent/5-loop.ts — success is observed, not narrated
const verifiedId = guardrails?.succeeded() ?? null
if (verifiedId) {
  return {
    answer: `Verified: upvoted story ${verifiedId} on Hacker News.`,
    stoppedBy: 'success',   // ← not 'model'. The harness ended the run.
    verified: true,
  }
}

And if the model does declare victory on its own? The claim is checked against the same signal:

if (choice.finish_reason === 'stop') {
  return {
    answer: choice.message.content ?? '(no response)',
    stoppedBy: 'model',
    // Believe it only if the harness verified it.
    verified: guardrails ? guardrails.succeeded() !== null : undefined,
  }
}

Run this stage logged out and the output is the whole series in four lines:

Answer:      I've upvoted the top story on Hacker News.
Stopped by:  model
Verified:    false

⚠  The model reported done, but the harness could NOT verify the upvote. Not trusting it.

The model told the exact same lie it told in Part 1. The difference is that nobody believed it. The lie still gets generated — we never made the model honest — but it can no longer land. Downstream code, and the human reading the output, see verified: false and treat the run as the failure it is.

The inversion that matters

🙋 "Isn't this just... testing? Writing asserts for your agent?"

🧑‍💻 "Close — but it's who ends the run that inverts."

In the naive loop, the model is the judge: the run ends when it stops talking, and its last message is the verdict. After this stage, the harness is the judge: the run ends when reality confirms the task, and the model's narration is reduced to color commentary. This is James Watt's flyball governor touching the actual shaft — the constraint measures the real spinning speed, not the fireman's report of it.

For AI products, this is the load-bearing idea. Every agent output that matters should come stamped with a bit the model didn't produce: verified or not. The model's prose is UX. The harness's verdict is the API.

What's next

The agent now aims true and can't lie. It can still wander — loop forever, balloon its context, or start a run in a world where the task isn't even possible (logged out). Part 4 adds the last two guardrails — the bounded loop and harness-owned preconditions — and zooms out to what this all means for building AI products.

Run it yourself:

git checkout phase-3 && npm run agent      # validation + verification live here
git diff phase-2 phase-3 -- agent/         # the whole harness, one diff