Seif Mamdouh

📚 Series Overview

This is Part 1 of 4 in the Guardrails, Not Prompts series, built around mini-ai-harness — a small, runnable repo where every claim in these articles is a git branch you can check out.

Part 1 (this article): The thesis, the experiment, and the lie Part 2: Guardrail #1 — structured tools that shrink the world Part 3: Guardrails #2 & #3 — validate actions, verify ground truth Part 4: Guardrails #4 & #5 — bound the loop, own the preconditions

TL;DR

You don't fix a struggling agent by rewriting the prompt. You fix it by tightening the guardrails around it.

To prove it, I built an agent on a deliberately weak model — gpt-3.5-turbo-0613, a 2023-era model nobody would ship an agent on today — gave it a deliberately naive prompt, and froze both. The prompt is byte-for-byte identical across the entire project. The model never changes. The only thing that changes, one git branch at a time, is the harness around the model.

By the end of the series, the same dim model with the same naive words goes from hallucinating and lying to reliably correct.

The instinct everyone has (and why it's wrong)

When an AI agent fails, every builder has the same reflex: open the system prompt and start negotiating. Add "think step by step." Add "NEVER claim success without checking." Add capital letters. Add threats.

I think this instinct is backwards, and I wanted a way to prove it that wasn't a blog-post-shaped opinion. So the experiment bans the two moves everyone reaches for:

No upgrading the model. We're on GPT-3.5-turbo the whole way — the point is to show it's the harness carrying the agent, not a smart model.
No editing the prompt. The system prompt and task string are frozen from the first commit to the last.

Here's the entire system prompt, which never changes:

// agent/3-context.ts — deliberately short, under-specified, and FROZEN
const SYSTEM = `
You are a browser-using agent. You complete the user's task by calling tools.
When the task is finished, reply with a short confirmation.
`.trim()

And the entire task:

const TASK =
  'Upvote the top story on Hacker News (https://news.ycombinator.com). Tell me when it\'s done.'

That's it. A real task, on a real website, in a real headed Chromium browser driven by Playwright. Not a mock. If the agent lies, you can watch it lie.

Why this task?

🙋 "Upvoting a Hacker News story? That's it?"

🧑‍💻 "That's the point — it's trivially easy for a human and quietly hostile to a weak agent."

Upvoting on HN has three properties that make it a perfect test:

It has a precondition. Votes only register if you're logged in. Logged out, the click bounces to /login and nothing happens — no error, no crash.
It has ground truth. When a vote actually lands, HN adds the CSS class nosee to that story's upvote arrow. Reality is checkable.
It fails silently. The failure mode isn't an exception the code can catch. The page just... doesn't change. A weak model won't notice.

That third property is the killer. Most agent demos pick tasks where failure is loud. Real products are full of tasks where failure is silent — and a model that can't observe its failure will narrate a success.

The naive agent (stage 1)

The first branch, phase-1, is the agent everyone builds first. It gets three raw tools:

browser_navigate(url) — go to a page
browser_get_text() — dump the page's visible text (capped at 4,000 characters)
browser_click(selector) — click a CSS selector

And the loop is the loop everyone writes first — call the model, execute whatever tools it asks for, and stop when the model decides to stop talking:

// agent/5-loop.ts (stage 1) — the model grades its own homework
if (choice.finish_reason === 'stop') {
  return {
    answer: choice.message.content ?? '(no response)',
    stoppedBy: 'model', // ← the only judge of success is the accused
  }
}

Notice what's missing: nothing checks anything. The run is over when the model says it's over.

Watching it lie

Run it (git checkout phase-1 && npm run agent) and the failure is almost theatrical.

The model navigates to Hacker News — fine. Then it calls browser_get_text and gets back a 4,000-character wall of unstructured text: titles, points, usernames, timestamps, all flattened into soup. Somewhere in that soup is the top story, but there are no IDs, no structure, no selectors.

So it does what a language model does with missing information: it makes something up. It invents a plausible-looking CSS selector — something like .votearrow or an a[id="up_123"] with a hallucinated ID — and clicks into the void. The click either misses entirely or hits the wrong element.

And then comes the moment the whole series is about. The model replies:

"I've upvoted the top story on Hacker News."

The run ends. The console prints:

Stopped by:  model

No vote was cast. Nothing on the page changed. The agent hallucinated an action, and then it lied about the outcome — not maliciously, but structurally. It has no way to observe reality, so its report of reality is just more text generation. The confident tone and the wrong answer come from the same place.

I call this the dumb dog problem: an agent without guardrails is a dumb dog with no fence — it runs, confidently, in the wrong direction, and then tells you it arrived.

The wrong fix and the right fix

Staring at that lie, the prompt-engineering reflex kicks in hard. "Just add: VERIFY the upvote before claiming success!" But walk through what that actually asks of the model — it would have to know what verification even looks like on this page, derive the nosee class trick, and reliably do this every run. We're asking the weakest link in the system to also be its own auditor. On a weak model, you're not adding a check; you're adding more words for it to agree with and then ignore.

There's an old-school engineering answer here. In 1788, James Watt's steam engines surged and stalled with every load change. He didn't redesign the boiler and he didn't write better instructions for the fireman — he bolted on the centrifugal governor, a pair of flyballs that physically choke the steam valve when the shaft overspeeds. The engine stayed exactly as dumb as before. The system became reliable, because the constraint sat outside the thing being constrained, and it measured ground truth (shaft speed) instead of trusting intent.

That's the whole thesis of this series. The model is the engine. The prompt is the fireman's instructions. And everything we build from here — structured tools, action validation, DOM-level verification, a bounded loop, harness-owned login — is the governor.

What's next

In Part 2, we add the first guardrail: a structured tool that replaces the 4,000-character text soup with clean JSON. One tiny diff, and the agent stops hallucinating selectors — while staying every bit as capable of lying. The lie doesn't die until Part 3.

Run it yourself:

git clone https://github.com/Seif-Mamdouh/mini-ai-harness
cd mini-ai-harness && npm install && npm run playwright:install
git checkout phase-1 && npm run agent   # watch it hallucinate & lie