Seif Mamdouh

📚 Series Overview

This is Part 2 of 4 in the Guardrails, Not Prompts series, built around mini-ai-harness.

Part 1: The thesis, the experiment, and the lie Part 2 (this article): Guardrail #1 — structured tools that shrink the world Part 3: Guardrails #2 & #3 — validate actions, verify ground truth Part 4: Guardrails #4 & #5 — bound the loop, own the preconditions

TL;DR

The naive agent hallucinated CSS selectors because we handed it 4,000 characters of text soup and asked it to guess. Guardrail #1 is one new tool — browser_get_stories — that returns clean JSON with real story IDs. The hallucination stops immediately. Tool design is the prompt engineering that actually works, because a tool result is ground truth the model can't argue with, while a prompt is advice it's free to ignore.

The model is still gpt-3.5-turbo-0613. The prompt is still frozen. And the agent can still lie about success — that's Part 3's problem.

Where Part 1 left off

Our agent read the Hacker News front page as a flattened wall of innerText, invented a selector that didn't exist, clicked into the void, and announced victory. Two separate failures were stacked on top of each other:

It aimed at nothing real (hallucinated selectors)
It declared success on its own word (no verification)

This article kills failure #1. It's worth killing on its own, because the fix teaches the single most useful lesson in agent-building.

The problem is the input, not the intelligence

🙋 "Why does the model make up selectors? Can't it just read the page?"

🧑‍💻 "Read what, exactly? Here's what the model actually saw."

browser_get_text returns something like this:

Hacker News new | past | comments | ask | show | jobs | submit
1. Show HN: I built a keyboard for my cat (catkeys.io)
   312 points by catperson 4 hours ago | 187 comments
2. The case against UUIDs (example.com)
   256 points by dbnerd 6 hours ago | 94 comments
...

There is no story ID in that text. There is no selector. The DOM attribute the agent actually needs — the id on each story row — doesn't survive the flattening to text. So when the model needs a selector to click, it can't retrieve one; it can only generate one. Hallucination here isn't a model defect. It's the only move we left available.

This reframe matters for every AI product: most "hallucinations" in agents are information-starvation problems wearing a model-quality costume. The reflex says "the model is dumb, upgrade it or scold it in the prompt." The diagnosis says "the model literally does not have the data it needs, and it's filling the gap the way language models fill gaps."

Guardrail #1: a structured view of the world

The fix is one new tool. Instead of making the model parse text, the harness parses the DOM itself — with code, which doesn't hallucinate — and hands the model exactly the facts it needs:

// agent/browser.ts — the harness does the parsing, in code, deterministically
async getStories(): Promise<string> {
  const stories = await this.page!.evaluate(() => {
    return Array.from(document.querySelectorAll('.athing')).map((row, i) => {
      const id = row.id
      const title = row.querySelector('.titleline a')?.textContent?.trim() ?? '(no title)'
      const upvoteEl = document.querySelector(`#up_${id}`)
      const alreadyVoted = upvoteEl?.classList.contains('nosee') ?? true
      return { rank: i + 1, id, title, alreadyVoted }
    })
  })
  return JSON.stringify(stories, null, 2)
}

Now the model calls browser_get_stories and receives:

[
  { "rank": 1, "id": "40203729", "title": "Show HN: I built a keyboard for my cat", "alreadyVoted": false },
  { "rank": 2, "id": "40203541", "title": "The case against UUIDs", "alreadyVoted": false }
]

Real IDs. Real ranks. Even the voted-or-not status. The model no longer has to imagine what the page contains — it's told, in a format it's extremely good at reading.

And one more trick hides in the tool's description, which teaches the exact selector recipe at the moment it's relevant:

description:
  'Get a structured list of Hacker News stories on the current page — rank, story ID, ' +
  'title, and whether you have already voted. Use this instead of browser_get_text. ' +
  'Upvote with the selector a[id="up_STORYID"].'

🙋 "Wait — isn't putting instructions in a tool description just prompt engineering with extra steps?"

🧑‍💻 "It's prompt engineering in the one place it actually pays rent."

The difference is where the words live and when they arrive. A system prompt is a pile of upfront advice the model must remember across the whole run. A tool description arrives attached to the capability itself, exactly when the model is choosing what to do — and the tool's JSON result is ground truth, not advice. The model can ignore an instruction; it can't ignore the fact that the only IDs it has ever seen are the real ones.

What changed when I ran it

git diff phase-1 phase-2 -- agent/ is tiny — one tool added to the registry. The frozen prompt would still fit in a tweet. But the behavior change is immediate:

The model calls browser_get_stories first (the description told it to)
It picks rank 1's real ID
It clicks a[id="up_40203729"] — a selector that exists

The dumb dog now runs in the right direction. We shrank the space of possible actions from "any string that looks like CSS" down to "one of thirty real IDs on the page," and the weak model handles the shrunken world just fine.

This is the interlocking-frame principle from Victorian railway signaling: after 1856, a signalman mechanically could not pull two conflicting levers — safety wasn't a rulebook entry, it was the shape of the machine. Guardrail #1 is the first lever of our interlocking: the model acts through tools, and the tools define what's even expressible.

The honest catch

Here's what stage 2 does not fix. Run it logged out and watch closely: the model picks the real top story, clicks the real upvote arrow... and HN silently bounces the click to /login. No vote registers. The model, having aimed perfectly, announces:

"I've upvoted the top story on Hacker News."

Still stoppedBy: model. Still a lie — just a better-aimed one. Structure made the agent accurate; nothing yet makes it honest. The agent still grades its own homework, and it still gives itself an A.

What's next

In Part 3, the harness stops trusting the model entirely: every click gets validated against IDs the agent has actually seen, and success gets verified against the live DOM. That's where the lie dies.

Run it yourself:

git checkout phase-2 && npm run agent
git diff phase-1 phase-2 -- agent/   # the whole guardrail, one diff