Day 3 — Run the lab | eval-harness-101

Time: 40–60 min

Lab pointer only. The runnable Python lab is offline in your study workspace. This site does not host the lab or execute suites—run it locally where your study materials live.

What it is

A deliberately tiny offline harness:

Fake orders store
Fake “agent” policies (rules, not an LLM — so it’s free/fast/deterministic)
Golden JSON suite + scorer → scorecard markdown

Why no LLM on Day 3? You’re learning harness shape. Swap in a real model later without changing golden ideas.

Commands (local lab)

cd path/to/your/eval-harness-lab
python3 run_suite.py
# optional:
python3 run_suite.py --agent baseline
python3 run_suite.py --agent sloppy   # should score worse

Do

Run baseline — note score
Open goldens.json — add one golden you designed Day 2
Re-run — confirm new row appears
Run --agent sloppy — confirm drop
Skim scorecard.md output

Stretch

Edit agents.py baseline policy: introduce a bug, watch suite catch it.

Log: did: Day 3 lab run baseline=X/Y sloppy=A/B · takeaway = what failed first.