Grade the explanation, not the diff.

Candidates use agents now. A take-home that grades the finished code measures something that no longer belongs to one person, and the industry’s answer — novel, unsearchable assignments, rotated every few weeks — is a treadmill you have to keep running forever. It costs real money and it never finishes.

So don’t grade the artefact. Record the session, and afterwards ask the candidate to explain their own moments. If they found the fix in a blog post, or the agent handed it to them whole, they still have to say at 2:14 why an atomic beat a mutex here. Someone who understands the answer can do that wherever it came from. Someone who cannot, cannot — and the gap is obvious in seconds.

That dissolves the cheating problem instead of fighting it. Assignments can be public, reusable and boring. And it means the replay is not a review surface an interviewer skims: it is the interview script.

Full VS Code in the browser — file tree, integrated terminal, language servers, real go-to-definition — running against the same filesystem the agent operates on. A single textarea is fine for a fizzbuzz and useless for a service, and if the environment can’t carry a realistic assignment the whole premise falls over.

The agent sits in a panel beside the files. The candidate picks a model from an allowed set, and the choice is recorded; the harness itself is fixed, because sessions that ran different harnesses are not comparable. Looking things up is fine — HTTP, DNS and package managers are open. Time remaining and spend are both visible in the workspace header, so nobody hits a wall without seeing it coming.

The invite link opens the brief, not a spinner: reading the README is what hides provisioning, and the clock starts when they press Start.

One flat event log, both actors, ordered by time. The candidate’s half: prompts, interrupts, files opened and how long they stayed open, hand edits as diffs, go-to-definition trails, terminal commands and their exit codes. The agent’s half: replies, tool calls, tool results, tokens and cost.

The part that is ours to get right is attribution. When the agent writes a file, VS Code’s watcher fires and the candidate’s tab updates — so an agent write must never be recorded as editor.edit. Hand edits are the candidate’s, agent writes appear as the tool calls they were, and the hand-edit ratio stays meaningful. On submit the log is sealed and cannot be edited afterwards.

Scrub the replay, flag moments, and walk into the call with six specific, evidence-anchored questions instead of “so, talk me through your approach”. Four inferred proxies do the triage — post-write read latency, hand-edit ratio, interrupt latency, verification independence — and point you at the six minutes of a ninety-minute session worth watching.

None of them is a score. A proxy that looks bad is not a verdict; it is a question worth asking. The grading happens in the debrief, and everything before it exists to make that conversation specific.

How a session runs, step by step →