Grade the explanation, not the diff.
The candidate works in a real IDE next to an agent, and every move — theirs and the agent’s — lands on one timeline. Afterwards you scrub that timeline and ask them to explain their own moments.
Rotating assignments is a treadmill
Candidates use agents now. A take-home that grades the finished code measures something that no longer belongs to one person — and the industry’s answer, novel unsearchable assignments rotated every few weeks, is a treadmill you have to keep running forever.
So don’t grade the artefact. Record the session, then ask the candidate to explain their own moments: at 2:41, why did you stop the agent?
Someone who understands the answer can do that wherever it came from. Someone who cannot, cannot.
That dissolves the cheating problem instead of fighting it. Assignments can be public, reusable and boring. And the replay stops being a review surface an interviewer skims: it is the interview script.
A real IDE, not a textarea
A single textarea is fine for a fizzbuzz and useless for a service. If the environment cannot carry a realistic assignment, the whole premise falls over.
- A real editor
- File tree, integrated terminal, language servers, working go-to-definition — on the same filesystem the agent operates on.
- The agent beside the files
- They pick a model from an allowed set and the choice is recorded. The harness is fixed: sessions that ran different harnesses are not comparable.
- Nothing walled off
- HTTP, DNS and package managers are open. Looking things up is fine — that is the point.
- No hidden wall
- Time remaining and spend both sit in the workspace header, so nobody runs out without seeing it coming.
The invite link opens the brief, not a spinner. Reading the README is what hides provisioning, and the clock starts when they press Start.
Both actors, one timeline
One flat event log, ordered by time, sealed on submit.
- 41:02prompt.sentmake the limiter safe under concurrency
- 41:07tool.callread · internal/ratelimit/limiter.go
- 41:19tool.calledit · internal/ratelimit/limiter.go
- 41:24tool.resultok · 34 lines written
- 41:31editor.openlimiter.go · dwell 48s
- 42:26editor.editlimiter.go · +4 −7 by hand
- 42:58terminal.rungo test -race ./... · exit 1
- 43:10agent.interrupt9s into the turn
- The candidate’s half
- Prompts, interrupts, files opened and how long they stayed open, hand edits as diffs, go-to-definition trails, terminal commands and their exit codes.
- The agent’s half
- Replies, tool calls, tool results, tokens and cost.
The part that is ours to get right is attribution. When the agent writes a file, VS Code’s watcher fires and the candidate’s tab updates — so an agent write must never be recorded as editor.edit. Hand edits are the candidate’s, agent writes appear as the tool calls they were, and the hand-edit ratio stays meaningful.
Six questions, not “talk me through your approach”
Scrub the replay, flag moments, and walk into the call with specific, evidence-anchored questions. Four inferred proxies point you at the six minutes of a ninety-minute session worth watching.
- Post-write read latency
- Did they open what the agent wrote, and how long did they stay?
- Hand-edit ratio
- How much of the agent’s output did they change themselves?
- Interrupt latency
- Did they stop a turn, and how far into it?
- Verification independence
- Did they run anything the agent had not proposed?
None of them is a score. They point at the minutes worth watching, and the timeline shows what happened there. The grading happens in the debrief, and everything before it exists to make that conversation specific.
Request access
Ctillion is early. It is a working system rather than a finished product: one assignment shipped as a built-in, customer-authored assignments built from an uploaded Dockerfile, and no pricing page — because there is no pricing yet.
Accounts are issued by hand, one at a time, so every one comes with a conversation. Tell us what you hire for and what your current take-home fails to tell you.
Screenshots are the running application on the built-in assignment