How a session runs
Five steps, one of which is the interview. The first four exist to make the fifth specific.
Session lifecycle
Invite
You pick an assignment and we mint a single-use link. Nothing is provisioned and nothing is billed until the candidate opens it. An invite can only be created against an assignment that has built successfully — nobody gets sent a link into an environment that will not boot.Open the brief
The link lands on the assignment README, the rules and the time limit, with a Start button. That page load is what fires provisioning: a machine is created, the repo is cloned, the IDE boots, and a session-scoped gateway key is minted with a spend cap and your allowed model set. The candidate reads for twenty to forty seconds, which is exactly the window the environment needs — cold start hides behind something they wanted to do anyway. The clock starts on Start, not on the page load.Work
The candidate edits, runs tests in a real terminal, and prompts or interrupts the agent. Both halves stream into the event log as they happen, with an on-disk outbox behind them so a dropped connection becomes a reconnect rather than a hole in the recording. The gateway enforces the spend cap independently of our application logic, and the candidate is warned before they reach it.Submit
The candidate submits, or the clock runs out. We snapshot the tree, revoke the session’s model key, seal the event log against further writes so the record cannot be edited after the fact, and destroy the machine.Debrief — where the grading actually happens
You scrub the replay and flag moments; the output is a set of timestamped “explain this” questions, which is the interview. You pick the moments — nothing is chosen for you. Export the whole thing as an immutable report for the archive. Everything before this step exists to make this step specific.
Reading a session
Tool approvals are auto-allowed, because confirmation modals every ninety seconds are not how anyone actually works. That costs us the explicit accept/reject record, so review behaviour has to be inferred instead. Four proxies carry it, and each one doubles as a debrief prompt — which is the point.
| Proxy | Derived from | Reads as |
|---|---|---|
| Post-write read latency | Gap between an agent write and the next editor.open on that path | Under a few seconds and never reopened: they did not look. Opened with real dwell time: they read it. |
| Hand-edit ratio | editor.edit lines ÷ agent-written lines | Zero is a flag on its own. Small and surgical is the strongest signal in the product. |
| Interrupt latency | agent.interrupt minus the start of the turn | Early interrupts beat long corrective prompts. Both beat silence. |
| Verification independence | terminal.run commands the agent did not suggest | Do they check the work, or take the agent’s PASS on trust. |
None of these is a score. There is no composite number, no pass mark and no ranking, and we would not trust one if we computed it. They are a triage layer: they point you at the six minutes of a ninety-minute session actually worth watching, and then at the questions worth asking about them. The judgment stays yours.
The model is selectable. The harness is not.
You choose the allowed model set; the candidate chooses within it, and the choice is recorded. Reaching for something cheap and fast on a mechanical task is itself worth grading — and it gives the debrief an easy opening question when someone burned frontier-model budget on a rename.
The agent loop, though, is the same for everybody. If one company’s candidates ran their own tooling and another’s ran ours, the sessions would not be comparable — and comparability across candidates is the entire product. A trace has to read as evidence about the person, not about which tool they happened to be handed.
Try it on one real loop
Ctillion is early. It is a working system rather than a finished product: one assignment shipped as a built-in, customer-authored assignments built from an uploaded Dockerfile, and no pricing page — because there is no pricing yet.
Accounts are issued by hand, one at a time, so every one comes with a conversation. Tell us what you hire for and what your current take-home fails to tell you.