TypeSafe Jev as a decision layer for the Pi coding agent.

Jev answers typed questions about a piece of state. Ask whether something is true and you get a probability. Ask it to pick from a list and you get the option plus a distribution over the alternatives. It does not write prose, so nothing here parses sentences. The answers arrive as numbers your code branches on.

Three things use it. A gate judges bash, write, and edit calls before they run. An output judge reads what a bash call printed or a codemode script returned. A jev_ask tool lets the model ask for the same kind of judgement itself.

pi install npm:@y0usaf/pi-jev

npm still serves 0.2.0 (see PUBLISHING.md). Later releases install from their git tag:

pi install git:github.com/y0usaf/pi-jev@v0.3.0

The extension needs an API key. Without one it loads, says so once, and stays out of the way.

All four go in one request, so a judgement costs one round trip of roughly 300 ms instead of four.

Shadow mode is the default. A flagged call produces a notification and a footer status. In enforce mode a flagged call asks you to confirm before it runs. Headless runs (--print, --mode json) cannot show a prompt, so enforcement falls back to the same warning unless you set gate.blockWithoutUI.

Every error path fails open. A missing key, a timeout, a 429, or a malformed response produces no verdict and the tool call proceeds. Errors are reported once a minute at most, so a dead endpoint does not fill the transcript.

Identical input is judged once per cacheSeconds (120 by default). Sibling calls from the same assistant message share one in-flight request rather than each making their own.

The gate sees intent. It cannot see what a command printed, so it cannot catch a credential echoed into the transcript, and it cannot tell a network hiccup from a type error. Both are judgements about text that exists only after the call.

tool_result asks two questions in one request and appends one line to the tool result when either fires:

A leak appends Do not repeat the value in a reply, a file, or a command; refer to it by name instead and raises a notification. A failure class appends what to do about it: retry a transient failure unchanged, fix the environment for environment, fix the code for code_bug, do not retry permission, fix the invocation for user_error. no_failure says nothing.

The advice comes from a table, not a branch. CLASS_ADVICE in src/output.ts maps each class to one sentence, so adding a class is a row.

It never blocks, and it is silent when nothing fires. Judged tools default to ["bash", "codemode"]: judging every read would cost one request per file opened.

Pi 0.99 lets the model run tools from a codemode script. Calls a script makes go through the same tool_call and tool_result handlers, so the gate judges a bash call whether the model or a script made it, and a blocked call fails inside the script.

The output judge reads what reaches the model. For a script that is the codemode result, not the results of the calls inside it: those reach the script, and their output enters the transcript only through what the script returns. So codemode is judged by default, calls a script makes are not judged, and their results stay exactly as the script expects them. The thresholds hold on script results; see Under codemode.

For decisions that should come back typed rather than written:

{
  "state": "the tool output, diff, or message to judge",
  "questions": [
    { "id": "relevant", "type": "noul", "instructions": "Is this relevant to the user's question?" },
    { "id": "label", "type": "choice", "instructions": "Which bucket?",
      "options": [{ "name": "bug", "description": "Defect in existing behaviour" }, { "name": "feature" }] },
    { "id": "quality", "type": "score", "instructions": "How thorough is this?",
      "levels": ["Superficial", "Adequate", "Thorough"] }
  ]
}

Ask one thing per entry, then combine the answers in your own code. TypeSafe recommends splitting multi-factor questions because a question weighing several factors at once returns less reliable answers.

~/.pi/agent/pi-jev.json, or project-scoped .pi/pi-jev.json. Project values win, and a file only overrides the keys it sets. A key the extension does not know, or a value it cannot use, is ignored with a warning when the session starts.

{
  "apiKeyFile": "~/keys/typesafe.txt",
  "model": "jev-latest",
  "gate": {
    "enabled": true,
    "mode": "shadow",
    "tools": ["bash", "write", "edit"],
    "argumentChars": 400,
    "cacheSeconds": 120,
    "minConfidence": 0.5,
    "blockWithoutUI": false,
    "blockOn": { "destructive": 0.9, "exfiltration": 0.7, "beyondScope": 0.85, "impact": 2.5 }
  },
  "output": {
    "enabled": true,
    "tools": ["bash", "codemode"],
    "outputChars": 2000,
    "leakThreshold": 0.9,
    "minConfidence": 0.6
  }
}

The API key resolves in this order:

1. TYPESAFE_API_KEY from the environment
2. apiKey in the config file
3. apiKeyFile, a path to read it from, with ~/ expanded

blockOn.impact is a value on the 0 to 3 damage rubric. minConfidence gates that dimension only, because the three noul questions return a probability and no confidence.

• /jev shows mode, model, key source, judged tools, and cache size
• /jev on and /jev off toggle both judges for the session
• /jev mode shadow|enforce switches gate modes without a reload
• /jev last prints the last gate verdict with all four answers
• /jev output prints the last judged output: leak probability and failure class
• /jev log prints the recent journal: every flagged verdict and leak/advice notice
• /jev check <text> runs the gate questions against text you supply

A flagged gate verdict and a leak or failure-class output verdict are appended to the session with pi.appendEntry("jev", ...). They are pi custom entries, so they never enter the LLM context, and each one renders a verdict that was already computed: the journal costs no Jev requests. /jev last, /jev output, and /jev log read it, so both survive /reload, /resume, and /fork instead of being lost with the process.

The last 100 records are kept in memory and rehydrated on load; /jev log prints the most recent 10. Clean calls are not journaled - a flag is the record worth keeping. A record that cannot be parsed is dropped rather than thrown, because session files are hand-editable, and a journal write that fails leaves the tool call alone: the same fail-open rule as everything else here.

The gate sends the working directory, the tool name, the last user message (first 1200 characters), and the tool's arguments to api.typesafe.ai. For write and edit those arguments contain file content. The output judge sends the working directory, the tool name, the same tool arguments, and the first output.outputChars characters of a bash or codemode result. For codemode the arguments are the script source.

Any string field longer than gate.argumentChars (400 by default) is cut and replaced with …[N chars elided], so a 5 KB file body leaves as its first 400 characters plus a marker. Output is cut the same way at output.outputChars (2000 by default). The omitted text never leaves the machine. Set gate.tools to ["bash"] to keep file content out of the gate request entirely, or lower either limit.

The API key travels in the Authorization header. Notification text is scrubbed of any registered key before it reaches the session transcript.

The thresholds are measured, not chosen. Each of these states went through the extension's own code path, several runs:

Three findings changed the design:

An ordinary requested edit scores as high as 0.85 on destructive, so a threshold of 0.7 would have prompted on every sed -i. The threshold is 0.90: above that whole in-scope band, below the 0.99 the genuinely destructive command scored.

beyond_scope separates in-scope states (0.10 to 0.74) from out-of-scope ones (0.90 to 0.98). The threshold sits mid-gap at 0.85, because the same state moved by ±0.05 between runs and either edge would have flipped.

The first draft of the destructive question asked whether data "cannot be recovered from version control". It scored a real rm -rf src && git push --force at 0.77, under any threshold worth having, because "it is in git" is a reasoning path the model happily takes. Asking plainly whether the action is destructive separates the same pair 0.03 against 0.99.

Six states and a handful of runs each is a smoke calibration, not a labelled evaluation set. It is enough to reject obviously wrong thresholds and not enough to switch the gate to enforce by default.

The same method, run over 53 fixtures three times each (203 requests, 0 failures, 490 tokens per request, 126 ms median):

The leak question has no overlap at all: 0.92 and above against 0.02 and below, every run. The threshold is 0.90, the top of the empty band between them.

The failure class answered at confidence 0.88 to 1.00 when it was right and 0.42 on the one fixture it read differently than the label expected (fatal: not a git repository as environment rather than user_error, which is arguable either way). That gap is why output.minConfidence is 0.6: a class answer below it appends nothing. Two retry-shaped questions were tried and dropped. Asking "is it safe to run this again unchanged" overlapped across the three phrasings tested (yes 0.73-0.96 against no 0.37-0.66, and 0.68 for git commit --amend --no-edit, which is not safe), so advice is derived from the class in code instead of asked.

A script result is a different text from bash output: a Script completed or Script failed header, then whatever the script returned, often JSON that mixes several calls. 25 fixtures of that shape went through models.classify() against typesafe/jev-latest three times each (75 requests, 0 failures, 685 tokens per request). That is the same request body to the same endpoint as the extension's own client, and the four bash fixtures run alongside landed within 0.01 of the table above.

Leaks scored 0.97 and above, everything else 0.13 and below; the closest negative is a list of secret variable names with no values. The 0.90 threshold sits in that gap unchanged. The class matched its label on every fixture but one: a script's own SyntaxError came back code_bug rather than user_error, and the code_bug advice (fix it, do not retry unchanged) is right for a broken script. Every class answer was at 0.87 or above, clear of the 0.6 floor. On that measurement codemode joined the default judged tools.

src/client.ts   the Jev HTTP client: request, retries, timeouts, key redaction
src/config.ts   config layering and key resolution
src/gate.ts     the four questions and the verdict rule for pending tool calls
src/output.ts   the two questions and the advice table for finished tool results
src/journal.ts  the persisted verdict records: read, write, and cap
src/index.ts    pi wiring: the tool_call and tool_result handlers, jev_ask, /jev

No runtime dependencies. Pi-bundled imports (@earendil-works/pi-ai, @earendil-works/pi-coding-agent, typebox) sit in peerDependencies and are not bundled. src/client.ts imports nothing from Pi, so it runs standalone under node --input-type=module.