RunEvals is a native studio for giving a small model one bounded job, generating the post-training data and evals to teach it, composing agents into hybrids, and freezing a deterministic gate before anything ships. It runs on your machine. No model calls, no network, no repository writes until you make that boundary explicit.
Press ⌘⇧P for anything. Deterministic gates. Ships as an Agent Plugin.
Most useful agent work is repeated and bounded: triage this, classify that, propose the narrow action. RunEvals turns that into a contract you can inspect: what the agent may touch, what good looks like, and what it may never do on its own.
Every tool is read-only, human-approval, or production-write. Production stays locked until a real quality gate passes and a person approves.
Fail-closed checks catch missing scope, thin data, no held-out set, or an unlocked write, before a single token is spent.
Everything is one versioned .runevals.json. Open it, diff it, hand it off, or publish it.
A package format ships a capability. It never tells you whether the capability is any good. RunEvals is where you make it good: expand a few seed decisions into a diverse, leak-free dataset, freeze the adversarial cases as gates, and prove a small model replaced the frontier one at a known token budget.
Grow seed decisions into normal, edge, recovery, and adversarial cases structured around the task's real failure modes, with clean train / valid / test splits.
Held-out and adversarial cases become frozen gates. Deterministic, reproducible, auditable. An open way to measure capability, not a closed lab.
Model the work as a deterministic flow; spend model tokens only at the nodes that need judgement. The per-run budget is visible and small.
Every agent is a portable contract that carries its own gates. Adopt one, or tick several and compose them into a hybrid: a union of tools, examples, and flow, with production writes still locked. You inherit a re-runnable test, not a claim.
Open plugin standards solve distribution: a manifest, some skills, an MCP server, portable across clients. They deliberately say nothing about quality, specialization, or measurement. RunEvals owns that layer, and exports to the standard so your work stays shippable.
Clone, verify, and package a signed local build. Verification is framework-free, so you do not need a full Xcode install.
Requires macOS 14 or newer on Apple Silicon. The local build is ad-hoc signed, not notarized for distribution.