ZevoUser Manual

01Overview#

An experiment starts from something you can score. Zevo takes that contract, freezes everything about it that could move a number, and then spends its budget searching for a model that scores better against it. What comes back is one saved checkpoint and the record of how it was reached.

Agent Architecture#

The system combines six configurable LLM Agents with one deterministic Evaluation Runner.

Fixed Evaluation Contract#

Anything that affects measurement is fixed at Baseline and reused. Agents may explore how to improve the model, but they may not move the ruler used to measure it.

That single rule is what the rest of the system is arranged around. It is why the Validation package is frozen after the first Data Ticket, why inference_config.yaml is written once and reused exactly, and why Evaluation is a Bash and Python runner rather than an Agent with an opinion.

02Core concepts#

Five words carry most of the system: Task, Setting, Run, Ticket and Heartbeat. They are worth reading once in order, because every page of the UI and every command in the CLI is named after one of them.

Task, Setting, and Run#

These objects intentionally own different kinds of state:

target is the metric direction: Max means higher is better and Min means lower is better. It is part of the Task because it is coupled to the evaluator, not a cosmetic display preference.

Workflow Control#

M1 through M4 describe how much of the end-to-end workflow Zevo runs. Each mode states whether Zevo runs the full workflow, whether the User or Zevo creates the evaluation, and whether the Agents run without User constraints. This dimension is independent of L1 through L4 Optimization Control: any complete workflow mode can use any canonical optimization level.

Optimization Control#

Optimization Control follows a sequential ownership ladder across training data, base model, and training method.

• L1 · Entry Autonomous. The User fixes the training data, base model, and training method. Zevo iterates within those choices.
• L2 · Constraint Autonomous. Zevo can collect, create, and prepare the training data. The User still fixes the base model and training method.
• L3 · Partially Autonomous. Zevo controls the training data and training method. The User fixes the base model.
• L4 · Fully Autonomous. Zevo chooses the training data, base model, and training method from the objective.

Iteration, Ticket, and Heartbeat#

• Iteration 0 is the Baseline: it has Inference and Evaluation, but no Train stage.
• Iteration 1+ produces and measures trained candidates.
• A Ticket is one durable unit of work owned by a role. It stores its lane, iteration, typed payload, artifact bindings, repair state, summary, and lifecycle status.
• A Heartbeat is one activation of a Ticket. A Ticket can have multiple heartbeats for follow-up or repair without becoming a second logical task.

The two execution lanes are optimization and held_out_test.

Orchestrated pipeline Tickets use typed inputs; Single Stage Tickets use the free-form input format. A failed Ticket does not automatically mean the Run has failed: the engine may repair the same Ticket or return an upstream problem to the Orchestrator. The Run becomes terminal only when its own lifecycle policy says so.

Conversation messages, system notices, typed results, work products, execution events, and resolved runtime configs are stored separately from the Ticket row. This keeps the Ticket lifecycle compact without hiding its transcript, artifacts, or training telemetry.

03Improvement Workflow#

Zevo runs Specialists sequentially. Every visible Specialist completion returns control to the stable Orchestrator Ticket; the Orchestrator is woken only when the preceding stage has settled. Auto adds one Data scoping Ticket before this sequence; after the scoring contract settles, execution is the same as Standard.

Auto only
  -> Data: scope the metric and select or create the held-out Test contract
  -> Engine: validate and privately settle Test + Validation

Orchestrator
  -> Infrastructure
  -> Orchestrator
  -> Data 0: prepare the training data
  -> Orchestrator
  -> Baseline Inference: write inference_config.yaml + generate Validation predictions
  -> Orchestrator
  -> Evaluation Runner: execute Validation evaluator
  -> engine-private held-out Test chain
  -> Orchestrator: write Baseline journal and decide

Iteration N >= 1
  -> Orchestrator: choose one coherent method/data direction
  -> Data N only when the training-data recipe must change; otherwise reuse Data
  -> Orchestrator
  -> Train N: bind the selected parent, write train_config.yaml, train candidate
  -> Orchestrator
  -> Inference N: reuse the Baseline inference_config.yaml exactly
  -> Orchestrator
  -> Evaluation Runner: execute Validation evaluator
  -> engine-private held-out Test chain
  -> Orchestrator: write journal, forecast the next loop, continue or stop

Stop
  -> Registry: save the Validation champion once
  -> Orchestrator: finalize the Run

Data preparation#

Data 0 materializes the training dataset, deterministic preparation script, and data_recipe.json. It receives the objective and training-side guidance, but it does not inspect Validation or Test answers when selecting examples. The recipe contains a fingerprint of the actual source bytes, and the resulting data_signature becomes the exact identity Train must bind. Agents report these artifacts through typed fields; they do not invent their keys or reconstruct paths from names.

Baseline and frozen inference contract#

Data runs before Baseline Inference so Inference can use answer-free schema and profile information without seeing held-out answers. Baseline Inference then inspects the model, tokenizer, backend, and task mapping; selects every unpinned inference detail; writes a complete inference_config.yaml; and generates predictions in the same heartbeat.

The file records the realized configuration, including:

• prompt framing, model reasoning type, system prompt, and chat-template identity;
• tokenizer and special-token behavior;
• input/output mapping and answer parsing;
• generation backend, maximum new tokens, temperature, top_p, top_k, repetition penalty, and seed;
• an example of the fully rendered inference input.

All later candidate inference reuses that same file exactly. Candidate Inference also reuses the Baseline predict.py when it remains compatible, and regenerates it only for a genuine candidate artifact-mode incompatibility. Train reads the YAML and aligns its prompt/template/tokenizer behavior with it. A blank system prompt under chat framing canonicalizes to You are a helpful assistant.. Thinking-capable and non-thinking models use their appropriate template class; Zevo does not invent a third “empty thinking block” mode.

Training iterations#

For each trained iteration, the Orchestrator recommends one coherent experimental direction while Train retains control of detailed compatible hyperparameters. If several parameters change together, Train must explain how they implement the same hypothesis.

Train writes train_config.yaml before execution. It includes the method, selected parent, exact data signature, loss contract, hyperparameters, implementation and software versions, prompt alignment, and an example rendered training row.

If the user pins a training method, every iteration must use it. Otherwise Train may explore different installed, data-compatible methods when justified; it does not rotate methods blindly. Installed Skills currently cover full_sft, lora_sft, dpo, cpo, orpo, kto, gkd, grpo, rloo, rft, and online_dpo. Auxiliary teacher or reward models are currently supplied as Hugging Face model IDs.

An iteration normally continues from the preceding candidate when testing the same hypothesis. Before seeing the result, the Orchestrator may instead choose the Baseline, another successful same-Run checkpoint, or a bounded verified weights-only intermediate checkpoint when that parent better isolates the proposed change. Cross-Run memory and checkpoints are not used implicitly.

Data is reused when its recipe is unchanged. Data runs again only when the next direction changes the subset, filtering, sampling, weighting, transformation, field mapping, method-required row shape, or seed. Train-only hyperparameter changes do not trigger another Data Ticket.

Train streams phases and bounded progress points into execution events so the UI can plot training and trainer-Validation loss without storing every step. It may retain a small number of verified weights-only intermediate checkpoints as optional branch points. Loss curves can inform the next hypothesis, but the Task’s deterministic Validation metric remains the only champion-selection authority.

Validation and held-out Test#

Validation and Test have different authority:

If no separate Validation set is supplied, the engine deterministically moves 20% of Test into Validation and leaves the remaining 80% held out. The derived Validation population must have at least 200 rows, so Test must contain at least 1,000 rows; otherwise the user must supply Validation separately.

Test-derived Validation inherits Test’s metric type, metric, direction, evaluator, answer fields, and sample-submission schema. A separately supplied Validation set may use its own independent built-in or custom metric contract.

A sample submission defines prediction columns, their order, and representative formatting. Its example row count does not have to equal the scoring population. At scoring time, Zevo validates the actual prediction row count and stable row identity against the corresponding questions-only dataset.

The held-out chain runs only after the corresponding Validation evaluation. The trusted UI may display its live state and results to the user, but optimization Agents receive no held-out paths, answers, predictions, summaries, or scores while the Run is active.

The final model is never selected by searching held-out Test scores. Zevo first selects the champion by the Task’s Validation metric and direction, then saves that model and reports its Test score. Improvement is the direction-normalized Test-score difference between the Baseline and evolved champion; proportion metrics are displayed as percentage points in the UI.

Run Journal#

After Baseline and every trained iteration, the Orchestrator records four concise fields: action, result, analysis, and next. The journal describes model performance and the next improvement hypothesis, not Ticket transitions or configuration bookkeeping. When another iteration is planned, next names the concrete changes and their before/after values while keeping the overall direction to one sentence.

Stopping and registration#

The Run may be bounded by:

• trained-iteration count (0 means unlimited);
• total cost in USD (0 means unlimited);
• wall-clock hours (0 means unlimited and is never saved in a Setting);
• an optional Validation metric threshold in the Task’s native scale;
• lack of credible improvement directions.

Before starting an expensive next loop, the Orchestrator refreshes elapsed time and cost and forecasts whether the full Data-if-needed → Train → Inference → Evaluation cycle fits. It should stop and register the current champion instead of knowingly starting a loop that cannot finish; hard budget/time exhaustion remains the emergency backstop.

When repeated experiments on one lever or direction produce only materially insignificant changes, the Orchestrator should switch direction. When credible directions are exhausted, it should stop.

Registry runs once, after stopping, for the Validation champion. It copies the durable checkpoint, writes Ticket-local registry metadata, and assigns M-<first eight characters of run_id> as the model tag.

Configuration ownership#

In Auto and full_pipeline, blank training values mean “the owning Specialist decides”; the Orchestrator does not fill that blank with arbitrary learning rates, epochs, batch sizes, temperatures, or sampling parameters. In customized_pipeline, supported user pins become hard constraints.

04Quick Start#

Docker with the Compose plugin and Git must already be available. Zevo itself needs no host Python environment: the first step creates the complete application as five containers.

Create the five containers#

Clone the repository, create the bind-mounted environment file from its template, then ask Compose to build the Zevo images and create all services:

git clone https://github.com/Zesearch/Zevo.git
cd Zevo
cp .env.example .env
docker compose up -d --build
docker compose ps

The first build may take several minutes. With the Compose project name fixed to zevo, the five default container names are:

In docker compose ps, all five services should be running; after startup probes complete they should report healthy. Database migrations run automatically in the backend. If a service is not healthy, inspect it before continuing:

docker compose logs --tail=200 postgres backend scheduler holdout-scheduler web

Install CLI#

./install.sh
zevo

The shim opens an interactive zevo › shell backed by the running container. Enter help for all commands, <command> --help for one command, and exit to leave.

./install.sh --uninstall

Open and check the Web UI#

The Web UI is the primary and recommended way to use Zevo. Compared with the CLI, it provides a more user-friendly interface for configuring Agents and compute, creating Tasks and Settings, launching Runs, and monitoring progress.

Open http://localhost:5173 in a browser. The Zevo Dashboard and left navigation should render even before Agent or GPU credentials are configured. This confirms that web can reachbackend and that the backend can reach PostgreSQL.

You can also check the two public endpoints from the repository root:

curl -fsS http://localhost:8001/health
curl -I http://localhost:5173

If the browser cannot open the page, first confirm that zevo-web-1 and zevo-backend-1 are healthy, then read their logs with docker compose logs --tail=200 web backend.

Configure Agent drivers#

Configuration has two parts. First, configure the drivers used by the LLM Agents. A complete driver setup has two layers: its credential in Settings, then its driver and model selection on the Agents page. Compute is configured separately in the next section.

1. Open Settings → Agent API. Choose at least one driver that you can authenticate. Each row explains where to obtain its credential and the expected value format.
2. On the required row choose set, paste the value, optionally use the eye button to verify it, then choose save. A malformed value is rejected before the file is changed.
3. The row changes to set with the value redacted. Use replace to rotate it or clear to remove it.
4. Recreate the three Python services so their process environments load the newly saved values, then refresh Settings:

docker compose up -d --force-recreate backend scheduler holdout-scheduler

5. Refresh Settings. A green readiness chip means the driver credential is loaded and the driver can authenticate.
6. Open Agents. Open the configuration for Orchestrator, Infrastructure, Data, Train, Inference, and Registry. For each Agent, choose a ready driver, choose a model supported by that driver, then choose save.
7. Leave Evaluation without a driver or model. Evaluation is a deterministic runner and does not make an LLM call.

Under Settings → Others, HF_TOKEN is optional unless a model or dataset is gated/private. Weights & Biases is optional; when used, set entity, project, and API key together.

Configure compute#

Second, configure where Train and Inference execute. A Run uses one of three compute modes. Cloud uses a GPU provider credential. Instance and Cluster use a verified SSH connection.

For Cloud, open Settings → GPU Providers and set a Vast.ai or Lambda.ai API key. Recreate backend and schedulers with the command used in the Agent driver section, then refresh Settings.Available compute must show the provider as available before it appears in the Launch run form.

For your own machine, open Settings → GPU Providers → SSH connections → add connection, then:

1. Enter a unique Name and choose Instance for a directly reachable GPU host or Cluster for Slurm.
2. Enter Host, SSH Port, User, an absolute remote parent directory, and an Environment command that activates the remote runtime. Zevo creates/appends zevo under the remote parent.
3. Provide exactly one authentication method: a private-key path on the host or an SSH password. The key is read through the read-only host ~/.ssh mount; its contents are never copied.
4. For Cluster, optionally provide a container image. Optionally upload a site-specific SKILL.md containing scheduler, storage, container, and policy instructions but no credentials.
5. Choose save & verify. Zevo authenticates, creates the remote workspace, runs the Environment command, and probes GPU and, for Cluster, Slurm. Only a verified managed connection is available in the Launch run form.

How the UI updates .env#

The initial cp .env.example .env creates the host file. Compose bind-mounts that exact file at /app/.env in the backend. Saving a credential in Settings updates the repository-root.env while preserving unrelated comments and key order. New keys are appended; values containing spaces or # are quoted. Clearing a key leaves a visible commented placeholder such as # OPENAI_API_KEY=.

A UI-managed configuration appears in .env approximately like this:

# Agent credential saved in Settings
OPENAI_API_KEY=sk-...redacted...

# Cloud credential saved in Settings
VASTAI_API_KEY=...redacted...

# SSH connection: My cluster (cluster)
ZEVO_SSH_CONNECTION_A1B2C3D4E5F60708_ID=...
ZEVO_SSH_CONNECTION_A1B2C3D4E5F60708_NAME="My cluster"
ZEVO_SSH_CONNECTION_A1B2C3D4E5F60708_CATEGORY=cluster
ZEVO_SSH_CONNECTION_A1B2C3D4E5F60708_SSH_HOST=login.example.edu
ZEVO_SSH_CONNECTION_A1B2C3D4E5F60708_SSH_PORT=22
ZEVO_SSH_CONNECTION_A1B2C3D4E5F60708_SSH_USER=my-user
ZEVO_SSH_CONNECTION_A1B2C3D4E5F60708_SSH_KEY=/root/.ssh/id_ed25519
ZEVO_SSH_CONNECTION_A1B2C3D4E5F60708_REMOTE_DIR=/scratch/my-user/zevo
ZEVO_SSH_CONNECTION_A1B2C3D4E5F60708_ENV_SETUP="source /opt/miniconda/bin/activate zevo"

05Launching a Run#

Standard and Customized Runs can be started from the UI or interactive CLI. Auto is currently launched from the UI, and Single Stage uses the UI form or agent task.

Prepare and save the Task files#

Save the files that define the evaluation contract before creating the Task. Open Files, choose New file, and create a named file set. The Task form can then select these saved files directly from the Files catalogue.

1. In Files → New file, enter a stable file set name and an optional description of what the files contain.
2. Under Files, select the destination before each upload. Put Task evaluation assets under test. Use train or validation for files that will later belong to a Setting.
3. Choose Upload files, then drag files into the upload area or select them from your computer. Several related files can be stored in the same file set.
4. Confirm that the file list contains every required asset, then choose save. Open the resulting Files card and use its details view to preview the data and verify the names.

A minimal classification contract can look like this:

id,question,answer
q-001,"Which option is correct?",A
q-002,"Which option is correct?",C

id,prediction
q-001,

Here the answer field is answer. Inference receives only id and question, then must produce a CSV with exactly id,prediction in that order and one row per Test example.

Create a Task#

A Task defines the problem and its held-out Test contract. It does not choose training data, a model, a training method, compute, or a budget. After saving the evaluation files, open Tasks, choose New task, and complete these steps:

1. Give the Task a stable name and write an objective that states what the improved model must do and how success is judged.
2. Choose Built-in or Custom, select or name the Test metric, and choose Max when a larger score is better or Min when a smaller score is better.
3. Select the held-out Test set containing both inputs and ground truth. This file is isolated from the optimization loop.
4. Enter every ground-truth column or JSON key in Test answer fields, separated by commas. Zevo removes exactly these fields to make the questions-only copy passed to Inference.
5. Select a CSV sample submission. Its header fixes the exact prediction columns and their order. It may contain only example rows; its row count does not need to equal the Test set.
6. For a custom metric, also select a Python evaluation script. Review the form and choose Create task.

Task field reference#

Create a Setting#

A Setting is an optional, reusable way to attack one Task. Open the Task card, add a Setting, give it a unique name of at most 32 characters, and fill only the choices you want to reuse. One Task can have many Settings for different models, data, methods, or budgets.

Launch from the web UI#

Open http://localhost:5173 and choose New Run. The safest workflow is to select a mode first, complete the visible Required section from top to bottom, open the optional panels you need, and finish by expanding Checklist.

1. Choose Auto, Standard, Customized, or Single Stage.
2. Enter a Run name. This identifies one execution and is separate from the reusable Task name.
3. Select an existing Task or name a new one as the selected mode allows. An existing Task fills its objective and Test contract.
4. In Standard or Customized, select a verified GPU backend and complete the Test contract. Optionally load a Saved Setting.
5. Open Training, Validation Setup, and Others to add pins, guidance, independent Validation, compute limits, or stopping conditions.
6. Expand Checklist. Required still missing must saynone; verify every resolved default under Optional values, then launch.

Quick mode guide: use Single Stage for one Specialist task, Customized for per-Agent constraints, Standard for the normal complete workflow, and Auto when Zevo should create the evaluation contract from a new objective. See Workflow Control for the full M1 through M4 definitions.

Launch field reference#

Training fields and Optimization Control#

The canonical ownership patterns are L1 = data/model/method pinned; L2 = model/method pinned; L3 = model pinned; L4 = none pinned. Any other valid combination is shown as Custom.

Validation fields#

Leave Validation Setup empty for the default deterministic split: 20% of Test becomes Validation with at least 200 rows, while the remaining 80% stays held out. The Validation lane inherits the Test answer fields, sample submission, metric, direction, and evaluator.

Compute, limits, and stopping#

Exact values and guidance queries#

CLI: save a Task#

This is one-time setup and does not launch a Run. Save the held-out Test assets in a named file set, create the reusable Task contract, and inspect it:

file add domain-qa \
  ./test.jsonl \
  ./test_sample_submission.csv

task add domain-qa \
  "Answer the supplied domain questions; maximize token-level F1." \
  --test-set domain-qa/test.jsonl \
  --answer-fields answer \
  --test-sample-submission domain-qa/test_sample_submission.csv \
  --metric-type builtin \
  --metric token_f1 \
  --target Max

task show domain-qa

For a custom scorer, upload its .py file, set --metric-type custom, give --metric its task-specific label, and pass --evaluation-script. Use task edit domain-qa --objective "..." and the matching field flags to update a Task. The Task name itself is immutable.

CLI: launch a saved Task#

Once the Task exists, run create needs only its name, a new Run name, and the Run-specific values you want to set:

task list
task show domain-qa
task setting list domain-qa

run create \
  --task domain-qa \
  --run-name domain-qa-01 \
  --gpu-provider instance \
  --num-gpus 1 \
  --iterations 3 \
  --max-cost 50 \
  --max-runtime-hours 10

Add --setting SETTING_NAME when the Task has a saved Setting you want to reuse. Other flags override that Run’s effective values without editing either saved definition.

CLI: add Run-specific inputs#

To control training or supply an independent Validation contract, upload only those additional files. The Task created above already owns the held-out Test files.

file add domain-qa-training \
  ./train.jsonl \
  ./validation.jsonl \
  ./validation_sample_submission.csv

run create \
  --task domain-qa \
  --run-name domain-qa-l1-01 \
  --dataset domain-qa-training/train.jsonl \
  --validation-set domain-qa-training/validation.jsonl \
  --validation-answer-fields answer \
  --validation-sample-submission domain-qa-training/validation_sample_submission.csv \
  --validation-metric-type builtin \
  --validation-metric token_f1 \
  --validation-target Max \
  --base-model Qwen/Qwen3-0.6B-Base \
  --training-method full_sft \
  --gpu-provider instance \
  --num-gpus 1 \
  --generation-backend vllm \
  --iterations 2 \
  --max-cost 50

The saved Task continues to supply Test data, answer fields, sample submission, metric, and target. These flags affect only this Run.--attach is a shortcut for uploading and using a local training dataset. If Validation is omitted, Zevo derives it from Test and inherits the complete Test contract.

Complete CLI launch option reference#

The source of truth for the installed version remains run create --help. The tables above explain every option currently exposed by that command rather than only listing its name.

Single-Agent task#

agent task data \
  "Convert this PDF into a compact instruction-tuning JSONL dataset" \
  --task notes-to-data \
  --run-name notes-to-data-01 \
  --attach ./notes.pdf

Or attach a new Ticket to an existing Run:

agent task infrastructure \
  "Inspect the current allocation and summarize idle GPU capacity" \
  --run 01234567

06Web UI#

The UI exposes the same durable state as the CLI. Its visible navigation contains seven pages.

Page by page#

Dashboard

Use Launch run to choose Auto, Standard, Customized, or Single Stage. The rest of the page summarizes active Runs, saved models, Run time, cost, and Task-separated model improvement. The command palette opens with ⌘K on macOS or Ctrl+K elsewhere.

Runs

Search and sort launch history or group it by Task. Open a Run to inspect Iterations, then Timeline, Artifacts, and Per-ticket. A newly launched Run opens on Timeline automatically. Iterations show the training dataset and rendered examples, model, method parameters, Validation/Test predictions, scores, and W&B telemetry. Cancel stops active local and exact matching remote work; delete removes a terminal Run only after confirmation.

Agents

Inspect each role’s effective driver and model, assembled instructions, discovered Skills, assigned Tickets, and heartbeat history. Evaluation is shown as a deterministic role and has no configurable LLM model.

Tasks

Create reusable objectives and Test contracts. Test Setup combines metric type, metric name, target direction, built-in or custom evaluator, answer fields, Test data, and sample submission. Open a Task to add, edit, duplicate, clear, or remove reusable Settings. A Setting owns training choices, queries, optional separate Validation, and reusable limits.

Files

Upload local files, register Hugging Face or URL-backed datasets, add a description, inspect schema and row counts, preview tabular data, refresh a profile, or remove individual entries and whole file sets.

Models

Inspect each registered champion’s model path, source Run and iteration, base model, training method, Validation and Test results, model card, and lineage. Zevo reports the durable remote or shared artifact path instead of downloading the model to the host automatically.

Settings

Configure Agent APIs (Anthropic, OpenAI, Bedrock, OpenRouter), cloud GPU providers (Vast.ai, Lambda Cloud), and other integrations (Hugging Face and Weights & Biases). Add, verify, edit, or delete Instance and Cluster SSH connections in the GPU Providers section. The status indicators show which drivers and compute backends are ready for a Run. After credential changes, recreate the backend and schedulers using the command shown at the top of the page.

The Timeline#

The Timeline uses role names such as Infrastructure, Data, Train, Inference, Registry, and Orchestrator as its primary labels. The suffix number is the number of times that Ticket has heartbeated so far. A waiting Slurm stage shows both what it is waiting for and its queue duration; that duration stops contributing to active Run time. Evaluation remains visible in the loop, with a note that Bash runs it deterministically without an Agent call.

07CLI Map#

The CLI mirrors the UI and exposes lower-level diagnostics. Start it with zevo, then enter every command below at the zevo › prompt without repeating the zevo prefix.

The command areas#

Driver and model, per Agent#

Change an Agent’s driver or model with:

agent set train --driver claude_cli --model MODEL_ID
agent set all --driver claude_cli --model MODEL_ID
agent status

Replace MODEL_ID with the model identifier accepted by that driver. all applies only to the six configurable LLM Agents. Evaluation is excluded. A change takes effect on each Agent’s next heartbeat; an already active heartbeat keeps its resolved model. Changing configuration during a Run can therefore produce a mixed-model Run, so changes are normally made between Runs.

Common CLI recipes#

Prepare and inspect files

file add my-data ./train.jsonl ./test.jsonl ./sample_submission.csv
file add-remote my-data organization/dataset --split train
file list
file show my-data --rows 10
file profile my-data --refresh

Inspect reusable experiment definitions

task list
task show <task>
task setting list <task>
task setting show <task> <setting>

Use task add --help for Test-contract fields and task setting add --help for the JSON shape used to add a reusable Setting.

Monitor and intervene

run list --all
run show <run-id>
run watch <run-id>
ticket list --run <run-id>
ticket show <ticket-id>
ticket message <ticket-id> "<comment>"
ticket retry-status <ticket-id>
ticket rerun <ticket-id> --strategy from_checkpoint
heartbeats list
heartbeats tail <heartbeat-id>
run cancel <run-id>

Inspect models and compute

models
model show <model-tag>
model card <model-tag>
model compare <first-tag> <second-tag>
hardware
gpu-search vastai --help
gpu-search lambda --help

Configure credentials without shell-history leakage

settings
set-secret ANTHROPIC_API_KEY
set-cloud-backend vastai

set-secret reads the value from a hidden prompt. Use set-secret <key> --clear to remove it, then recreate backend and scheduler services before launching new work.

08Reliability and Recovery#

Validation layers#

Zevo does not ask an Agent to guess an undocumented result shape. Each typed stage receives its schema and canonical examples before execution. Results then pass through several distinct checks:

1. Payload validation checks the Ticket before activation.

2. Typed result validation checks the Agent’s structured response.

3. Artifact validation checks generated YAML/JSON, files, schemas, signatures, and provenance.

4. Runner cross-checks compare the claimed result with the files and execution evidence that actually exist.

5. Cross-stage hard checks protect model lineage, user pins, held-out isolation, evaluator identity, frozen inference reuse, and Train/Inference prompt alignment.

Non-result-affecting differences become notices when safe. Identity, evaluation, isolation, or realized-configuration drift remains a hard failure.

Repair and failure routing#

Failures are routed by ownership:

Repair reuses verified expensive work whenever possible; it does not blindly retrain or regenerate. A Ticket is marked failed only after its bounded repair path is exhausted or a terminal condition is reached.

Run-scoped memory#

Memory is explicit, durable, and keyed by Run, Agent, and lane. It is not a model-provider conversation feature and is never shared across Runs.

• agent_local memory helps the same Specialist avoid repeating a pitfall or reuse a verified runtime fact later in the Run.
• shared_candidate memory exposes only verified facts, experiment findings, or recommendations that may help the Orchestrator or another Specialist.
• Runtime pitfalls and artifact references remain Agent-local.

Remote jobs and queue waiting#

Cluster Train and Inference stages first finish their executable, configuration, data bindings, output paths, and finite train.sbatch or predict.sbatch. Submission is the last preparation step. Zevo records the returned Slurm job id, posts a waiting state while it is pending, changes to running after the scheduler starts it, and reports success only after the job is terminal and every required artifact validates.

The watcher queries Slurm with low-frequency adaptive backoff rather than high-frequency polling. It checks short Inference work sooner than long Train work, but accelerates around observed state changes so the UI does not remain stale after a job starts or finishes. Pending time is bounded by max_queue_wait_hours and excluded from active Run duration.

A Train job approaching its wall-time limit saves a verified weights-only checkpoint when possible. Zevo may submit a new finite job and continue from that checkpoint after another queue wait. Native scheduler requeue is not treated as a second independent continuation mechanism. Instance stages run directly over SSH and therefore have process monitoring and checkpoint continuation but no Slurm queue. Cloud stages add rental creation and destruction around the same Train or Inference lifecycle.

Cancellation#

Cancelling a Train or Inference Ticket terminates its local Agent process group and the exact matching remote task or Slurm step identified by ZEVO_TICKET_ID. For instance, Zevo does not cancel the user’s allocation itself. System-owned cluster or cloud resources may be released.

09Developer Guide#

This section is for contributors who want to understand, modify, test, or extend the Zevo codebase.

Technical Architecture#

Two clients, one API, one database, and one shared data root. Every view of a Run reads the same rows, and everything a Run produces lands under the same root.

The web UI and the zevo CLI both talk to FastAPI, and only FastAPI writes to PostgreSQL and to the shared data/ root. The two clients differ in what they make convenient, never in what they can see.

Work moves on its own from there. The scheduler reads the database, consumes wakeups and activates the Tickets that are due. Each activation hands control to the run engine, which owns the DAG and calls either the LLM Agents, which in turn reach an instance, cluster or cloud GPU, or the deterministic Evaluation Runner. Both write their artifacts back into the same data/ root the API serves.

Three of those parts hold rules rather than state, and the rest of this document keeps returning to them:

• The engine owns the DAG, held-out isolation, typed validation, repair, cancellation, and stopping guards.
• The Playbook defines how each Agent should reason and act.
• Pydantic contracts and deterministic validators define the exact machine-readable inputs, results, and artifact invariants.

Data Root#

Runtime state is deliberately kept under one root:

data/
  files/       reusable file catalogue
  uploads/     uploaded attachments
  runs/        per-Run and per-Ticket artifacts

Repository Structure#

playbook/
  agents/      Agent identities, goals, typed guidance, and platform rules
  skills/      concrete Data, Train, Infrastructure, Inference, and Registry procedures
  runners/     deterministic Evaluation Runner manifest
src/zevo/
  api/         FastAPI routes
  cli/         zevo CLI
  contracts/   typed inputs, results, configs, and validators
  db/          database models and sessions
  engine/
    agent/     Agent loading and drivers
    run/       scheduler, DAG, runner, repair, retry, and cancellation
    method/    splits, metrics, framing, and experiment policy
    observe/   transcripts, telemetry, audit, and model cards
    cost/      token/GPU accounting and budget logic
  providers/   GPU provider clients
web/           React/Vite UI
alembic/       database migrations
ops/           Docker, deployment, and sandbox support
tests/         contract, workflow, API, CLI, and regression tests

Local Development#

Python source, Playbook files, migrations, and operations files are bind-mounted into backend and scheduler containers. Restart those services after Python changes:

docker compose restart backend scheduler

The web application is built into its image, so frontend changes require a rebuild:

docker compose build web
docker compose up -d web

Run backend tests:

docker compose run --rm backend pytest -q

Run the frontend build from web/ with Node installed:

cd web
npm ci
npm run build

Follow service logs with:

docker compose logs -f backend scheduler web

Stop the stack while preserving PostgreSQL data:

docker compose down

docker compose down -v

10Troubleshooting#

Start with service health and the Run’s own Timeline before changing a Task or rerunning expensive work:

docker compose ps
docker compose logs --tail=200 backend scheduler holdout-scheduler web

• The UI opens but a URL such as the database does not: only http://localhost:5173 is a browser page. PostgreSQL is a database service, while backend endpoints are consumed by the UI and CLI.
• A saved API key or SSH connection is not active: confirm it reached .env, then recreate backend, scheduler, and holdout-scheduler. Existing heartbeats keep the configuration resolved when they started; new values apply to later activations.
• SSH works in a host terminal but verification fails: verify the key path is inside host ~/.ssh, that the containers can see it under /root/.ssh, and that the non-interactive Environment command can find Slurm or activate the runtime without relying on an interactive shell profile.
• A Slurm stage is waiting: inspect its scheduler state and queue timer. Pending time does not consume active Run duration, but the per-submission queue limit still applies.
• Slurm finished but Zevo still says running: inspect scheduler logs and the Ticket heartbeat. Zevo waits for terminal scheduler evidence and validates the expected output, stderr, metrics, prediction, or checkpoint artifacts before posting success.
• Validation cannot be derived: Test must contain at least 1,000 rows for a 20% Validation split with at least 200 rows. Otherwise upload a separate Validation set and contract. A sample submission only defines schema; its example row count does not have to match Test.
• A Ticket failed: open Per-ticket, read the latest heartbeat and error, and check whether it is repairing, waiting, or terminal. Use ticket retry-status before requesting a bounded rerun, and prefer from_checkpoint only when a verified compatible checkpoint exists.

Useful CLI diagnostics:

run list --all
run show <run-id>
run watch <run-id>
ticket show <ticket-id>
ticket retry-status <ticket-id>
heartbeats tail <heartbeat-id>
agent status
hardware