CLI Reference

The koji CLI manages clusters, processes documents, and runs benchmarks. Install via pip install koji-cli.

Project lifecycle

koji init

Scaffold a new Koji project. Creates koji.yaml and (optionally) a starter schema.

koji init                                    # bare koji.yaml in the current directory
koji init myproject                          # new project directory with koji.yaml
koji init myproject --template invoice       # scaffold from a bundled template
koji init myproject --quickstart             # alias for --template invoice
koji init --list-templates                   # show all available templates

Flag	Description
project_dir (positional)	Optional. Directory name to create. Defaults to the current directory.
--template, -t	Scaffold from a bundled template. Run --list-templates to see all options.
--quickstart, -q	Alias for --template invoice.
--list-templates	List available templates and exit.

Bundled templates: invoice, receipt, contract, insurance, form. Each ships with a working schema and a sample document so you can run extraction immediately.

---

koji doctor

Check that your environment is ready to run Koji. Verifies Docker, Docker Compose, the Koji configuration file, port availability, and required environment variables.

koji doctor
koji doctor -c ~/proj/koji.yaml   # check a project outside the current directory

Koji Doctor

  ✓ Docker installed (Docker version 27.x.x)
  ✓ Docker Compose available
  ✓ Docker daemon running
  ✓ koji.yaml found
  ✓ koji.yaml valid (project: myproject)
  ✓ Ports available (base: 9400)
  ✓ OPENAI_API_KEY set

7 passed, 0 warning, 0 failed

Run this any time something looks wrong. It's the fastest way to diagnose setup issues.

---

Cluster lifecycle

koji start

Start the cluster defined in koji.yaml. By default, pulls pre-built images from ghcr.io/getkoji.

koji start                          # pull pre-built images and run (default)
koji start --dev                    # build images from local source (for contributors)
koji start -c ~/proj/koji.yaml      # start a project outside the current directory

Flag	Description
--dev	Build images from the local source tree instead of pulling. Required when developing on Koji itself.
--clean	Destroy existing data and start fresh (equivalent to koji destroy + koji start).
--config, -c	Path to koji.yaml (default: ./koji.yaml). The project's .koji/ state lives next to the config file, so every other cluster command accepts the same flag.

First start with --dev takes a few minutes for the docling/torch image build. Default koji start pulls pre-built images and is usually under a minute once images are cached locally.

The dashboard comes up at http://127.0.0.1:9400 (or whatever cluster.base_port is set to in koji.yaml).

---

koji stop

Stop the running cluster.

koji stop
koji stop -c ~/proj/koji.yaml   # stop a project outside the current directory

Tears down all containers but preserves Docker volumes (model caches, etc.). Run koji start again to bring the cluster back up.

---

koji status

Show cluster health and per-service status.

koji status
koji status -c ~/proj/koji.yaml   # status of a project outside the current directory

Output shows each running service, its port, and health check result. Use this to verify the cluster is fully up before processing documents.

---

koji logs

Stream container logs for one or all services.

koji logs                       # tail all services (last 100 lines)
koji logs parse                 # tail just the parse service
koji logs parse --follow        # follow parse logs in real time
koji logs server --tail 500     # show last 500 lines of server logs

Flag	Description
service (positional)	Service name: server, parse, extract, ui, ollama. Omit to show all services.
--follow, -f	Follow log output (like tail -f). Press Ctrl-C to stop.
--tail, -t	Number of lines to show from the end of the log (default: 100).
--config, -c	Path to koji.yaml (default: ./koji.yaml).

---

Document processing

koji process

Run the full pipeline: parse a source document into markdown, then extract structured data using a schema.

koji process ./invoice.pdf --schema schemas/invoice.yaml
koji process ./documents/                                  # process a whole directory
koji process ./doc.pdf --schema schemas/invoice.yaml --output ./results/

Flag	Description
path (positional)	Path to a document file or a directory of documents.
--schema, -s	Path to an extraction schema YAML. If omitted, only the parse step runs.
--output, -o	Output directory (default: ./output/).
--config, -c	Path to koji.yaml (default: ./koji.yaml). Locates the running cluster when you're not in the project directory.

When --schema is provided, you get the full pipeline: parse → extract → JSON output. Without --schema, you get parsed markdown only — useful for inspecting how Koji sees a document before writing a schema.

---

koji extract

Skip the parse step and run extraction against an already-parsed markdown file. Much faster than koji process because parsing (Docling + OCR) is the slow step.

koji extract ./output/invoice.md \
  --schema schemas/invoice.yaml \
  --model openai/gpt-4o-mini

Flag	Description
path (positional)	Path to a markdown file (from a previous parse).
--schema, -s	Required. Path to an extraction schema YAML.
--model, -m	Model override. Format: provider/model-name. Examples: openai/gpt-4o-mini, openai/gpt-4o, ollama/llama3.2.
--output, -o	Output directory (default: ./output/).
--strategy	Extraction strategy: parallel (default, recommended) or agent.
--config, -c	Path to koji.yaml (default: ./koji.yaml). Locates the running cluster when you're not in the project directory.

This is the fastest feedback loop while iterating on a schema. Parse once, extract many times with different schemas or models.

---

Quality and benchmarking

koji test

Run regression tests against fixture files. Catches schema or pipeline changes that break extraction on documents you care about.

koji test --schema schemas/invoice.yaml
koji test --schema schemas/invoice.yaml --update           # snapshot mode: save current outputs as new baseline
koji test --schema schemas/invoice.yaml --json             # machine-readable output for CI

Flag	Description
--schema, -s	Required. Path to the schema being tested.
--model, -m	Model override.
--update	Snapshot mode: run extraction and save outputs as the new expected baseline. Use this for first-time setup or after intentional schema changes.
--json	Output machine-readable JSON results.
--strategy	Extraction strategy.

koji test looks for fixture files alongside your schema. Place markdown documents in <schema>.fixtures/ and corresponding <name>.expected.json files for ground truth. Field-level comparison: numbers and dates are matched semantically, strings case- and punctuation-insensitively (a formatting-only difference like CHARLOTTE, NC vs CHARLOTTE NC or 704-376-9896 vs 704.376.9896 is a match; a content difference still fails), arrays order-insensitively. Exit code is 0 on full pass, 1 on any regression.

Adversarial fixtures (expected: null): a field in the expected JSON that's explicitly set to null (or an empty string / empty list / empty dict) asserts that the model should not extract that field — either because the value isn't in the document or because the document is a trap meant to measure hallucination resistance. Both empty → pass ("correctly absent"); expected empty but actual populated → fail ("hallucinated"); expected populated but actual empty → fail ("missing"). Use this to build a trap corpus of documents where the right answer is "I don't know" and grade models on how often they correctly decline.

---

koji bench

Benchmark extraction accuracy across an entire validation corpus. Use this to measure accuracy before shipping schema changes, compare models, or produce numbers for an accuracy dashboard.

koji bench --corpus ./corpus                                # model from koji.yaml
koji bench --corpus ./corpus --model openai/gpt-4o-mini     # explicit model override
koji bench --corpus ./corpus --category invoices --limit 10
koji bench --corpus ./corpus --model openai/gpt-4o --output bench.json

Flag	Description
--corpus, -c	Required. Path to a corpus directory (with documents/, expected/, manifests/, schemas/ subdirectories per category).
--model, -m	Model override. Defaults to the extract step's model in koji.yaml — the same model the cluster runs — falling back to the server default.
--category	Only benchmark a single category (e.g., invoices).
--limit	Maximum documents to process per category. Useful for fast CI runs.
--json	Output machine-readable JSON.
--output, -o	Write JSON results to a file (always JSON, regardless of --json).
--config	Path to koji.yaml (default: ./koji.yaml). Long form only — bench uses -c for --corpus.

The corpus format is the convention used by getkoji/corpus. Per-category, per-document, and aggregate accuracy are reported. Exit code is 0 on full pass, 1 on any regression or error.

---

The schema loop (connected platform)

These commands drive the Build → Validate → Corpus workflow from the dashboard, but from the terminal. They talk to a running Koji platform (the same API the dashboard uses), so they need credentials: run koji login first, or set KOJI_API_URL + KOJI_API_KEY. Pass --profile to target a specific saved profile.

Every request is scoped to a project — the isolation boundary within a workspace. An API key is bound to the project it was created in and cannot reach any other project; to work in a different project, mint a key there. Setting the profile's project (koji login --project <slug>) or KOJI_PROJECT=<slug> with env-var credentials sends the x-koji-project header as an explicit assertion — useful to fail fast when the credentials don't match the project you meant to target.

Every command below accepts --json to emit raw machine-readable output instead of a table — handy for scripting and for driving the loop from an agent.

The inner loop is: edit the schema YAML → koji validate to backtest it (safely) against ground truth → drill into a failing doc with koji corpus diff → repeat → koji schema promote once it performs well.

koji validate

Backtest a schema against its corpus ground truth — safely. Snapshots your local YAML as a release candidate (v0.0.4-rc.N, deduped by content), then runs the platform's validation — re-extracting every corpus doc that has ground truth and scoring it — and prints overall + per-field accuracy, regressions, and failing docs. The run executes in the background on the server (each document as its own job) while the CLI shows a live progress bar — a large corpus or an expensive schema (e.g. enumerate_rows fields) never hits a request timeout. Array fields are scored by F1 and show a P/R (precision/recall) column so you can tell missed elements from spurious ones; --json includes precision/recall per array field. The candidate is not made live: iterating never touches the schema your pipelines run. The semver bump (major/minor/patch) is auto-derived from how the output shape changed.

koji validate insurance_policy                       # snapshot schemas/insurance_policy.yaml as a candidate + backtest
koji validate ./schemas/insurance_policy.yaml        # explicit path
koji validate insurance_policy --bump minor          # override the auto-derived bump
koji validate insurance_policy --no-push             # validate the version already live on the server
koji validate insurance_policy --watch               # re-run whenever the local file changes
koji validate insurance_policy --check               # exit non-zero if any field regressed (CI / loops)
koji validate insurance_policy --explain             # show WHY each field failed (routing diagnosis)
koji validate insurance_policy --json                # raw result for an agent to read

Flag	Description
--model	Override the extraction model (e.g. openai/gpt-4o-mini).
--bump	Override the auto-derived semver bump: major | minor | patch.
--no-push	Validate the version already live on the server; don't snapshot local edits.
--message, -m	Message for the candidate snapshot.
--watch, -w	Re-run whenever the local schema file changes.
--check	Exit non-zero if any field regressed (for CI / loops).
--explain	For each failing field, show which chunks the model saw, how they were routed, and whether the expected answer was even present in them.
--json	Emit raw JSON instead of a table.
--profile, -p	CLI profile to use.

The <schema> argument is either a slug (a local schemas/<slug>.yaml is found automatically) or a path to a YAML file. The slug is taken from the file's name: field. Promote a candidate to live with koji schema promote.

Diagnosing failures with --explain

A failing field has two very different causes, and telling them apart decides the fix. --explain (also present as routingDiagnosis on each fields[].failingDocs[] entry under --json) reports, per failing field, how its chunks were selected (source) and whether the ground-truth answer was even present in the chunks the model was shown (answerInRoutedChunks):

• answerInRoutedChunks: false — a routing miss. The correct answer never reached the model, so no prompt change and no bigger model can fix it. The fix is to steer chunk selection with the field's hints (look_in, prefer_contains, patterns, prefer_position, max_chunks).
• answerInRoutedChunks: true — a model misread. The answer was in front of the model and it still got it wrong; tighten the field description, type, or enum aliases. A larger model is a last resort, justified only here.
• answerInRoutedChunks: null — couldn't be determined (a non-scalar expected value, or the read-only GET path with no fresh routing data).

For failing array fields, --explain additionally prints an array element diagnostics table with the per-element diff behind the field's precision/recall: which extracted elements were spurious (FP — they hurt precision), which expected elements were missed (FN — they hurt recall), and how many matched by key but differ in a sub-field (~). Elements are labeled by their element_key value when the schema declares one, so a set-level diff reads at a glance instead of requiring a by-hand comparison against ground truth. The same data is on each element of fields[].failingDocs[].diff.elements under --json (status: matched / changed / missing / extra, plus key). FP points at over-enumeration — tighten section_anchor / skip_row_when; FN points at routing/enumeration recall — check per_section / enumerate_rows.

koji run

Run one corpus document through a schema and show the extraction — the Build tab's Run button. Uses your local schema YAML if a file is found (so you can iterate without committing a version), otherwise the server's latest version.

koji run insurance_policy "10th street townes.pdf"   # match a doc by filename
koji run insurance_policy 561c6e69                    # …or by id (prefix is fine)
koji run insurance_policy 561c6e69 --provenance       # show the source snippet per value
koji run insurance_policy 561c6e69 --no-cache         # force a fresh parse (bypass the parse cache)
koji run insurance_policy 561c6e69 --json             # raw extraction for an agent

Flag	Description
--model	Override the extraction model.
--no-cache	Force a fresh parse, bypassing and refreshing the parse cache. The cache is keyed per parse provider, so switching providers already re-parses — this is the belt-and-suspenders override (e.g. re-parse the same provider from scratch).
--provenance	Show the source snippet each value came from.
--json	Emit raw JSON.
--profile, -p	CLI profile to use.

koji corpus

Manage a schema's validation corpus — documents and their ground-truth annotations.

koji corpus ls insurance_policy                      # list docs (id, filename, ground-truth?, source, tags)
koji corpus ls insurance_policy --no-gt              # only docs missing ground truth
koji corpus ls insurance_policy --tag edge-case      # filter by tag

koji corpus diff insurance_policy 561c6e69           # extracted vs ground truth, field by field
koji corpus diff insurance_policy 561c6e69 --run     # extract fresh first, then diff

koji corpus get insurance_policy 561c6e69 -o doc.pdf # download the source file to read it
koji corpus get insurance_policy 561c6e69 --markdown # …or the parsed markdown text

koji corpus add insurance_policy ./new-doc.pdf       # upload a doc into the corpus
koji corpus rm insurance_policy 561c6e69             # remove a doc (wrong type / skewing validation)
koji corpus tag insurance_policy 561c6e69 --add edge-case --remove synthetic

koji corpus gt show insurance_policy 561c6e69        # show current ground truth
koji corpus gt accept insurance_policy 561c6e69      # promote the latest extraction to ground truth
koji corpus gt set insurance_policy 561c6e69 --from truth.json

A document is addressed by corpus-entry id (a unique prefix is enough — the id shown by corpus ls is truncated) or by filename (exact, or a unique substring). All corpus subcommands accept --json and --profile.

koji corpus get downloads the document so you can read it directly — the source file by default (PDFs, images, etc.), or the parsed markdown with --markdown. This is how you settle a disagreement between an extraction and ground truth: pull the document, read what it actually says, and correct ground truth with koji corpus gt set.

koji corpus rm removes a document from the corpus when it doesn't belong — e.g. a doc that isn't really of this schema's type and is skewing your validation numbers. It's a soft-delete (the row and file are retained for recovery and the entry drops out of every read path); it prompts for confirmation unless you pass --yes. Re-add a removed doc with koji corpus add.

koji corpus gt accept reads the document's latest extraction (run koji run first) and saves those values as ground truth — the fast path for "this extraction is correct." koji corpus gt set --from <file> sets ground truth from a JSON file of {field: value}.

koji review

Inspect the human-review queue and promote reviewed documents into the corpus. Documents land in this queue when a pipeline routes them for review — a field's confidence fell below the pipeline's review threshold, a validation rule failed, etc. These are the highest-signal documents to add to your corpus, because they're exactly the ones the current schema struggles with.

koji review stats                                    # true queue counts (count(*), server-side)
koji review stats --urgent-below 0.5 --json          # adjust the urgent threshold

koji review ls                                       # pending items, worst confidence first
koji review ls --status completed                    # resolved items (ready to promote)
koji review ls --reason low_confidence               # filter by routing reason
koji review ls --limit 50 --json                     # raw rows for an agent to read

koji review show <id>                                # full context: flagged field, why it
                                                     #   routed, the doc's whole extracted record
koji review promote <id>                             # resolved+approved → corpus ground truth
koji review promote <id> --to edge-case              # …and tag the new corpus entry
koji review promote <id> --provisional --gt-from label.json   # agent draft label (needs approval)

koji review stats reports pending / urgent / completed / reviewedToday computed server-side — use it for queue size and burn-down progress. koji review ls is a page, not the queue: it returns at most --limit rows (default 100), so counting its output caps every number at the fetch limit.

koji review promote closes the review → corpus loop. By default it requires the item to be resolved and approved (in the dashboard); the human's corrected record becomes ground truth that koji validate scores immediately. With --provisional, an agent-supplied label is written as a draft that stays out of validation until a human approves it in the dashboard Corpus tab. A review item is addressed by its id (a unique prefix is enough). All review subcommands accept --json and --profile.

The full loop — promote the flagged docs, then fix the schema so they stop getting flagged — is encoded in the review-corpus-loop Claude skill (which hands off to schema-loop for the schema-improvement half).

koji schema

Manage schema versions — list the released lineage and candidates, and promote/release. Versions use semver: koji validate snapshots candidates (v0.0.4-rc.N); promotion graduates one to a release (v0.0.4) and makes it live for pipelines. Promotion is manual and gated by the schema:deploy permission.

koji schema versions insurance_policy                # released lineage + candidates, scores, which is live
koji schema promote insurance_policy                 # graduate the latest candidate to a release + make it live
koji schema promote insurance_policy --version v0.0.4-rc.7      # promote a specific candidate
koji schema promote insurance_policy --require-no-regressions   # refuse if the candidate's latest run regressed
koji schema release insurance_policy                 # release a schema directly, skipping the rc loop
koji schema release ./schemas/insurance_policy.yaml  # …from a local file

koji schema release is the early-stage path: when there's nothing in the corpus to backtest yet, skip candidates and release straight to a full version. All schema subcommands accept --json and --profile.

The full validate → promote loop is encoded in the schema-loop Claude skill.

koji classify

Run and manage classifiers — the same lifecycle CLI as koji schema, plus a run verb that classifies a single document. A classifier assigns a document a label (its type) using a cost-ordered cascade of tiers (keyword match → windowed heuristics → LLM → vision), stopping at the first tier confident enough to answer.

koji classify run document_type ./invoice.pdf         # classify a doc: label, confidence, method, tier
koji classify run document_type ./invoice.pdf --json  # raw result for an agent
koji classify run document_type ./invoice.pdf --draft # run the unreleased candidate instead of the release
koji classify run ./classifiers/document_type.yaml ./invoice.pdf   # …with a local config (iterate without pushing)
koji classify run document_type ./big-scan.pdf --max-pages 0       # send the whole doc (default: first 3 pages)

koji classify versions document_type                  # released lineage + candidates
koji classify promote document_type                   # graduate the latest candidate to a release + make it live
koji classify release document_type                   # release directly, skipping the rc loop
koji classify release ./classifiers/document_type.yaml  # …from a local file
koji classify delete document_type                    # delete a classifier + all its versions

koji classify run drives the standalone POST /api/classify primitive and persists nothing. By default it runs the classifier's released version — the exact version the ingestion pipeline runs — so its result is a faithful proxy for how the pipeline will route the document. It prints which config it used (released v0.0.2, draft, or local file …), then the assigned label, the confidence, the method and tier that produced it, and the evidence page. A document that matches no class comes back as unknown.

• --draft runs the latest unreleased candidate instead of the release — for checking an edit before you release it.
• Passing a file path (./classifiers/document_type.yaml) runs that local YAML, for iterating before you push at all.

For a large multi-page PDF, only the first --max-pages pages (default 3) are uploaded — classification reads the masthead, and this keeps big scans under the API's upload size limit; pass --max-pages 0 to send the whole file.

koji classify versions / promote / release mirror their koji schema equivalents: versions lists the released lineage and candidates; promote graduates the latest candidate to a live release; release skips the candidate loop and releases directly (from the server draft, or a local file if you pass one). Promotion and release are gated by the deploy permission. All classify subcommands accept --json and --profile.

koji classify delete <slug> removes a classifier and all its versions (confirmation prompt; --yes to skip). Pipelines that reference the slug will fail to resolve it until it's recreated — use it to clean up a test classifier or to recreate one from scratch.

To register a classifier as a named resource (so pipelines can reference it and it gets a version history), give the file kind: classifier and koji push it — the same way you push schemas and pipelines. koji push -d . scans a classifiers/ subdirectory alongside schemas/ and pipelines/.

koji pipeline

Inspect pipelines and control which schema version each one runs. Every pipeline tracks schema versions in one of two modes: auto (default) always runs the schema's current live release, so it picks up a promotion immediately; pinned holds a specific version until you bump it — useful for canary / staged rollout (pin a critical pipeline, let the rest auto-follow a promotion, verify, then bump the pinned one).

koji pipeline ls                                     # pipelines with their schema + status
koji pipeline deploy policies --version v0.0.26      # pin this pipeline to v0.0.26
koji pipeline deploy policies --auto                 # unpin → follow the live release again

Pinning is gated by the schema:deploy permission. The version is addressed by its semver label (or an id prefix) and must belong to the pipeline's schema. All pipeline subcommands accept --json and --profile.

koji pipeline run — run documents through a pipeline

Submit one or more documents to a pipeline and get the extraction back. This is the same path the dashboard's manual run uses (POST /api/pipelines/<slug>/run): each document is parsed, extracted, and routed exactly as production ingestion does, creating a real job. Point it at a file, several files, or a directory.

koji pipeline run invoices ./inbox/acme.pdf          # run one doc, wait, print extraction
koji pipeline run invoices ./inbox/                  # run every file in a directory
koji pipeline run invoices a.pdf b.pdf --provenance  # also show the source snippet per field
koji pipeline run invoices ./inbox/ --group march    # tag all docs with a grouping key

By default the command waits for every document to reach a terminal state (completed / failed / review) and prints the result. Pass --no-wait to submit and return immediately — useful for an agent that wants to fire documents off and poll later:

koji pipeline run invoices ./inbox/ --no-wait --json # submit, print job slugs, exit

--timeout <seconds> caps how long the sync path waits (default 600). The pipeline must have a deployed schema version (simple pipelines) or a saved DAG definition; otherwise the run is rejected. Requires the job:run permission.

koji pipeline result — fetch a submitted job's results

Read the documents + extraction for a job created by koji pipeline run --no-wait (reads GET /api/jobs/<slug>/documents — the same data the dashboard's job view shows). Pass --wait to block until every document finishes.

koji pipeline result cool-otter-1a2b                 # print current status + extraction
koji pipeline result cool-otter-1a2b --wait --json   # block until done, emit JSON

koji pipeline test — dry-run and see the routing decision

Dry-run a document through a pipeline without persisting anything (no job is created) and see exactly how it routes. This wraps POST /api/pipelines/<slug>/test — the same dry-run the dashboard's Test button uses. It parses the document (via the tenant's parse provider, matching production), then walks the pipeline's steps and prints, for each one: a classify step's chosen label / confidence / method, which route matched at every branch (✓/✗), the path taken end-to-end, and the final extraction.

koji pipeline test family-router ./doc.pdf -p prod    # show the route + extraction
koji pipeline test family-router ./doc.pdf --json     # raw result (steps[], path, edgeEvaluations)

This is the tool for validating a router — a pipeline with classify steps that branch to different schemas. koji pipeline run gives you the real, persisted run; koji pipeline test shows you why a document went where it did (which classifier fired, which branch, which schema ran) so you can debug routing before sending real traffic through. Gated by the pipeline:write permission.

Example output for a two-tier router (line → carrier → schema):

doc-router test  acme.pdf
path: classify_kind → classify_fin → extract_invoice

▸ classify_kind (classify)
    label: financial  conf 100%  method: keyword
    ✓ output.label == 'financial' → classify_fin
    ✗ output.label == 'other' → extract_other
▸ classify_fin (classify)
    label: invoice  conf 100%  method: keyword
    ✓ output.label == 'invoice' → extract_invoice
    ✗ output.label == 'receipt' → extract_receipt
▸ extract_invoice (extract)
    schema: invoice  2/2 fields  conf 100%

koji pipeline bench — run a whole corpus through a pipeline and score it

Where koji bench scores a corpus against a single schema, koji pipeline bench runs every corpus document through a pipeline (DAG) and scores two things: did each document route to the right schema, and did it extract correctly once there. It runs each doc through POST /api/pipelines/<slug>/test (the same dry-run as koji pipeline test), so nothing is persisted — no jobs are created.

The corpus layout is identical to koji bench (per category: documents/ schemas/ expected/ manifests/). No new labels are needed: each document's manifest already names the schema it belongs to, so that slug is the expected route, and its .expected.json is the extraction ground truth. Point it at a mixed corpus — documents that legitimately route to different schemas — to exercise routing.

koji pipeline bench doc-router --corpus ./corpus                    # score routing + extraction
koji pipeline bench doc-router --corpus ./corpus --category invoices
koji pipeline bench doc-router --corpus ./corpus --limit 10 -p prod
koji pipeline bench doc-router --corpus ./corpus --json > bench.json # machine-readable

Extraction is scored only for correctly-routed documents — a mis-route makes field-level scores meaningless, so a mis-routed doc counts as a routing failure and is excluded from the extraction numbers rather than silently averaged in. Extraction accuracy is broken out per terminal schema, because outputs vary with the path a document takes through the DAG and a single blended number hides which branch is weak.

Array fields (a document's coverages, line_items, …) are scored element-wise by F1 — the same partial-credit semantics koji validate uses — so a coverages array with four of five elements right contributes ~0.8, not a full miss. The EXTRACTION line reports F1-weighted accuracy with the exact-match count in parentheses; each mismatched array shows its element F1 in the failure detail.

koji pipeline bench — doc-router
corpus: ./corpus

  ok inv_01.md → invoice_basic (4 fields, 812ms)
  MISROUTE rec_03.md: routed to invoice_basic, expected receipt_basic
  -- inv_07.md → invoice_basic: 3/4 fields (905ms)
       line_items: array items differ (80% element F1)

============================================================
ROUTING: 18/20 docs to correct schema (90.0%)
  receipt_basic: 2→invoice_basic

CLASSIFY: 20 steps — keyword 6, llm 14

EXTRACTION (correctly-routed only): 87.9% F1 over 72 fields (61 exact)
  invoice_basic: 91.4% F1 (40.2/44 fields)
  receipt_basic: 82.1% F1 (23.0/28 fields)

The CLASSIFY line tells you whether the routing score means anything. Each classify step reports the method that produced its label. A step that ran reports keyword, llm, or vision; a step that never inspected the document at all reports why — no_classifier (the referenced slug doesn't resolve in this project), no_version (the pinned version doesn't exist), no_file, or no_provider (no model endpoint was reachable).

That distinction matters because both a failed classifier and a genuine unknown send a document down its pipeline's default edge — from the routed schema alone they are indistinguishable. When a classify step fails to run, the bench says so loudly and tells you not to trust the routing number:

  MISROUTE policy_04.pdf: routed to policy_generic, expected policy_harford_mutual
       classify: classify_line=package(llm)  classify_carrier=unknown(no_classifier)
       ! classify_carrier never ran (no_classifier): Classifier 'family_carrier' has no released version in this project

============================================================
ROUTING: 0/30 docs to correct schema (0.0%)

CLASSIFY: 60 steps — llm 30, no_classifier 30

  !! 30 of 30 documents had a classify step that never inspected the document (no_classifier).
     These docs fell to their pipeline's default route. The ROUTING score above reflects a broken pipeline, not classifier accuracy — fix this before reading it.

--json includes the same detail: a classify.method_counts rollup, classify.docs_with_classifier_failures, and a per-document classify array carrying each step's method, reasoning, classifier, and classifier_version. Extraction scoring is exposed as extraction.field_credit (the F1-weighted numerator) alongside passed_fields (exact matches), a per-schema credit, a per-document credit, and a score on each failure.

Gated by the pipeline:write permission (same as koji pipeline test).

---

Projects

A project is the intra-tenant boundary that scopes schemas, pipelines, classifiers, sources, and jobs. Your profile carries a default project (the x-koji-project header sent on every request); manage them with koji project.

koji project list                                  # projects your key can see (● marks the active one)
koji project create rnd --name "R&D"               # create a project (tenant:admin)
koji project create rnd --use                       # …and switch the active profile to it
koji project use rnd                                # scope the active profile to an existing project
koji project delete old-project                     # delete a project (tenant:admin; --yes to skip the prompt)

create and delete require the tenant:admin permission; list/use need tenant:read. use (and create --use) update the active profile's default project in ~/.koji/credentials — the same field koji login --project <slug> sets — so subsequent commands run against that project without re-authenticating. create --name defaults to the slug; slugs are lowercase letters, numbers, and hyphens (2–64 chars).

API keys are bound to a single project. create and list are tenant-level (a key can create a project and see every project in the tenant), but an API key can only operate in the one project it's bound to — scoping a request to any other project returns 404 Project not found. So the key that creates a new project usually can't push to it: create --use detects this and tells you to mint a key for the new project instead of pinning your profile to a project every command would 404 on. To get a working key for a fresh project, create an API key from within that project in the dashboard (Settings → API Keys), then koji login --api-key <key> --project <slug>.

---

Misc

koji version

Print the installed Koji version.

koji version
# koji 0.26.0

---

Global options

Flag	Description
--help	Show help for any command. Pass --help to a subcommand for details.
--install-completion	Install shell completion for your shell.
--show-completion	Show shell completion script.

---

What's missing here?

If you find a command, flag, or behavior in this doc that doesn't match what koji --help shows, please open an issue. The CLI is the source of truth — this document follows it.