Decisions, confidence & review

A successful run can still contain a weak decision. For example, a model may return confident: false, causing application code to replace a research plan with a default. The run can finish successfully even though the plan changed substantially. Record confidence and fallback behavior so you can find these cases without comparing paths by hand.

The original example behind this guide was a travel-planning run with 34 steps instead of an earlier run's 40. One low-confidence decision replaced four planned lookups with a generic question. Its output said "1 of 1 advisory lookups answered." The counts describe that historical example, not a benchmark or current live result.

The examples below illustrate the implementation and records captured for this feature. To repeat requests against a secured server, send the ATFGraph key in X-API-Key; use Authorization separately for a platform proxy credential when required.

What a decision is

A decision is a capability for a typed judgment, usually from a model, with an explicit confidence criterion. The server dispatches it like a traversal. The decision kind labels the record and Console view; it does not enforce the quality of the judgment or change its scheduling.

@router.decision()
async def plan_context_lookups(situation: str, model: str | None = None) -> dict:
    ...

Capability kinds are traversal, skill and decision. Each recorded step retains the declared kind:

[{"kind": "decision", "status": "succeeded", "target": "docs.plan"}]

Step 1: say what confident means

A confidence flag needs a written meaning. Without a criterion, the model must invent its own standard, while downstream code may treat that flag as a reason to discard the answer.

Describe confidence for the specific decision and explain the consequence of being wrong:

class ContextPlan(BaseModel):
    lookups: list[PlannedLookup] = Field(description="Only the lookups this brief needs. At most four.")
    confident: bool = Field(description=(
        "False ONLY if you cannot name a single worthwhile lookup. False DISCARDS every lookup "
        "you listed and replaces the whole plan with one generic fare question, so a plan you are "
        "merely unsure about should still be confident. Uncertainty belongs in the questions, not here."
    ))

Step 2: close the enums

Use a closed type for fields with a fixed set of valid values. A list of values in descriptive prose does not prevent an unexpected string from reaching downstream code.

ContextEngine = Literal["fare_insights", "destination_context", "map_context", "web_context", "reviews_context"]

class PlannedLookup(BaseModel):
    engine: ContextEngine = Field(description="Which advisory engine answers this question.")

Literal produces an enum in JSON Schema. Providers that support the schema can constrain generation; SDK validation rejects values outside it rather than silently filtering them out.

Step 3: record the outcome

Pass confidence_field to record the parsed result's confidence with the model call's usage:

return await app.ai(schema=schema, system=system, user=user, confidence_field="confident")

The application chooses the field name; the SDK does not require confident. app.ai has the parsed result when it records usage, before returning to the handler. This matters because the server deduplicates on (traverser, attempt, call_id): reposting the same usage row to add confidence would be dropped rather than merged.

The stored usage row includes fields such as:

{"capability": "plan", "confident": false, "fallback": "model reported confident=false after proposing 3 lookup(s)",
 "input_tokens": 1412, "output_tokens": 96, "model": "verify-model", "operation": "chat", "source": "llm"}

These fields are nullable. Missing confidence means the call did not report it; false means it explicitly reported doubt. Keep those cases separate, just as unknown cost differs from zero cost.

Record usage where the model call happens. A resumed handler starts again from the top, so unconditional handler-level recording can count a call that a checkpoint actually skipped. Calls through app.ai record usage when executed; your replay logic must avoid repeating the provider call when reusing its saved result.

Step 4: aggregate it

Query group_by=capability to see which traversals, skills or decisions incurred usage and how often they reported uncertainty or fallback:

GET /api/v2/usage/totals?group_by=capability

{"groups": [{"key": "plan", "calls": 2, "unconfident_calls": 2, "fallback_calls": 1,
             "input_tokens": 2824, "output_tokens": 192, "total_tokens": 3016,
             "priced_calls": 0, "unpriced_calls": 2, "cost_usd": null, "cost_complete": false}],
 "group_by": "capability", "scanned": 2}

unconfident_calls and fallback_calls answer different questions. A low-confidence result may still be used; a provider failure may trigger fallback without returning a confidence value. Both counts appear for every grouping, including none, model, run and day.

Step 5: measure it, where the provider lets you

If the provider supports token probabilities, request them alongside the self-reported confidence flag:

export ATFGRAPH_MODEL_LOGPROBS=1

This is opt-in because some gateways reject logprob options. atfgraph.model.boolean_field_probability identifies the token for a boolean field and records its probability in usage extra as confident_p. It returns no value when the evidence is missing or ambiguous, including absent logprobs, absent fields or unrecognized boolean tokens.

This measures the probability of the emitted value, not the probability the judgment is correct. A high probability for false means the model strongly selected the low-confidence answer. Do not treat it as a calibrated correctness score.

Step 6: calibrate before you threshold

scripts/calibrate-decisions.py --server http://127.0.0.1:18880

The read-only calibration script summarizes confidence rate, fallback rate and mean measured probability per capability from existing usage records. It creates no additional storage.

The label: retrieval feedback

To measure correctness, you need labeled outcomes. Retrieval feedback adds labels about items a decision consumed; a search response's scanned and strategy fields alone do not establish whether its results were useful.

Record feedback when the handler uses a retrieved item:

await app.memory.feedback(item_key, "useful")       # or irrelevant | outdated | wrong

The four verdicts form a closed set for aggregation. Feedback is append-only: an item can be useful at one time and outdated later, and both observations remain. Inside a traversal, the server derives capability identity from the lease, ignoring a caller-supplied capability. That gives feedback and usage a shared, server-checked attribution.

POST /api/v2/retrieval/feedback/totals   {"group_by": "capability"}

{"groups": [{"key": "plan", "useful": 2, "irrelevant": 0, "outdated": 1, "wrong": 1, "total": 4}],
 "group_by": "capability", "scanned": 4}

Totals include every verdict, even when its count is zero. Once feedback exists, calibration reports labelled: true and adds judged and useful columns.

Retrieval feedback labels the input material. It does not establish that the resulting judgment was correct; use evaluation cases for that. Do not choose an approval threshold from confidence frequency or useful-item rates alone.

You can gather repeated observations using real model decisions with fixture suppliers. In the travel example, set CO_CONSULT_DECISIONS=model CO_CONSULT_MODE=fixture. Model calls still cost money, but supplier calls use fixtures. Low confidence is expected when synthetic offers lack the requested context. A failed decision records a fallback message in usage; releases from 0.3.4 retain the error message, while earlier ones retained only its type.

Step 7: gate it on a human

When replacing a decision would have a material consequence, pause for a human to review the proposed action:

verdict = await app.request_approval(
    {"question": "run this plan anyway?", "proposed": lookups},
    operation_key="plan_context_lookups",
    checkpoint={"plan": result.model_dump()},
)
if verdict["status"] == "approved":
    return {"lookups": lookups, "confident": True}

request_approval saves a pause on the server and releases the worker lease. The traverser remains waiting across worker/server restarts. List requests with GET /api/v2/approvals; decide them with POST /api/v2/approvals/{id}/decide using approved or rejected. On resume, the handler starts from the top with a new lease, and the same approval call returns the recorded decision.

Make replay use the same proposal and approval:

• Save the proposal. Read its checkpoint first and consult the model only if missing. Otherwise resume could produce a plan different from the one the reviewer approved. See checkpoints for revision checks.
• Keep `operation_key` stable. The same key retrieves the existing approval instead of creating another request.

Until you have calibrated a numeric threshold, request review for a concrete event such as discarding a decision. Limit approvals to cases someone will actually review. In the travel example they remain off unless CO_CONSULT_GATE_DECISIONS names a capability.

Build the process in this order

Each stage gives you evidence for the next:

1. Record the confidence and fallback outcome.
2. Aggregate it to find patterns.
3. Collect probability measurements where supported.
4. Calibrate against observed, labeled outcomes.
5. Add human review where the consequences justify it.

Adding a threshold first gives it little evidential basis. Recording without aggregation also leaves problems hard to find: the original example already contained confidence in its path, but there was no query surfacing the pattern.

A note on vector storage

Memory vectors use base64-encoded, little-endian float32 storage. The historical measurements below compare that encoding with the earlier JSON representation using exact linear search. Current PostgreSQL deployments can also use pgvector/HNSW indexes; SQLite and PostgreSQL without an applicable index retain linear search. Check the returned strategy before comparing performance.

The original real-daemon experiment used dimension 1024 (databricks-gte-large-en). These are historical before/after measurements, not fresh results from this rewrite:

In that experiment, parsing the earlier 21,153-character JSON vectors accounted for 87% of the measured per-vector cost; arithmetic accounted for 13%. The encoding reduced parsing overhead without changing linear search's complexity.

Writes round values to float32, so callers supplying float64 values read back roughly seven significant digits. Typical float32 model embeddings retain their existing precision. The contract describes this on MemoryVectorWrite.embedding.

For the linear strategy, work grows with the vectors compared, reported in scanned. The historical test suggested sub-second searches at roughly 15,000 vectors instead of 2,500, but that depends on hardware, dimension and workload. For larger corpora, evaluate the supported PostgreSQL vector indexes and their approximate recall rather than extending that estimate.

Verify it yourself

scripts/verify-decisions.py

The verification script starts an isolated server with a temporary database, exercises the feature stages and prints pass/fail results. It leaves the server available for inspection. It uses no model or supplier calls, so it requires no provider credentials. Running it checks the fixture journey, not model judgment quality.

Where to see it in the console

In the Console, decision appears in the path's Kind column. Confidence remains on the usage record and is aggregated by capability; it is not a correctness badge on each step.

Use Monitor over HTTP for equivalent requests and Observability for the stored usage and metadata interfaces.