Evaluations API
Operations grouped under this contract tag.
Operations
listEvalCaseSets
GET /api/v2/evals/case-sets
Requires: invoke role
Case sets visible to the caller, newest first, optionally filtered by exact target. Each item carries case_count and, when the set has been run, last_run. Pass next_cursor back as cursor; limit defaults to 50 and is at most 500.
Generated Python client internals: atfgraph._generated.api.evaluations_api.EvaluationsApi.list_eval_case_sets()
Parameters
| Parameter | In | Required | Type |
|---|
cursor | query | no | string |
limit | query | no | integer |
target | query | no | string |
Responses
Example response 200
{
"items": [
{
"case_count": 0,
"created_at": 0,
"description": "string",
"id": "string",
"last_run": {
"case_set_id": "string",
"cost_complete": false,
"cost_usd": 0.0,
"created_at": 0,
"duration_ms_p50": 0,
"errored": 0,
"failed": 0,
"finished_at": 0,
"id": "string",
"label": "string",
"pass_rate": 0.0,
"passed": 0,
"pending": 0,
"revision": 0,
"skipped": 0,
"started_at": 0,
"status": "string",
"target": "string",
"total": 0,
"total_tokens": 0,
"version": "string"
},
"name": "string",
"principal": "string",
"revision": 0,
"target": "string",
"updated_at": 0,
"version": "string"
}
],
"next_cursor": "string"
}
createEvalCaseSet
POST /api/v2/evals/case-sets
Requires: invoke role
Create a case set for one `label.capability` target, optionally with its first cases (at most 500). version pins an immutable agent version; absent or empty means the newest registered version at run time. The set belongs to the calling principal: admins see every set in the namespace, other callers only their own, and foreign ids answer 404 eval_set_not_found. Namespaces hold at most 1024 sets; names are at most 256 characters, descriptions 4096.
Generated Python client internals: atfgraph._generated.api.evaluations_api.EvaluationsApi.create_eval_case_set()
Request body
EvalCaseSetCreate (required)
{
"name": "string",
"target": "string"
}
Responses
Example response 201
{
"case_count": 0,
"created_at": 0,
"description": "string",
"id": "string",
"last_run": {
"case_set_id": "string",
"cost_complete": false,
"cost_usd": 0.0,
"created_at": 0,
"duration_ms_p50": 0,
"errored": 0,
"failed": 0,
"finished_at": 0,
"id": "string",
"label": "string",
"pass_rate": 0.0,
"passed": 0,
"pending": 0,
"revision": 0,
"skipped": 0,
"started_at": 0,
"status": "string",
"target": "string",
"total": 0,
"total_tokens": 0,
"version": "string"
},
"name": "string",
"principal": "string",
"revision": 0,
"target": "string",
"updated_at": 0,
"version": "string"
}
getEvalCaseSet
GET /api/v2/evals/case-sets/{id}
Requires: invoke role
One case set with its case count and most recent run. A set owned by another principal is 404 eval_set_not_found rather than 403.
Generated Python client internals: atfgraph._generated.api.evaluations_api.EvaluationsApi.get_eval_case_set()
Parameters
| Parameter | In | Required | Type |
|---|
id | path | yes | string |
Responses
Example response 200
{
"case_count": 0,
"created_at": 0,
"description": "string",
"id": "string",
"last_run": {
"case_set_id": "string",
"cost_complete": false,
"cost_usd": 0.0,
"created_at": 0,
"duration_ms_p50": 0,
"errored": 0,
"failed": 0,
"finished_at": 0,
"id": "string",
"label": "string",
"pass_rate": 0.0,
"passed": 0,
"pending": 0,
"revision": 0,
"skipped": 0,
"started_at": 0,
"status": "string",
"target": "string",
"total": 0,
"total_tokens": 0,
"version": "string"
},
"name": "string",
"principal": "string",
"revision": 0,
"target": "string",
"updated_at": 0,
"version": "string"
}
updateEvalCaseSet
PATCH /api/v2/evals/case-sets/{id}
Requires: invoke role
Compare-and-set edit of name, description, target and version. expected_revision must equal the current revision or the request fails with 409 revision_mismatch; unknown fields are rejected. Existing runs keep the target and version they were started with.
Generated Python client internals: atfgraph._generated.api.evaluations_api.EvaluationsApi.update_eval_case_set()
Parameters
| Parameter | In | Required | Type |
|---|
id | path | yes | string |
Request body
EvalCaseSetUpdate (required)
{
"expected_revision": 0
}
Responses
Example response 200
{
"case_count": 0,
"created_at": 0,
"description": "string",
"id": "string",
"last_run": {
"case_set_id": "string",
"cost_complete": false,
"cost_usd": 0.0,
"created_at": 0,
"duration_ms_p50": 0,
"errored": 0,
"failed": 0,
"finished_at": 0,
"id": "string",
"label": "string",
"pass_rate": 0.0,
"passed": 0,
"pending": 0,
"revision": 0,
"skipped": 0,
"started_at": 0,
"status": "string",
"target": "string",
"total": 0,
"total_tokens": 0,
"version": "string"
},
"name": "string",
"principal": "string",
"revision": 0,
"target": "string",
"updated_at": 0,
"version": "string"
}
deleteEvalCaseSet
DELETE /api/v2/evals/case-sets/{id}
Requires: invoke role
Delete a set together with its cases, runs and results in one transaction. Traversers the runs admitted are untouched. expected_revision travels as a query parameter and, when given, must match (409 revision_mismatch).
Generated Python client internals: atfgraph._generated.api.evaluations_api.EvaluationsApi.delete_eval_case_set()
Parameters
| Parameter | In | Required | Type |
|---|
id | path | yes | string |
expected_revision | query | no | integer |
Responses
Example response 200
{
"deleted": false
}
listEvalCases
GET /api/v2/evals/case-sets/{id}/cases
Requires: invoke role
Page the cases of one set in creation order. cursor is the opaque position of the previous page's last case; a null next_cursor is the last page. limit defaults to 50, at most 500.
Generated Python client internals: atfgraph._generated.api.evaluations_api.EvaluationsApi.list_eval_cases()
Parameters
| Parameter | In | Required | Type |
|---|
id | path | yes | string |
cursor | query | no | string |
limit | query | no | integer |
Responses
Example response 200
{
"items": [
{
"assertions": [
{
"path": "string",
"type": {},
"value": {}
}
],
"case_set_id": "string",
"created_at": 0,
"id": "string",
"input": {},
"name": "string",
"revision": 0,
"tags": [
"string"
],
"updated_at": 0
}
],
"next_cursor": "string"
}
createEvalCase
POST /api/v2/evals/case-sets/{id}/cases
Requires: invoke role
Add a case to a set: a name, the capability input (canonical JSON at most 65536 bytes), at most 64 validated assertions and at most 16 tags. An empty assertion list is the implicit `status: succeeded`. Sets hold at most 500 cases (422 eval_set_full). Regex patterns and JSON Schemas are compiled on write so a malformed assertion is a 400 invalid_input here rather than a verdict later.
Generated Python client internals: atfgraph._generated.api.evaluations_api.EvaluationsApi.create_eval_case()
Parameters
| Parameter | In | Required | Type |
|---|
id | path | yes | string |
Request body
EvalCaseInput (required)
{
"input": {},
"name": "string"
}
Responses
Example response 201
{
"assertions": [
{
"path": "string",
"type": "equals",
"value": {}
}
],
"case_set_id": "string",
"created_at": 0,
"id": "string",
"input": {},
"name": "string",
"revision": 0,
"tags": [
"string"
],
"updated_at": 0
}
startEvalRun
POST /api/v2/evals/case-sets/{id}/run
Requires: invoke role
Start a run of the set (or of case_ids) against the requested version, the set's version, or the newest registered version, in that order of preference; the resolved immutable version is recorded on the run. The agent and capability are resolved before anything is written (404 agent_not_found / capability_not_found). Then one root traverser per case is admitted through the ordinary submit path with idempotency key `eval:<run_id>:<case_id>`, so access policies, capacity and the worker protocol apply unchanged; an admission fault marks that case errored immediately. Returns 202 with the run in status running; poll getEvalRun, or read the traversers directly.
Generated Python client internals: atfgraph._generated.api.evaluations_api.EvaluationsApi.start_eval_run()
Parameters
| Parameter | In | Required | Type |
|---|
id | path | yes | string |
Request body
EvalRunRequest (required)
{
"case_ids": [
"string"
],
"label": "string",
"version": "string"
}
Responses
Example response 202
{
"case_set_id": "string",
"cost_complete": false,
"cost_usd": 0.0,
"created_at": 0,
"duration_ms_p50": 0,
"errored": 0,
"failed": 0,
"finished_at": 0,
"id": "string",
"label": "string",
"pass_rate": 0.0,
"passed": 0,
"pending": 0,
"revision": 0,
"skipped": 0,
"started_at": 0,
"status": "string",
"target": "string",
"total": 0,
"total_tokens": 0,
"version": "string"
}
evalScores
GET /api/v2/evals/case-sets/{id}/scores
Requires: invoke role
Finished runs of one set aggregated per version, newest version first: run and case counts, verdict counts, pass_rate, the median duration over every settled result, token and cost totals (cost_usd null unless some result was priced; cost_complete when every run was fully priced), and the last run's id and finish time. Running runs are settled first and excluded.
Generated Python client internals: atfgraph._generated.api.evaluations_api.EvaluationsApi.eval_scores()
Parameters
| Parameter | In | Required | Type |
|---|
id | path | yes | string |
Responses
Example response 200
{
"case_set_id": "string",
"items": [
{
"cases": 0,
"cost_complete": false,
"cost_usd": 0.0,
"duration_ms_p50": 0,
"errored": 0,
"failed": 0,
"last_finished_at": 0,
"last_run_id": "string",
"pass_rate": 0.0,
"passed": 0,
"runs": 0,
"skipped": 0,
"total_tokens": 0,
"version": "string"
}
]
}
updateEvalCase
PATCH /api/v2/evals/cases/{id}
Requires: invoke role
Compare-and-set edit of name, input, assertions and tags. expected_revision must equal the current revision (409 revision_mismatch); unknown fields are rejected. Runs already started keep the assertions they were started with.
Generated Python client internals: atfgraph._generated.api.evaluations_api.EvaluationsApi.update_eval_case()
Parameters
| Parameter | In | Required | Type |
|---|
id | path | yes | string |
Request body
EvalCaseUpdate (required)
{
"expected_revision": 0
}
Responses
Example response 200
{
"assertions": [
{
"path": "string",
"type": "equals",
"value": {}
}
],
"case_set_id": "string",
"created_at": 0,
"id": "string",
"input": {},
"name": "string",
"revision": 0,
"tags": [
"string"
],
"updated_at": 0
}
deleteEvalCase
DELETE /api/v2/evals/cases/{id}
Requires: invoke role
Delete one case. Results it produced in earlier runs are kept. expected_revision, when given, must match (409 revision_mismatch). A case of another principal's set is 404 eval_case_not_found.
Generated Python client internals: atfgraph._generated.api.evaluations_api.EvaluationsApi.delete_eval_case()
Parameters
| Parameter | In | Required | Type |
|---|
id | path | yes | string |
expected_revision | query | no | integer |
Responses
Example response 200
{
"deleted": false
}
listEvalRuns
GET /api/v2/evals/runs
Requires: invoke role
Runs visible to the caller, newest first, optionally restricted to case_set_id (404 for a foreign or unknown set). Running runs are settled before the page is read, so counters reflect every traverser that has ended. Pass next_cursor back as cursor.
Generated Python client internals: atfgraph._generated.api.evaluations_api.EvaluationsApi.list_eval_runs()
Parameters
| Parameter | In | Required | Type |
|---|
case_set_id | query | no | string |
cursor | query | no | string |
limit | query | no | integer |
Responses
Example response 200
{
"items": [
{
"case_set_id": "string",
"cost_complete": false,
"cost_usd": 0.0,
"created_at": 0,
"duration_ms_p50": 0,
"errored": 0,
"failed": 0,
"finished_at": 0,
"id": "string",
"label": "string",
"pass_rate": 0.0,
"passed": 0,
"pending": 0,
"revision": 0,
"skipped": 0,
"started_at": 0,
"status": "string",
"target": "string",
"total": 0,
"total_tokens": 0,
"version": "string"
}
],
"next_cursor": "string"
}
getEvalRun
GET /api/v2/evals/runs/{id}
Requires: invoke role
One run with its per-case results, settled first: every pending case whose traverser has ended is evaluated (assertions, duration, model usage from the traverser's usage rows) and the run counters are recomputed. When no case is pending the run is failed if any case failed or errored, otherwise succeeded. Results are ordered by case name.
Generated Python client internals: atfgraph._generated.api.evaluations_api.EvaluationsApi.get_eval_run()
Parameters
| Parameter | In | Required | Type |
|---|
id | path | yes | string |
Responses
Example response 200
{
"results": [
{
"assertions": [
{
"actual": {},
"message": "string",
"passed": false,
"path": "string",
"type": {}
}
],
"case_id": "string",
"case_name": "string",
"cost_usd": 0.0,
"duration_ms": 0,
"error": {},
"finished_at": 0,
"output": {},
"run_id": "string",
"status": "string",
"total_tokens": 0,
"traverser_id": "string"
}
],
"run": {
"case_set_id": "string",
"cost_complete": false,
"cost_usd": 0.0,
"created_at": 0,
"duration_ms_p50": 0,
"errored": 0,
"failed": 0,
"finished_at": 0,
"id": "string",
"label": "string",
"pass_rate": 0.0,
"passed": 0,
"pending": 0,
"revision": 0,
"skipped": 0,
"started_at": 0,
"status": "string",
"target": "string",
"total": 0,
"total_tokens": 0,
"version": "string"
}
}
cancelEvalRun
POST /api/v2/evals/runs/{id}/cancel
Requires: invoke role
Cancel a running run: its pending results become skipped, the run becomes cancelled with finished_at set, and every pending traverser is cancelled through the ordinary cancel path (unfinished descendants included). Cancelling a run that already ended is a no-op that returns the run.
Generated Python client internals: atfgraph._generated.api.evaluations_api.EvaluationsApi.cancel_eval_run()
Parameters
| Parameter | In | Required | Type |
|---|
id | path | yes | string |
Responses
Example response 200
{
"case_set_id": "string",
"cost_complete": false,
"cost_usd": 0.0,
"created_at": 0,
"duration_ms_p50": 0,
"errored": 0,
"failed": 0,
"finished_at": 0,
"id": "string",
"label": "string",
"pass_rate": 0.0,
"passed": 0,
"pending": 0,
"revision": 0,
"skipped": 0,
"started_at": 0,
"status": "string",
"target": "string",
"total": 0,
"total_tokens": 0,
"version": "string"
}