Resilience verification suite

Directory examples/resilience

Run sixteen failure scenarios against a real graph server with fixture workers and a local mock model endpoint. They cover backend errors/stalls, capacity, worker crashes, cancellation and timeout. No external provider or credential is needed; communication stays on loopback. These replace the older issue-316 resilience checks.

Agent node label test-slow, version 1.0.0.

Capabilities

Non-default capability options: timeout_then_fail: timeout_seconds=5; ask: max_attempts=2, timeout_seconds=4; bounded_task: timeout_seconds=3; slow_task: timeout_seconds=600

Commands

Taken from this example's own README and module docstrings.

python -c 'import asyncio; from atfgraph import Client, TraversalError
python -m atfgraph.worker --module examples.resilience.agent_flaky:app
python -m atfgraph.worker --module examples.resilience.agent_healthy:app
python -m atfgraph.worker --module examples.resilience.agent_slow:app
python -m examples.resilience.mock_llm_server --port 14000
python -m examples.resilience.run_checks
python -m examples.resilience.run_checks            # temporary daemon + mock + 3 workers, then teardown
python -m examples.resilience.run_checks --server http://127.0.0.1:18880 --mock-url http://127.0.0.1:14000
python -m examples.resilience.run_checks --server http://127.0.0.1:18880 --mock-url http://127.0.0.1:14000 --no-workers

Files

• examples/resilience/README.md
• examples/resilience/__init__.py
• examples/resilience/agent_flaky.py
• examples/resilience/agent_healthy.py
• examples/resilience/agent_slow.py
• examples/resilience/mock_llm_server.py
• examples/resilience/run_checks.py

README

Rendered from examples/resilience/README.md; the source file is not modified.

Run sixteen failure scenarios against a real graph server with fixture workers and a local mock model endpoint. They cover backend errors/stalls, capacity, worker crashes, cancellation and timeout. No external provider or credential is needed; communication stays on loopback. These replace the older issue-316 resilience checks.

python -m examples.resilience.run_checks            # temporary daemon + mock + 3 workers, then teardown

  PASS basic execution
  PASS model backend healthy
  PASS backend failure detection: status=failed attempts=2 of 2
  PASS failure classification: code=retryable message='model backend failed on attempt 2: model endpoint returned HTTP 500 ...'
  PASS backend recovery
  PASS concurrency limits: peak running=3 (capacity 3), queued observed=True, all completed=True
  PASS node visibility
  PASS worker kill / restart: while dead: status stayed 'queued' ...; after restart: the same queued run succeeded
  PASS event stream: 3 events: ['submitted', 'attempt.started', 'succeeded']
  PASS backend freeze (issue #316): status=failed code=handler_timeout after 5.4s
  PASS post-recovery execution
  PASS crash visibility: code=RuntimeError message='Agent crashed: test crash'
  PASS error details in records
  PASS client timeout: WaitTimeout after 2 s, then cancel -> cancelled
  PASS budget enforcement: status=failed code=handler_timeout after 4.2s
  PASS worker death mid-run: status=failed code=interrupted ...
passed 16, failed 0

What is checked, and which vNext mechanism carries it

The runner also points the daemon's model-health monitor at the mock endpoint and prints its startup confirmation. Current servers expose model-health state through /api/v2/model-health; these scenarios primarily check worker retries and timeout outcomes rather than proving every monitor/admission behavior.

Pieces

Manual use

python -m examples.resilience.mock_llm_server --port 14000
export ATFGRAPH_MODEL_URL=http://127.0.0.1:14000/v1 ATFGRAPH_MODEL_NAME=mock-model
target/debug/atfgraphd
python -m atfgraph.worker --module examples.resilience.agent_healthy:app
python -m atfgraph.worker --module examples.resilience.agent_slow:app
python -m atfgraph.worker --module examples.resilience.agent_flaky:app

# Freeze the backend (the #316 scenario), watch a run hit its budget, thaw it:
curl -X POST http://127.0.0.1:14000/freeze
python -c 'import asyncio; from atfgraph import Client, TraversalError
async def m():
    async with Client("http://127.0.0.1:18880") as c:
        try: print(await c.traverse("test-healthy.ask", {"prompt": "hi"}, timeout=60))
        except TraversalError as e: print(e.status, e.code, e.error)
asyncio.run(m())'
curl -X POST http://127.0.0.1:14000/unfreeze

# Run the checks against what you started (skips the two kill/restart checks):
python -m examples.resilience.run_checks --server http://127.0.0.1:18880 --mock-url http://127.0.0.1:14000 --no-workers

Differences from the original, and open gaps

• Worker capacity queues excess work rather than applying the parent's concurrency rejection behavior. Other configured rate/access rules can still refuse admission. Caller limits are separate.
• Inspect /api/v2/workers for the current fleet, and use a real capability call to verify service. Registration alone does not show a worker can execute work.
• Work without a serving worker remains queued until a worker returns or the run is cancelled; its execution deadline starts at the first claim.
• Errors use vNext envelopes such as retryable, handler_timeout, timed_out, interrupted and cancelled.
• The formerly blocking flaky task now awaits, and the Python runner replaces the old shell script.
• The migrated-operations suite exercises these checks with a real local daemon. It does not verify live providers or every model-health circuit setting.