Retry & recovery

Directory examples/python/features/resilience

Exercise retries, timeouts and cancellation with a local fixture provider. The scenarios require no external supplier, model credential or non-loopback service.

Agent node label disruption, version 1.0.0.

Capabilities

Capability	Kind	Signature	Module	Summary
`rebooking_note`	skill	`(flight: str, model: str = FLAKY_MODEL) -> dict`	`examples/python/features/resilience/agent.py`	Ask the provider for a note; a provider failure becomes another server attempt.
`stalled_note`	skill	`(flight: str) -> dict`	`examples/python/features/resilience/agent.py`	Deliberately outrun the capability's budget against the stalling fixture model.
`watch`	traversal	`(flight: str, seconds: float = 30.0, slice_seconds: float = 0.1) -> dict`	`examples/python/features/resilience/agent.py`	Poll for a rebooking window until it is cancelled, reporting progress as it goes.

Non-default capability options: rebooking_note: max_attempts=3; stalled_note: timeout_seconds=2; watch: timeout_seconds=600

Commands

Taken from this example's own README and module docstrings.

python -m atfgraph.worker --module examples.python.features.resilience.agent:app
python -m examples.python.features.resilience.call
python -m examples.python.features.resilience.fixture_model --port 18899

Files

examples/python/features/resilience/README.md
examples/python/features/resilience/__init__.py
examples/python/features/resilience/agent.py
examples/python/features/resilience/agentnode.yaml
examples/python/features/resilience/call.py
examples/python/features/resilience/fixture_model.py

README

Rendered from examples/python/features/resilience/README.md; the source file is not modified.

Exercise retries, timeouts and cancellation with a local fixture provider. The scenarios require no external supplier, model credential or non-loopback service.

python -m examples.python.features.resilience.fixture_model --port 18899
export ATFGRAPH_MODEL_URL=http://127.0.0.1:18899 ATFGRAPH_MODEL_NAME=fixture-fast
target/debug/atfgraphd
python -m atfgraph.worker --module examples.python.features.resilience.agent:app
python -m examples.python.features.resilience.call

The local /chat/completions fixture selects behavior by model name: fast replies, two initial 503s per user message, or a stall. Schema requests receive schema-derived values such as fixture <field>, 2, true and the first enum. This verifies error handling rather than model reasoning or travel data.

What each capability shows

Capability	Declared	Outcome
`rebooking_note`	`max_attempts=3`	attempts 1 and 2 raise `RetryableError`, the server requeues, attempt 3 returns
`stalled_note`	`timeout_seconds=2`	the budget ends the call: `failed`/`handler_timeout`
`watch`	`timeout_seconds=600`	`cancel()` stops the poll between slices: `cancelled`

The server retries only retryable envelopes, within max_attempts. Use RetryableError or HandlerError(..., retryable=True) for recoverable failures; ordinary exceptions fail terminally. Each attempt has a fresh lease and attempt number. The driver prints attempt.started and requeued events so you can inspect the sequence.

The worker enforces handler timeout and reports failed with handler_timeout. Separately, the server tracks attempt deadlines and can mark a nonresponsive worker's run timed_out. The driver prints the outcome that occurred.

Cancellation reaches the worker at lease renewal, sets its context flag and cancels its task. Short awaits and app.cancel_requested() checks let a handler react promptly; blocking code cannot respond until it yields. A progress write racing with cancellation may return 409 cancelled.

The extended-feature tests run all three scenarios against a real daemon and a local fixture model on temporary ports. They do not establish live-provider reliability.