/\ ATFGraph /docs
README Downloads
Examples / ATFGraph
Framework reference

This article is retained from the existing framework docs. Self-hosting commands and server-admin access differ from the managed CLI workflow. Start with Get started for the installed product.

Retrieval quality

Directory examples/python/retrieval_quality

Compare answers using faithfulness, relevance, hallucination checks and configured principles. Quick mode uses one assessment per metric, standard adds critic/defender assessments, and thorough adds an independent judge. Live runs assess configured principles with domain-specific weights. Scores are judgments about supplied cases, not correctness guarantees.

Agent node label retrieval_quality, version 1.0.0.

Capabilities

CapabilityKindSignatureModuleSummary
evaluate_metrictraversal(metric: str, question: str, context: str, response: str, perspective: str = 'reviewer', principles: list[dict] | None = None) -> dictexamples/python/retrieval_quality/main.py
evaluate_rag_responsetraversal(question: str, context: str, response: str, mode: str = 'quick', domain: str = 'general', use_ml: bool = False) -> dictexamples/python/retrieval_quality/main.py

Commands

Taken from this example's own README and module docstrings.

ATFGRAPH_SERVER=http://127.0.0.1:18880 npm run dev -- --hostname 127.0.0.1
EXAMPLE_MODE=fixture python -m atfgraph.worker --module examples.python.retrieval_quality.main:app
atfg install ./examples/python/retrieval_quality
atfg run retrieval_quality
atfg server start
atfg traverse retrieval_quality.evaluate_rag_response --in '{"question": "When does the desk open?", "context": "The desk opens at 9am.", "response": "The desk opens at 9am.", "mode": "quick"}'
atfg traverse retrieval_quality.evaluate_rag_response --in '{"question":"When does the desk open?","context":"The desk opens at 9am.","response":"The desk opens at 9am.","mode":"quick"}'
docker build -f deploy/docker/Dockerfile.server -t atfgraph-vnext/graph-server:local .
docker compose -f examples/python/retrieval_quality/compose.yaml down
docker compose -f examples/python/retrieval_quality/compose.yaml up --build -d
python -m examples.python.retrieval_quality.call

Files

  • examples/python/retrieval_quality/Dockerfile
  • examples/python/retrieval_quality/Dockerfile.dockerignore
  • examples/python/retrieval_quality/README.md
  • examples/python/retrieval_quality/__init__.py
  • examples/python/retrieval_quality/agentnode.yaml
  • examples/python/retrieval_quality/call.py
  • examples/python/retrieval_quality/compose.yaml
  • examples/python/retrieval_quality/config/constitution.yaml
  • examples/python/retrieval_quality/config/presets/financial.yaml
  • examples/python/retrieval_quality/config/presets/legal.yaml
  • examples/python/retrieval_quality/config/presets/medical.yaml
  • examples/python/retrieval_quality/main.py
  • examples/python/retrieval_quality/ml_services/__init__.py
  • examples/python/retrieval_quality/ml_services/embeddings.py
  • examples/python/retrieval_quality/ml_services/ner.py
  • examples/python/retrieval_quality/ml_services/nli.py
  • examples/python/retrieval_quality/node.py
  • examples/python/retrieval_quality/requirements-ml.txt
  • examples/python/retrieval_quality/requirements.txt
  • examples/python/retrieval_quality/support.py
  • examples/python/retrieval_quality/ui/Dockerfile
  • examples/python/retrieval_quality/ui/app/api/graph/[...path]/route.ts
  • examples/python/retrieval_quality/ui/app/globals.css
  • examples/python/retrieval_quality/ui/app/layout.tsx
  • examples/python/retrieval_quality/ui/app/page.tsx
  • examples/python/retrieval_quality/ui/components/ClaimBreakdown.tsx
  • examples/python/retrieval_quality/ui/components/EvaluationForm.tsx
  • examples/python/retrieval_quality/ui/components/LoadingState.tsx
  • examples/python/retrieval_quality/ui/components/MarkdownContent.tsx
  • examples/python/retrieval_quality/ui/components/MetricCard.tsx
  • examples/python/retrieval_quality/ui/components/OverallScore.tsx
  • examples/python/retrieval_quality/ui/components/PoweredBy.tsx
  • examples/python/retrieval_quality/ui/components/eval/DashboardShell.tsx
  • examples/python/retrieval_quality/ui/components/eval/InputPanel.tsx
  • examples/python/retrieval_quality/ui/components/eval/ResultsPanel.tsx
  • examples/python/retrieval_quality/ui/components/ui/badge.tsx
  • examples/python/retrieval_quality/ui/components/ui/button.tsx
  • examples/python/retrieval_quality/ui/components/ui/card.tsx
  • examples/python/retrieval_quality/ui/components/ui/collapsible.tsx
  • examples/python/retrieval_quality/ui/components/ui/dropdown-menu.tsx
  • examples/python/retrieval_quality/ui/components/ui/input.tsx
  • examples/python/retrieval_quality/ui/components/ui/label.tsx
  • examples/python/retrieval_quality/ui/components/ui/popover.tsx
  • examples/python/retrieval_quality/ui/components/ui/progress.tsx
  • examples/python/retrieval_quality/ui/components/ui/scroll-area.tsx
  • examples/python/retrieval_quality/ui/components/ui/select.tsx
  • examples/python/retrieval_quality/ui/components/ui/separator.tsx
  • examples/python/retrieval_quality/ui/components/ui/tabs.tsx
  • examples/python/retrieval_quality/ui/components/ui/textarea.tsx
  • examples/python/retrieval_quality/ui/components.json
  • examples/python/retrieval_quality/ui/hooks/useEvaluation.ts
  • examples/python/retrieval_quality/ui/lib/api.ts
  • examples/python/retrieval_quality/ui/lib/presets.ts
  • examples/python/retrieval_quality/ui/lib/types.ts
  • examples/python/retrieval_quality/ui/lib/utils.ts
  • examples/python/retrieval_quality/ui/next.config.js
  • examples/python/retrieval_quality/ui/package-lock.json
  • examples/python/retrieval_quality/ui/package.json
  • examples/python/retrieval_quality/ui/postcss.config.js
  • examples/python/retrieval_quality/ui/public/atfgraph-icon-dark.svg
  • examples/python/retrieval_quality/ui/public/atfgraph-icon-light.svg
  • examples/python/retrieval_quality/ui/public/atfgraph-logo-dark.svg
  • examples/python/retrieval_quality/ui/public/atfgraph-logo-light.svg
  • examples/python/retrieval_quality/ui/scripts/smoke.py
  • examples/python/retrieval_quality/ui/tailwind.config.ts
  • examples/python/retrieval_quality/ui/tsconfig.json

README

Rendered from examples/python/retrieval_quality/README.md; the source file is not modified.

Compare answers using faithfulness, relevance, hallucination checks and configured principles. Quick mode uses one assessment per metric, standard adds critic/defender assessments, and thorough adds an independent judge. Live runs assess configured principles with domain-specific weights. Scores are judgments about supplied cases, not correctness guarantees.

Run locally

With the SDK installed, run from the repository root:

atfg server start
EXAMPLE_MODE=fixture python -m atfgraph.worker --module examples.python.retrieval_quality.main:app
# In a second terminal:
python -m examples.python.retrieval_quality.call

Or install and run the folder as a managed node package:

atfg install ./examples/python/retrieval_quality
atfg run retrieval_quality

A copied project also works through node.py. By default it uses http://127.0.0.1:18880; inspect its runs at /ui/.

Call with the CLI

atfg traverse retrieval_quality.evaluate_rag_response --in '{"question": "When does the desk open?", "context": "The desk opens at 9am.", "response": "The desk opens at 9am.", "mode": "quick"}'

Providers

Fixture mode uses deterministic local values with no external model/search calls. It checks the scoring workflow rather than model evaluator quality.

For live evaluations, set EXAMPLE_MODE=live and the worker's model URL/name/key before startup. Missing settings and provider errors fail explicitly rather than returning fixture judgments.

Python client

from atfgraph import Client

async with Client("http://127.0.0.1:18880") as client:
    result = await client.traverse(
        "retrieval_quality.evaluate_rag_response",
        {'question': 'When does the desk open?', 'context': 'The desk opens at 9am.', 'response': 'The desk opens at 9am.', 'mode': 'quick'},
        timeout=300,
    )
    print(result)

Evaluation dashboard

The optional Next.js dashboard submits traversers and displays results/events. Its server proxy holds the graph API key, while the worker selects the configured model. The page does not accept arbitrary provider models per request.

cd examples/python/retrieval_quality/ui
npm ci
ATFGRAPH_SERVER=http://127.0.0.1:18880 npm run dev -- --hostname 127.0.0.1
# Open http://127.0.0.1:3000

Use the dashboard on a trusted local machine. Its proxy acts under one configured server identity; add application user authentication and authorization before sharing it.

Docker Compose

Build the server image and choose a local API key. Stop any existing server using port 18880 before starting this stack:

docker build -f deploy/docker/Dockerfile.server -t atfgraph-vnext/graph-server:local .
export ATFGRAPH_API_KEY="$(python -c 'import secrets; print(secrets.token_urlsafe(32))')"
docker compose -f examples/python/retrieval_quality/compose.yaml up --build -d
atfg traverse retrieval_quality.evaluate_rag_response --in '{"question":"When does the desk open?","context":"The desk opens at 9am.","response":"The desk opens at 9am.","mode":"quick"}'
docker compose -f examples/python/retrieval_quality/compose.yaml down

Open the dashboard on 3000 and the server API/Console on 18880. The named Compose volume retains data across container replacement.

Local ML checks and principles

Edit config/constitution.yaml to define principles and weights. Its domain overrides are selected by the API's domain; neighboring domain preset files are examples rather than automatically active configuration.

For optional NLI checks, install requirements-ml.txt or build with INSTALL_ML_DEPS=true, then request use_ml=true. Model downloads happen on first use and need network access. NLI results remain separate from fixture/model scores. The NER and embedding utilities provide additional local experiments. Fixture mode needs none of these downloads. Evaluate the resulting diagnostics against labeled cases before using them for decisions.

ATFGraph documentationHosted workflow · Framework reference