Registration benchmark

Directory examples/benchmarks/scale_100k

Use this synthetic benchmark to measure local Python Agent construction, handler registration and memory use. It does not measure server throughput or predict production capacity. The directory replaces the older benchmarks/100k-scale name with an importable module name.

Commands

Taken from this example's own README and module docstrings.

python -m examples.benchmarks.scale_100k.analyze
python -m examples.benchmarks.scale_100k.analyze                    # table + results/benchmark_summary.png
python -m examples.benchmarks.scale_100k.crewai_bench.benchmark --tools 1000 --json
python -m examples.benchmarks.scale_100k.langchain_bench.benchmark --tools 1000 --json
python -m examples.benchmarks.scale_100k.python_bench.benchmark --handlers 5000 --iterations 10 --warmup 2 --json
python -m examples.benchmarks.scale_100k.python_bench.benchmark --handlers 5000 --json > results/ATFGraph_Python.json
python -m examples.benchmarks.scale_100k.python_bench.benchmark --handlers 5000 --output examples/benchmarks/scale_100k/results/ATFGraph_Python.json

Files

examples/benchmarks/scale_100k/README.md
examples/benchmarks/scale_100k/__init__.py
examples/benchmarks/scale_100k/analyze.py
examples/benchmarks/scale_100k/common.py
examples/benchmarks/scale_100k/crewai_bench/__init__.py
examples/benchmarks/scale_100k/crewai_bench/benchmark.py
examples/benchmarks/scale_100k/langchain_bench/__init__.py
examples/benchmarks/scale_100k/langchain_bench/benchmark.py
examples/benchmarks/scale_100k/python_bench/__init__.py
examples/benchmarks/scale_100k/python_bench/benchmark.py
examples/benchmarks/scale_100k/requirements.txt
examples/benchmarks/scale_100k/run_benchmarks.sh

README

Rendered from examples/benchmarks/scale_100k/README.md; the source file is not modified.

Run from the repository root with the SDK installed:

examples/benchmarks/scale_100k/run_benchmarks.sh --handlers 5000 --iterations 10 --warmup 2 --json
# or
python -m examples.benchmarks.scale_100k.python_bench.benchmark --handlers 5000 --output examples/benchmarks/scale_100k/results/ATFGraph_Python.json
python -m examples.benchmarks.scale_100k.analyze

Check setup with a small run such as --handlers 500 --iterations 2 --warmup 1 --requests 2000. Compare measurements on the same machine with the same toolchain/settings, and save the JSON alongside those details. Historical results from the previous runtime are excluded.

What is measured

Metric	How
`agent_init_time_*_ms`	`Agent(node_id, graph_server=...)` alone; nothing is registered with a server until it serves
`registration_time_*_ms`, `registration_time_per_handler_ms`	`@agent.traversal(name)` for N handlers on an already-constructed Agent; the cost is dominated by deriving each handler's JSON input/output schema
`agent_memory_mean_mb`, `memory_per_handler_bytes`	`tracemalloc` after a forced GC, Agent and handlers measured separately
`cold_start_*_ms`	Agent plus one handler
`request_latency_*_us`, `theoretical_single_thread_rps`	the raw `await handler(...)` call, as the original measured
`local_call_latency_*_us`	`agent.call_local(...)`: JSON input coercion, handler, output encoding, still no server

Run analyze.py to compare suites in results/. If matplotlib is installed, it also writes results/benchmark_summary.png.

Optional baselines

Optional langchain_bench/ and crewai_bench/ suites construct comparable tool counts and use the same JSON result format. Install their respective requirements (langchain-core or crewai). They make no model calls and report missing libraries rather than substituting results.

Differences from the original

Construct Agents with graph_server=; construction itself does not register them.
Handlers use @agent.traversal(f"handler_{j}").
agent.call_local measures SDK argument binding separately from a raw function call.
Historical Go/TypeScript SDK benchmarks, the Mastra baseline and previous-runtime result files were not carried over. Current generated clients have a different purpose and are not measured here.

The migrated-operations test suite runs small benchmark counts and checks that the analyzer reads their output. That validates the benchmark flow, not large-scale production performance.