Two nodes on Databricks

Directory examples/databricks/two_node

This example separates intake and pricing into two Python workers that can run as Databricks Apps. Intake extracts a travel request with a model and calls the pricing skill for a deterministic quote. Its scheduled probe submits a fixed request every fifteen minutes. Both workers connect outbound to the graph server; their App ports serve status pages.

Agent node label pricing, version 1.0.0.

Capabilities

Non-default capability options: probe: timeout_seconds=180; triage: timeout_seconds=180

Commands

Taken from this example's own README and module docstrings.

python -m atfgraph.worker --module examples.databricks.two_node.intake.agent:app
python -m atfgraph.worker --module examples.databricks.two_node.pricing.agent:app
python -m examples.databricks.two_node.call "Two people, four nights in Lisbon"
python -m examples.python.features.resilience.fixture_model --port 18899

Files

• examples/databricks/two_node/README.md
• examples/databricks/two_node/__init__.py
• examples/databricks/two_node/call.py
• examples/databricks/two_node/intake/__init__.py
• examples/databricks/two_node/intake/agent.py
• examples/databricks/two_node/intake/agentnode.yaml
• examples/databricks/two_node/intake/databricks.yml
• examples/databricks/two_node/intake/requirements.txt
• examples/databricks/two_node/pricing/__init__.py
• examples/databricks/two_node/pricing/agent.py
• examples/databricks/two_node/pricing/agentnode.yaml
• examples/databricks/two_node/pricing/databricks.yml
• examples/databricks/two_node/pricing/requirements.txt

README

Rendered from examples/databricks/two_node/README.md; the source file is not modified.

This example separates intake and pricing into two Python workers that can run as Databricks Apps. Intake extracts a travel request with a model and calls the pricing skill for a deterministic quote. Its scheduled probe submits a fixed request every fifteen minutes. Both workers connect outbound to the graph server; their App ports serve status pages.

                 +----------------------+   edge    +----------------------+
  request -----> | intake.triage        | --------> | pricing.quote        |
  (or emitter)   | traversal . AGENT    |           | skill . TOOL         |
                 +----------------------+           +----------------------+
                   app atfgraph-intake                app atfgraph-pricing
                            |   register, claim, renew leases, sync emitters  |
                            +------------->  graph server  <-----------------+
                                       (Compose / Kubernetes / Helm)

Each directory contains node code, an install/package manifest, a Databricks bundle and dependencies: agent.py, agentnode.yaml, databricks.yml and requirements.txt.

Run the same two nodes locally

The same node code runs locally. Use a graph server and an OpenAI-compatible provider; the bundled fixture provider lets you check wiring offline:

python -m examples.python.features.resilience.fixture_model --port 18899
export ATFGRAPH_MODEL_URL=http://127.0.0.1:18899 ATFGRAPH_MODEL_NAME=fixture-fast
target/debug/atfgraphd
python -m atfgraph.worker --module examples.databricks.two_node.pricing.agent:app
python -m atfgraph.worker --module examples.databricks.two_node.intake.agent:app
python -m examples.databricks.two_node.call "Two people, four nights in Lisbon"

With a real model, a correctly extracted request should produce a result of this shape:

{"status": "quoted", "destination": "Lisbon", "nights": 4, "travellers": 2, "currency": "EUR", "nightly_rate": 140, "total": 1120}

The path should contain intake.triage followed by pricing.quote. Fixture output instead uses schema-derived values: fixture destination, two nights, two travellers and total 520 at the default rate. It does not verify natural-language extraction. A low-confidence model result stops with needs_review; a quote below one night raises ValueError.

For local tracing, install the MLflow extra and enable ATFGRAPH_MLFLOW_TRACING=true on both workers. Use the same experiment and tracking URI. Set ATFGRAPH_MLFLOW_LINK_CHILDREN=metadata to join traces across the two worker processes; the default links children only within a process. Expect Agent, model, child-call and pricing-tool spans. See the MLflow guide for details.

Before you deploy

Prepare a server reachable from the workspace over HTTPS, its API key in a secret scope, and an intake model credential as required by the chosen preset. For MLflow, install mlflow>=3.11,<4 in both Apps and give their service principals CAN_EDIT on the selected experiment (default /Shared/atfgraph/two-node). See the deployment guide and neighboring presets.

Build, validate and deploy the two Apps

# 1. Build both bundles locally (wheel, node package, requirements, entrypoint checks; no workspace)
scripts/product/build-databricks-bundle.sh --node-dir examples/databricks/two_node/pricing \
    --package pricing --module pricing.agent:app --label pricing
scripts/product/build-databricks-bundle.sh --node-dir examples/databricks/two_node/intake \
    --package intake --module intake.agent:app --label intake

# 2. Put each node's own bundle file in place (the build script writes the generic template)
cp examples/databricks/two_node/pricing/databricks.yml target/dist/databricks/pricing/databricks.yml
cp examples/databricks/two_node/intake/databricks.yml  target/dist/databricks/intake/databricks.yml
.venv/bin/python scripts/product/check_databricks_bundle.py target/dist/databricks/pricing
.venv/bin/python scripts/product/check_databricks_bundle.py target/dist/databricks/intake

# 3. Validate, deploy and start each App with an explicit profile (nothing here picks one)
(cd target/dist/databricks/pricing && \
  databricks bundle validate -t dev --profile <PROFILE> --var server_url=https://<graph-server> --var api_key_scope=atfgraph && \
  databricks bundle deploy   -t dev --profile <PROFILE> --var server_url=https://<graph-server> --var api_key_scope=atfgraph && \
  databricks bundle run agent_node -t dev --profile <PROFILE>)
(cd target/dist/databricks/intake && \
  databricks bundle validate -t dev --profile <PROFILE> --var server_url=https://<graph-server> --var api_key_scope=atfgraph \
    --var model_url=https://<workspace-host>/serving-endpoints --var model_name=<serving-endpoint> --var model_key_scope=atfgraph && \
  databricks bundle deploy   -t dev --profile <PROFILE> --var ... && \
  databricks bundle run agent_node -t dev --profile <PROFILE>)

The Apps do not exchange callback URLs: both contact the graph server. Check each status page for registered: true, then verify workers can claim fresh work rather than relying on registration alone.

Walk the edge

export ATFGRAPH_SERVER=https://<graph-server> ATFGRAPH_API_KEY=<the key in the secret scope>
python -m examples.databricks.two_node.call "Two people, four nights in Lisbon"

What to look at

1. In the Console, inspect the two-step run and the probe emitter's next due time.
2. If tracing is enabled, inspect model, traversal and pricing spans in MLflow. Cross-process joining requires metadata linking on both workers.
3. Wait for a scheduled intake.probe run and verify that it calls pricing with the fixed request.
4. Read /api/v2/traversers/<id>/path. Use identity operations separately if you want to sign and verify it; reading a path does not sign it automatically.

Differences from the original

• Workers replace inbound node servers and callback URL exchange. The shared entrypoint uses the App port only for status.
• These bundles use secret-bound ATFGRAPH_API_KEY and model credentials. They do not demonstrate all current SDK authentication options.
• Model settings use ATFGRAPH_MODEL_URL, ATFGRAPH_MODEL_NAME and ATFGRAPH_MODEL_API_KEY rather than litellm prefixes.
• MLflow tracing is opt-in through the environment rather than the old mlflow_autolog flag.
• The graph server itself can also deploy as a Databricks App; Compose, Kubernetes and Helm are other supported targets.
• The example's bundles were built and checked offline. Databricks bundle validation, deployment and live execution were not verified.

The migrated-operations suite runs both nodes against a real local daemon with a fixture model and checks bundle files offline. It does not establish a live Databricks deployment.