Multimodal input
Use five capabilities to try image understanding, speech generation/transcription, image generation and streaming. The media-parity tests cover their request/response handling with fixtures; live providers require their own configuration.
Capabilities
| Capability | Kind | Signature | Module | Summary |
|---|---|---|---|---|
describe | traversal | (image_url: str, question: str = 'What is in this image?') -> dict | examples/python/features/multimodal/agent.py | Vision: image content part + question, structured answer, usage recorded on the execution. |
narrate | traversal | (text: str, voice: str = 'alloy', model: Optional[str] = None, format: str = 'mp3') -> dict | examples/python/features/multimodal/agent.py | Text to speech via the media router; the audio is saved under ATFGRAPH_MEDIA_DIR. |
narrate_stream | traversal | (text: str) -> dict | examples/python/features/multimodal/agent.py | Streaming text completion: deltas are forwarded as progress notes, the final text is returned. |
poster | traversal | (prompt: str, model: Optional[str] = None, size: str = '1024x1024') -> dict | examples/python/features/multimodal/agent.py | Image generation; inline results are saved, URL results are returned as-is (no implicit download). |
transcribe | traversal | (audio_b64: str, format: str = 'wav', model: Optional[str] = None, language: Optional[str] = None) -> dict | examples/python/features/multimodal/agent.py | Speech to text: inline audio is uploaded as multipart (OpenAI) or a data: URL (fal). |
Commands
python -m atfgraph.worker --module examples.python.features.multimodal.agent:app
python -m examples.python.features.multimodal.callFiles
examples/python/features/multimodal/README.mdexamples/python/features/multimodal/__init__.pyexamples/python/features/multimodal/agent.pyexamples/python/features/multimodal/agentnode.yamlexamples/python/features/multimodal/call.py
README
Use five capabilities to try image understanding, speech generation/transcription, image generation and streaming. The media-parity tests cover their request/response handling with fixtures; live providers require their own configuration.
| Traversal | Surface | Notes |
|---|---|---|
describe(image_url, question) | app.ai(text, Image.from_url(...), schema=Findings) | image content part + structured output; Anthropic gets an image block, OpenAI-compatible gets image_url |
narrate(text, voice, model, format) | app.ai_generate_audio | tts-1 default; minimax/speech-*, fal-ai/..., openrouter/... by prefix; inline audio saved under ATFGRAPH_MEDIA_DIR |
transcribe(audio_b64, format, model) | app.ai_transcribe_audio | multipart upload to /audio/transcriptions (whisper-1), or data: URL to fal-ai/whisper |
poster(prompt, model, size) | app.ai_generate_image | dall-e-3/gpt-image-1, openrouter/google/..., fal-ai/flux/..., minimax/image-01; URL outputs are never downloaded implicitly |
narrate_stream(text) | app.ai(..., stream=True) | SSE deltas forwarded as app.note progress events |
Agent media helpers record usage progress events with per-call data and execution totals; the driver reads them using UsageEvent.from_event. Known cost can come from provider billing fields or a configured price table, labeled by source. Missing prices remain None, and calls_without_price identifies incomplete totals. Provider fields vary, so token/timing data alone does not establish a charge.
python -m atfgraph.worker --module examples.python.features.multimodal.agent:app
python -m examples.python.features.multimodal.call