Overview

Design voice-first assistants with the same Agency Swarm agents you already use.

Agency Swarm reuses your existing Agent definitions for voice. This page shows how to adapt agents for spoken conversations.

Try it locally

Save this as voice_demo.py. It starts the packaged browser client on your computer, so you can speak with your agent before you host anything.

from agency_swarm import Agency, Agent
from agency_swarm.ui.demos.realtime import RealtimeDemoLauncher

voice_agent = Agent(
    name="Voice Concierge",
    instructions="You are a friendly concierge. Keep answers brief.",
    voice="marin",
)

RealtimeDemoLauncher.start(Agency(voice_agent), host="127.0.0.1")

1. Install the voice extra: pip install "agency-swarm[voice]".
2. Set OPENAI_API_KEY, then run python voice_demo.py.
3. Open http://127.0.0.1:8000, click Connect, and allow microphone access.

When you are ready to host the browser client or connect phone calls, use the Deployment guide.

What you can build

• Phone receptionist — answers calls, routes to specialists, captures caller details.
• Live support triage — gathers context, lets callers interrupt, and escalates to a human or another agent.
• Language coach — listens, corrects pronunciation, and keeps the dialogue short and encouraging.

Prerequisites

• Access to a supported realtime provider:
  • OpenAI (gpt-realtime-2 is the Agency Swarm default; see OpenAI’s realtime model catalog for other models and current pricing)
  • xAI (grok-voice-think-fast-1.0 is the Agency Swarm default; see xAI’s Speech to Speech guide for current model availability)
• The voice extra:

pip install "agency-swarm[voice]"

Define your agent (same API)

You keep using the standard Agent class—voice agents are regular agents with the same tools, handoffs, and instructions.

from agency_swarm import Agent, function_tool

@function_tool
def lookup_order(order_id: str) -> str:
    """Return a short order status by ID."""
    return f"Order {order_id} has shipped and will arrive soon."

voice_agent = Agent(
    name="Voice Concierge",
    instructions=(
        "You are a friendly concierge. Answer in one or two sentences and offer to look up order "
        "details when the caller mentions a number."
    ),
    tools=[lookup_order],
    voice="marin",
)

Keep using the same agent definitions—add voice= only when you care about the spoken persona.

Set voice to any supported realtime token:

• OpenAI voices: alloy, ash, ballad, cedar, coral, echo, marin, sage, shimmer, verse
• xAI voices: ara, eve, leo, rex, sal

One voice per call

A call uses a single voice from start to finish. Once the model has spoken, the realtime API refuses to switch voices—OpenAI answers with a cannot_update_voice error. Agency Swarm therefore picks the voice as the call starts and keeps it, even when the call hands off to another agent.

The voice comes from the first place that sets one:

1. The voice= you pass to run_realtime() or the demo launcher.
2. The voice= on your entry agent—the agent that answers the call.
3. The provider’s own default when neither is set.

A voice= on any other agent is not used for that call. It is still worth setting: that agent uses its own voice whenever it is the one answering.

If you want variety without picking voices by hand, build the agency with randomize_agent_voices=True. Every agent without an explicit voice gets a deterministic random pick from the provider’s list, and the entry agent’s pick becomes the voice for the call. Pass voice_random_seed= to keep the picks stable between runs.

Add handoffs (optional)

Handoffs work exactly as they do in text mode. Register your flows once and they will carry over to voice sessions.

from agency_swarm import Agency, Agent, Handoff

billing = Agent(name="Billing", instructions="Handle billing questions briefly.")
faq = Agent(name="FAQ", instructions="Answer frequently asked questions.")

concierge = Agent(
    name="Concierge",
    instructions="Greet the caller, collect intent, then hand off when a specialist is needed.",
    voice="marin",
)

agency = Agency(
    concierge,
    communication_flows=[
        (concierge > billing, Handoff),
        (concierge > faq, Handoff),
    ],
)

When the concierge hands off, the specialist takes over the conversation with its own instructions and tools. The caller keeps hearing the concierge’s marin voice for the rest of the call, so give the specialists a different name or greeting if you want the switch to be obvious.

Next steps

• Try the realtime browser demo
• Deploy your agents for phone calls using Twilio.
• Review OpenAI’s realtime Quickstart and Guide for protocol details—Agency Swarm builds on those primitives.