Building voice agents with LiveKit and Rime

A practical Python walkthrough for building and deploying Nudge, a document follow-up agent with streaming speech and interruption recovery.

TL;DR

Nudge voice agent gives teams a deployable foundation for automating document follow-up while preserving human approval. Built with LiveKit and Rime, it combines natural, streaming speech with explicit workflow checks and interruption recovery. The modular reference implementation includes tests and latency measurements, helping engineering leaders assess behavior and integration effort before connecting real account data and business systems.

Introduction

I asked Nudge for a few more days to submit my documents. It recorded the date I requested, told me the request needed review, and asked if I needed anything else.

That sounds like a small interaction. But there is quite a bit happening inside it. The agent has to understand what I said, decide which action to take, preserve the date, and explain the result without accidentally promising approval. It also has to sound like someone I can comfortably have that conversation with.

I built Nudge at Rime while working on a real customer POC to work through those details. It runs on LiveKit Cloud and connects to a real phone number. For this walkthrough, it uses a fictional business account and records local request previews, so you can inspect the workflow before connecting your own backend.

In the accompanying video, you can hear the call and see the main implementation choices. Here, we'll build up the explanation from the beginning, run the project, and follow that extension request through the code. You'll need some familiarity with Python and a terminal. You don't need to have built a voice agent before.

Watch the video · Download the complete project

Start with a conversation worth having

Think about an application waiting on one last document. Someone has to follow up, explain what is missing, and deal with the response. The person might have already uploaded it. They might be stuck on the upload page. Or they might need until Friday.

Nudge handles that kind of follow-up. You could adapt the same flow to onboarding documents, insurance paperwork, or another process where a missing detail holds up the next step.

For our example, the caller represents Northstar Labs. The fictional account needs August bank statements and second-quarter financials, with an original deadline of September 22, 2026. We'll request an extension through September 25. These are fixed sample dates in the project, so update them when you adapt the example.

The useful outcome is a recorded request and a clear next step. Approval belongs to the reviewing team. That distinction will show up in the prompt, the tool, and the words the caller hears.

What happens between listening and speaking?

A voice agent needs a way to hear you, decide what to do, and speak back. Nudge uses separate components for those jobs.

• Deepgram Flux: Converts the caller's speech into text. This is speech-to-text, or STT.
• Gemini 3.1 Flash Lite: Uses the conversation and available tools to choose a response or request an action.
• Python workflow tools: Check the request, record its result, and provide the acknowledgment.
• Rime Coda, with Wawona: Turns the response text into speech. This is text-to-speech, or TTS.
• LiveKit: Connects the audio session, coordinates these components, and manages turns and interruptions.

For an extension request, the path looks like this:

Caller asks for more time
    → Deepgram transcribes the request
    → Gemini selects the extension tool
    → Python checks and records the request
    → Rime speaks the acknowledgment
    → LiveKit delivers the audio to the caller

This arrangement is often called a cascaded voice agent. The separate stages make it easier to inspect a problem. If the date was transcribed correctly but recorded incorrectly, you know to look after transcription. If the recorded date is right but sounds awkward when spoken, you can examine the text sent to TTS and listen to its output.

In this version, Deepgram, Gemini, and voice activity detection use LiveKit Inference. Rime connects directly through the LiveKit Rime plugin with a separate API key.

Voice activity detection, or VAD, detects speech in the incoming audio. It helps the agent notice when you start talking over it. Turn detection answers a different question: have you finished what you wanted to say? A pause halfway through a date should not make the agent rush in with a reply.

Get Nudge running on your laptop

You'll need uv to install the Python environment, a LiveKit Cloud project, and a Rime API key for the Coda endpoint. You can obtain a Rime key through the Rime dashboard. The project selects Python 3.14.6 locally and locks LiveKit Agents and the Rime plugin to version 1.8.3.

Download the source ZIP above and save it as nudge-source.zip. In the directory containing it, run:

unzip nudge-source.zip
cd nudge
cp .env.example .env.local

Open .env.local and fill in these four values with your credentials:

LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=your-livekit-api-key
LIVEKIT_API_SECRET=your-livekit-api-secret
RIME_API_KEY=your-rime-api-key

Keep the remaining settings from the template. They select the models, the Wawona voice, and the worker name nudge-demo. A worker is the running Python process that accepts a conversation. LiveKit uses its name to send the right agent into a call.

The two sets of credentials serve different connections. Your LiveKit credentials connect the worker to your project and its inference services. RIME_API_KEY authenticates the TTS connection. Both are needed here.

Install the locked dependencies and start a local microphone-and-speaker session:

uv sync --locked
uv run python -m nudge_voice_agent.main console

--locked keeps the install tied to the supplied dependency versions. Console mode lets you test the conversation before adding a browser client or a phone number. The inference and TTS requests still use their online services.

When Nudge asks which business you are calling about, say "Northstar Labs." Then try "I need until September 25, 2026 to upload the documents." It should record an extension request and tell you that it has not been approved.

Try a second conversation where you ask for more time without naming a date. Nudge should ask which date you want. We'll look at the code that makes that detail necessary in a moment.

To use a browser client instead, stop console mode and run:

uv run python -m nudge_voice_agent.main dev

Connect your client or LiveKit playground to the same project and explicitly dispatch nudge-demo. An empty room alone won't select this named worker. LiveKit's voice AI quickstart covers the client connection if this is your first session.

Find the parts you'll change

The project is split so you can change the conversation without hunting through audio plumbing.

• demo_context.py: Change the business, documents, deadline, or upload instructions.
• assistant.py: Change how Nudge handles the conversation and when it calls tools.
• workflow.py: Change what an action checks, records, and says.
• workflow_agent.py: Inspect how an interrupted acknowledgment takes priority on the next turn.
• runtime.py: Inspect the model connections and turn-handling settings.
• main.py: Follow the worker entrypoint and session startup.

An AgentSession holds the running conversation. Nudge's factory in runtime.py wires STT, the language model, and TTS into that session. The TTS part is small:

from livekit.agents import AgentSession
from livekit.plugins import rime

session = AgentSession(
    tts=rime.TTS(
        websocket_url="wss://api.rime.ai/coda/ws",
        speaker="wawona",
        lang="en",
    ),
)

This is the TTS configuration excerpt, not a second application to run. The supplied runtime.py also configures transcription, Gemini, voice activity detection, and turn handling. /coda/ws selects Coda; this configuration does not take an additional model argument. The plugin handles the connection. The endpoint-specific Coda integration reference describes this interface.

Keep the supplied lockfile when following this article. Older plugin versions may not accept websocket_url.

Turn "I need more time" into a recorded request

A tool gives the model a Python function it can ask the application to execute. For Nudge, the model selects the extension tool and supplies the caller's requested date. The function decides whether it can record that request.

The inbound flow first asks the caller to name the business. A tool compares that answer with our fictional record. Subsequent account actions check that the match succeeded. This is a demonstration access check; a business name is not sufficient authentication for access to real account data.

Here is the extension method from workflow.py, with its tool decorator omitted so we can focus on the action:

async def record_extension_request(
    self,
    ctx: RunContext,
    requested_submission_date: str,
) -> str | None:
    """Record an extension request for Underwriting review."""
    self._require_access()
    self._require_no_pending_result()
    requested_date = _required_extension_date(requested_submission_date)
    self.record_preview(
        ConversationDisposition.EXTENSION_REQUESTED,
        requested_submission_date=requested_date,
    )
    return await self._speak_required_result(
        ctx,
        f"I've recorded your extension request through {requested_date} for "
        "Underwriting to review. It is not approved yet. Do you need anything "
        "else?",
    )

First, it checks access. Then it checks whether an earlier action still has an acknowledgment waiting to be spoken. Finally, it validates the requested date before recording anything.

That date check came from a rehearsal. An extension request was recorded with "unknown" as the date. The prompt now tells the model to ask for the missing information, and the function rejects empty values and placeholders such as unknown and TBD. If the model still supplies one, the tool returns an error without recording the request.

The validation here checks that a date detail was supplied. It does not establish that the string is a valid calendar date or that the requested extension meets your business rules. Those checks belong in the integration you build around your own records.

You can inspect the payload without making a call or invoking a model:

uv run nudge-zendesk-preview \
  --disposition extension_requested \
  --request-id nudge-tutorial \
  --requested-submission-date 2026-09-25

These are the fields to look at in the output:

{
  "conversation_disposition": "extension_requested",
  "demo_only": true,
  "details": {
    "requested_submission_date": "2026-09-25"
  },
  "suggested_route": "Underwriting",
  "write_performed": false
}

The date is attached to an extension request, and the suggested review route is Underwriting. write_performed is false because this example creates a local preview. It does not write a Zendesk ticket or change an account deadline. The offline command shows the payload in isolation; the conversation tool separately enforces the access check.

In your application, replace the preview step with the backend operation. Speak a success acknowledgment only after that operation succeeds. If the write fails, the caller needs to hear that the request wasn't saved. A confident voice makes accurate status especially important.

Give the acknowledgment a voice

Some replies can be generated freely. An explanation of the upload steps can adapt to the caller's question. The result of an action needs more consistency.

For the extension request, Nudge constructs the acknowledgment in Python and passes it to session.say(). The date comes from the tool, and the review status is part of the fixed wording. The model doesn't have to rewrite that result before Rime speaks it.

Listen to Wawona's reply in the actual phone call:

Play the call excerpt

The phrasing is what I want you to pay attention to here. Hear the pauses around the date and the review status, then the change in delivery when Nudge asks whether I need anything else. Those small choices help an administrative conversation feel easier to follow.

When choosing a voice for your own agent, listen to the sentences it will actually say. Include names, dates, an explanation, and a question. A greeting tells you very little about how a voice handles an entire document follow-up call.

The stored record and the spoken wording can also differ without changing the underlying fact. Nudge keeps Q2 2026 in its data, for example, and expands it to "second quarter 2026" for speech. You can inspect that conversion in demo_context.py.

Stream the reply as it becomes available

There are two places to look when someone says their agent is "streaming."

On the text side, the language model produces a reply over time. The TTS integration can start receiving that text before the whole reply is ready. On the audio side, TTS produces audio chunks, which the application can forward while synthesis continues.

Nudge does both. For generated replies, LiveKit passes text into the Rime plugin as the model produces it. The pinned plugin buffers complete sentences locally and sends successive sentences into one Coda synthesis context. Audio comes back while later text is still arriving. It does not send every individual token straight to the server.

That shared context is useful for the delivery across a reply. Successive sentences belong to the same synthesis operation. The WebSocket connection can also stay open for later turns, but each turn gets its own synthesis context. Reusing the connection and preserving context within a reply are two different things.

We checked the playback behavior directly. In one warm-connection diagnostic, the first audio reached our local output at 234 ms, while synthesis completed at 3.65 seconds. That observation confirmed that the application was forwarding audio before synthesis finished. It is a separate diagnostic from the latency measurements below.

Our earlier LiveKit Inference implementation also streamed audio. Switching to the direct Coda endpoint did not turn streaming on for the first time, and we haven't established an old-versus-new latency improvement from that migration.

For this project, keep the plugin's native streaming path. Avoid adding an adapter that turns each sentence into a separate synthesis request, and avoid collecting the entire generated reply before passing it to TTS. The endpoint's streaming lifecycle reference explains how a context stays open as more text arrives.

The fixed acknowledgment we just heard already has complete text. It benefits from streaming audio output, but it has no LLM text generation to overlap. That overlap matters when Nudge generates a new reply.

Measure the pause the caller experiences

I wanted numbers alongside the listening example. We measured two different parts of the system on September 24, 2026.

• Wawona ready text to first audio. Median: 138 ms. P95: 146 ms. 20 sequential requests on a warmed connection, from complete text entering the plugin to the first PCM audio frame received by our client.
• Simulated conversation turn gap. Median: 1.68 s. P95: 3.02 s. 10 gaps from the end of the simulated caller's speech to the start of the agent's audio in a LiveKit room.

PCM is decoded audio data. Receiving its first frame tells us when audio becomes available to our client; it doesn't tell us when a person hears it through a phone.

The first check used five short replies, repeated four times at 24 kHz. All 20 warm requests succeeded. The slowest took 271 ms. A separate first request on a cold connection took 401 ms; one cold request is not enough to characterize cold-start behavior. P95 uses the nearest-rank method.

Those TTS timings include the plugin, network, and provider work. They exclude transcription, LLM generation, telephone transport, and audible playback. Each client issued one request at a time, although part of the run overlapped a separate check on the same account. Network and shared-service load were uncontrolled.

The room measurement covers more of the conversation. It still isn't a telephone measurement. Its ten gaps include a failed workflow scenario, so the timing results should be read alongside the behavior results in the next section. These are small integration checks, not a load test or a comparison with other providers.

You can inspect every first-audio measurement and download all the audio samples. To run the same first-audio check with your own key, use a new output directory:

uv run python evaluations/coda_first_audio_check.py \
  --output /tmp/nudge-first-audio-check

The separate input-streaming check compares sending scheduled text as it arrives with waiting for all of that text, using the same Coda endpoint and voice:

uv run python evaluations/coda_streaming_check.py \
  --output /tmp/nudge-input-streaming-check

In three pairs, the median time from the first scheduled text chunk to first received audio was 0.67 seconds when streaming and 3.03 seconds when waiting for all text. That shows the effect of overlapping work in this synthetic schedule. Its clock includes text arrival, so don't combine it with the 138 ms ready-text measurement.

For your actual calls, record when the caller stops, when the agent has usable text or a tool result, when TTS returns its first audio, and when playback reaches the listener. Then inspect a slow turn using those timestamps from the same call. An endpoint name or a fast TTS number alone won't explain the whole pause.

Let the caller interrupt without losing the result

Try interrupting Nudge halfway through its extension acknowledgment. The speech should stop. But the extension request has already been recorded.

There are two responsibilities here. LiveKit stops playback when it detects the interruption, and the Rime plugin cancels unfinished synthesis. The application then needs to remember which parts of the conversation still matter.

Nudge checks whether the acknowledgment finished playing:

async def _speak_tool_result(ctx: RunContext, message: str) -> bool:
    """Speak a grounded result and report whether its playout completed."""
    await ctx.wait_for_playout()
    speech = ctx.session.say(message, allow_interruptions=True)
    await speech.wait_for_playout()
    return not speech.interrupted

The first wait lets any preceding speech finish. session.say() starts the new speech. The second wait tells the application when playback has completed. It does not make playback wait for the entire TTS response to finish generating.

If the caller interrupts, the required result stays pending. On the next model turn, WorkflowAgent restricts the available tools to finishing that result or ending the call. Both paths handle the pending acknowledgment. Finishing it replays the required message; it does not record the extension again.

This matters if the caller heard "I've recorded your extension request" but interrupted before "It is not approved yet." Canceling speech must not silently erase that distinction.

The behavior also has a limit worth seeing. In one cloud rehearsal, three of four scenarios passed. In the failed scenario, an informational document reminder was interrupted, and Nudge answered the caller's new question without completing the missing documents and deadline. A repeat passed without a code change. Required action acknowledgments have the recovery mechanism described above; that mechanism does not guarantee recovery of every interrupted informational reply.

The project includes tests for missing dates, blocked account actions, interrupted acknowledgments, and repeated recovery. Run them locally:

uv run pytest --no-cov -q

These tests use playback doubles to check application behavior. The separate streaming check also verifies that canceling synthesis still allows the next turn to produce audio. Neither tells you how quickly a caller hears an interruption over a telephone connection. Listen to real calls as well.

Connect a phone number

Once the local conversation works, you can run the same worker on LiveKit Cloud. We'll use an inbound route so you can call your agent. The opening video call is outbound; reproducing that direction also needs an outbound SIP trunk and dispatch setup. The source keeps outbound calling disabled by default.

Install the LiveKit CLI, then run these commands from the extracted nudge directory:

lk cloud auth
lk project list
lk project set-default "YOUR_PROJECT_NAME"

Choose the same project used for your local rehearsal. The downloaded source excludes our cloud deployment ID. LiveKit will create a configuration for your deployment.

Pass the .env.local you already configured through the deployment's secrets mechanism. LiveKit supplies the hosted worker's LiveKit connection credentials automatically and imports the other populated settings, including your Rime key. The flag below skips the template's unused empty settings:

lk agent create --secrets-file .env.local --ignore-empty-secrets
lk agent status

The project includes a Dockerfile. The create command builds and deploys it; status lets you check that the worker is running. LiveKit's deployment quickstart explains the generated livekit.toml and subsequent deployments.

Next, connect an inbound number in the same LiveKit project. The phone-number setup walks through renting a LiveKit number and assigning it to a dispatch rule. If you already use a carrier, follow LiveKit's SIP trunk setup for that route.

A dispatch rule decides which room receives a call and which agent joins it. In the dashboard's dispatch-rule JSON editor, this is the configuration for our named worker:

{
  "name": "Nudge inbound",
  "rule": {
    "dispatchRuleIndividual": {
      "roomPrefix": "nudge-call-"
    }
  },
  "roomConfig": {
    "agents": [
      { "agentName": "nudge-demo" }
    ]
  }
}

Assign your number to that rule. Each incoming call gets a separate room, and the rule dispatches nudge-demo into it. LiveKit's dispatch-rule reference covers the dashboard and carrier-specific settings.

Stop your local dev worker before testing the cloud deployment with the same agent name. Then call your number, name the fictional business, and request an extension. Inspect the matching session's transcript and request preview. You should be able to follow the same date from what you said to what the tool recorded and what Nudge spoke.

If the call connects but the agent never joins, check the project and dispatch name first. If it transcribes your speech but produces no voice, inspect the Rime connection and key. If a request is recorded but its acknowledgment is cut short, inspect playback and pending-result recovery.

Make it useful for your workflow

You now have a running conversation, a tool whose output you can inspect, and a phone route you can call. Start adapting it by changing the fictional record and listening to the resulting conversation. Then connect a single backend action and test what Nudge says when that action succeeds or fails.

Keep the requested date and the approval status separate in that integration. Give repeated requests an idempotency key so a retry can't create duplicate work. Replace the business-name match with account authentication and authorization appropriate to your application.

We'll cover deployment and operating considerations in the next episode, including monitoring, failure handling, and what changes when real customer traffic arrives. For now, try the extension flow with your own wording. Interrupt it. Leave out the date. Ask whether it has actually been approved. Those conversations will tell you much more about your agent than a successful greeting.

‍