Building AI Voice Agents with Live Web Access

By Nicholas St. Germain — May 4, 2026

A voice agent that can't browse the web is an oracle frozen in time. Ask it about today's weather, the score of last night's game, or whether a flight is delayed, and the best it can do is apologize. To be genuinely useful, the agent has to reach out to live data - and it has to do it fast enough that the conversation doesn't stall.

This guide covers the architecture for adding real-time web access to a voice agent: SERP scraping for discovery, a web unlocker for fetching arbitrary pages, and the proxy layer that keeps both reliable under production load.

Why Voice Agents Need Live Data

Text agents can paper over latency with streaming tokens and "thinking..." indicators. Voice agents can't. If the round-trip from microphone to speaker exceeds about 1.5 seconds, the user notices, and beyond 3 seconds the conversation feels broken. Every web request the agent makes has to fit inside that budget alongside speech-to-text, LLM inference, and text-to-speech.

That changes the design constraints in three ways:

• Tool calls have to be cheap. A 4-second scrape of a JavaScript-heavy site is unusable. The web access layer has to return parsed data in well under a second on the happy path.
• Failures have to be invisible. A voice agent can't show a retry spinner. If a request fails, the agent has to fall back gracefully - usually by acknowledging it can't get the information rather than going silent.
• Freshness matters more than coverage. A user asking "is my flight delayed?" needs an answer based on data from the last few minutes, not a cached result from an hour ago.

Architecture Overview

The pipeline has four moving parts:

Mic → STT → LLM (with tools) → TTS → Speaker
                  │
                  ├─ search(query)     → SERP API → top results
                  └─ fetch(url)        → Web unlocker → clean HTML/markdown
                                          │
                                          └─ Residential proxy pool

The LLM exposes two tools to the voice loop: search for discovery and fetch for retrieval. The SERP API answers "what URLs are relevant" and the unlocker answers "what does that URL actually say right now."

Defining the Tools

A minimal tool schema for a Claude or OpenAI function-calling model:

[
  {
    "name": "search",
    "description": "Search the web for current information. Returns the top 5 results with titles, URLs, and snippets.",
    "input_schema": {
      "type": "object",
      "properties": {
        "query": { "type": "string" }
      },
      "required": ["query"]
    }
  },
  {
    "name": "fetch",
    "description": "Fetch the contents of a specific URL as clean markdown. Use after search() to read a result in full.",
    "input_schema": {
      "type": "object",
      "properties": {
        "url": { "type": "string" }
      },
      "required": ["url"]
    }
  }
]

Keep the tool surface small. Voice agents that expose ten tools spend half their token budget deciding which to call.

Implementing search()

search hits a SERP scraper that returns Google results as structured JSON. Doing this yourself with a residential proxy pool looks roughly like:

import httpx
from urllib.parse import quote_plus

PROXY = "http://user-session-serp:pass@proxy.statproxies.com:3128"

async def search(query: str) -> list[dict]:
    url = f"https://www.google.com/search?q={quote_plus(query)}&num=10"
    async with httpx.AsyncClient(proxy=PROXY, timeout=4.0) as client:
        response = await client.get(url, headers={
            "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) ...",
        })
    return parse_serp(response.text)[:5]

The 4-second timeout is deliberate. If Google takes longer than that, you've already missed the latency budget - better to fail fast and let the model say "I couldn't reach the web just now" than to leave the user listening to silence.

Implementing fetch()

fetch is where most voice-agent web tools fall apart. The URLs the model wants to read are exactly the ones with the most aggressive bot protection - news sites, airlines, ticket brokers, retail. A bare httpx.get works on roughly 40% of them.

A web unlocker handles the awkward parts in one call: rotating residential exits, solving challenges, rendering JavaScript when needed, and returning clean markdown:

async def fetch(url: str) -> str:
    async with httpx.AsyncClient(timeout=6.0) as client:
        response = await client.post(
            "https://api.statproxies.com/unlock",
            headers={"Authorization": "Bearer " + API_KEY},
            json={"url": url, "format": "markdown", "render_js": "auto"},
        )
    response.raise_for_status()
    return response.json()["content"][:8000]

Truncating at ~8KB keeps the result inside the LLM's effective context for a fast turn. Voice agents rarely need the full page - they need the first few sections, which is almost always where the answer lives.

Wiring It Into the Voice Loop

Cartesia, LiveKit, and Pipecat all expose the same hook: an async function the framework calls when the LLM emits a tool call. The voice loop continues holding the conversation open while the tool runs:

from livekit.agents import llm

@llm.ai_callable(description="Search the web for current information")
async def search_tool(query: str) -> str:
    results = await search(query)
    return "\n".join(f"{r['title']} - {r['url']}\n{r['snippet']}" for r in results)

@llm.ai_callable(description="Fetch the contents of a URL as markdown")
async def fetch_tool(url: str) -> str:
    return await fetch(url)

While the tool runs, emit a short filler phrase ("let me check on that...") so the user hears something. A 600 ms filler buys you a 2-second tool call without the conversation feeling broken.

Why Proxies Are the Bottleneck

The interesting failure mode in production isn't the LLM - it's the web layer. After a few hundred queries, the same patterns appear:

• Google starts returning consent walls or sorry pages from the agent's IP
• News sites serve a paywall variant that the model dutifully reads aloud
• Airline status pages return a generic "we'll be right back" because the request looked like a bot

All three are proxy problems. Residential and ISP exits in the user's region defuse most of them; for the rest, the unlocker's render-on-demand layer handles JavaScript challenges that a pure HTTP fetch can't.

For a deeper look at why residential ISP networks outperform datacenter pools for agent traffic, see our post on building reliable proxy infrastructure for AI agents.

Latency Budget in Practice

A workable end-to-end budget for a single conversational turn that includes a web call:

The tool call is the largest single line item. If your unlocker or SERP fetch consistently runs over 1.2 seconds, the entire experience feels sluggish even when nothing is technically broken. Optimize that stage before anything else.

Conclusion

Giving a voice agent live web access is a proxy problem dressed up as an AI problem. The model architecture is the easy part - two tools, a small JSON schema, a callback. The hard part is making search and fetch fast and reliable enough that the agent can actually use them mid-conversation. Residential exits, a web unlocker for the awkward sites, and a strict latency budget on every tool call are what turn a demo into something a user will keep talking to.