Case study
KOTA: restaurant phone calls in, confirmed orders out. In English and 中文.
A real-time voice agent that answers the call, takes the order against the restaurant’s own menu data, records it, and confirms it to the caller.
The problem
Restaurants take phone orders at the moments staff are busiest. Many serve Mandarin-speaking callers whom an English-only answering service can’t help. A missed or garbled order is lost revenue and a frustrated customer.
What it does
The caller presses 1 for English or 2 for Chinese. The agent answers menu questions from the menu database and takes the order: standard items, combos, required options, quantity changes, removals, and cart review. It texts a menu link on request, places the order as a structured record, and texts the caller a confirmation. Pressing 0 at any time transfers the call to a staff phone.
How a call becomes an order
Understand the input
Streaming speech-to-text on Twilio media streams: Deepgram for English, ByteDance for Mandarin.
Retrieve information
Menu, prices, options, and combos come from PostgreSQL. Exact answers must come from tool calls.
Apply business rules
Required options, combo structure, and cart changes are enforced in code, not by prompt.
Call the right tools
Typed tool calls: menu search, option selection, cart edit and remove, order summary, menu-link SMS, place order.
Check the action worked
The agent only says “I texted your order” if the messaging API reports success.
Handle the unexpected
No exact menu match? It offers real menu items instead of guessing. Messaging failures are caught and logged without losing the order.
Leave a record
Every call, utterance, and placed order is stored with the call ID.
Hand off to a person
Pressing 0 redirects the live call to a staff phone.
Engineering decisions
The model is never the source of truth
Prices, items, and order state are reached only through deterministic tools against Postgres. A compact menu summary orients the model; anything price-sensitive must come from a tool call.
Menu onboarding is validated extraction
Menu photos or text go through LLM extraction, then grounding checks remove pairings, recommendations, or aliases that point at items not on the menu. A refinement step must preserve item sets, option structure, prices, and availability before anything reaches the database.
Multi-restaurant by design
Each call carries a restaurant ID. Prompt, persona, and menu are assembled per restaurant from the database.
Tested where it breaks
111 automated tests cover the tool contract, language routing, order options, menu grounding, schema and search, recommendations, and WhatsApp messaging.
Stack
- Twilio voice and media streams, SMS, WhatsApp
- Deepgram streaming agent for English
- ByteDance speech-to-text and text-to-speech, with OpenAI tool calling, for Mandarin
- Python and FastAPI WebSocket runtime, one handler per call
- PostgreSQL for restaurants, menus, calls, utterances, and orders
- Docker Compose for development, Kubernetes manifests for Google Cloud
Not built yet
POS integration, kitchen printing, a restaurant-side order dashboard, staff notification, payments, calendar booking, and CRM sync. Each one is a bounded integration on top of the structured order record, the same kind of work we do for clients.
Have a process like this?
Phone, inbox, leads, or documents: tell us what happens today and what a good result looks like.