Machines That Answer

A voice agent that works in a demo can still fail on a real phone line. This book shows how to build one that holds up.

The caller's audio arrives in real time, and a recognizer and a voice share a budget of under a second. The agent may act only through tools that check what the caller heard and agreed to, and the systems of record say what actually happened. Each part can work and the whole call can still fail, at the seams where one part takes another's word.

Machines That Answer builds three voice agents, one for each grade of difficulty: a plumber's booking line, a regional utility's service line, and an airline's storm desk. All twenty-nine chapters end in a lab that runs with Python alone.

Inside:

• The phone line and every other way in: the carrier, SIP, WebSockets, and WebRTC for web and mobile apps
• Hearing, turn-taking, and speaking, so the agent neither cuts callers off nor leaves them waiting in silence
• Speech-to-speech in depth, and when a fast model in front of a stronger one is worth it
• Agent architecture: the turn loop, state, policy, instructions, and knowledge
• Tools that confirm before they commit, so the agent never says "booked" before the system of record does
• Guardrails, handoff to people, and other AI agents on the line, including how to tell when the caller is a machine
• Trust on the live call: disclosure, card numbers, and threats
• Metrics the business will accept, testing with simulated callers, and rates you can trust
• Deployment, observability, capacity, recovery from failure, and the years of maintenance after launch

Every chapter follows one order. It sets out the decision, with the options in one table and a recommendation, then the build and the lab. The swap shows how to change the provider at that layer without starting again, and the chapter closes with the edge cases and what to measure.

The open-source companion repository on GitHub holds every lab and the Knowledge Line, a working voice agent app that runs the plumber's appointment line end to end. The labs run with no key and no network.

The book is for engineers, technical founders, solution architects, conversation designers, AI and ML engineers, test engineers, and the operators who are paged when a line goes quiet at 2 am. You need to read Python. You do not need to have built for the phone before.

The companion volume, Every Call Answered, covers the same ground for the leaders who decide, fund, and buy, with the same businesses, numbers, and eighty-six checkpoints. Each book stands alone. Vendors appear only as dated examples, and this edition's facts were checked in September and October 2026.

About the author: Jay Gopalan builds and deploys AI-native products for businesses at production scale. His work includes fine-tuned and sovereign models, and the compliance that lets them go live. In recent engagements he built and ran a client's voice AI, which answers tens of millions of calls a year. He wrote both volumes from what those calls taught him.

Part of the Taming Your AI series.