Speech Synthesis and Recognition (TTS & ASR)

Open a recording of someone saying "hello world" and look at what a computer actually sees.

A phone samples sound 16,000 times per second. Two seconds of speech = 32,000 floating-point numbers. Nobody trains a network to spit out 32,000 precise values one by one.

Both speech tasks are really about squashing that river of numbers into symbols and back:

• TTS (text-to-speech, synthesis): text → a waveform. Expanding 11 letters into 32,000 numbers that sound human.
• ASR (automatic speech recognition, transcription): a waveform → text. Compressing 32,000 numbers into 11 letters, correctly, at speed.

They are the same sequence-to-sequence problem pointed in opposite directions.

Carry one analogy: two embassies, Text-land and Sound-land, exchanging mail across a border.

• The outbound ambassador (TTS) turns a letter into spoken sound.
• The inbound ambassador (ASR) turns heard sound into a letter.
• Historically each embassy had its own bizarre internal bureaucracy of hand-designed departments. The last decade of speech AI is the story of replacing those departments with neural nets — and the current chapter is the two embassies merging into one bilingual brain.

Two scoreboards matter throughout. ASR is judged by WER (Word Error Rate) — align the transcript against the truth and count substitutions + deletions + insertions over the reference length (100 correct words with 7 wrong → 7% WER) — plus latency for live use. TTS is judged by MOS (Mean Opinion Score): humans rate naturalness/intelligibility 1-5, averaged — and by speaker similarity when cloning a specific voice.