Capstone — Live Multilingual Travel Translator#

Speech in, speech out, in another language — a three-model pipeline where every stage can fail, and your job is to make the whole thing not fall over.

⏱ ~6–8 hours 🔗 needs: Speech AI · FastAPI · Structured Output

Chaining STT → translation → TTS is easy to demo and hard to make reliable: errors compound, latency stacks, and each stage has its own failure mode. This capstone is graded on the engineering around the models.

⚖️ Voice is biometric personal data. Use your own voice, synthetic audio, or openly-licensed clips. Don’t upload recordings of other people without their consent, don’t retain audio longer than a request needs, and don’t build voice cloning of a real person.

What you’re building#

flowchart LR
    A["Audio upload<br/>(wav/mp3)"] --> V{"Validate:<br/>type, size, duration"}
    V -->|Reject| E["400 + clear reason"]
    V -->|OK| S["STT → text + confidence"]
    S --> T["Translate → target language"]
    T --> TTS["TTS → audio"]
    TTS --> R["Return audio + BOTH transcripts"]
    S -.->|"low confidence"| W["Warn, don't silently proceed"]

Requirements#

1. A FastAPI endpoint. POST /translate taking an audio file plus a target_language, returning the synthesised audio and both the source transcript and the translated text. Returning only audio is a failure — users must be able to see what it heard, because that’s where errors originate.

2. Validate input properly. Reject wrong content types, oversized files, and zero-length audio with useful 4xx messages. Cap duration; a 40-minute upload should be refused, not processed.

3. STT with confidence. Use faster-whisper (or equivalent). Surface the detected language and its probability. If confidence is low, say so in the response rather than confidently mistranslating noise.

4. Translate. Any LLM or translation API, with a schema-validated response (Structured Output) — never a raw free-text blob you then regex.

5. TTS. Hosted (OpenAI TTS, ElevenLabs) or local (Piper). Handle the case where the target language has no available voice — degrade gracefully to text-only with an explicit flag.

6. Engineer for failure. Every stage gets a timeout. A stage failure returns a clear error naming which stage failed, never a 500 with a stack trace. Log per-stage latency with a request ID (observability).

7. Measure quality honestly. Take 10 test clips, at least three in an Indian language, and report per clip:

clip | true text | STT output | WER | translation | fluent? (your judgment) | total latency

Then write which stage contributes the most error. Usually it’s STT on accented or noisy audio — and demonstrating that you measured it is the point.

Deliverables#

Grading#

Common failure modes#

Stretch goals#

• Stream results so the transcript appears before the audio finishes.
• Add a glossary so place names and proper nouns survive translation.
• Compare Whisper against Parakeet on your Indian-language clips and report the WER difference.
• Cache by audio hash so repeated clips skip the pipeline entirely.