Sermon audio in the pastor's voice — Tagalog TTS via voice cloning
Tagalog is a low-resource language for speech synthesis, and no commercial API offered Filipino voice cloning at a volunteer's budget. So I built a hybrid pipeline: MMS-TTS for authentic Tagalog, then RVC voice conversion to match our pastor.
Python 3.11 / PyTorch / Apple MPS / MMS-TTS (facebook/mms-tts) / RVC (Retrieval-based Voice Conversion) / Coqui TTS / ffmpeg
Context
The natural next step after live caption translation was audio: sermon recordings in Tagalog, in the voice our congregation actually hears on Sunday. Commercial TTS APIs either don't offer Filipino or don't offer voice cloning — and the ones that do charge enterprise rates for a church project funded by enthusiasm.
Constraints
- Tagalog is low-resource for TTS. Off-the-shelf multilingual voices render it with English phonology; it sounds wrong immediately to any native listener.
- The pastor's voice had to be matched from existing sermon recordings — no studio sessions, no clean reads. Real audio with real background.
- Training runs locally on Apple Silicon (MPS), not on rented GPUs.
- The TTS stack is pinned to Python 3.11 — a hard constraint from the libraries, not a choice.
Decisions
Hybrid synthesis over monolithic. A single model that is good at Tagalog and good at voice matching doesn't exist at this budget. Splitting the problem does: MMS-TTS (Meta's massively multilingual speech model) produces authentic Tagalog phonology in a generic voice, then RVC converts that output into the pastor's timbre. Each stage is a well-understood tool doing one job.
Mine the dataset we already had. The caption corpus from the live translation stack turned out to be a ready-made training set: ~908 utterances, roughly 54,000 characters of real sermon Tagalog, already segmented. The projects compose — capture, translate, and now speak the same corpus.
Numbered pipeline scripts, not a monolith. Extract utterances → TTS → train RVC → convert → merge. Each stage is independently inspectable and re-runnable, which matters when a stage takes an hour and you want to know which stage broke.
What shipped
A working end-to-end pipeline: captions in, cloned-voice Tagalog sermon audio out, with a fast test path (3,000 characters, no RVC) for iteration and the full voice-cloned path for production. Alongside it, honest documentation of the Python version trap and the MPS-specific setup — the kind of README I want to find when I land in someone else's ML repo.
What I'd do differently
RVC training data quality is the ceiling on the whole pipeline. I'd invest in line-level source recordings (cleaner than sermon captures) for the conversion model, and add an objective evaluation — even a crude listener score across candidate checkpoints beats choosing by vibes. A comparison against newer two-stage models would tell me whether the hybrid approach is still the right architecture or just the first one that worked.