The problem
Speech recognition for Lingala, Shona, and Luganda brings together limited training data, differences in spelling conventions, and recognizers with uneven language coverage. The challenge becomes more demanding when incoming audio has no language label. A system that relies on a filename prefix or known language can look convincing in development and fail when that information disappears.
I built a competition research workflow around that constraint: take an audio clip, produce a transcription, and evaluate the complete decision path without giving it information that would be unavailable at inference time.
Building the recognition pipeline
The implementation separates data manifests, model training, inference, and transcript combination. CTC recognizers provide one family of predictions, while a Whisper branch adds an autoregressive approach for Lingala and Shona. Shared scripts sit underneath the notebook launchers so experiments can move between GPU environments without duplicating the scientific logic.
The ensemble aligns alternative transcriptions and combines their word-level evidence. ROVER voting provides a deterministic consensus, while language routing controls when a specialist recognizer can contribute. The routing decision is learned from ordinary training data and applied through an anchor transcription. It does not depend on a hidden language field in the evaluation set.
Each ensemble run can emit an audit of the participating predictions and word-level choices. That makes disagreements inspectable and helps distinguish a useful complementary model from one that merely produces different text.
Making evaluation representative
The development arena mixes the languages and anonymizes identifiers to reproduce the conditions of unlabeled incoming audio. Word error rate and character error rate capture different aspects of transcription quality, so both are part of the evaluation workflow.
Speaker separation is another central concern. Shared split utilities use the dataset publisher's speaker-disjoint assignments to build validation manifests and exclude those examples from training. Training configurations can require evidence of that separation before a run starts. A separate label-access guard blocks the prohibited test-label loading path.
New recognizers and transcript rules are assessed against fixed baselines. The code also distinguishes raw-token-preserving consensus from experimental character-level synthesis, where an ensemble can construct a spelling that no individual recognizer produced.
Outcome and scope
The result is a research pipeline for comparing recognizers, combining their predictions, and inspecting why a transcription changes. Its contribution is as much the evaluation discipline as the model selection: language uncertainty, speaker overlap, and transcript provenance are explicit parts of the system.
This is competition research rather than a deployed speech service. The work reinforces a practical lesson for low-resource language modeling: a larger ensemble is useful only when its improvements survive the conditions the final system will actually face.