I am currently working on training an Automatic Speech Recognition (ASR) model for Nepali, and this is the first of three posts on it. Training an ASR system is not trivial, because the training data is simply audio and transcript pairs. Without alignment information, it is hard to know which parts of an audio clip correspond to which characters in the transcript. Additionally, Nepali is a morphologically rich and low-resource language, which adds to the challenge in training an ASR for Nepali. In this post, I will cover how audio becomes tokens, the alignment problem that makes ASR training hard, and the three families of models that solve it.
Audio Waveforms to Features
Raw audio is typically sampled at 16 kHz and is split into short overlapping frames, commonly 25 ms windows with 10 ms hops. The raw audio frames are converted into a frequency-domain power spectrum using the Fourier transform, which is then projected onto mel-spaced filter banks. This mimics how the human ear resolves pitch. A log operation is then applied to approximate how it perceives loudness.
The mel-inputs have high temporal resolution, around 100 frames/sec, which is far more than what is required by text. Acoustic encoders downsample the mel-input sequence to extract high-level phonetic and contextual representations, while improving memory and latency.
Self-supervised models such as wav2vec 2.0 and HuBERT extract features from raw waveforms using convolutional encoders. These methods have been shown to outperform other methods, particularly for low-resource speech recognition. Once these acoustic features are extracted, the model maps them to text tokens. For a morphologically rich language like Nepali, these tokens are typically sub-word units or character-level graphemes that capture complex script characters like क्ष or त्र.
The Alignment Problem
The main challenge in training an ASR system is that the training data contains no alignment between input frames and transcript characters. A typical utterance has far more input frames (T) than transcript labels (U), T ≫ U, so mapping frames to output labels is not straightforward.
Over the next sections, we will look at how modern end-to-end methods either build alignment into the model, like Connectionist Temporal Classification (CTC) and transducer models or learn it implicitly, through attention, like attention-based encoder-decoders (AED).
Connectionist Temporal Classification (CTC)
The CTC model is designed for tasks where the alignment between input and output is not available or expensive to annotate, provided that alignment is monotonic. Phone durations vary intrinsically, silences and pauses occupy frames with no corresponding label, and speaking rate varies across and within speakers. Hence a fixed rule-based speech-to-text mapping fails. CTC handles the alignment issue by introducing a blank token (∅) that lets a label span multiple frames or none, and then summing over all valid alignments rather than committing to one, learning the durational structure from data.
In speech-to-text, each output label spans several input frames, and one of the core steps in CTC is merging adjacent identical labels back down to one. This causes a problem when the same character genuinely appears twice in a row in the target. Take ममता for example. With four frames, the alignment म-म-त-ा collapses to मता. The two म are merged into one, resulting in the wrong word. The blank token solves this: a blank between the two म blocks the merge, since the collapse rule only joins labels that are directly adjacent. The alignment म-∅-म-त-ा survives as ममता. Blanks are removed after the merge step, so the ε leaves no trace in the output.
CTC assumes that every output is conditionally independent of the other outputs given the input. This assumption leads to the absence of a native language model in CTC model, which can lead to these models producing incoherent sequences as getting grammatical agreement becomes challenging. This is often more pronounced for long sequences and morphologically rich languages like Nepali. This issue is addressed by the families of models we are going to look at next.
Transducer
The transducer is a sequence-to-sequence model proposed in Recurrent Neural Network Transducer (RNN-T) and is a popular method used in speech recognition. Like CTC, Transducers work by generating the probability of all possible paths and aggregating the probabilities to get the output label sequence. However, transducers introduce a separate prediction network (which acts as a language model) and a joint network to model autoregressive dependencies across output tokens. Because the likelihood of the next token depends on both the acoustic frame and the text predicted immediately before it, this results in a 3D grid of size T × U × V, where T is the audio frame, U is the number of text labels predicted so far, and V is the vocabulary size, while CTC produces a 2D grid of size T × V.
Additionally, transducers use a blank token as a routing mechanism to solve the alignment issue. The blank token is used to decide when to move forward in the input audio time steps and when to predict more text tokens at the current time step, which is different from how CTC uses its blank token.
While transducers have an internal language model and are better at handling the issue with grammatical agreement over long sequences, it is hard to train for a low-resource language like Nepali, as the joint network and prediction network in transducers have many parameters that require substantial data to learn robust language priors.
Attention-based Encoder-Decoder (AED)
Attention-based Encoder-Decoder (AED) models learn all the components of speech recognition jointly similar to transducers. While transducers move forward by input audio time steps, AED models step forward by output tokens, not time. Instead of a frame-by-frame timeline, the encoder processes the entire audio sequence, and the decoder uses an attention mechanism to create a 2D alignment map over the full audio. It uses this attention to look back at the entire audio feature map at every single time step, learning to align the speech to text by highlighting the relevant audio segments for each token.
While CTC and Transducers are much faster at decoding speech to text and are preferred for streaming use cases, the AED architecture is slower and used mostly in offline/batch settings. Additionally, because the decoder relies on text-driven steps rather than strict audio timestamps, it can occasionally experience issues with repetition or skipped text when encountering background noise. However, AED models are excellent at transcribing long, clean audio sequences.
Conclusion & What’s Next
Ultimately, CTC, Transducers, and AED models represent three different paths to handle the alignment problem to convert input audio frame to output tokens. CTC is the fastest of the three but lacks internal language model. Transducers add a time-aware language model for real-time interaction. AED models use a global cross-attention mechanism for high-quality transcription. However, each of these systems have certain trade-offs. CTC struggles to capture subject-verb agreement in morphologically rich language like Nepali. Transducers require a large amount of training data and the model can easily overfit for a low-resource language. AED can be very slow for streaming use cases. When building an ASR system for a morphologically rich, low-resource language like Nepali, deciding between these architectures requires a deliberate trade-off between linguistic accuracy, data availability, and application intent.
As I mapped out in my previous post on Nepali Speech Benchmarks and Resource Gaps, we only have a handful of open-source ASR datasets for Nepali. So we need to know exactly what we are working with before we begin training an ASR for Nepali. In my next post, I will share a deep-dive Exploratory Data Analysis (EDA) of the three public Nepali speech corpora: OpenSLR-54, IndicVoices, and Common Voice. We will look at audio duration distributions, speaker diversity, and the unique textual quirks of the Devanagari transcripts that our models will have to navigate. Stay tuned!