Benchmark and Resource Gaps for Nepali Speech Tasks
Even with the breakthroughs in Large Language Models (LLMs) there are significant performance gaps in Natural Language Processing (NLP) systems for low-resources languages. To bridge this gap we need more high quality data in the target languages. Given the low digital footprint of low-resource languages, the availability of data, both labelled and unlabelled, remains one of the most critical challenges when developing LLMs/NLP systems for these languages.
In my previous post, I introduced Joshi et al., 2020’s language taxonomy classification, which defines six classes of languages from 0 to 5 based on their resource availability for NLP tasks with 0 or “The Left-Behinds” being the class languages with exceptionally limited data and 5 or “The Winners“ being the ones with dominant online presence, have massive corporate and government investments in the development language technologies. While Class 0 languages benefit the least the from the LLM breakthrough and the author claim that “it is probably impossible to lift them up in the digital space.”
Low-Resource NLP in the Era of LLMs - Introduction
There has, undoubtedly, been drastic shifts in the landscape of Natural Language Processing (NLP) research and development with the breakthrough of Large Language Models (LLMs) like ChatGPT. However the majority of LLMs are optimized for a few high-resource languages such as English. This is because these LLMs are pretrained with large corpora of text f…
Based on “The Six Kinds of Languages”, Nepali falls into Class 1 “The Scraping Bys,” languages with some unlabelled data but where collecting labelled data remains a challenge. In fact, according to the W3Tech Surveys, Nepali is used in less than 0.1% of websites and while it has ticked upward compared to last year, it remains very small in comparison.
In addition, W3Techs' market-position chart below, shows Nepali's web content footprint is concentrated in a handful of major high-traffic platforms and institutions, which suggest lack of diversity in content.
The usage and market position data tells us how little of the web contains Nepali content, and the web content that does exist is almost all unlabelled text and is already scarce. This scarcity further deepens for speech, particularly annotated speech data that is useful for NLP. In the rest of this post, I will map out the available benchmarks and discuss resource gaps for Nepali speech tasks.
Speech Tasks and Resources
Broadly, speech tasks in NLP include Automatic Speech Recognition (ASR), Speech Synthesis (TTS), Speech Translation (ST), Spoken Language Understanding (SLU), Speaker Diarization, and Speaker Profiling.
Automatic Speech Recognition (ASR)
ASR, or speech-to-text, involves using AI models to transcribe speech into text. Notably, there are four major open-source Nepali speech-to-text datasets: OpenSLR-54 contains about 165 hours of transcribed Nepali speech with 157,905 utterances and 527 speakers. This dataset was collected by Google through crowdsourcing. Mozilla Common Voice is another crowdsourced dataset with 1739 utterances and 49 speakers. IndicVoices by AI4Bharat is an ASR dataset for 22 Indic languages. For Nepali, it contains 496 speakers and 100 hours in the training split, additionally, the held out set contains 114 speakers and 5 hours of speech. Seamless by Meta contains 153 hours of ASR data for Nepali. Additionally, Paudel et al. mentions Nep_DS, an additional dataset with 6031 recordings consisting of Nepali phrases scraped from the web including E-kantipur, Setopati, Hamro Patro, Nepali Me and English sentences from BBC and translated to Nepali using Google Translate.
Speech Synthesis (TTS)
Speech synthesis, or Text-to-Speech (TTS), refers to the synthesis of speech from text. I found two publicly available TTS datasets for Nepali: OpenSLR-43 built by Google is a TTS dataset for Nepali. It contains 2,064 audio files totaling 2.8 hours of speech by 19 female speakers. Another open-source Nepali TTS dataset is OpenSLR-143, which was developed by Khadka et al. because OpenSLR-43 did not meet the prosody requirements. It contains 1.2 hours of data with 666 sentences spoken by 2 speakers (557 female recordings and 109 male recordings). While these are decent-sized datasets, they fall short of diversity needed to build a general purpose Nepali TTS system.
Speech Translation (ST)
Speech Translation models translate speech from one language into either speech (S2ST) or text (S2TT) in another language. FLEURS by Google is a multi-way parallel speech dataset that includes 12 hours of Nepali speech. BhasaAnuvaad, by AI4Bharat, is a speech translation dataset for 13 Indic languages, including Nepali. It is built by combining existing corpora, a replicated version of Meta's SeamlessAlign mining pipeline, and large-scale synthetic data generated from IndicVoices transcripts. It contains about 465 hours of English to Nepali translation and there is about 538 hours of Nepal to English translation. These numbers exclude the FLEURS dataset which is included in this dataset. An important thing to note is that over 85% of these dataset in both directions is synthetically generated. Finally, the original Seamless dataset by Meta contains 129 hours of Nepali to English speech-to-text data and 1,714 hours of English to Nepali speech-to-text data.
Spoken Language Understanding (SLU)
Spoken Language Understanding (SLU) covers a wide array of language understanding tasks directly from speech, including intent classification, information extraction, emotion recognition, and question answering. While there are some resources for Nepali textual language understanding tasks, I found no dataset with clear origin specifically for Nepali speech understanding tasks. There are a few Kaggle uploads that I am excluding from this list, as those lack details on how and where the data was sourced.
Speaker Diarization
Speaker Diarization helps identify who spoke when in multi-speaker audio clips. This Nepali-inclusive speaker diarization study combines multiple datasets from four sources: LibriSpeech (921 English speakers), VoxCeleb (1,211 speakers), Hindi audio (100 speakers), and OpenSLR-43 (19 native Nepali speakers, the same TTS dataset discussed above).
Speaker Profiling
Speaker Profiling tasks generate detailed and descriptive characterizations of speakers. I found no dedicated Nepali speaker-profiling study. The closest adjacent findings come from the multi-speaker voice cloning work, where they curated datasets with diverse speaker characteristics across varying domains and acoustic conditions. This study also uses the OpenSLR-54, OpenSLR-43 and OpenSLR-143 in addition to the self-collected dataset.
The Gap
The dataset coverage across speech tasks for Nepali is sparse and uneven. Speech paired resources are almost non-existent for some of the tasks. The only task with significant data volume is ASR. TTS has high quality but a small set of datasets. Speech Translation dataset is substantial in raw hours, but over 85% of the translation it is synthetic. SLU and Speaker Profiling have no dedicated data or research in Nepali.
Nepali speech tasks are under-resourced and under-researched. I am interested in improving the translation task. If you are someone who is interested in working on Nepali speech technology, building datasets, training models, or just thinking through the problem, I would love to hear from you. Let's figure out how to close this gap together.