Whisper: Robust Speech Recognition via Large-Scale Weak Supervision (Whisper)Permalink

IntroductionPermalink

Briefing Document: Whisper: Robust Speech Recognition via Large-Scale Weak Supervision (Whisper)

Source: Excerpts from “https://arxiv.org/pdf/2212.04356”

Date: December 2022 (Based on arXiv preprint date)

Authors: Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, Ilya Sutskever

Organization: OpenAI

What Problem is Addressed?Permalink

This paper is about Speech to text transcription.

• Existing automatic speech recognition (ASR) models lack robustness and generalization. They perform well on data similar to their training data but struggle on different datasets or conditions (out-of-distribution data), even showing “superhuman” performance on specific benchmarks.
• Many state-of-the-art ASR systems require fine-tuning on specific datasets to achieve high quality. Whisper aims to work well zero-shot, without needing dataset-specific fine-tuning.
• There is a limited amount of easily available, high-quality supervised speech data. Whisper tackles this by using a much larger scale of weakly supervised data (680,000 hours).
• Traditional speech processing often involves multiple separate components for tasks like voice activity detection or translation. The paper seeks to create a single model that can perform the entire pipeline, handling transcription, translation, language identification, and voice activity detection.
• Transcribing long audio recordings is challenging for models trained on short segments. The paper develops strategies for buffered transcription of long audio to make it practical for real-world use.

Executive Summary:Permalink

This paper introduces Whisper, a speech processing system trained on a massive dataset of 680,000 hours of diverse, weakly supervised audio and their transcripts from the internet. The core finding is that simple scaling of weakly supervised pre-training has been significantly underappreciated for speech recognition. Unlike many recent large-scale speech recognition models that rely heavily on unsupervised pre-training or self-training, Whisper achieves strong performance across various tasks (multilingual speech recognition, speech translation, language identification, voice activity detection) and generalizes remarkably well to standard benchmarks in a zero-shot setting, without the need for fine-tuning. When compared to humans, the models approach their accuracy and robustness, particularly in out-of-distribution scenarios where traditional supervised models struggle. The authors release models and inference code to promote further research in robust speech processing.

Key Themes and Ideas:Permalink

1. The Power of Large-Scale Weak Supervision: The central tenet of the paper is that training a model on a vast quantity of weakly supervised data can yield highly capable and robust speech processing systems. The dataset of 680,000 hours of audio is significantly larger and more diverse than typical academic supervised datasets.

   • “We study the capabilities of speech processing systems trained simply to predict large amounts of transcripts of audio on the internet. When scaled to 680,000 hours of multilingual and multitask supervision, the resulting models generalize well to standard benchmarks and are often competitive with prior fully supervised results but in a zero-shot transfer setting without the need for any fine-tuning.”
   • “Our work suggests that simple scaling of weakly supervised pre-training has been underappreciated so far for speech recognition.”

2. Zero-Shot Transfer and Robustness: A major focus is on the ability of Whisper models to perform well on various datasets and tasks without specific fine-tuning on those datasets (“zero-shot transfer”). This contrasts with many prior approaches that required dataset-specific fine-tuning. This zero-shot capability leads to significantly improved robustness to out-of-distribution data and noise compared to models trained on smaller, in-distribution datasets.

   • “The goal of a speech recognition system should be to work reliably “out of the box” in a broad range of environments without requiring supervised fine-tuning of a decoder for every deployment distribution.”
   • “When compared to a human in Figure 2, the best zero-shot Whisper models roughly match their accuracy and robustness.”
   • Table 2 demonstrates that despite similar performance on the LibriSpeech benchmark, the zero-shot Whisper model achieves an “average relative error reduction of 55.2%” on other datasets compared to a supervised LibriSpeech model.
   • Figure 5 illustrates Whisper’s robustness to additive white noise and pub noise, showing that while other models might perform better at low noise levels, Whisper significantly outperforms them under higher noise conditions.

3. Multilingual and Multitask Training: Whisper is trained to be both multilingual and multitask. The dataset includes data for 97 languages and supports tasks beyond simple transcription, such as translation, language identification, and voice activity detection. The authors find that training on this broad scope is beneficial for sufficiently large models.

   • “In addition to scale, our work also focuses on broadening the scope of weakly supervised pre-training beyond English-only speech recognition to be both multilingual and multitask. Of those 680,000 hours of audio, 117,000 hours cover 96 other languages. The dataset also includes 125,000 hours of X→en translation data.”
   • “We find that for sufficiently large models there is no drawback and even benefits to joint multilingual and multitask training.”

4. Simple Encoder-Decoder Transformer Architecture: To isolate the impact of data scaling and weak supervision, the authors used a standard encoder-decoder Transformer architecture, avoiding complex model innovations.

   • “Since the focus of our work is on studying the capabilities of large-scale supervised pre-training for speech recognition, we use an off-the-shelf architecture to avoid confounding our findings with model improvements. We chose an encoder-decoder Transformer (Vaswani et al., 2017) as this architecture has been well validated to scale reliably.”

5. Novel Multitask Training Format: A unique token-based format is used to specify different tasks and conditioning information as input to the decoder, enabling a single model to handle various speech processing functions.

   • “We use a simple format to specify all tasks and conditioning information as a sequence of input tokens to the decoder.”
   • This format includes special tokens like <|startoftranscript|>, language tokens (e.g., EN), task tokens (<|transcribe|>, <|translate|>), timestamp tokens, <|nospeech|>, and <|endoftranscript|>.
6. Performance Scaling with Model and Dataset Size: The performance of Whisper models improves consistently with increasing model size across various tasks and languages (Figure 8). Increasing the dataset size also leads to improved performance (Table 6), although diminishing returns were observed when scaling from 54,000 hours to the full 680,000 hours, suggesting potential for further improvement with longer training or larger models.

7. State-of-the-Art Results (Zero-Shot): Whisper achieves competitive or state-of-the-art performance on various benchmarks in a zero-shot setting, including English speech recognition, multilingual speech recognition, and speech translation.

   • On English speech recognition benchmarks, zero-shot Whisper models often outperform supervised models when evaluated out-of-distribution (Table 2).
   • Whisper achieves a “new state of the art of 29.1 BLEU zero-shot” on the X→en subset of CoVoST2 for speech translation.
8. Comparison to Human Performance: The study includes a comparison of Whisper’s robustness to human performance, finding that the best zero-shot Whisper models approach human accuracy and robustness. This suggests that the robustness gap observed in prior models on out-of-distribution data can be closed with large-scale weak supervision.
9. Strategies for Long-Form Transcription: The paper describes heuristics developed to improve the reliability of transcribing long audio segments, which relies on accurate timestamp prediction. These include using beam search with temperature fallback, voice activity detection, and previous text conditioning.

10. Text Normalization: The authors acknowledge the challenge of evaluating zero-shot models on diverse datasets with varying transcription styles and introduce a text normalizer designed to mitigate the impact of innocuous formatting differences on metrics like Word Error Rate (WER). They release the code for this normalizer.

    • “While this poses a problem for all transcribers, it is particularly acute for zero-shot models like Whisper, which do not observe any examples of specific datasets transcript formats.”
    • “We are releasing the code for our text normalizer to allow for easy comparison and to help others study the performance of speech recognition systems in out-of-distribution settings.”

11. Open-Source Release: The authors emphasize their commitment to advancing the field by releasing their models and inference code, facilitating further research on robust speech processing.

    • “We are releasing models and inference code to serve as a foundation for further work on robust speech processing.”
    • “To serve as a foundation for further research on robust speech recognition, we release inference code and models at the following URL: https://github.com/openai/ whisper.”

Important Facts and Data Points:Permalink

• Training Data Size: 680,000 hours of audio and corresponding transcripts.
• Data Composition: Includes 438,218 hours of English speech recognition, 117,113 hours of multilingual speech recognition (96 other languages), and 125,739 hours of X→en translation data.
• Model Architecture: Encoder-decoder Transformer.
• Model Sizes: A suite of models with parameters ranging from 39M (Tiny) to 1550M (Large).
• Zero-Shot Performance: Whisper models perform comparably to or better than prior fully supervised models on standard benchmarks when evaluated in a zero-shot setting.
• Robustness Improvement: Zero-shot Whisper achieved an average relative error reduction of 55.2% on other datasets compared to a supervised LibriSpeech model with similar in-distribution performance (Table 2).
• Speech Translation SOTA: Achieved 29.1 BLEU zero-shot on the X→en subset of CoVoST2.
• Multitask Capabilities: Performs multilingual transcription, speech translation, language identification, and voice activity detection within a single model.
• Release: Models and inference code released on GitHub.

Areas for Future Research:Permalink

• Investigating the specific contributions of the encoder and decoder to Whisper’s robustness.
• Exploring the integration of unsupervised pre-training or self-training techniques to potentially further enhance results.
• Characterizing “scaling laws” for speech recognition to better understand performance improvements with dataset size.

This briefing document summarizes the key findings and important details presented in the provided source about the Whisper speech processing system.