DUAP: Disentanglement-Based Universal Adversarial Perturbations for Robust Multilingual Speech Privacy Protection
Universal multilingual adversarial perturbations for speech privacy against Whisper
Abstract Abstract
This work addresses privacy risks from multilingual automatic speech recognition models such as Whisper through DUAP, a universal adversarial perturbation method based on language feature disentanglement. The method first separates and reconstructs language-related features in latent space, then uses gradient optimization to disrupt language identification, providing protection across languages, model sizes, and digital and physical settings.

Language feature disentanglement (LFD): separating language and content features, then reconstructing exchanged features to generate Mel spectrograms with target-language features.
Key takeaway and scope
DUAP (Disentanglement-Based Universal Adversarial Perturbations, by Qianli Ma et al., IEEE Transactions on Information Forensics and Security, Vol. 21, pp. 3703–3718, 2026) is a universal speech privacy protection framework targeting multilingual automatic speech recognition (ASR) models such as OpenAI Whisper. Whereas conventional monolingual universal adversarial perturbations (UAPs) lose effectiveness outside English, DUAP combines Language Feature Disentanglement (LFD) with gradient optimization against Whisper’s language identification module to prevent unauthorized automatic transcription across languages, model scales, lossy codecs, and physical over-the-air capture.
Research background: why monolingual UAPs fail on multilingual Whisper
End-to-end multilingual ASR systems such as Whisper route spoken audio through unified language identification tokens before autoregressive text decoding, making mass automated transcription of intercepted or hidden-microphone recordings practical across dozens of languages. Prior adversarial speech privacy methods are predominantly optimized against single-language (mostly English) acoustic transcription losses, creating two structural bottlenecks:
- Entangled acoustic representations limit cross-lingual transfer: Spoken utterances entangle lexical content with language-specific phonetic cues, causing perturbations fitted to English token distributions to degrade sharply on non-English speech.
- Unaddressed language-routing bottleneck: Whisper’s decoder conditions heavily on its front-end language identification stage; perturbations that do not systematically shift latent language representations allow the decoder to recover sensitive text from contextual redundancy.
| Aspect | Monolingual UAPs | DUAP (IEEE TIFS 2026) |
|---|---|---|
| Core mechanism | Direct perturbation on single-language ASR loss | Language Feature Disentanglement (LFD) + language-ID gradient attack |
| Cross-lingual transfer | Primarily English or training language | Multilingual (> 95% WER on English, > 85% WER on other languages) |
| Scale transferability | Tied to a single surrogate checkpoint | Transfers across 3 Whisper model sizes |
| Channel robustness | Fragile under codecs and room acoustics | > 87% WER over the air; resilient to AAC and MP3 compression |
Method: Language Feature Disentanglement (LFD) and two-stage attack
As illustrated in Fig. 3, DUAP couples a latent Language Feature Disentanglement (LFD) swap-reconstruction network in Stage 1 with universal gradient optimization in Stage 2:
1. Dual-encoder disentanglement of language and content
Given a source Mel spectrogram (Source.Mel, the sensitive utterance to protect) and a target-language Mel spectrogram (Target.Mel), LFD decomposes acoustic features into two orthogonal latent streams:
- Language Encoder (): Extracts utterance-level language representations and invariant to specific words.
- Content Encoder (): Extracts linguistic and prosodic content representations and stripped of source language identity.
2. Swap concatenating reconstruction and three loss functions
A shared decoder concatenates () language and content codes along the channel axis to run two symmetric branches: self-reconstruction ( as Recon.Source.Mel and as Recon.Target.Mel) and bidirectional cross-language swap reconstruction producing (Transformed.Source.Mel, which embeds target-language cues into the source speech carrier as an initial adversarial spectrogram) alongside its symmetric counterpart (Transformed.Target.Mel). Training optimizes three complementary objectives shown in Fig. 3:
- Language Attack Loss (
lang Loss, ): Minimizes the cross-entropy loss through Whisper’s language identification module so swapped spectrograms are routed toward target language . - Self-Reconstruction Loss (
reg Loss, ): Enforces an -norm reconstruction constraint between original inputs () andRecon.Source.Mel/Recon.Target.Melto preserve encoder-decoder completeness. - Cross-Transformation Fidelity Loss (
rec Loss, ): Bounds the -norm distance between original Mel spectrograms andTransformed.Source.Mel/Transformed.Target.Melso perceptual speech quality for human listeners stays intact while machine language identification flips.
3. Stage 2: universal gradient optimization on language identification
Building on the LFD-initialized adversarial spectrograms, Stage 2 aggregates gradients directly against Whisper’s language identification module across utterances to synthesize a utterance-agnostic Universal Adversarial Perturbation (UAP). A single precomputed perturbation can then protect arbitrary unseen speech across speakers, languages, and Whisper model scales without per-utterance optimization.
Quantitative evaluation and robustness
DUAP is evaluated across digital multilingual benchmarks, physical over-the-air playback-and-recording environments, and lossy audio compression pipelines across three Whisper model sizes (higher Word Error Rate indicates stronger protection against unauthorized ASR transcription):
| Evaluation setting | Test condition | Whisper Word Error Rate (WER) |
|---|---|---|
| English speech privacy | Digital audio across 3 Whisper model sizes | > 95% |
| Multilingual privacy | Non-English cross-lingual speech evaluation | > 85% |
| Physical scenarios | Over-the-air speaker playback to microphone | > 87% |
| Lossy compression | Post-processing under AAC and MP3 codecs | Retains high attack effectiveness |
Technical Q&A and threat model scope
Why target Whisper’s language identification module instead of token-level transcription logits alone?
Multilingual Whisper conditions every downstream decoding step on its detected language token. Redirecting the front-end language identification representation triggers cascading decoding failures across the entire utterance at a tighter perturbation budget, yielding substantially stronger transfer across non-English languages and unseen Whisper model sizes.
Which real-world speech privacy threat scenarios does DUAP address?
DUAP addresses two passive eavesdropping threat models: (1) digital channel interception, where voice messages or cloud recordings undergo AAC or MP3 lossy compression before bulk ASR scraping, and (2) physical acoustic eavesdropping, where playing or mixing a universal perturbation in the environment disrupts unauthorized transcription by nearby hidden microphones.
Publication details
- Journal: IEEE Transactions on Information Forensics and Security (IEEE TIFS)
- Volume and pages: Volume 21, pages 3703–3718 (2026)
- DOI: 10.1109/TIFS.2026.3671687
- Lead author and persistent IDs: Qianli Ma (ORCID: 0009-0005-2060-4172, Google Scholar, Author research blog)
The bibliographic metadata and architecture figure (Fig. 3) on this page are supplied and verified by the first author and cross-checked against IEEE Xplore, Crossref, OpenAlex, and Semantic Scholar. Dynamic citation counts are intentionally omitted from static repository source files.
Keywords
- speech privacy
- adversarial perturbation
- multilingual ASR
- Whisper
- language disentanglement