NVIDIA Nemotron 3 Diarization Overview
NVIDIA has introduced Nemotron 3 Diarization, an open-weight, 100M-parameter model designed to accurately assign speaker activity timestamps in both live and recorded conversations. The model has achieved a notable ranking of #1 on VoiceArena’s initial Diarization-Bench, with a diarization error rate (DER) of 14.72% across 139 English conversations, totaling approximately 22 hours, and including evaluations of overlapping speech.
Model Design and Input Specifications
The model processes 16 kHz single-channel audio, converting it into Mel-spectrogram features with a 10 ms frame step. These features are stacked by a factor of eight to produce 80 ms frames, which are then analyzed by a 31-layer Transformer encoder utilizing rotary positional embeddings (RoPE). The output is a default [T, 8] floating-point tensor, representing time steps across up to eight anonymous speaker channels. Each value indicates the probability of activity for a given channel at that frame, allowing for the representation of overlapping speech.
Memory Mechanisms for Streaming
NVIDIA has implemented streaming memory mechanisms to maintain stable speaker assignments across audio chunks. This includes an Arrival-Order Speaker Cache (AOSC) that records speakers based on their first appearance and a FIFO queue that retains recent frame context. The model allows for adjustments to right-context audio, enabling a trade-off between latency and accuracy. Recommended input-buffer latencies are set at 30.4, 1.04, 0.64, and 0.32 seconds, with 0.32 seconds being the lowest recommended setting.
Training Data and Performance Improvements
The training of Nemotron 3 utilized both public and licensed speech data, including multispeaker-annotated conversations from David AI. Notably, the inclusion of David AI data led to a reduction in the compound DER by 0.77 absolute points, decreasing from 11.19% to 10.42% at specific operating points. The model supports up to eight speaker channels, which are represented as anonymous labels (e.g., speaker_2) and do not include identity verification. Downstream systems can map these channel IDs to specific identities using meeting metadata or verification models.
Benchmark Results and Throughput Metrics
In the initial Diarization-Bench results from VoiceArena, Nemotron 3 achieved a DER of 14.72%, outperforming the next-ranked system, which recorded a DER of 19.3%, representing a relative reduction of approximately 24%. Compared to NVIDIA’s previous streaming baseline, the diar_streaming_sortformer_4spk-v2.1, Nemotron 3 demonstrated a reduction in DER across various evaluation sets while expanding support to eight speakers. Performance metrics reported on an NVIDIA RTX PRO 5000 (using BF16 and the NeMo PyTorch backend) indicate a batched real-time factor speedup (RTFx) of 15,113× at a 30.4-second buffer (batch size 32, torch.compile) compared to 2,619× for the baseline. At a 1.04-second configuration, the model reached an RTFx of 865× versus 136× for the baseline. It is important to note that these results reflect batched throughput measurements and do not represent single-stream end-to-end latencies.
Live Demonstration and Developer Integration
NVIDIA provides a live demo of Nemotron 3 Diarization and offers integration with Argmax Pro SDK 3, allowing developers to explore streaming diarization, multilingual live microphone scenarios, and pre-diarized transcription workflows. VoiceArena has indicated that its results may be subject to change as it completes its Version 1 evaluation and statistical analysis.
Original source: Hugging Face Blog