Published October 20, 2026, NVIDIA introduced Cosmos3-H-Dreams, a research simulator that generates interactive surgical video conditioned on instrument kinematics recorded during clinical cholecystectomy on the CMR Surgical Versius platform. NVIDIA says this is the first real-time generative surgical simulator trained on clinical cholecystectomy recordings to support live kinematic control.
From tabletop tasks to clinical scenes
Cosmos3-H-Dreams builds on earlier work that demonstrated interactive simulation in tabletop robotics. Inside the body, small tool motions produce complex tissue deformation, occlusion and changing reflections, so temporal consistency and adherence to controls are essential. NVIDIA frames visual realism as a necessary but not sufficient validation for biomechanical accuracy and emphasizes that fidelity to tool commands is required for interactive environments.
Training, distillation and runtime
The model is specialized from Cosmos3-H-Embodied-Nano, a 16-billion-parameter model pretrained on Open-H-Embodiment. Training used 832×480 video with actions aligned at 10 Hz and a 28-dimensional action vector covering pose and gripper for both arms plus energy controls and thumbstick axes/buttons. A bidirectional teacher was trained on horizons of 24 then 72 frames (2.4 and 7.2 seconds at 10 Hz) and distilled into causal students via a teacher-to-student pipeline that uses Causal Teacher Forcing and Self-Forcing.
Two student checkpoints are provided: a four-step denoising student and a separately trained two-step student, giving researchers a trade-off between visual quality and inference cost. Streaming generation is accelerated with FlashDreams; an adapted FadeMem module consolidates the growing attention cache during long rollouts. NVIDIA reports the system runs interactively on a single NVIDIA RTX PRO 6000 GPU.
Data, adaptation and availability
NVIDIA notes Cosmos3-H-Dreams can generalize to unseen instrument trajectories and supports adaptation to other datasets and controllers that provide synchronized video and compatible action representations. A step-by-step recipe for teacher fine-tuning and Self-Forcing distillation, model checkpoints, and code repositories are provided as named artifacts for researchers. NVIDIA also lists research directions including modeling the full procedure, combining learned visuals with numerical simulation, surgeon evaluation, and expanding instrument support. The release is described as a research simulator, not a validated system for patient care.
Original source: Hugging Face Blog