Introduction to ToyBlocks
Published on November 6, 2026, ToyBlocks is introduced as the first FoundationWorld, a controlled simulated environment designed for researchers to vary training data, repeat tests, and analyze failures in robot learning. The setup features a tabletop with a robot arm and marked blocks, enabling tasks such as pushing, flipping, stacking, and searching.
How ToyBlocks Operates
ToyBlocks is built around four core requirements: realistic physics and appearance, automatic generation of tasks and demonstrations at scale, challenges that integrate reasoning with dexterity, and extensive task coverage within the domain. The framework utilizes reinforcement learning (RL) and motion planning within Nvidia’s Isaac Lab to generate short-horizon behaviors like non-prehensile pushing and flipping.
It leverages privileged information from the simulator, such as exact poses, to create skills and subsequently produces labeled demonstrations by sequencing these skills with a task-and-motion planner (TAMP). The system can execute hundreds of plans in parallel on a single GPU, achieving up to 24 times real-time performance.
Domain Randomization and Generalization
Domain randomization is employed to test spatial generalization by varying block sizes, table heights, and other physical properties. This approach allows researchers to evaluate how well models generalize across different scenarios.
Research Findings and Benchmarks
ToyBlocks supports hierarchical configurations where a high-level model, Molmo 2, selects skills, and a low-level controller, ManiFlow, executes them based on sensor observations. Initial experiments on vision-language-action (VLA) models indicate poor generalization despite extensive pretraining, though improvements have been noted with techniques like low-rank adaptation (LoRA) fine-tuning.
Agentic Experiments
Agentic experiments utilize frontier models through the Model Context Protocol (MCP). Using Claude Opus 5.5 and Claude Fable 5.1, the team conducted a search-and-spell task where an agent locates blocks to construct three-letter stacks. The results show that the agent successfully solved or correctly declared infeasible 63% of tasks, with errors primarily due to letter confusion (13%) and incorrect orientation (15%).
In classical planning evaluations, the agent completed all 30 stacking tasks in a closed-loop setting. Additionally, when agents generated their own skill descriptions, success rates improved from 24% with default descriptions to 63%, compared to 36% with human-written descriptions.
Interactive Capabilities
The framework also includes a natural-language web interface that connects frontier models to the ToyBlocks environment, facilitating interactive experiments.
Conclusion
ToyBlocks is positioned as a valuable open simulation framework for systematic studies of data, model design, and the transfer of learning between simulation and physical robots.
Original source: Hugging Face Blog