Physics in games is a lie we tell convincingly. Every crate you knock over, every ragdoll that tumbles down stairs, every explosion that sends debris scattering—it's all hand-tuned by engineers who spend months tweaking friction coefficients and collision meshes. And the moment you want a character with four arms, a robot that crawls, or a creature that moves like nothing on Earth? The physics break. You start over.
But what if physics didn't need to be coded at all? What if it could be learned by watching videos?
That's the premise behind CLAP (Cross-embodiment Latent Action Pretraining) , a method developed by researchers at the University of Toronto, Vector Institute, and NVIDIA. CLAP learns physical dynamics from human egocentric videos—people cooking, cleaning, grabbing objects—and then applies that knowledge to robots with completely different bodies. Zero fine-tuning. Zero paired data. Just a latent action space that encodes how things move and interact.
For game developers, this isn't just an academic curiosity. It's a potential end to the physics pipeline as we know it. Here are seven ways CLAP is transforming physical simulation, and what that means for the games you build.
The core problem in physical simulation is representation. How do you describe "pushing a cup" in a way that applies to a human hand, a robotic claw, and a fantasy creature with tentacles? Traditional approaches require explicit labels—joint angles, forces, torques—which are different for every body type.
CLAP sidesteps this entirely. It uses a variational autoencoder (VAE) to compress video frames into low-dimensional latent action vectors. Think of it as a compression algorithm for motion. The model watches a video, identifies what's changing between frames, and encodes that change as a compact vector. No labels. No annotations. Just the raw visual information of things moving.
The training data comes from Epic-Kitchens, a dataset of over 100,000 video clips of humans performing everyday activities—chopping vegetables, opening drawers, pouring drinks. CLAP ingests all of it and learns a latent action space that is:
Gaming application: Imagine NPCs that learn movement from real human footage. Instead of hand-authoring every animation transition, you train a model on YouTube videos of people performing tasks, then drive your characters with the learned action vectors. The result is motion that feels authentically human because it is authentically human—just filtered through a latent space.
Key Takeaway: CLAP learns a universal "action language" from videos, compressing physical interactions into vectors that work across any body type.
Here's where CLAP gets genuinely interesting. In traditional robotics, you train a model on one robot, then painstakingly adapt it to a different robot. The embodiment gap—the difference in limb lengths, joint limits, and mass distribution—usually requires new training data or careful domain adaptation.
CLAP doesn't do any of that. Because its latent actions are embodiment-agnostic, you can train on human videos and deploy on a Franka arm, a mobile manipulator, or a quadruped—all without fine-tuning.
The paper reports an 80% success rate on zero-shot manipulation tasks across embodiments. In one experiment, a Franka arm trained on human kitchen videos picks up a cup it has never seen, in a configuration it has never encountered. In another, a mobile manipulator navigates a cluttered environment and pushes objects aside using only latent actions learned from human demonstrations.
Gaming application: This is the holy grail for character variety. In a game with diverse body types—a hulking brute, a nimble rogue, a floating sentinel—you want them to interact with the world consistently. CLAP lets you train one model on real human motion and apply it to all of them. The brute's massive arms and the rogue's slender limbs use the same action vectors, interpreted through their different morphologies.
Key Takeaway: CLAP transfers learned skills across completely different body types with zero retraining, closing the embodiment gap that plagues traditional simulation.
Traditional physics engines are rule-based. You define collision shapes, mass, friction, restitution, gravity—then solve equations every frame. It works, but it only handles the interactions you explicitly model. Soft bodies, fluid dynamics, deformable objects, or novel materials all require bespoke systems.
CLAP takes a different approach. It's a video world model—a neural network trained to predict future frames given current frames and latent actions. The physics isn't coded; it's learned from observation.
When CLAP predicts the next frame, it implicitly encodes:
Because the model learns from real videos, it handles unstructured environments naturally. Cluttered tables, irregular objects, and partial occlusions don't break it—they're just variations in the training distribution.
Gaming application: For procedurally generated worlds, hand-authoring physics for every possible object combination is impossible. A learned world model adapts to whatever geometry appears. Drop a thousand unique objects into a scene, and CLAP predicts how they'll interact without a single physics rule being written.
Key Takeaway: CLAP replaces rule-based physics with learned dynamics, handling complex interactions that traditional engines struggle with.
Physics engines have a strict requirement: they must run in real time. A simulation that takes two seconds to compute one second of motion is useless for interactive games.
CLAP is surprisingly lean. The model uses 120 million parameters—small by modern AI standards—and can predict 5 future frames at 224x224 resolution at 10 fps. That's not just fast enough for offline planning; it's fast enough for model-predictive control in live settings.
The architecture is efficient because it doesn't generate full video sequences. It predicts a limited horizon of frames, encodes them, and discards them. The latent action space is the real output—the video prediction is just a means to evaluate candidate actions.
Gaming application: Real-time physics for interactive simulations becomes feasible on consumer hardware. A game can run CLAP's world model to predict the consequences of player actions, enabling dynamic physics without pre-baked animations or canned destruction sequences.
Key Takeaway: At 120M parameters and 10 fps inference, CLAP is light enough for real-time use in interactive applications.
Imitation learning—teaching an AI by showing it examples—has a dirty secret: it usually requires the demonstrator and the imitator to have the same body. You can't directly copy a human's arm trajectory to a robot with different joint limits.
CLAP's latent action space solves this. Because actions are embodiment-agnostic, they serve as a shared interface for imitation. A robot watches a human video, extracts the latent actions, and replays them with its own body. The action intent transfers even though the kinematics don't match.
The paper shows an average 15% improvement over prior methods in cross-embodiment transfer tasks. That's significant because prior methods often required some paired data or shared structure. CLAP does it with zero paired examples.
Gaming application: AI that learns from watching players. Imagine an NPC that observes your playstyle—how you approach obstacles, how you interact with objects—and then applies those learned behaviors to its own body. Or a game that ingests real-world footage of martial arts and teaches its characters those techniques through latent action imitation.
Key Takeaway: CLAP enables imitation learning across different body types, with a 15% average improvement over prior transfer methods.
Planning is hard. An agent needs to anticipate the consequences of its actions before committing to them. Traditional planning in games uses search trees or scripted behaviors. CLAP offers something more elegant: search over latent action sequences.
Here's how it works:
This is model-predictive control (MPC) powered by a learned simulator. It's zero-shot in the sense that the world model was never trained for the specific task—it just knows how physics works from watching videos.
The paper demonstrates zero-shot planning in unseen environments: a robot enters a room it has never seen, observes the scene, and plans a multi-step interaction—pushing an obstacle aside, grasping an object, and moving it to a target location—all without task-specific training.
Gaming application: NPCs that plan multi-step interactions with objects. Instead of scripted "pick up, carry, place" routines, characters can reason about novel situations. A guard sees a crate blocking a door, plans to shove it aside, and executes—because it understands physics, not because a designer anticipated that specific scenario.
Key Takeaway: CLAP enables zero-shot planning by searching over latent action sequences and simulating outcomes with the learned world model.
CLAP is not the endgame—it's a proof of concept. The paper acknowledges significant limitations:
But the direction is clear. Video world models are becoming viable physical simulators. The trend suggests a future where games don't ship with physics engines at all—they ship with learned models that generate physics on demand, trained on whatever data is relevant to the game world.
For open-world games, this could mean dynamic storytelling where objects behave consistently without hand-authoring. For multiplayer games, it could mean physics that adapts to player-created content. For simulation games, it could mean worlds that respond to emergent behavior in ways no rule-based engine could predict.
Call to action: The paper is on arXiv (2410.10773), and the approach is designed to be reproducible. If you're building systems that need physical interaction, start experimenting with video world models now. The infrastructure is still raw, but the direction is inevitable.
Key Takeaway: CLAP is a stepping stone toward general-purpose physical simulators, with clear limitations but a clear trajectory.
CLAP (Cross-embodiment Latent Action Pretraining) is a method that learns a shared latent action space from human videos using a variational autoencoder. It compresses video frames into action vectors, then uses a video world model to predict future frames conditioned on those actions. This enables physical simulation without explicit physics modeling.
Because latent actions are learned to be embodiment-agnostic, they describe "what happens" rather than "how a specific body does it." A grasping action vector means the same thing for a human hand and a robotic claw, so the model transfers without retraining.
CLAP is trained on Epic-Kitchens, a dataset of over 100,000 egocentric video clips of humans performing everyday activities. The model learns physical dynamics and action primitives from this diverse set of interactions.
Yes. At 120 million parameters, CLAP predicts 5 future frames at 224x224 resolution at 10 fps, making it suitable for model-predictive control in real-time applications.
The main limitations are data diversity (trained primarily on kitchen activities), limited prediction horizon (5 frames), and untested performance on highly novel embodiments. Long-horizon tasks and unusual morphologies remain open challenges.
Traditional simulators use explicit rules and equations, which are precise but require manual tuning and break with novel interactions. CLAP learns dynamics from data, handling unstructured environments naturally but with less precision for well-understood physical scenarios.
The paper is publicly available on arXiv (2410.10773). The research is from the University of Toronto, Vector Institute, and NVIDIA, and the method is designed to be reproducible.
CLAP can drive NPC animation from real human motion, enable cross-body skill transfer for diverse character types, provide learned physics for procedurally generated worlds, and power real-time planning for interactive AI behaviors.
The seven ways CLAP is transforming physical simulation converge on a single idea: physics can be learned, not coded.
For decades, game developers have been hand-tuning physics engines, fighting with collision meshes, and scripting interactions that break the moment a player does something unexpected. CLAP offers an alternative—a world model that understands how objects move and interact because it has watched hundreds of thousands of videos of real physical interactions.
The implications extend beyond gaming. If video world models can serve as physical simulators for robots, they can also serve as simulators for virtual worlds. The same technology that lets a Franka arm pick up a cup in a zero-shot setting can let a game character knock over a stack of crates with physics-plausible motion, learned from reality rather than approximated by equations.
The limitations are real, but they're limitations of scale and data, not of fundamental approach. As video datasets grow broader and world models become more sophisticated, the line between "physical simulator" and "video generator" will blur until it disappears.
The next physics engine might not be an engine at all. It might be a neural network that watched a lot of YouTube.
Ready to see CLAP in action? Dive into the full paper on arXiv and start experimenting with video world models in your own projects!