Building physical intelligence through interpretability

Scaling data and model size has produced increasingly capable language models. Efficient scaling of physical intelligence, however, remains an open problem. Current models still fail to generalise to unfamiliar physical conditions, from video models mispredicting motion at unseen speeds to robots struggling with unfamiliar objects, limiting their robustness, safety and capabilities.

At World Mechanics, our core hypothesis is that these limitations arise from inaccurate internal abstractions of the physical world (the concepts and relationships models use to represent it). We argue that interpretability is the core tool for understanding and overcoming these limitations. By examining what models learn and use, and where and how they fail, we can identify what needs to change in data, architecture and training. Our goal is to develop training recipes that make physical generalisation scale more efficiently, enabling more capable, robust and safer foundation models.

The scaling laws paper for LLMs by Kaplan et al. showed that language-model capabilities scale predictably with model size, training data and compute. [1] The corresponding challenge for physical intelligence is to find training recipes that turn more data, model capacity and compute into increasingly general models of the physical world. However, the results so far show that physical generalisation remains an open problem.

For example, Kang et al. trained diffusion-based video generation models to predict simulated 2D object motion and collisions. Increasing model size and scaling training data from 30,000 to three million videos did not resolve failures at held-out speeds or object sizes; the models mimicked similar training examples rather than learning the underlying physical laws. Scaling did improve combinatorial generalisation, but not extrapolation to these unseen physical conditions. [2] In EgoDyn-Bench, Qwen3-VL models from 2B to 8B parameters stayed at 37–40% balanced accuracy when inferring acceleration, braking and turning from driving video. Supplying the 8B model with explicit kinematic time series raised this to 62.2%, pointing to visual grounding, not just model size, as a bottleneck. [3] In robotic manipulation, π₀'s average success dropped from 72.3% with familiar objects and placements to 58.5% with unseen objects in randomised poses. [4] These examples expose different limitations in extracting motion information from video and transferring manipulation skills to unfamiliar objects.

We believe LLM scaling has benefited from learning on data that humans have already abstracted into language. In coding and mathematics, inexpensive verification has also supported gains from reinforcement learning. [6, 7] These advantages do not carry over directly to models of the physical world. Unlike human-written text, which is produced to communicate meaning, raw video and sensor streams contain noisy, high-dimensional measurements of continuous processes. Long stretches may show little change between events such as collisions or equipment failures. [8] Many physical tasks also involve open-ended environments where reliable behaviour is difficult to verify and collecting experience requires time, costly equipment and potentially unsafe experiments. [9]

These results suggest that efficient scaling requires going beyond larger training budgets to understand and address the underlying limitations of physical generalisation. We need methods to identify which shortcomings more training can resolve and which call for changes to data, architecture or the training recipe.

We believe a central limitation lies in the internal abstractions these models learn, including how they represent physical variables, entities, relationships and dynamics. Different abstractions can fit the same observations while behaving differently in new situations. For example, a model could learn to use an object's size as a proxy for its mass across the training distribution. It may then fail for a large, lightweight box during deployment, mispredicting how it moves when pushed. We care about whether an abstraction preserves the physical and semantic distinctions needed for prediction and action, rather than merely fitting familiar observations.

It is still unclear which architecture and training setup is best suited to learning transferable physical abstractions. Generative video models, latent-space predictors such as V-JEPA, world-action and vision-language-action models for robotics, and time-series models take different approaches, but they all have the shared goal to learn broadly generalising representations of the physical world that allow these approaches to scale. Because of this shared goal, we remain agnostic to any specific method and study the representations and failure modes of each to understand both common limitations and the architectural and training choices that lead to successful generalisation.

Figure 01

We propose to use interpretability as a central tool to connect the analysis of existing models to how we improve their robustness, safety and scalability. Using white-box evaluations, we identify what internal representations a model has and how it uses them. Collecting these insights across model architectures, training recipes and data distributions informs how to design and train models with transferable abstractions. Such improvement loops have already been shown to improve the capabilities and safety of LLMs, for example by improving multi-hop reasoning in looped transformers [19] and improving LLM alignment through synthetic persona pretraining [18], both informed by interpretability insights.

1. White-box evaluations

Black-box benchmarks expose failure modes, but do not by themselves explain what went wrong internally. Examining what a model represents in examples where it fails, and how it uses those representations, allows us to connect failures to missing or conflated variables, incorrect relationships, or relevant information that is present but unused. These diagnoses help distinguish what needs to be learned from what the model already knows but fails to use.

New tools for physical representations

These evaluations require post-hoc interpretability methods adapted to models of the physical world. Differences in architecture and training change how features are organised. For example, V-JEPA 2 applies its predictive loss only to masked patches, while visible tokens can aggregate scene-wide information, complicating local attribution. [10, 11] Our goal is to test whether particular physical variables and relationships are represented and used correctly.

• Structure-aware featurizers. We develop methods to recover continuous variables, semantic concepts and object–property relationships in video and time-series models, including how variables bind to objects and persist over time.
• Decoders. Some models of the physical world do not produce directly legible outputs. JEPA models, for example, emit an encoding. Diffusion decoders map these representations back to pixel space, making them useful for inspecting their content and the effects of interventions. [5]
• Causal interpretability. By intervening on features, we can inspect their effects on decoded observations and on the original model's predictions. For continuous variables such as direction of movement, inspecting the top-activating examples alone does not establish what a feature represents or how the model uses it.

Example: finding physical variables in V-JEPA 2

In our internal V-JEPA 2 experiment, manifold steering worked better than linear steering, but changing one variable still affected representations of others. Building on Joseph et al.'s probe-removal results [12], we fitted direction, speed and acceleration with spline-based manifold steering [13]. These variables follow low-dimensional manifolds, motivating featurizers that go beyond linear directions. The remaining cross-effects also show the challenge of disentangling conflated variables.

Figure 02

2. Model improvement loop

White-box results provide the basis for a model improvement loop. We use interpretability to identify fixable shortcomings and guide changes to data, architecture and training. The aim is to spend capital and compute on changes that address the underlying error, instead of executing larger training runs that are not guaranteed to scale. Understanding which changes matter should reduce both the resources needed to learn a capability and the effort required to find an effective training recipe.

Work on influence functions and training-data attribution provides a precursor at the data level by tracing predictions back to training examples and helping identify dataset errors. [14, 15] We want to extend this principle across the full training stack, investigating how architectural and training choices shape what a model learns and uses.

Figure 03

Example: windshield wipers and collision prediction

In an internal research project, we investigated the representations used by BADAS, a V-JEPA-based collision predictor trained on dashcam video. [16] Attribution methods showed that the model attended to cars and road markings, but also identified an unexpected predictor, namely active windshield wipers. We initially suspected a spurious correlation with rainy weather, but found that several features responding to vehicles crossing into the camera's view also responded to the moving wiper. This suggested that the model treated the motion of the wiper, a component of its own car, like the motion of an external vehicle crossing into its trajectory.

Figure 04

The diagnosis suggests a targeted change to the training mixture, for example diversifying examples with active wipers. If the same part–whole error persists across varied examples, it could instead motivate changes to how the model represents objects and their components. Re-evaluation would test whether the intended representation changed and collision predictions improved under new conditions. The efficiency question is whether this targeted correction works better than spending the same resources on broader data collection or longer training.

Beyond individual corrections, we want to understand how training methods shape physical abstractions, when those abstractions emerge, and which choices lead to successful generalisation. Studies of LLM representation geometry [17] and early pretraining interventions [18] motivate investigating these questions in physical models. Learning from both failures and successful abstractions should let each iteration inform subsequent training, rather than require a separate correction for every failure.

3. Towards more efficient scaling

Over time, we expect the improvement loop to inform how we build physical structure into models. Our hypothesis is that actively designing parts of this structure, instead of relying on training alone to discover it, can improve learning efficiency and robustness while giving models a clean internal structure by design.

Figure 05

• Structured latent spaces. We aim to develop models with explicit internal structure for objects, their properties, motion and relationships, while retaining capacity for higher-order phenomena that may not have a simple interpretation. This would allow us to test aspects of a model's physical understanding directly from its representations, rather than only from its outputs.
• Explanation traces for deployment. We will investigate how models with known causal structure can trace predictions through the physical variables and relationships used in their computation. In manufacturing and medical applications, these traces should help engineers and clinicians inspect and document the basis for a decision, including when investigating failures or questions of liability.
• Zero-shot readout and simulation. We will test whether known latent variables allow physical-state questions to be answered directly from video or sensor data, without a separately labelled dataset for each question. Interventions on those variables would allow us to simulate alternative outcomes, with both readouts and predicted effects checked against the physical system.

Our central goal is to use interpretability to unlock a new generation of physical foundation models that generalise more reliably, learn more efficiently, and scale to increasingly complex environments. By understanding and deliberately shaping their internal representations, we aim to make physical intelligence more capable, robust, and safe as models and training compute scale.

We believe understanding and improving the abstractions models learn is central to more efficient scaling of physical intelligence. Interpretability is the core tool we propose for this, connecting white-box evaluations to model development across data, architecture and training. By incorporating physical structure into model design and training, we hope to build a new generation of highly capable, efficient and safe foundation models for the physical world.

We thank Suraj Srinivas, Junpu Wang, David Klindt, Benno Krojer, and Dileep George for thoughtful discussions that helped shape the ideas in this post. We also thank our team members for their support in drafting and refining the draft. We’re grateful to the Astera Institute, Laude Institute, and Laude Ventures for their early support, which has helped make our work possible.