\( \mathcal{N}_{0} \)−TWAM

Overview

Contact-rich tasks are often decided by states that an external camera cannot resolve: a plug may be aligned or jammed, a grasp may be secure or slipping, and a gripper may close without actually loading the object. Vision-language-action policies do not explicitly model the next moment, while existing world-action models usually predict only video. Neither treats contact as part of the future the policy must anticipate.

We present \( \mathcal{N}_{0} \)-TWAM, a tactile-native world-action model for contact-rich manipulation. Rather than regress actions from the present observation, it learns one generative model that rolls forward what the robot will see, what it will feel, and what it should do. Future video and future touch are generated jointly, and the action is then denoised from that predicted multimodal future.

\( \mathcal{N}_{0} \)-TWAM closes that gap with an asymmetric Mixture-of-Transformers, a dual-pathway tactile design, and tactile-aware execution. It is trained on NeoData spanning six embodiments and 450 tasks. Across UniVTAC, NeoSim, and eight real-robot tasks, it outperforms strong VLA policies and vision-only world-action models, reaching 84.5%, 49.4%, and 46.3% average success respectively.

A Tactile-Native World Action Model

The model predicts three coupled streams for each chunk: future video, future tactile, and action. A frame-id causal cascade first lets the video and tactile experts co-generate the coming scene and contact, then lets the action expert denoise a unified end-effector action conditioned on that just-predicted pair. No token can read a future chunk, so the complete predict-then-act factorization is implemented inside one shared attention mask.

The Mixture-of-Transformers isolates modality capacity in private weights rather than by restricting attention. The 5B video expert retains the full 3,072-dimensional width and is paired with 1,024-dimensional action and tactile experts. All three remain mutually attentive through one full-width shared self-attention layer at every block.

This asymmetric design contains 7.16B trainable parameters, roughly half the approximately 15B parameters required by three full-width experts. During inference, predicted video and tactile keys and values are cached before action denoising, so each action step reruns only the slim action expert. A rolling cache and asynchronous chunk generation hide most of the remaining prediction latency behind robot execution.

One Tactile Stream, Two Roles

The predicted pathway treats tactile video as part of the future. Tactile streams use the same frozen video VAE and latent-frame flow-matching objective as scene video, with learned sensor identifiers supporting up to four tactile views. The expert predicts the residual from the initial tactile frame, concentrating capacity on contact onset, slip, loading, and release rather than reproducing a near-static sensor background.

The observed pathway handles the contact already happening. On real robots, a frozen estimator converts each InTac S1 image into a dense three-axis force map and a NeoForce encoder turns that physical signal into tokens. A zero-initialized cross-attention port injects those tokens immediately before the action head. Predicted touch therefore provides anticipatory contact in VAE latent space, while observed touch provides a reactive, sensor-grounded reading in force space.

Training deliberately separates these roles. Large-scale pretraining enables only predicted touch, forcing the model to learn contact dynamics instead of taking current tactile as a shortcut. Task-specific post-training then activates observed conditioning and fine-tunes NeoForce with the action objective. For simulation environments, the same conditioning interface processes simulated tactile observations.

Learning Contact at Scale

Pretraining is based on NeoData across six embodiments and 450 tasks, yielding approximately 7.5 million training clips. Each clip aligns multi-view RGB, per-finger tactile video, language, and a canonical 20-dimensional bimanual end-effector action. A single frozen causal video VAE encodes both camera and tactile streams, while robot-specific action normalization keeps heterogeneous embodiments compatible.

Video, tactile, and action are trained with equal flow-matching weights. The curriculum first learns from real-robot data and then mixes \( \mathcal{N}_{0} \)-TacUMI demonstrations into roughly 60% of each batch. Pretraining runs for 30,000 steps on 128 H800 GPUs and jointly optimizes the video, tactile, and action experts under the multimodal flow-matching objective.

During post-training, language and tactile conditions are independently dropped on 10% of samples. Tactile dropout removes the tokens entirely rather than replacing them with zeros, so tactile-free data, deliberate dropout, and a missing sensor share the same interface. The resulting policy learns a vision-only fallback instead of treating an absent sensor as an out-of-distribution input.

Table 1. Private expert capacity with a 30-layer shared-attention backbone.

Tactile-Punctuated Execution

Long-horizon manipulation fails when one episode-level instruction cannot tell the policy which stage it is currently executing. \( \mathcal{N}_{0} \)-TWAM uses contact as punctuation. Sharp changes in tactile magnitude mark contact onset, secure grasp, release, seating, or loss of load; gripper aperture provides a fallback when the tactile event is weak. These events segment demonstrations into short subtask clips; a language model proposes a concise instruction for each clip, and human review corrects the remaining labels and boundaries.

At inference, a lightweight scheduler keeps a queue of those subtasks. The model runs on the current prompt while the predicted tactile stream anticipates the next contact event. A predicted event can trigger the transition one step early, but the corresponding onset or release in observed touch must confirm it before the switch is committed. A failed grasp can close the gripper without producing a force rise, so the same signal that marks a boundary also verifies physical completion.

Evaluation

We evaluate on eight UniVTAC tasks, twelve NeoSim tasks, and eight contact-rich real-robot tasks on PiPER and Flexiv. Each simulation task uses 100 randomized trials and each real task uses 20 trials.

\( \mathcal{N}_{0} \)-TWAM leads all three suites with 84.5% on UniVTAC, 49.4% on NeoSim, and 46.3% on real robots. The real-robot average improves over \( \pi_{0.5} \) at 30.0%, LingBot-VA at 21.9%, and FastWAM at 14.4%, with the clearest gains on tasks where grasp security, load, slip, or seating is visually ambiguous.

NeoSim separates four single-arm tasks from eight dual-arm tasks. \( \mathcal{N}_{0} \)-TWAM is strongest overall at 49.4% mean success and leads the dual-arm subset by a wide margin, while remaining competitive with the strongest VLA baseline on single-arm tasks.

Under unseen objects, unseen positions, and visual perturbations, \( \mathcal{N}_{0} \)-TWAM also achieves the strongest average generalization score at 51.7%.

Conclusion

\( \mathcal{N}_{0} \)-TWAM turns touch from an auxiliary input into a native part of the future a robot models. Joint visual-tactile prediction provides foresight, force-space conditioning supplies immediate correction, and tactile punctuation connects both signals to long-horizon execution. The 7.2B asymmetric design keeps this predictive loop practical by rerunning only the lightweight action expert during action denoising. Future work will extend the predicted horizon, increase control rate, and broaden tactile-sensor coverage across more embodiments.