\( \mathcal{N}_{0} \)−VTLA

Overview

We present \( \mathcal{N}_{0} \)-VTLA, a vision-tactile-language-action foundation model for fine-grained contact-rich manipulation and offline policy improvement from stored deployment data. Its training recipe combines large-scale visuo-tactile pretraining, staged integration of a predictive tactile pathway, and advantage-conditioned offline reinforcement learning.

Current VLA systems remain weak when the outcome depends on contact. Tactile frames are noisy, nearly empty away from contact, and most informative during the brief moments when contact forms or changes, so treating touch as another camera stream wastes a prefix built for information-dense visual observations. Injecting only the current tactile frame is also reactive: it describes contact that earlier actions have already produced.

\( \mathcal{N}_{0} \)-VTLA instead predicts the net tactile change expected over the next action chunk and conditions the action expert on that estimate. Large-scale NeoData pretraining supplies diverse multi-platform contact experience, while ALTER, Advantage Labeling from Trajectory Events and Relative Progress, learns from failures, object drops, human corrections, and recoveries that clean demonstrations do not contain.

Data and Action Space

NeoData provides a large-scale collection of synchronized RGB, tactile, state, and action data across single-arm robots, dual-arm robots, and an \( \mathcal{N}_{0} \)-TacUMI handheld collection gripper. Every participating gripper finger carries a visuo-tactile sensor, and each episode begins with at least 0.5 seconds of open-gripper, zero-contact observations that establish the tactile baseline used throughout the trajectory.

All embodiments share a fixed-width state and action representation with consistent slots for arm motion and gripper control. Action chunks are expressed relative to their first pose, allowing real and simulated single- and dual-arm trajectories to train one common objective.

Predictive Touch

The base policy combines a pretrained vision-language backbone with a flow-matching action expert that denoises a future action chunk. For each active tactile view, the current image is differenced against its episode-start baseline and processed by a frozen tactile visual encoder with a trainable projection. The resulting compact contact tokens preserve spatial structure while suppressing static gel appearance and mount-specific bias.

A lightweight predictor distills the current tactile tokens together with contextualized scene and language into latent tactile tokens \( z \). Its future target encodes the tactile change from the current frame to the end of the action horizon, averaged across the active tactile views. Symmetric contrastive matching identifies the correct future, while an auxiliary L1 reconstruction anchors the latent to the coarse spatial layout of contact.

The resulting \( z \) tokens are prepended to the noisy action suffix. Current-contact tokens never enter the vision-language prefix or action expert directly; action generation receives touch only through the predictor. This makes touch an internal estimate of contact the next action chunk will create rather than a reaction to contact already made.

The training process below shows how \( \mathcal{N}_{0} \)-VTLA progressively turns the tactile latent from a grounded prediction target into a signal the action expert can use.

Step 1 Step 2 Step 3

‹

›

ALTER: Learning from Deployment Data

ALTER stands for Advantage Labeling from Trajectory Events and Relative Progress. Given a fixed deployment corpus, it performs advantage-conditioned offline RL without additional environment interaction. The corpus combines clean teleoperation, autonomous rollouts, human-in-the-loop corrections, and staged recovery episodes.

Clean demonstrations provide dense progress supervision from duration-calibrated stage intervals. Tactile contact changes identify candidate boundaries, complemented by robot motion, gripper state, and visual events. Imperfect trajectories contribute sparse before-and-after preferences around tactile-detected object drops and logged HIL corrections. These signals supervise a pretrained VLA-based pairwise progress model, but are not inputs to that model.

Comparing a frame with the episode start estimates task phase; comparing it with the frame one action chunk later estimates local execution change. Samples are ranked within each predicted stage, converted into positive or negative advantage text conditions, and used for offline flow-matching policy learning. The top 30% within each stage receive Advantage: positive; all other retained samples receive Advantage: negative. Deployment always uses the positive condition.

Under ALTER, \( \mathcal{N}_{0} \)-VTLA reaches 95%, 80%, and 75% success on Towel Folding, Bag Packing, and Cardboard Box Folding, compared with 90%, 75%, and 60% for the \( \pi_{0.5} \) baseline. The stronger tactile-pretrained checkpoint retains its advantage under the same offline RL procedure.

Evaluation

Evaluation uses closed-loop success on the target robot or simulator; NeoReal also reports a 100-point milestone score. Across nine NeoReal tasks, \( \mathcal{N}_{0} \)-VTLA beats the strongest baseline on every success rate, averaging 47.2% against 29.4% for a pretrained \( \pi_{0.5} \) baseline, and raises mean progress from 42.3 to 56.8. Socket Plugging reaches 85% against 60%, while Board Insertion reaches 25% where both baselines remain at zero.

Across eight UniVTAC and twelve NeoSim tasks, the model averages 63.8%, more than 19 points above the strongest overall baseline at 44.0%. It reaches 83.1% on UniVTAC against 67.1% for InternVLA-A1, and 50.8% on the harder NeoSim suite against 45.8% for \( \pi_{0.5} \).

‹

›

Representative trajectories show the behavioral effect of predictive touch. The policy lifts, realigns, and retries after unexpected rim contact during insertion; regulates gripper aperture on compliant objects; and changes its action path at firm contact while remaining close to the vision-only path in free space.

Conclusion

\( \mathcal{N}_{0} \)-VTLA connects large-scale tactile pretraining, predictive latent tactile tokens, and ALTER offline RL. The three-stage recipe grounds future contact before the action expert learns to consume it, and ALTER converts stored demonstrations, failures, and human corrections into stage-relative advantage conditions. Together they produce a policy that anticipates contact during action generation and continues improving from a fixed deployment corpus without further environment interaction.