\( \mathcal{N}_{0} \)−Foundation

Overview

Large-scale robot learning is still dominated by visual observations, even though many manipulation outcomes are decided by local force, friction, incipient slip, deformation, and contact transitions. These states are weakly observable from external or wrist cameras but directly determine whether an insertion, grasp, fold, wipe, or handover succeeds.

\( \mathcal{N}_{0} \)-Foundation is a tactile-centric foundation for embodied manipulation. It integrates tactile sensing hardware, large-scale multimodal data, hardware-agnostic tactile representation learning, and standardized evaluation into one system, creating a scalable path from contact acquisition to tactile-aware policy learning.

The foundation connects four components. Durable vision-based tactile sensors and \( \mathcal{N}_{0} \)-TacUMI provide a synchronized acquisition stack; NeoData supplies more than 30,000 hours of synchronized visual-tactile interaction across six embodiments; NeoForce converts device-specific tactile images into dense three-axis force fields and learns transferable contact features; and the NeoReal and NeoSim suites standardize evaluation in the physical world and simulation.

Contributions

• Tactile infrastructure: durable vision-based tactile sensors and \( \mathcal{N}_{0} \)-TacUMI make synchronized visual, tactile, pose, and gripper-aperture collection scalable.
• NeoData: a cross-embodiment corpus with more than 30,000 hours, 1.4M episodes, 3.3B timesteps, 8B RGB frames, and 10B tactile frames.
• OpenNeoData: a 5,000-hour open-source subset spanning six embodiments, more than 250 tasks, and over 200 skills.
• NeoForce: a visuo-tactile model that learns transferable representations from hardware-agnostic three-axis force-field supervision.
• Standardized evaluation: NeoReal and NeoSim provide aligned physical and simulated evaluation for contact-rich manipulation.

Data Acquisition Hardware

The collection stack begins with a camera-based visuo-tactile sensor that measures contact through optical deformation. A wear-resistant glass panel protects the surface; an elastic, textured sensing layer responds to pressure and shear; and an embedded RGB camera with a compact controller records the resulting spatial deformation during repeated interaction.

\( \mathcal{N}_{0} \)-TacUMI integrates two of these tactile sensors with a 160-degree fisheye wrist camera, low-drift infrared 6-DoF tracking, and magnetic gripper-aperture sensing in a handheld parallel gripper. Its wrist images, left and right tactile streams, aperture, and relative end-effector motion use conventions aligned with five instrumented robot platforms, connecting scalable robot-free collection with executable robot demonstrations.

NeoData

NeoData contains more than 30,000 hours of interaction collected by nearly 100 human operators. Its 1.4M episodes contain 3.3B timesteps, 8B RGB frames, and 10B tactile frames, with external vision, wrist vision, left and right touch, robot state, and action streams kept in temporal alignment throughout each manipulation.

The corpus spans handheld \( \mathcal{N}_{0} \)-TacUMI demonstrations and five robot platforms: Franka Research 3, Piper, ARX X5, UR5e, and Flexiv Rizon 4s. More than 450 tasks cover deformable-object handling, precision assembly, transport, surface interaction, pouring, opening, semantic manipulation, and bimanual coordination. \( \mathcal{N}_{0} \)-TacUMI contributes 57% of the trajectories, while the five robot platforms anchor the corpus in embodiment-specific execution dynamics.

Each episode passes checks for stream completeness, motion quality, task completion, video validity, and robot transfer feasibility. A four-level hierarchy organizes complete task (L3), subtask (L2), action (L1), and atomic segment (L0): a VLM proposes a task template, a human verifies it, and signal-grounded boundaries localize each fixed interval.

‹

›

NeoForce

Tactile hardware is fragmented: optical gels, capacitive arrays, and piezoresistive skins produce signals with different units, resolutions, and sensing geometries. \( \mathcal{N}_{0} \)-Foundation grounds touch in a device-independent physical quantity instead. Every observation is mapped to a dense three-axis force field with two tangential shear components and one normal pressure component over the sensing surface.

NeoForce learns temporally structured visuo-tactile features on this common force space. Synchronized chunks of RGB frames and force fields are patchified and fused by a shared transformer. A reconstruction head recovers force fields and contact masks, while teacher-guided latent prediction and cross-modal alignment require the representation to infer contact across masked regions, neighboring frames, and visual context.

Table 1. NeoForce force-field reconstruction ablation on held-out tactile contacts.

Evaluation

The evaluation asks a practical question: does touch change what a manipulation policy can do? NeoReal evaluates 10 physical tasks with controlled initial-state distributions, reset procedures, and binary success criteria. A progressive score also records partial milestone completion, separating meaningful task progress from total failure.

NeoSim extends the benchmark with 12 tasks, including four single-arm and eight dual-arm settings. Its tactile simulation renders per-contact force fields in the same three-axis format used by NeoForce, enabling reproducible comparisons across pouring, plugging, delicate grasping, insertion, handover, stacking, and unstacking.

‹

›

Tactile-Aware Policy Evaluation

NeoReal first evaluates imitation, vision-language-action, video-action, and world-action policies without tactile integration. Even the strongest policy leaves substantial room for improvement: \( \pi_{0.5} \) reaches 26.5% mean success and a 38.1 progressive score, showing that visual competence alone does not resolve the contact decisions required by these tasks.

A controlled study keeps the \( \pi_{0.5} \) backbone fixed and changes only how touch enters the policy. Raw tactile image concatenation offers a small gain, direct action-expert conditioning is stronger, and NeoForce force-field conditioning performs best at 32.5% mean success and 47.5 progressive score. The ordering favors physical contact state over device-specific tactile appearance.

Table 2. Success rate / progressive score across all ten NeoReal tasks.

NeoSim exposes the same gap at larger breadth. \( \pi_{0.5} \) reaches 45.8% mean success, followed by LingBot-VA at 32.1%; sustained mutual contact in dual-arm tasks remains the main bottleneck. Together, the two suites measure both policy progress and the value of tactile information under physical and reproducible simulated contact dynamics.

Conclusion

\( \mathcal{N}_{0} \)-Foundation establishes an integrated path from tactile acquisition to policy evaluation. Its hardware makes touch scalable, NeoData supplies cross-embodiment experience, NeoForce learns transferable contact features from a shared force-field space, and NeoReal with NeoSim measures whether that information improves contact-rich behavior. Future work will extend the representation to dexterous hands and non-camera transducers, scale policy learning over the full corpus, and tighten the connection between simulated and real contact.