Hybrid future masking
Patch-mask preserves local future context; full-mask enforces strict past-to-future prediction. Together they align V-JEPA pre-training with planning.
V-JEPA-native world-action modeling
WA-JEPA
Rethinking the Video JEPA Paradigm for World-Action Modeling in Autonomous Driving
1Afari Intelligent Drive 2University of Electronic Science and Technology of China 3Southeast University 4Beijing University of Posts and Telecommunications 5Tianjin University
Closed-loop driving
Plans that shape the next observation.
WA-JEPA runs zero-shot in HUGSIM. Each predicted trajectory moves the ego vehicle, changes the camera observations, and becomes the context for the next planning step.
Abstract
Predict future semantics, then act.
Video Joint Embedding Predictive Architecture (V-JEPA) learns powerful spatiotemporal representations from video through self-supervised latent feature prediction. However, V-JEPA is built around random-mask completion and deterministic regression, making it fundamentally ill-suited for autonomous driving planning that demands future-directed prediction tightly coupled with action. To address this, we rethink the V-JEPA paradigm and present WA-JEPA, a V-JEPA-native world-action model designed for autonomous driving planning. Instead of random spatiotemporal masking, WA-JEPA employs hybrid future-masked pre-training, where the model infers future latents from observed context. Departing from deterministic regression, we recast future prediction as conditional flow matching over latent futures, which substantially improves the model's ability to generate plausible future latents for downstream planning. Finally, a joint future-action predictor is proposed to denoise future scene tokens and ego trajectories together in a unified spatiotemporal latent space, allowing action supervision to directly shape planning-relevant world representations. Pre-trained on nuPlan videos and fine-tuned on NAVSIM, WA-JEPA reaches 91.7 EPDMS on NAVSIM-v2, surpassing the strongest end-to-end and world-action baselines by 1.6 and 1.3 EPDMS, and, without HUGSIM-specific fine-tuning, attains the best HD-Score of 0.4462 on the closed-loop HUGSIM benchmark under the same evaluation protocol. These results validate V-JEPA-native world-action modeling as a powerful and scalable paradigm for autonomous driving planning.
A new modeling paradigm
Rich semantics, future rollout, and planning.
Video-generation WAMs can roll out futures but inherit reconstruction-oriented latent spaces. V-JEPA offers rich semantics but lacks causal rollout and direct planning. WA-JEPA unifies all three capabilities.
Method
A two-stage path from video to action.
Stage 1 adapts V-JEPA to multi-view future prediction. Stage 2 couples future scene generation and ego planning through a shared flow predictor.
Latent flow matching
Future scene tokens are generated through conditional flow matching instead of deterministic regression, preserving richer temporal dynamics.
Joint future-action prediction
Scene and action streams interact inside one predictor, allowing action supervision to shape planning-relevant world representations.
Results
State of the art, from open loop to closed loop.
#1 NAVSIM-v2
91.7
EPDMS on navtest
#1 HUGSIM
0.4462
HD-Score across 436 scenarios
WA-JEPA exceeds the strongest end-to-end baseline by 1.6 EPDMS and the strongest world-action baseline by 1.3 EPDMS on NAVSIM-v2. In HUGSIM, it achieves the best NC, DAC, TTC, PDMS, RC, and overall HD-Score under the shared evaluation protocol.
Qualitative results
Planning across turns, forks, stops, and traffic.
Predicted trajectories remain aligned with road geometry and navigation intent across representative NAVSIM scenarios.
Citation
Build on WA-JEPA.
@article{wang2026wajepa,
title={WA-JEPA: Rethinking the Video JEPA Paradigm for
World-Action Modeling in Autonomous Driving},
author={Wang, Xinlin and Xiang, Yujiao and Zhou, Yuheng and
Wang, Jingqi and Huang, Minqing and Huang, Jiajie and
Wei, Dongxu and Zhou, Tingguang and Wang, Xiyang and
Chen, Gong and Xu, Zhi and Tan, Feiyang and
Zhou, Hangning and Yang, Mu},
journal={arXiv preprint arXiv:2608.20974},
year={2026}
}