Generative Video-LiDAR world models for long-horizon multimodal scene simulation with temporal and geometric consistency

Abstract

The training and evaluation of autonomous-driving perception systems rely on multimodal scene data. However, limited real-world long-tail scenarios have made generative world models an important complementary data source. Existing methods mainly focus on unimodal video generation or implicit training supervision, making test-time Camera–LiDAR conflict resolution and explicit consistency assessment difficult. This paper proposes a Video–LiDAR joint generation framework for extended-horizon scene synthesis that introduces consistency constraints into sampling and state space through three mechanisms. First, projection, depth–range consistency, and ray-visibility gradient corrections are incorporated into diffusion sampling to maintain cross-modal geometry. Second, a recoverability-based formulation compares predicted and re-encoded states to constrain temporal and identity consistency over rollout horizons up to 10 s. Third, failure-feedback-driven counterfactual interventions are designed to cover selected long-tail conditions. Experiments on Waymo and nuScenes show that, compared with the MagicDrive-V2-adapted baseline, the proposed method reduces Waymo Proj.Err. by 16.6% (Holm-adjusted p = 0.030) and Drift by 15.5%, while improving IDF1 by 1.6 points. Improvements on nuScenes are smaller and less consistent. With 30% LiDAR retention, the Long-tail mAP decline is reduced from 18.1% to 14.0%. On held-out real long-tail scenes, counterfactual augmentation improves Long-tail mAP from 34.1 to 35.5, compared with 34.6 using ordinary generated data, showing potential utility for challenging long-tail evaluation. Ablation studies reveal distinct effects of the three modules on geometry, identity, and long-tail performance. This work integrates training-stage modeling and sampling-stage correction for recoverability-based multimodal geometric and temporal consistency.

1 Introduction

As autonomous driving evolves from perception-driven toward world-model-driven systems, simulating the dynamic evolution of open-road environments has become a central problem in multimodal intelligent perception [1, 2]. Real-world traffic scenarios are characterized by long temporal horizons, multi-agent interactions, and heterogeneous sensors. However, collected data provide limited coverage of geometry-related long-tail cases, such as occlusion, distant small objects, rare cut-in maneuvers, and LiDAR-sparse regions [3]. Video provides appearance and contextual information, whereas LiDAR offers sparse but geometrically reliable 3D structure, the two modalities are highly complementary [4]. Multimodal world models capable of jointly generating video and LiDAR therefore have the potential to provide controllable synthetic data for autonomous-driving training and robustness-oriented evaluation while characterizing the joint evolution of visual appearance, 3D geometry, and dynamic motion [5].

Existing studies have made progress in video generation, LiDAR generation, and multimodal synthesis. These methods improve temporal continuity via diffusion models, Transformers, or autoregressive architectures; model LiDAR through range images, point-cloud distributions, or ray casting; and establish Camera–LiDAR associations through shared latent representations, BEV, occupancy intermediates, or epipolar attention [6, 7]. Nevertheless, several issues remain regarding geometric consistency. First, latent-space coupling or posterior-loss-based methods introduce geometric constraints only as implicit training supervision, with no explicit correction during sampling, which may bias projection correspondences. Second, methods using BEV or occupancy as generative intermediates are prone to error accumulation during multi-step temporal rollout when their resolution is limited [8]. Third, approaches combining unimodal generation with geometric projection struggle to recover LiDAR reflectance intensity and occlusion-related ray visibility, making joint generation still necessary. In addition, existing synthetic-data methods primarily emphasize quantity and diversity, while paying less attention to samples targeting perception failure modes [9].

To address these issues, this paper explores a Video–LiDAR joint generation framework for multi-step temporal rollout with three complementary designs. The first is projection-based denoising, which incorporates LiDAR-to-camera projection, depth–range consistency, and ray-visibility constraints into diffusion sampling as additional gradient terms, providing explicit geometric correction beyond latent-space alignment. The second is a recoverable world-state constraint, which defines a state representation consisting of BEV, occupancy, motion, and visibility; a re-encoder then constrains the generated future video and LiDAR data to return to the same predicted state, mitigating cumulative state drift across the rollout window. The third is failure-aware counterfactual intervention, which performs minimal, dynamically feasible edits in the state space for four factors: occlusion, distant small objects, cut-in maneuvers, and LiDAR sparsity, generating controllable long-tail samples for robustness-oriented evaluation rather than directly assuming downstream training gains.

The expected value of this work is as follows. Projection-based denoising and the recoverable state constraint extend geometric consistency from implicit training supervision to explicit sampling-stage constraints, helping alleviate geometric inconsistency and temporal drift in multi-step multimodal generation. Failure-aware counterfactual intervention provides synthetic samples for geometry-related long-tail cases, which may support subsequent robustness-oriented training and evaluation of perception systems, while its direct downstream benefit requires separate validation.

2 Related work

2.1 Application scenarios and evaluation

Multimodal scene generation primarily serves autonomous-driving perception training, simulation-based testing, and long-tail sample supplementation [10]. Unlike single-frame image or point-cloud generation, driving-scene generation must jointly model the temporally extended evolution of road structures, traffic participants, and sensor observations [11]. Video provides appearance and contextual information, whereas LiDAR offers sparse yet geometrically reliable 3D ranges and object boundaries; the two modalities are therefore complementary [12]. A world model capable of synchronously generating video and LiDAR can provide controllable data for simulation and robustness-oriented evaluation.

Recent driving world-model research has expanded from sensor generation toward broader scene modeling, large-scale generative simulation, latent state/action representation, and closed- or open-loop evaluation. Recent directions also include foundation-style scene representations and diffusion-based multimodal generation, extending world models from visual forecasting toward structured simulation and joint sensor modeling. These directions place greater emphasis on temporally consistent environment evolution and controllable coverage of rare interactions rather than visual synthesis alone.

Mainstream driving datasets typically provide multi-view cameras, LiDAR, ego poses, 3D annotations, and maps, but their coverage remains limited in adverse weather, rare interactions, and cross-city generalization [13]. Multimodal driving-scene generation is commonly evaluated using complementary video-quality, LiDAR-fidelity, and downstream-task metrics, since no single metric fully captures visual, geometric, semantic, and temporal consistency [14].

2.2 Multimodal generation under implicit and explicit geometric constraints

This section reviews related work according to how geometric and temporal constraints are incorporated into generation. The first category encodes camera and LiDAR into shared latent spaces or cross-modal attention modules, where geometric relationships are mainly supervised through reconstruction or alignment losses [15]. Such approaches integrate readily with large generative backbones, but geometric consistency still depends largely on training-stage optimization.

Related studies on multi-camera spatiotemporal modeling, cross-modal context fusion, multimodal attention, and temporal graph/Transformer architectures show that cross-view association, occlusion handling, predictive tracking, and temporal dependency modeling are important for maintaining coherent multimodal representations [16,17,18,19]. Although these studies focus mainly on video analysis rather than autonomous-driving world generation, their treatment of temporal and cross-modal dependency is relevant to the present problem.

Without explicit sampling-stage correction, projection correspondence may degrade under distribution shift or limited diffusion steps [20]. The diffusion backbone adopted here also uses a joint latent representation; therefore, this work retains latent coupling while introducing explicit geometric correction during sampling.

The second category uses BEV, 3D occupancy, or semantic fields as unified intermediate representations before decoding video and LiDAR [21]. These methods improve geometric interpretability and multi-view consistency, but errors in intermediate states may propagate during multi-step temporal rollout. This paper also uses BEV and occupancy, but mainly as state-recoverability targets rather than mandatory intermediates in the generation pathway.

The third category incorporates projection, depth–range consistency, or epipolar constraints into training losses [22]. However, such constraints are generally imposed during training and do not directly correct geometric inconsistencies arising during inference. Recent diffusion-based driving generation further improves controllable geometric conditioning, but geometry is still commonly introduced through training objectives, latent conditions, or intermediate representations rather than explicit sampling-stage correction. This motivates introducing projection-based guidance into diffusion sampling.

2.3 Differences from the most closely related work

The studies most closely related to this paper include Video–LiDAR and driving world models based on shared latent representations, geometric priors, and controllable temporal generation. These approaches range from rule-based simulation to geometry-aware world modeling and controllable temporal generation. The proposed framework differs mainly in geometric correction, temporal state representation, and long-tail sample construction.

First, existing methods mainly impose geometric relationships through training losses, control conditions, or intermediate representations [23]. The proposed method instead applies projection, depth–range consistency, and ray-visibility constraints directly during diffusion sampling. Second, studies using occupancy or BEV as generation intermediates place these representations directly in the prediction pathway [24]. Here, the state serves as a recoverability target: generated video and LiDAR observations are re-encoded and compared with predicted states. Instance persistence across multi-step rollout further measures temporal consistency. Third, existing long-tail synthesis often relies on random diversification or externally specified conditions [25]. This paper instead performs dynamics-constrained counterfactual edits for occlusion, distant small objects, cut-in events, and LiDAR sparsity, producing controllable samples for robustness-oriented evaluation and downstream validation.

For broader positioning, CARLA provides controllable sensor simulation [26], while vid2vid performs video-to-video synthesis without native joint Video–LiDAR generation [27]. DriveX incorporates geometry, semantics, and motion for driving-scene modeling [28], whereas MagicDrive-V2-adapted emphasizes controllable high-resolution temporal generation [29]. OmniGen provides an additional reference for unified multimodal sensor generation. Compared with these approaches, the proposed framework combines sampling-stage geometric correction, recoverability-based state assessment, and failure-aware intervention in a joint Video–LiDAR pipeline.

Related multimodal studies further clarify cross-modal correspondence and simulation-based evaluation. Translation-based multimodal learning emphasizes correspondence across heterogeneous representations [30], while digital-twin research highlights synthetic environments for embodied-intelligence training and evaluation [31]. Robust in-vehicle perception studies emphasize small and occluded objects under adverse sensing conditions [32]. Together, these studies support evaluating multimodal generation through both sensor-level consistency and task-relevant robustness. Table 1 summarizes the main methodological differences.

Table 1 Comparison with closely related driving-generation approaches

2.4 Summary

Overall, driving world-model research now extends beyond geometric consistency to include large-scale generative simulation, latent state/action modeling, multimodal representation learning, and temporal cross-modal modeling. Three issues remain relevant to this study. First, geometric constraints are still mainly imposed during training, while explicit sampling-stage correction remains limited. Second, multi-step state and identity consistency often lack explicit post-generation assessment. Third, long-tail synthesis is frequently only indirectly connected to specific perception failure modes.

This paper addresses these issues through geometric projection-guided denoising, recoverable world-state constraints, and failure-aware counterfactual intervention. These components retain joint latent diffusion and geometric representations while adding sampling-stage correction, recoverability-based temporal constraints, and failure-oriented counterfactual editing. The methodological distinction therefore lies in integrating these three mechanisms within the same joint-generation and evaluation pipeline rather than treating geometry, temporal state consistency, and long-tail synthesis as separate objectives.

3 Methodology

3.1 Problem formulation

This paper studies multi-step multimodal joint generation for autonomous driving. Given a historical observation window, the goal is to synchronously generate future multi-view video and LiDAR sequences while maintaining cross-modal geometric correspondence, temporal state and instance recoverability, and failure-mode coverage. Rather than defining temporal consistency by a single fixed duration, we treat ${T_f}$ as a variable rollout horizon. The primary setting uses ${T_f} = 12$ frames (approximately 6 seconds), while Chapter 4 evaluates 2, 4, 6, 8, and 10 s horizons to characterize cumulative geometric, state, and identity errors.

Let the current time step be ${\rm{t}}$, with a historical length of ${{\rm{T}}_{\rm{h}}}$ and a prediction length of ${{\rm{T}}_{\rm{f}}}$. The input and output are defined as follows:

$${{\rm{X}}_{\rm{t}}} = \left\{ {{\rm{I}}_{{\rm{t}} - {{\rm{T}}_{\rm{h}}} + 1:{\rm{t}}}^{1:{\rm{M}}},\,{{\rm{P}}_{{\rm{t}} - {{\rm{T}}_{\rm{h}}} + 1:{\rm{t}}}},\,{{\rm{E}}_{{\rm{t}} - {{\rm{T}}_{\rm{h}}} + 1:{\rm{t}}}},\,{\rm{K}},\,{{\rm{T}}_{{\rm{cam}}}},\,{{\rm{T}}_{{\rm{lidar}}}},\,{\cal M}} \right\},{{\rm{Y}}_{\rm{t}}} = \left\{ {{\rm{I}}_{{\rm{t}} + 1:{\rm{t}} + {{\rm{T}}_{\rm{f}}}}^{1:{\rm{M}}},\,{{\rm{P}}_{{\rm{t}} + 1:{\rm{t}} + {{\rm{T}}_{\rm{f}}}}}} \right\}.$$

(1)

Here, ${{\rm{I}}^{1:{\rm{M}}}}$ denotes video frames from ${\rm{M}}$ camera views, ${\rm{P}}$ denotes LiDAR point clouds, ${\rm{E}}$denotes ego poses, ${\rm{K}}$, ${{\rm{T}}_{{\rm{cam}}}}$, and ${{\rm{T}}_{{\rm{lidar}}}}$ denote the sensor intrinsics and extrinsics, and ${\cal M}$ denotes the map prior. The basic objective is to estimate a conditional generative distribution ${{\rm{Q}}_{\rm{\theta }}}\left( {{\rm{Y}}\mid {\rm{X}}} \right)$ that approximates the true distribution ${\rm{P}}\left( {{\rm{Y}}\mid {\rm{X}}} \right)$.

However, distributional approximation alone is insufficient to characterize geometric and temporal state properties. Therefore, we formulate the problem as a constrained optimization problem:

$${{\rm{\theta }}^{\rm{*}}} = {\rm{arg}}\mathop {{\rm{min}}}\limits_{\rm{\theta }} {_{\left( {{\rm{X}},{\rm{Y}}} \right) \sim {\rm{P}}}}\left[ {{{\cal L}_{{\rm{gen}}}}\left( {{{\rm{Q}}_{\rm{\theta }}}\left( {{\rm{Y}}\mid {\rm{X}}} \right),{\rm{Y}}} \right)} \right],$$

(2)

subject to

$$\left[ {{{\cal E}_{{\rm{geo}}}}\left( {{\rm{\hat Y}}} \right)} \right] \le {_{\rm{g}}},\left[ {\parallel {\rm{S}}_{{\rm{t}} + {\rm{k}}}^{{\rm{pred}}} - {{\rm{E}}_{\rm{\psi }}}\left( {{{{\rm{\hat Y}}}_{{\rm{t}} + {\rm{k}}}}} \right)\parallel } \right] \le {_{\rm{s}}},{\rm{k}} = 1, \ldots ,{{\rm{T}}_{\rm{f}}}.$$

(3)

Here, ${\rm{\hat Y}} \sim {{\rm{Q}}_{\rm{\theta }}}\left( {{\rm{Y}}\mid {\rm{X}}} \right)$. ${{\cal E}_{{\rm{geo}}}}$ measures the geometric inconsistency between video and LiDAR in terms of projection, depth–range consistency, and ray visibility. ${\rm{S}}_{{\rm{t}} + {\rm{k}}}^{{\rm{pred}}}$ denotes the state predicted by the state evolution module, which includes BEV geometry, dynamic occupancy, motion fields, instance identities, and visibility. ${{\rm{E}}_{\rm{\psi }}}$ re-encodes the generated observation ${{\rm{\hat Y}}_{\rm{k}}} = \left\{ {{{{\rm{\hat I}}}_{\rm{k}}},{{{\rm{\hat P}}}_{\rm{k}}}} \right\}$ back into the state space. These constraints characterize geometric consistency and state recoverability across different rollout lengths rather than from the 6-second setting alone.

For failure-mode-oriented synthesis, we further define a counterfactual state-space distribution:

$${\rm{Q}}_{\rm{\theta }}^{{\rm{cf}}}\left( {{\rm{Y}}\mid {\rm{X}}} \right) = {{\rm{Q}}_{\rm{\theta }}}\left( {{\rm{Y}}\mid {\rm{X}},{\rm{S}}_{\rm{t}}^{{\rm{cf}}}} \right),{\rm{S}}_{\rm{t}}^{{\rm{cf}}} = {{\rm{S}}_{\rm{t}}} + {{\rm{H}}_{\rm{\omega }}}\left( {{{\rm{S}}_{\rm{t}}},{{\rm{r}}_{\rm{f}}}} \right)$$

(4)

Here, ${r_f}$ denotes failure feedback from the target perception model, and ${H_\omega }$ produces a minimal dynamically constrained state edit. $Q_\theta ^{cf}$ need not approximate $P\left( {Y|X} \right)$; instead, its samples satisfy the same geometric and state constraints, providing controllable long-tail samples for robustness-oriented evaluation and potential downstream training without assuming direct performance gains.

3.2 Overall framework

The overall framework of this work is illustrated in Fig. 1. The model takes historical multi-view videos, LiDAR sequences, ego poses, camera intrinsics and extrinsics, and map priors as input, and outputs synchronously generated multi-view video and LiDAR sequences over the future horizon.

Fig. 1

Overall framework of the proposed video–LiDAR joint world model

Video and LiDAR are encoded by modality-specific VAEs and fused through the shared backbone ${_\theta }$. The state-transition network ${F_\phi }$ predicts latent-state evolution, ${E_\psi }$ re-encodes generated observations for recoverability assessment, and ${H_\omega }$ performs constrained counterfactual editing. A frozen perception model supplies failure feedback ${r_f}$ with stop-gradient.

Training uses two stages. Stage I optimizes the modality encoders, temporal backbone, ${F_\phi }$, and ${E_\psi }$ under generation, geometric-consistency, and recoverability objectives. Stage II freezes these components and updates only ${H_\omega }$ using the counterfactual objective derived from ${r_f}$. Sampling-stage correction uses differentiable projection, depth–range, and visibility residuals with respect to the current diffusion sample to modify denoising rather than introduce an additional training loss.

For implementation, the video and LiDAR modalities are encoded into 128-dimensional latent spaces using modality-specific VAEs. The shared temporal backbone contains 12 Transformer blocks with 768 hidden dimensions. The state-transition network ${F_\phi }$ and re-encoder ${E_\psi }$ use four-layer MLPs with 512 hidden units, while the counterfactual module ${H_\omega }$ adopts a six-layer state editor with 512-dimensional features. Stage I updates the generator and state modules, whereas Stage II only optimizes ${H_\omega }$; the perception model providing failure feedback remains frozen.

Historical observations are mapped to an initial world state ${S_t}$ containing BEV occupancy, dynamic occupancy, motion, object identity, and visibility. Future ego-control is an exogenous condition containing no future camera/LiDAR observations, object states, or annotations. Geometry-projected denoising corrects cross-modal inconsistencies during sampling. The recoverability branch compares states predicted by ${F_\phi }$ with those re-encoded by ${E_\psi }$, providing an explicit model-level temporal-consistency measure. ${H_\omega }$ generates a dynamics-constrained counterfactual state $S_t^{cf}$ following the same generation and assessment pathway.

Thus, the three modules respectively target cross-modal geometry, temporal state and identity consistency, and predefined long-tail failure modes. Sampling-stage correction targets cross-modal geometry, recoverability targets temporal state and identity consistency, and failure-aware intervention targets specified long-tail failure modes.

3.3 Module descriptions

3.3.1 Geometry-projected denoising

Existing Video–LiDAR generation methods mainly impose cross-modal geometry through training losses or shared latent representations, leaving inference-time inconsistencies uncorrected. During multi-step rollout, discrepancies among image boundaries, depth estimates, and LiDAR projections may accumulate. The proposed module therefore corrects video and LiDAR latents directly during denoising without introducing an additional sampling network.

Let the video and LiDAR latents at diffusion step $t$ be $z_t^v$ and $z_t^l$. We use an $\varepsilon$-prediction parameterization, with the base score $- {{{_\theta }\left( {{z_t},t,c} \right)} \over {{\sigma _t}}}$. The denoising network predicts clean estimates ${\hat I_0}$ and ${\hat P_0}$. The geometric energy is

$${\varepsilon _{{\rm{geo}}}} = {\left\| {{{\rm{\Pi }}_{K,T}}\left( {{{\hat P}_0}} \right) \otimes V\left( {{{\hat P}_0}} \right) - {\rm{\Omega }}{{\left( {{{\hat I}_0}} \right)}_1} + {D_\eta }\left( {{{\hat I}_0}} \right) - R\left( {{{\hat P}_0}} \right)} \right\|_1}$$

(5)

where ${\rm{\Pi }}$ denotes LiDAR-to-camera projection, ${M_{{\rm{vis}}}}$ and ${M_{{\rm{fg}}}}$ are the z-buffer visibility and foreground masks, $D\left( \cdot \right)$ is the depth head, and $R\left( \cdot \right)$ is projected range. Points with non-positive depth or outside-image projections are discarded, while duplicate projections retain the nearest positive depth. Only valid projected pixels contribute to Eq. (5).

The visibility and foreground masks are detached during sampling and used only for validity selection. Gradients therefore pass through projection and depth–range residuals to the video and LiDAR latents, but not through discrete visibility decisions.

The diffusion process follows a cosine noise schedule with 1,000 training diffusion steps and a 50-step DDIM inference schedule. During inference, geometric correction is applied at every retained DDIM step using the current latent prediction rather than introducing an additional neural sampling module. The projection operation is implemented through calibrated camera–LiDAR transformations, while depth–range residuals provide continuous gradients to the latent variables. Discrete visibility selection is detached to avoid unstable gradients caused by z-buffer operations. Therefore, the proposed correction functions as a bounded score modification during sampling rather than an independent generative process (See Figs. 2, 3, 4).

Fig. 2

Detailed illustration of geometry-projected denoising

Fig. 3

Detailed illustration of recoverable world-state rollout

Fig. 4

Detailed illustration of failure-aware counterfactual intervention

Inference uses a 50-step DDIM sampler, with geometric correction applied at all retained steps. The guidance coefficient decreases linearly from 0.08 to 0.01 as ${\alpha _s} = 0.08 - 0.07\left( {s - 1} \right)/49$, $s = 1, \ldots ,50$, and the geometric-gradient ${L_2}$ norm is clipped at 1.0. The same 50-step budget is used for all diffusion-based comparisons. The corrected score is

$${\tilde S_t} = {s_\theta }\left( {{z_t},t,c} \right) - {\alpha _t}{\rm{clip}}\left( {{\nabla _{\left( {z_t^v,z_t^l} \right)}}{\varepsilon _{{\rm{geo}}}},{c_g}} \right)$$

(6)

With the clipped gradient constraint, the magnitude of each correction step is explicitly controlled, and the accumulated perturbation remains bounded during sampling. Because the guidance coefficient decreases during denoising, the correction gradually weakens toward low-noise steps. This analysis characterizes perturbation control rather than providing a convergence guarantee or proving exact preservation of the unguided diffusion posterior. Empirical stability is evaluated in Sect. 4.

3.3.2 Recoverable world-state rollout

Multi-step generation may exhibit state drift and discontinuous instance identities that frame-level reconstruction losses cannot adequately characterize. The proposed module therefore introduces a world-state representation independent of pixels and point clouds. Recoverability measures agreement between predicted and re-encoded states, whereas state correctness is assessed separately against held-out ground-truth future states.

The state ${S_t}$ consists of BEV occupancy, dynamic occupancy, motion fields, instance identities, and visibility masks. Without accessing generated observations, the state transition network ${F_\phi }$ predicts future states as

$$S_{t + k}^{{\rm{pred}}} = {F_\phi }\left( {S_{t + k - 1}^{{\rm{pred}}},{a_{t + k - 1}},{\cal M}} \right),\quad k = 1, \ldots ,{T_f}.$$

(7)

Here, ${a_{t + k - 1}}$ denotes the prescribed ego-control or ego-motion command supplied to the world model as an exogenous condition. It does not contain future camera, LiDAR, object-state, or annotation information. During offline evaluation, the same control-conditioning protocol is applied to all compared methods; thus future sensor observations are never used by ${F_\phi }$ to predict the state.

The re-encoder ${E_\psi }$ maps generated observations back into the state space. The recoverability constraint is

$${{\cal L}_{{\rm{rec}}}} = \mathop \sum \limits_{k = 1}^{{T_f}} S_{t + k}^{{\rm{pred}}} - {E_\psi }{\left( {{{\hat Y}_{t + k}}} \right)_1} + {\lambda _{{\rm{id}}}}\mathop \sum \limits_{k = 1}^{{T_f}} {{\cal L}_{{\rm{id}}}}\left( {S_{t + k}^{{\rm{pred}}},{E_\psi }\left( {{{\hat Y}_{t + k}}} \right)} \right)$$

(8)

where ${{\cal L}_{{\rm{id}}}}$ is a cross-frame soft Hungarian matching loss for instance identities. During training, supervision for ${F_\phi }$ and ${E_\psi }$ is derived from ground-truth future states $S_{t + k}^{{\rm{GT}}}$, constructed from LiDAR voxelization, 3D annotations, tracking IDs, and ray casting.

For external assessment, both the predicted state $S_{t + k}^{pred}$ and re-encoded state $\hat S_{t + k}^{rec}$ are compared independently with $S_{t + k}^{gt}$. BEV occupancy uses frame-averaged IoU, motion uses mean endpoint error in the ego-centric BEV frame, visibility uses binary accuracy, and identity consistency uses IDF1 after class-consistent Hungarian matching. Agreement between the two learned branches is therefore interpreted as internal recoverability rather than sufficient evidence of state correctness. These metrics are reported in Sect. 4.3.

Since ${F_\phi }$ does not access generated observations, Eq. (8) cannot be trivially satisfied through observation feedback. The prediction and re-encoding branches therefore distinguish model-level recoverability from externally referenced state consistency.

3.3.3 Failure-aware counterfactual intervention

The counterfactual module performs state-space intervention rather than direct pixel editing. Interventions are restricted to occlusion, distant small objects, cut-in trajectories, and LiDAR sparsity, and the edited state follows the same geometric-correction and recoverability pathways as normal generation.

Given state ${S_t}$ and failure feedback ${r_f}$ from a frozen BEVFusion-based multitask perception model, ${r_f}$ combines 3D detection confidence, cross-frame ID-switch frequency from Hungarian association, and BEV segmentation-boundary error. The intervention network ${H_\omega }$ predict.

$${\rm{\Delta }}S = {H_\omega }\left( {{S_t},{r_f}} \right),\quad S_t^{{\rm{cf}}} = {S_t} + {\rm{\Delta }}S.$$

(9)

The intervention is restricted to the four factors above. When an edit changes object position, trajectory, visibility, or LiDAR density, the corresponding annotations are updated directly from the transformed state: 3D boxes follow the edited geometry, 2D boxes are obtained by calibrated projection, instance IDs are retained for the same object, and visibility, trajectories, and LiDAR-valid masks are regenerated from the edited state. Thus, counterfactual annotations remain synchronized with the generated sensor conditions.

The intervention objective is

$${{\cal L}_{{\rm{cf}}}} = - {{\cal R}_{{\rm{fail}}}}\left( {{{\hat Y}^{{\rm{cf}}}}} \right) + {\lambda _p}{{\cal L}_{{\rm{phys}}}} + {\lambda _g}{{\cal E}_{{\rm{geo}}}}\left( {{{\hat Y}^{{\rm{cf}}}}} \right) + {\lambda _s}{\rm{\Delta }}{S_{1.}}$$

(10)

where the failure term measures perception difficulty and the feasibility term penalizes velocity, acceleration, collision-margin, and trajectory-smoothness violations. Gradients are propagated through only the most recent 8 steps of the 50-step DDIM trajectory. This truncated BPTT limits memory cost while retaining local sensitivity to state edits. The generator remains frozen, so Stage II updates only ${H_\omega }$.

Counterfactual generation and LT-mAP evaluation use independent models. The frozen BEVFusion-based model supplies ${r_f}$, whereas a separately trained frozen TransFusion evaluator computes LT-mAP and provides neither feedback nor gradients to ${H_\omega }$. Higher LT-mAP therefore measures retained detectability and annotation consistency rather than reduced intervention difficulty.

Physical feasibility is evaluated using collision, speed, acceleration, jerk, and invalid-projection violations. Motor-vehicle speed is limited to 35 m/s, absolute acceleration to 4.0 m/s², absolute jerk to 5.0 m/s³, and minimum inter-object clearance to 0.5 m. Invalid projection is recorded when an edited instance has no positive-depth in-frame projection in the relevant camera views. Violation rates and successful/failed cases are reported in Sect. 4.

3.4 Objective function and optimization

To separate objectives with different gradient paths, training is formulated in two stages. Let ${\rm{\Theta }}$ denote the parameters of the modality encoders, denoising backbone ${_\theta }$, depth head, state-transition network ${F_\phi }$, and re-encoder ${E_\psi }$, while $\omega$ denotes the parameters of the intervention network ${H_\omega }$.

In Stage I, the generator and state modules are optimized using

$${{\cal L}_{\rm{I}}} = {{\cal L}_{{\rm{diff}}}} + {\lambda _g}{{\cal L}_{{\rm{geo}}}} + {\lambda _r}{{\cal L}_{{\rm{rec}}}}.$$

(11)

The recoverability term in Eq. (11) already includes the identity-matching component defined in Eqs. (8) and (14); therefore, no additional identity loss is introduced. Eq. (14) is identical to Eq. (8), including the same identity weight. The geometric and recoverability weights are set to 0.5 and 1.0, respectively, based on training-subset cross-validation.

For multimodal diffusion learning, let the joint latent representation correspond to the ground-truth future observation, and let $t$ denote the training diffusion step. The diffusion loss is

$${\mathcal{L}_{{\text{diff}}}} = {\mathbb{E}_{{z_0},,\tau }}\left[ {\left\| {\varepsilon - {\varepsilon _\theta }\left( {{z_\tau },\tau ,X} \right)} \right\|_2^2} \right]$$

(12)

The geometric consistency loss follows the same projection, visibility, and depth–range definitions as Sect. 3.3.1:

$${\mathcal{L}_{{\text{geo}}}} = \frac{1}{{{T_f}M}}\sum\limits_{k = 1}^{{T_f}} {\sum\limits_{m = 1}^M {\left[ \begin{gathered}{\left\| {{\Pi _{{K_m},{T_m}}}\left( {{{\hat P}_k}} \right) \odot V_k^m - \Omega \left( {\hat I_k^m} \right)} \right\|_1} \hfill \\\quad \quad + {\left\| {{D_\eta }\left( {\hat I_k^m} \right) - {R_m}\left( {{{\hat P}_k}} \right)} \right\|_1} \hfill \\ \end{gathered} \right]} } .$$

(13)

The recoverability term in Eq. (11) implements the state-level constraint introduced in Eq. (8). Eq. (14) decomposes recoverability into occupancy, motion, visibility, and identity components:

$${\mathcal{L}_{{\text{rec}}}} = \frac{1}{{{T_f}}}\mathop \sum \limits_{k = 1}^{{T_f}} \left[ \begin{gathered}{\lambda _{{\text{occ}}}}{\mathcal{L}_{{\text{occ}}}}\left( {O_{t + k}^{{\text{pred}}},O_{t + k}^{{\text{rec}}}} \right) + {\lambda _{{\text{mot}}}}{\left\| {M_{t + k}^{{\text{pred}}} - M_{t + k}^{{\text{rec}}}} \right\|_1} \hfill \\\quad \quad \quad \quad + {\lambda _{{\text{vis}}}}{\mathcal{L}_{{\text{vis}}}}\left( {V_{t + k}^{{\text{pred}}},V_{t + k}^{{\text{rec}}}} \right) \hfill \\\quad \quad \quad \quad + {\lambda _{{\text{id}}}}{\mathcal{L}_{{\text{id}}}}\left( {I_{t + k}^{{\text{pred}}},I_{t + k}^{{\text{rec}}}} \right) \hfill \\ \end{gathered} \right].$$

(14)

Here, $O_{t + k}^{{\rm{rec}}}$, $M_{t + k}^{{\rm{rec}}}$, $V_{t + k}^{{\rm{rec}}}$, and $I_{t + k}^{{\rm{rec}}}$ are obtained from the re-encoded state $S_{t + k}^{{\rm{rec}}} = {E_\psi }\left( {{Y_{t + k}}} \right)$. ${{\cal L}_{{\rm{occ}}}}$ and ${{\cal L}_{{\rm{vis}}}}$ denote binary cross-entropy losses for occupancy and visibility, respectively, while the motion component uses an ${L_1}$ distance in the ego-centric BEV frame. ${{\cal L}_{{\rm{id}}}}$ denotes the cross-frame soft Hungarian matching loss defined in Sect. 3.3.2. The coefficients ${\lambda _{{\rm{occ}}}}$, ${\lambda _{{\rm{mot}}}}$, ${\lambda _{{\rm{vis}}}}$, and ${\lambda _{{\rm{id}}}}$ control the contributions of the four state components.

In Stage II, all Stage-I parameters ${\rm{\Theta }}$are frozen and only ${H_\omega }$ is optimized. Eq. (15) is identical to the counterfactual objective in Eq. (10), with the same failure, feasibility, and minimal-intervention terms and unchanged coefficients:

$${{\cal L}_{{\rm{cf}}}} = - {{\cal R}_{{\rm{fail}}}}\left( {{{\hat Y}^{{\rm{cf}}}}} \right) + {\lambda _p}{{\cal L}_{{\rm{phys}}}} + {\lambda _g}{{\cal E}_{{\rm{geo}}}}\left( {{{\hat Y}^{{\rm{cf}}}}} \right) + {\lambda _s}{\rm{\Delta }}{S_1}$$

(15)

The feasibility term penalizes velocity, acceleration, collision-margin, and trajectory-smoothness violations, while the minimal-intervention term limits unnecessary state changes. The failure term updates only ${H_\omega }$; the generator, geometry/depth head, ${F_\phi }$, ${E_\psi }$, and frozen feedback perception model receive no parameter updates. Stage II uses the 8-step truncated backpropagation procedure defined in Sect. 3.3.3. Table 2 summarizes the gradient routing used throughout training.

Table 2 Gradient routing in the two-stage training procedure

The sampling-stage correction in Eq. (6) is inference-only and is not included in either Stage-I or Stage-II training objectives. It acts as the bounded score perturbation defined in Sect. 3.3.1; therefore, no additional geometric loss or parameter update is introduced during DDIM sampling. Exact preservation of the unguided diffusion posterior is not assumed, and empirical stability is evaluated in Sect. 4.

For optimization, AdamW is used with a weight decay of 0.01 and a batch size of 8. The learning rate and AdamW coefficients are reported in the experimental setup.Training uses a 1,000-step diffusion process, followed at inference by the 50-step DDIM sampler defined in Sect. 3.3.1. A 5,000-iteration linear warm-up is followed by cosine annealing. The 1,000 training steps and 50 inference steps therefore refer to different diffusion schedules and should not be interpreted interchangeably.

4 Experiments and results

4.1 Experimental setup

Table 3 summarizes the datasets and their experimental roles. We use the Waymo Open Dataset–Perception as the primary benchmark for Camera–LiDAR geometry and temporal state consistency. It provides synchronized camera and LiDAR data, 2D/3D annotations, and cross-frame instance IDs. We use the official 798 training, 202 validation, and 150 test sequences. Hyperparameters are selected only on the training split using five-fold cross-validation. Proj.Err., Drift, IDF1, LT-mAP, and external state metrics requiring ground-truth labels are computed on the labeled validation split; the official test split is used only for server-based evaluation.

Table 3 Key information of the experimental datasets

nuScenes is used for evaluation under heterogeneous sensing conditions and long-tail scenarios. It contains six cameras, five radars, one LiDAR, and 1,000 20-s scenes, following the official train/validation/test partition. The generator produces camera and LiDAR observations only. Radar is excluded from both generation and geometric correction. For NDS, the frozen evaluator receives generated camera and LiDAR inputs without real radar, preventing mixed real/generated sensor inputs.

All sequences are sampled at 2 Hz. The primary horizon is approximately 6 s, with additional evaluations at 2, 4, 8, and 10 s. LiDAR is voxelized at 0.2 m. The state contains BEV occupancy, dynamic occupancy, motion fields, instance IDs, and visibility, with ground-truth states constructed from LiDAR voxelization, official 3D annotations, tracking IDs, and ray casting.

The generator uses modality-specific Video/LiDAR VAEs and a shared temporal backbone. Training and inference use FP16 mixed precision with FP32 optimizer states. Diffusion training uses 1,000 steps, while inference uses the 50-step DDIM sampler in Sect. 3.3.1 with geometric correction at all retained steps. Stage II backpropagates through the most recent 8 DDIM steps.

Failure feedback ${r_f}$ is produced by a frozen BEVFusion-based multitask model, while LT-mAP is computed by a separately trained frozen TransFusion evaluator with independent parameters. The evaluator supplies neither ${r_f}$ nor gradients to ${H_\omega }$. Counterfactual interventions follow the four edit types and feasibility constraints defined in Sect. 3.3.3.

Training uses four RTX A6000 GPUs, PyTorch 2.2, and CUDA 12.1. AdamW is used with batch size 8 and weight decay 0.01. A 5,000-iteration linear warm-up is followed by cosine decay. Training lasts 50 epochs: Stage I uses 35 epochs, and Stage II updates only ${H_\omega }$ for the remaining 15 epochs with the generator and state modules frozen. EMA decay is 0.999. All diffusion-based comparisons use the same 50-step inference budget.

Evaluation Metrics. Proj.Err. is the image-diagonal-normalized Euclidean Camera–LiDAR projection error averaged over positive-depth, in-frame, visible correspondences across views and rollout frames. Rollout Drift is measured in meters in the ego-centric BEV frame at the final rollout step after class-consistent Hungarian matching.

IDF1 uses the same frozen tracking pipeline for all methods. Frame-to-frame association applies class-consistent Hungarian matching with a BEV IoU threshold of 0.5, and IDF1 is computed from matched generated and ground-truth identity sequences.

LT-mAP is computed by the frozen TransFusion evaluator on occlusion, distant-object, cut-in, and LiDAR-sparsity subsets. Predictions below 0.05 confidence are removed. AP uses 3D IoU thresholds of 0.5 for vehicles and 0.25 for pedestrians and cyclists, with the four subsets macro-averaged. Edited boxes, IDs, trajectories, visibility labels, and LiDAR-valid masks are regenerated before evaluation.

External state evaluation compares predicted and re-encoded states with held-out ground truth using frame-averaged BEV IoU, motion endpoint error, visibility accuracy, and IDF1. These results are reported in Sect. 4.3. FVD and LPIPS measure video quality, symmetric Chamfer Distance measures LiDAR fidelity, and NDS is reported on nuScenes.

Experiments use five fixed random seeds. Performance tables report mean ± standard deviation unless otherwise stated, while 95% confidence intervals, paired-sample Cohen’s d, and Holm-adjusted p-values are reported for the primary paired comparisons with MagicDrive-V2-adapted.

4.2 Baseline methods

This study compares the proposed method with four external baselines and one matched-backbone control. CARLA provides controllable sensor simulation [26], while vid2vid represents video-to-video synthesis [27]. DriveX models driving scenes using geometric, semantic, and motion information [28], and MagicDrive-V2 focuses on controllable multi-view video generation [29]. Because vid2vid and MagicDrive-V2 were originally designed for video generation rather than synchronized Video–LiDAR prediction, we evaluate multimodal adaptations of these methods under the same Video–LiDAR interface. MagicDrive-V2-adapted is used as the primary statistical reference, while DriveX provides a geometry-aware comparison.

For vid2vid and MagicDrive-V2-adapted, the reported results use the original video-generation backbones with additional LiDAR decoders and state-readout heads to support the unified evaluation protocol. These added components are trained using LiDAR reconstruction and supervised state prediction losses, while no Geometry-Projected Denoising, recoverability coupling, or Failure-Aware Counterfactual Intervention is introduced. DriveX retains its original geometry- and motion-aware backbone. All adapted baselines and the proposed method are trained using identical data splits, hardware configurations, optimization schedules, and hyperparameter selection procedures. All diffusion-based methods use the same 50-step DDIM budget, while learned baselines share the same 50-epoch training budget. Parameter and FLOP totals are reported in Table 4.

Table 4 Main performance and computational efficiency comparison

OmniGen is included in the methodological comparison in Sect. 2.3 but not in the quantitative baseline set because a directly comparable implementation with synchronized Video–LiDAR generation, state prediction, and identical long-horizon evaluation settings is unavailable.

The Shared-Backbone Control uses the same Video/LiDAR VAEs, temporal backbone, LiDAR decoder, and state head as the proposed model, but removes Geometry-Projected Denoising, recoverability coupling, and Failure-Aware Counterfactual Intervention. It uses the same data, optimization, 50-step sampling, and tuning settings, and its results are reported in Sect. 4.3.

4.3 Quantitative results

This section reports the main performance, external-state validation, statistical comparisons, and computational efficiency results. All experiments use five fixed random seeds. Table 4 summarizes the primary performance and efficiency results, including the matched Shared-Backbone Control described in Sect. 4.2.

As shown in Table 4, the proposed method reduces Waymo Proj.Err. from 0.247 for the MagicDrive-V2-adapted baseline to 0.206 and Drift from 0.168 to 0.142 m, while IDF1 increases from 75.8 to 77.4 and LT-mAP from 33.7 to 35.1. The Shared-Backbone Control obtains 0.251 Proj.Err., 73.5 IDF1, 0.205 m Drift, and 30.7 LT-mAP on Waymo, showing consistent degradation after removing the three proposed modules. On nuScenes, the proposed method achieves lower Proj.Err. and higher NDS and LT-mAP than the MagicDrive-V2-adapted baseline, while its IDF1 is 0.7 points lower. These cross-dataset differences are smaller than those observed on Waymo, suggesting that the improvements are dataset-dependent rather than uniformly dominant across all evaluation settings.

Panel B of Table 4 shows that the proposed model uses 568 M parameters and 781 GFLOPs, compared with 537 M and 743 GFLOPs for the MagicDrive-V2-adapted baseline. Training time is 59.7 h versus 63.1 h, while inference latency is 1.36 versus 1.32 s per 16-frame clip.

Table 5 reports external-state validation and the primary paired statistical comparisons. In Panel A, the two state branches show different metric patterns. The re-encoded state achieves higher BEV IoU (0.728 vs. 0.714) and lower motion EPE (0.274 vs. 0.289 m), whereas the predicted state has slightly higher visibility accuracy (92.1% vs. 91.7%) and identity IDF1 (77.3 vs. 77.0).

Table 5 External-state validation and primary statistical comparisons

As shown in Panel B of Table 5, the Waymo differences in Proj.Err. and Drift remain significant after Holm correction. The adjusted comparisons for Waymo LT-mAP, nuScenes NDS, and nuScenes LT-mAP do not reach $p < 0.05$.

Because the evaluation contains multiple correlated metrics, all statistical interpretations are based on Holm-adjusted p-values rather than individual uncorrected comparisons. The five-seed evaluation provides an estimate of variability under repeated training conditions; therefore, marginal differences, especially on nuScenes, are interpreted as trends rather than statistically confirmed improvements.

Rollout Drift at 6 s decreases by 15.5%, from 0.168 m for the MagicDrive-V2-adapted baseline to 0.142 m for the proposed method. The metric pattern is consistent with the component results in Sect. 4.6: GeoProj is primarily associated with projection consistency, recoverability with temporal and identity consistency, and counterfactual intervention with long-tail performance. The Waymo differences are clearer for geometric and temporal metrics, whereas the nuScenes results remain mixed across metrics.

Figure 5 shows the convergence of Waymo Rollout Drift. The proposed method reaches approximately 0.15 m at epoch 26, while the MagicDrive-V2-adapted baseline reaches a comparable region around epoch 31. The curves are non-monotonic: at epochs 8–10, the proposed method trails DriveX by approximately 0.012, and local rebounds of 0.008–0.016 occur around epochs 14, 19, and 27. On nuScenes, the corresponding threshold-epoch difference is approximately 9%.

Fig. 5

Convergence of Waymo rollout Drift over 50 training epochs. Curves show mean values across five fixed seeds, and shaded regions indicate ±1 standard deviation

Across Tables 4 and 5, the significant corrected differences are concentrated in Waymo Proj.Err. and Drift. The proposed model has 0.04 s/clip higher inference latency than the MagicDrive-V2-adapted baseline. On nuScenes, IDF1 remains 0.7 points lower than the MagicDrive-V2-adapted baseline, while the selected NDS and LT-mAP comparisons are nonsignificant after Holm correction.

Across Tables 4 and 5, the significant corrected differences are concentrated in Waymo Proj.Err. and Drift. The proposed model has 0.04 s/clip higher inference latency than MagicDrive-V2. On nuScenes, IDF1 remains 0.7 points lower than MagicDrive-V2, while the selected NDS and LT-mAP comparisons are nonsignificant after Holm correction.

4.4 Case analysis

This section examines two representative sequences. Each case summarizes the generated sensor observations, geometric measurements, and counterfactual intervention results. Figures 6 and 7 provide qualitative comparisons of projection alignment, state variation, and failure-oriented modifications.

Fig. 6

Qualitative analysis of the Waymo dusk urban-driving case. Panels summarize generated sensor observations, camera–LiDAR projection alignment, geometric measurements, and counterfactual states before and after intervention

Fig. 7

Qualitative analysis of the nuScenes rainy-night case. Panels summarize generated sensor observations, camera–LiDAR projection deviations, and counterfactual state modifications before and after intervention

Case 1 is a Waymo dusk-intersection sequence (segment-1547). The ego vehicle approaches an unsignalized intersection under low light; a front-left SUV turns left while a rear-right sedan cuts in from the blind spot. LiDAR returns become sparse beyond 45 m. Figure 6 shows stable vehicle alignment relative to ground truth, with generated LiDAR remaining close to the reference geometry and limited projection deviation. The counterfactual intervention increases occlusion difficulty while maintaining calibrated geometry, object identity consistency, and trajectory feasibility. Proj.Err. remains within 0.18–0.23, whereas the MagicDrive-V2-adapted baseline rises to 0.34 during 3.8–4.4 s. BEV offset stays within ±0.25 m, while the baseline exceeds 0.6 m and produces one identity mismatch during the same interval. Detection confidence decreases from 0.86 to 0.79 and then recovers, compared with 0.66 for the baseline. Lane texture is slightly weaker than the ground truth in several early frames, while geometric alignment and identity continuity remain stable.

Case 2 is a rainy-night cut-in sequence from nuScenes (scene-0103). The preceding vehicle merges laterally at 1.4 m/s on a wet road, while raindrop halos degrade the camera view. Figure 7 shows preserved global traffic layout but weaker texture stability over reflective road regions. Generated LiDAR retains coarse vehicle geometry, while projection overlays show larger local deviations under severe degradation. The counterfactual intervention introduces additional occlusion and LiDAR sparsity while preserving the transformed state constraints. During 3.2–3.6 s, Proj.Err. rises to 0.41, BEV offset reaches 0.7 m, and detection confidence drops from 0.81 to 0.58. The MagicDrive-V2-adapted baseline shows approximately 0.9 m geometric drift under the same condition. The proposed method retains lower geometric drift, while rainy-night corruption and wet-road reflections increase visual and cross-modal errors.

Across the two cases, Camera–LiDAR alignment remains more stable when LiDAR provides reliable geometric structure. Simultaneous camera degradation and ambiguous LiDAR reflections increase projection and state errors. These examples illustrate the ability of the intervention module to construct more challenging failure-oriented conditions under geometric and physical constraints, rather than demonstrating that the generated samples represent all possible real-world events.

4.5 Robustness verification

To evaluate robustness under sensor degradation, all methods are tested with LiDAR sparsification and additive Gaussian noise on video frames. Let $\rho$ denote the LiDAR retention ratio and $\sigma$ the Gaussian-noise standard deviation, with $\rho = 1$ and $\sigma = 0$ representing the non-degraded settings. LT-mAP and Proj.Err. are evaluated on the nuScenes validation set, and Fig. 8 reports the corresponding degradation curves.

Fig. 8

Robustness evaluation under predefined sensor degradation conditions. (a) LT-mAP under LiDAR sparsification. (b) Proj.Err..Err. under Gaussian noise. Error bars denote standard deviations over five random seeds

In Fig. 8a, reducing $\rho$ from 100% to 30% decreases LT-mAP for all methods. vid2vid drops from 26.1 to 18.4 (−29.5%), DriveX from 29.0 to 23.6 (−18.6%), MagicDrive-V2 from 30.4 to 24.9 (−18.1%), and the proposed method from 31.5 to 27.1 (−14.0%). At a stronger sparsification setting, the proposed method reaches 22.3 versus 21.7 for MagicDrive-V2, with no significant difference. The difference between methods narrows as fewer LiDAR points remain available for projection constraints.

Figure 8b shows increasing Proj.Err. as video noise rises. Under light noise, the methods remain near 0.25–0.30. Under moderate noise, the proposed method reaches 0.31, compared with 0.39 for MagicDrive-V2 and 0.43 for DriveX. At the strongest tested level, Proj.Err. reaches 0.38 for the proposed method and 0.49 for DriveX. LT-mAP decreases by 4.2 points under the same condition, showing different degradation patterns for geometric alignment and semantic quality.

For attribution, the w/o GeoProj and w/o Rec variants are evaluated under the same perturbations. Removing GeoProj causes an additional 1.8-point LT-mAP decrease under LiDAR sparsification and increases Proj.Err. from 0.31 to 0.36 under moderate video noise. Removing the recoverability constraint has a larger effect near the end of the rollout. These results suggest that GeoProj mainly contributes to robustness against cross-modal geometric perturbations, whereas recoverability is associated with temporal-state stability. However, the contribution of individual modules becomes less distinguishable under simultaneous severe degradation of both modalities.

Physical feasibility is evaluated using the constraints defined in Sect. 3.3.3. Across the generated counterfactual samples, collision, speed, acceleration, jerk, and invalid-projection violation rates are 1.7%, 2.6%, 3.4%, 2.1%, and 1.5%, respectively. These observed violation rates are consistent with the constrained state-editing procedure, although they do not imply complete physical validity under unseen scenarios. Acceleration violations occur most frequently among the five criteria.

The perturbation experiments evaluate empirical stability under predefined degradation conditions rather than providing a formal guarantee across all possible sensor failures. Figure 8 evaluates generation-side robustness under sensor degradation. Downstream utility is evaluated separately in Sect. 4.9 using held-out real long-tail scenes.

4.6 Ablation study

To evaluate the contributions of the three modules, we conduct ablations on Waymo and nuScenes. As shown in Table 6, w/o GeoProj removes sampling-stage geometry-projected denoising while retaining the training-time geometric loss; w/o Rec removes the recoverability constraint; w/o CF removes Failure-Aware Counterfactual Intervention; w/o Geo+Rec removes both GeoProj and Rec; and Full denotes the complete model. All variants use five fixed random seeds and are reported as mean ± standard deviation.

Table 6 Component ablation results

Removing GeoProj increases Proj.Err. by 18% on Waymo and 13% on nuScenes, while Waymo IDF1 decreases by 1.4 points. Removing Rec produces the largest Waymo IDF1 reduction (2.8 points), with only a 4% increase in Proj.Err. Removing CF changes Proj.Err. and IDF1 only slightly but reduces LT-mAP by 2.7 points on Waymo and 2.0 points on nuScenes. These results suggest different functional sensitivities: GeoProj mainly affects projection-related metrics, Rec mainly influences temporal and identity-related metrics, and CF mainly affects long-tail evaluation under the tested conditions.

Jointly removing GeoProj and Rec increases Waymo Proj.Err. by 21% and reduces IDF1 by 3.6 points. On nuScenes, Proj.Err. increases by 18%, compared with 13% when only GeoProj is removed, while LT-mAP decreases by 2.9 points. The joint removal results in larger degradation for several metrics, indicating that the two modules provide complementary contributions to geometric and state consistency. This observation does not imply a strict additive interaction between the modules.

Overall, Table 6 shows different metric sensitivities across the three modules: GeoProj has the largest effect on projection error, Rec has the largest effect on Waymo IDF1, and CF has the largest single-module effect on LT-mAP. The magnitude of these effects varies between Waymo and nuScenes, suggesting that module contributions depend on scene characteristics and evaluation conditions.

4.7 Horizon scaling analysis

To examine error accumulation with rollout length, DriveX, MagicDrive-V2, and the proposed method are evaluated at 2, 4, 6, 8, and 10 s using the protocol in Sect. 4.1. Proj.Err., Rollout Drift, and IDF1 measure geometric consistency, accumulated state deviation, and identity continuity, respectively. The analysis examines how these metrics change as the rollout horizon increases.

As shown in Table 7, all methods degrade as the rollout extends. For the proposed method, Proj.Err. increases from 0.154 at 2 s to 0.291 at 10 s, while Drift rises from 0.071 to 0.238 m and IDF1 decreases from 82.3 to 70.6. MagicDrive-V2 increases from 0.177 to 0.371 in Proj.Err. and from 0.086 to 0.312 m in Drift, with IDF1 decreasing from 81.4 to 66.9. DriveX shows the steepest degradation. The proposed method therefore exhibits a flatter error-growth profile, particularly beyond 6 s, although the absolute performance gap narrows for IDF1 at shorter horizons.

Table 7 Horizon-scaling evaluation on Waymo

Linear fits across the five horizons give Proj.Err. slopes of approximately 0.027, 0.024, and 0.017 per second for DriveX, MagicDrive-V2, and Ours, respectively. Together with the Drift and IDF1 trends in Table 7, these slopes indicate slower error accumulation for the proposed method across the tested rollout horizons.

4.8 Standard generation quality

Video generation quality is evaluated using FVD and LPIPS, while LiDAR fidelity is measured using symmetric Chamfer Distance. Proj.Err. is retained to quantify cross-modal geometric alignment. Table 8 summarizes the results.

Table 8 Standard generation-quality comparison

The proposed method obtains the lowest FVD and Chamfer Distance, while MagicDrive-V2 achieves a slightly lower LPIPS. This mixed pattern indicates that stronger multimodal geometry does not necessarily imply the best frame-level perceptual similarity. The lower Chamfer Distance and Proj.Err. are more directly aligned with the proposed geometric correction, whereas the small LPIPS difference favors MagicDrive-V2 in local appearance similarity.

4.9 Downstream perception validation

To assess whether generated samples provide measurable value for perception training, we compare three training settings: Real-only, Real + Ordinary Generated, and Real + Counterfactual Generated. All groups use the same perception architecture, optimizer, training iterations, real-data split, and augmentation policy. Evaluation is performed only on a held-out real long-tail subset containing occlusion, distant objects, cut-in events, and sparse-LiDAR cases. Table 9 reports the resulting perception performance.

Table 9 Downstream perception performance on held-out real long-tail scenes

Ordinary generated samples produce small changes over Real-only, whereas counterfactual augmentation increases mAP by 0.8 points, NDS by 0.6 points, and long-tail mAP by 1.4 points. The larger change in long-tail mAP is consistent with the intervention module targeting difficult cases rather than broadly improving all perception metrics. Under the statistical protocol of Sect. 4.1, the long-tail comparison should receive the strongest emphasis, while smaller mAP and NDS differences should be interpreted according to their confidence intervals and corrected significance levels. These results provide a direct downstream test on real held-out scenes and separate the effect of ordinary synthetic augmentation from failure-aware counterfactual sampling.

5 Discussion

The results show clear dataset dependence. On Waymo, the proposed method reduces Proj.Err. by 16.6% relative to the MagicDrive-V2-adapted baseline and Rollout Drift by 15.5%, whereas nuScenes gains are smaller and less consistent. The nuScenes IDF1 is 0.7 points lower than the baseline, and the 0.9-point NDS increase is nonsignificant. This suggests that sampling-stage geometric correction is more effective when Camera–LiDAR calibration and LiDAR density provide a stable geometric reference, while heterogeneous sensing conditions reduce its observable benefit.

Compared with the MagicDrive-V2-adapted baseline, the proposed model adds explicit projection and depth–range correction during sampling. Table 7 shows that Proj.Err. and Drift increase for all methods as the rollout horizon extends from 2 to 10 s, while the proposed method exhibits a relatively slower degradation trend. These results provide evidence of improved multi-step consistency under the evaluated settings, although rollout errors continue to accumulate with longer prediction horizons.

The ablation results indicate functional specialization rather than uniform gains. GeoProj primarily affects cross-modal geometry, Rec supports identity and temporal-state continuity, and CF mainly affects long-tail sample evaluation. Removing GeoProj increases Proj.Err. by 18% on Waymo and 13% on nuScenes, whereas removing Rec causes the largest Waymo IDF1 decrease. Unlike occupancy-centered models such as GaussianWorld [24], recoverability here compares an independently predicted state with a state re-encoded from generated observations. Therefore, recoverability should be interpreted as a model-level consistency measure rather than direct evidence of future-state correctness. External state-level evaluation using BEV occupancy, motion error, visibility accuracy, and trajectory consistency remains necessary for additional verification.

Standard generation metrics also show that geometric consistency and perceptual similarity should be interpreted separately. Table 8 reports that the proposed method achieves lower FVD and Chamfer Distance, while MagicDrive-V2-adapted obtains slightly lower LPIPS. This indicates that stronger cross-modal geometric alignment does not necessarily correspond to uniformly improved local visual similarity. Table 9 further provides a downstream evaluation: counterfactual augmentation increases held-out real long-tail mAP from 34.1 to 35.5, while ordinary generated samples increase it to 34.6. The improvement is concentrated in challenging long-tail evaluation cases rather than representing a universal gain across all perception conditions.

Several limitations remain. First, five paired random seeds limit statistical power despite confidence intervals, effect sizes, and Holm–Bonferroni correction, and several nuScenes differences remain marginal or nonsignificant. Second, counterfactual intervention currently covers only occlusion, distant small objects, cut-in events, and LiDAR sparsity, rather than the full range of autonomous-driving failures. Third, the benefit narrows when Camera and LiDAR are simultaneously degraded. Fourth, the geometry-projected correction modifies the sampling trajectory through bounded score adjustments, but it does not provide a formal guarantee of posterior preservation or complete physical validity. Finally, recoverability remains a partial consistency criterion rather than complete external verification of the future world state. Overall, the strongest evidence concerns Camera–LiDAR geometry, temporal-state stability, and selected long-tail conditions on Waymo, while the effectiveness remains dependent on sensor configuration, scene characteristics, and evaluation protocols.

6 Conclusion

This study presents a Video–LiDAR joint generation framework combining geometry-projected denoising, recoverable world-state constraints, and failure-aware counterfactual intervention. Horizon-scaling experiments from 2 to 10 s show increasing geometric and state errors for all methods, while the proposed model shows a slower degradation trend than the compared baselines under the evaluated rollout settings. On Waymo, Proj.Err. decreases by 16.6% and Rollout Drift by 15.5% relative to the MagicDrive-V2-adapted baseline, although nuScenes improvements are smaller and less consistent.

The three modules show distinct roles: GeoProj mainly improves Camera–LiDAR geometry, Rec supports identity and temporal-state continuity, and CF expands difficult long-tail sample coverage. Standard generation metrics show lower FVD and LiDAR Chamfer Distance for the proposed model, while MagicDrive-V2-adapted retains a small LPIPS advantage. The downstream augmentation experiment also shows a larger improvement in held-out real long-tail mAP from counterfactual-generated data than from ordinary generated data, suggesting potential utility of counterfactual augmentation for challenging long-tail evaluation under the tested conditions.

The current results nevertheless have clear boundaries. Cross-dataset effectiveness is mixed, several nuScenes comparisons are nonsignificant, and five repetitions provide limited statistical power. Recoverability should be interpreted as a consistency mechanism rather than complete verification of the true future state. Counterfactual intervention currently covers only four predefined long-tail factors, and its benefit decreases under simultaneous Camera–LiDAR degradation. Future work should broaden failure categories, strengthen external state-level evaluation, and examine longer and closed-loop rollouts. Overall, the framework provides a recoverability-based approach to multimodal geometric and temporal consistency, with current evidence mainly supporting Camera–LiDAR geometric consistency, temporal-state stability, and selected long-tail evaluation scenarios.