Simscale Arterial Stenosis Cfd

SIMSCALE: Learning to Drive via Real-World Simulation at Scale

Hongyang Li2

1 MAIS, Institute of Automation, Chinese Academy of Sciences

E2E Planner

Sim.

Real

(a) Scalable Simulation Data and Sim-Real Co-training

Real Ego

Figure 1. Scaling up end-to-end planners by simulation. (a) We construct large-scale simulation data by perturbing ego trajectories, generating corresponding pseudo-expert demonstrations, and rendering multi-view observations in reactive environments. Combined with real-world data, this enables broad coverage of out-of-distribution states and supports sim–real co-training for any end-to-end planner.

(b) Across three representative planner families, including regression, diffusion, and vocabulary scoring, sim-real co-training consistently produces synergistic improvements in robustness and generalization, demonstrating clear and predictable simulation scaling trends.

Abstract

Achieving fully autonomous driving systems requires learning rational decisions in a wide span of scenarios, in- cluding safety-critical and out-of-distribution ones. How- ever, such cases are underrepresented in real-world cor- pus collected by human experts.

To Complement For The

lack of data diversity, we introduce a novel and scalable simulation framework capable of synthesizing massive un- seen states upon existing driving logs. Our pipeline uti- lizes advanced neural rendering with a reactive environ- ment to generate high-fidelity multi-view observations con- trolled by the perturbed ego trajectory. Furthermore, we ⋄Work done while interning at Xiaomi Embodied Intelligence Team.

† Project Lead. § Equal Advising. develop a pseudo-expert trajectory generation mechanism for these newly simulated states to provide action supervi- sion. Upon the synthesized data, we find that a simple co- training strategy on both real-world and simulated samples can lead to significant improvements in both robustness and generalization for various planning methods on challenging real-world benchmarks, up to +8.6 EPDMS on navhard and +2.9 on navtest. More importantly, such policy improve- ment scales smoothly by increasing simulation data only, even without extra real-world data streaming in. We further reveal several crucial findings of such a sim-real learning system, which we term SimScale, including the design of pseudo-experts and the scaling properties for different pol- icy architectures. Simulation data and code have been re-

Leased At Https://Github.Com/Opendrivelab/

SimScale.

1. Introduction

Data scaling is recognized as a foundational principle in modern deep learning across various domains, including language, vision, and multimodal modeling, underpinning steady performance improvements as data sizes increase [5, 38, 72, 92]. In autonomous driving, end-to-end (E2E) plan- ning learns to map raw observations to actions, offering a promising way to leverage large-scale driving data to enable the emergence of fully autonomous systems [29, 68, 98].

Nevertheless, real-world driving data from human ex- pert demonstrations are dominated by common scenarios, while non-trivial cases, e.g., safety-critical, are underrepre- sented [9, 62, 84, 87]. Moreover, planners trained on such data are confined to human driving distribution and struggle to generalize to rare or unseen situations, leading to distri- bution shift and causal confusion in deployment [20, 56].

Consequently, scaling real-world data only is inefficient for achieving deploy-ready autonomous driving. Simulation via neural rendering can generate high- fidelity driving scenarios and thus has the potential to pro- duce out-of-distribution (OOD) states deviated from hu- man demonstrations at scale, which is essential for closed- loop planning [23, 71]. Therefore, scaling simulation data presents an attractive alternative to solely relying on real- world data. However, planners require feasible correspond- ing demonstrations to learn how to handle OOD states, while current simulation methods fail to generate such demonstrations effectively. Moreover, the impact of scaling simulation data lacks in-depth analysis. In this work, we aim to derive a systematic recipe for scaling simulation data from limited real-world scenarios in end-to-end planning.

To conduct comprehensive experiments and analyses, this study is structured along three key questions: (1) what constitutes effective simulation data, (2) how well plan- ners benefit from it, and (3) whether this system scales pre- dictably upon fixed real-world corpus.

To this end, we formulate a scalable simulation data generation framework that extends the expert distribution from existing real-world training data to bootstrap end-to- end autonomous driving systems. We develop a 3D Gaus- sian Splatting (3DGS) simulation data engine, which allows controlling the temporal ego and other agent states and rendering multi-view videos from the ego’s perspec- tive. Concretely, we first sample a diverse set of plausi- ble perturbations to the ego trajectory, maximizing cover- age of the state space, e.g., off-center lane drifts, close in- teractions, among others. Then, we take the final state of each perturbed trajectory as the perturbed state and gener- ate corresponding demonstrations using pseudo-experts of two forms in comparison. The first, recovery-based expert retrieves trajectories that steer the policy within the human trajectory manifold, resulting in human-like yet cautious be- haviors. The second, privileged planner-based expert generates the trajectory that maximizes optimality, repre- senting an exploratory strategy with lower realism. To en- hance scalability and plausibility, the entire pipeline is ex- ecuted in a reactive environment , where surrounding agents interact with the ego vehicle responsively.

To thoroughly assess the effect of simulation data, we consider three types of end-to-end planners with various model scales, i.e., LTF for regression methods, Dif- fusionDrive for diffusion-based planners, and GTRS- Dense for vocabulary scoring.

We Employ A Sim-

ple yet effective sim-real co-training strategy [67, 69] to maintain human driving distribution while mitigating visual domain degradation. Furthermore, by keeping a constant amount of real data and progressively adding simulation data via non-overlapping samples, we investigate how di- verse planners can benefit from simulation data and the gen- eral scaling property. We utilize two real-world closed-loop benchmarks to evaluate the planners from multiple perspec- tives. navhard focuses on unseen, challenging scenar- ios to assess the impact of OOD states on planners, while navtest offers a broad set of diverse scenarios to test planners’ ability to handle varying situations.

The complete sim-real learning system, which comprises a scalable simulation data construction pipeline and an ef- fective sim–real co-training strategy as shown in Fig. 1, is termed SimScale. Rigorous experiments uncover crucial findings enabled by SimScale—including, but not limited

To, The Following:

• Scalable simulation with pseudo-expert unlocks the in- herent potential of available real-world driving data. • Sim-real co-training improves both robustness and gener- alization synergistically in diverse end-to-end planners.

• Exploratory expert and interaction environments improve the effectiveness of simulation data. • Planners with multimodal modeling capabilities exhibit more encouraging data scaling properties.

2. Methodology

We outline SimScale as follows. In Sec. 2.2, we briefly in- troduce our 3GDS simulation data engine supporting con- trollable multi-view video rendering. In Sec. 2.3, we pro- pose a pseudo-expert scene simulation pipeline to generate diverse simulation data containing OOD states with feasible demonstrations. In Sec. 2.4, we demonstrate a scalable sim- real co-training method with different end-to-end planners.

2.1. Preliminary

End-to-end planning models take observations within a his- tory window as input and output a predicted future trajec- tory. Each training scenario starts at a selected timestep and includes a history horizon of length T and a planning hori- zon of length H. The model processes the past T frames to

Real

Sim.

(A) Trajectory Perturbation

Sim.

(B) Pseudo-Expert Trajectory Generation

Sim.

𝑇+ 𝐻

reactive env. Figure 2. Pseudo-expert scene simulation pipeline. (a) Trajectory perturbation on T to T + H, (b) reactive environment rollout, and pseudo-expert trajectory generation from T + H to T + 2H under recovery-based and planner-based strategies.

predict the future H frames, resulting in a complete training sample spanning T + H timesteps.

2.2. 3Dgs Simulation Data Engine

To reduce the domain gap between real-world data and gen- erated observations from novel views in simulated scenar- ios, a photorealistic data engine is required.

Built Upon

3DGS assets reconstructed from real-world datasets,

I=1) Takes As In-

put the camera intrinsics Kt and extrinsics Et at timestep t, along with the positions and yaw angles (x, y, θ) of a non- ego vehicle i at the same timestep, and renders the corre- sponding RGB observations from the camera. The cam- era extrinsics Et can easily be obtained from the ego vehi- cle’s position and heading (x0,t, y0,t, θ0,t) with the ego-to- camera transformation, while the other camera parameters are directly adopted from the raw dataset.

Preprocessing. Following prior work , we apply expo- sure alignment across images captured simultaneously by multi-view cameras at time t, using projected LiDAR points at t as guidance. Moreover, with help of 3D bounding boxes provided by NAVSIM annotations, colored LiDAR points are divided into several groups, i.e., the static background and multiple vehicles, and further used as initialization for Gaussians to improve the reconstruction performance.

Block-wise Reconstruction. During scene reconstruction, increasing the number of images significantly raises compu- tational cost and runtime. Thus, we perform reconstruction in a block-wise manner, where each block corresponds to a spatio-temporal range. Following previous work [51, 86], backgrounds and foregrounds are reconstructed as separate models using the per-timestep locations and orientations of 3D bounding boxes, resulting in a static background asset and multiple movable vehicle assets. These assets could then be placed at desired locations and headings specified by inputs, rendering sensor data at novel views. Blocks with low average PSNR in novel view synthesis are excluded to ensure the quality of generated simulation data.

2.3. Pseudo-Expert Scene Simulation

Based on the data engine, we design a pseudo-expert scene simulation pipeline to generate diverse simulation data from existing real-world data, as shown in Alg. 1. The pipeline aims to produce feasible demonstrations from perturbed states paired with expert trajectories, as illustrated in Fig. 2.

Reactive Scene Reconstruction. For each training clip d, we perform two simulations of duration H: one for explor- ing perturbed states at t = T, and the other for generating expert trajectories at t = T +H. In each simulation, ego tra- jectory ˜at:t+H is simulated by LQR , while other agents are modeled using the IDM to interact with the ego, producing the corresponding future states ˜st:t+H. For the valid expert trajectories, the simulated ego–agent trajecto- ries cross two stage (˜aT :T +2H, ˜sT :T +2H) are then rendered into multi-view videos ˜oT :T +2H using the data engine Φ.

This decoupling of behavior simulation from sensor render- ing enables reactive environments where other agents re- spond plausibly to the ego’s behaviors, thereby enhancing both the realism and diversity of the simulation data.

Trajectory Perturbation. At the first simulation step (t = T), we perturb the ego trajectory ˜at:t+H so that the ego reaches a new terminal state at t = T + H, which then serves as the start state for the next rollout (Fig. 2 (a)). Our objective is to sample diverse yet plausible states. For di- versity, perturbations are drawn from a clustered human- trajectory vocabulary that densely covers the action space.

For plausibility, we restrict perturbations to remain near human behavior by thresholding longitudinal/lateral shifts (rlon, rlat) and heading change |∆θ|, and by removing tra- jectories that are physically invalid (collisions, off-road, un- stable renderings). We further apply spatially sparse sam- pling on trajectory endpoints using an interleaved grid with steps (δlon, δlat) to promote uniform coverage. Since reac- tive simulation is costly, we first filter infeasible trajectories

∈D, Perturb

action sets Aper, sensor engine Φ, pseudo policy πexp, reward functions f, reactive simulation T .

6:

else ˜at:t+H ←πexp(˜ot, ˜st).

9:

˜st:t+H ←T (˜at:t+H, ˜st).

11:

˜ot:t+H ←Φ(˜st:t+H).

13:

˜rt:t+H ←f(˜at:t+H, ˜st:t+H).

20: Return Dsim

non-reactively and run reactive checks only on the sparsed set. This produces a set of dynamically and physically fea- sible perturbations aper ∈Aper.

Pseudo-Expert Trajectory Generation.

At The Second

simulation t = T + H, for each perturbed state, we employ a non-human expert, referring as the pseudo-expert πexp, to generate a feasible corresponding trajectory ˜at:t+H. Since the pseudo-expert is not perfect and ˜at:t+H will be used as supervision, we apply stricter filtering in the second stage simulation. In addition to physical constraints, traffic rules and vehicle kinematic limits are also enforced, following Eq. 6. To investigate which strategy best serves as supervi- sion for end-to-end planners, we compared two strategies: a conservative recovery-based expert, and an exploratory planner-based expert, as detailed below.

(1) Recovery-based Expert. Recovery steers the policy within the human manifold after perturbation. To ensure robustness, our recovery expert πexp retrieves a human tra- jectory from a large vocabulary Vh that best matches the ego vehicle’s logged state at time t = T + 2H (Fig. 2 (b1)). For each candidate trajectory of horizon H, we summarize its initial and final poses with a compact matching vector:

H B F {

m } = [ \tilde { v}^x_t, \tilde {v}^y_t,\tilde {\theta }_t, \tilde {x}_{t+H-1}, \tilde {y}_{t+H-1}, \tilde {\theta }_{t+H-1}].

(1)

Given the ego’s perturbed state with target vector mr, πexp

\Tild E {A }_{

t:t+ H} = \ arg \min _{a \in \mathcal {V}_{\text {h}}} \|\mathbf {m}(a) - \mathbf {m}_{\text {r}}\|_1.

(2)

This yields a human-like yet conservative fallback behavior, stabilizing under distributional drifts. (2) Planner-based Expert. Following prior works [3, 97], we use a privileged planner P that leverages ground-truth states to generate reactive and optimized trajectory rollouts in simulation, as shown in Fig. 2 (b2). The planner-based expert πexp ←P is defined as: ˜at:t+H = P(˜st:t+H). Com- pared with the recovery policy, planner-based expert relies on rules or cost heuristics, trading human-likeness and real- ism of its behavior occasionally. Still, it offers strong opti- mization and diverse exploratory rollouts, enriching expert supervision beyond human data.

2.4. Scalable Sim-Real Co-Training

Strategies for Co-training. Sim-real co-training serves as a simple yet effective strategy [4, 67, 69, 87] that enables the integration of both real and simulated data for plan- ning. In our approach, we randomly sample from a mix- ture of real-world data D and simulation data Dsim during training, aiming to preserve the human driving distribution while mitigating visual domain degradation caused by po- tential simulation artifacts, such as subtle rendering incon- sistencies, temporal jitter, or unrealistic lighting and shad- ows [63, 64]. Our fully automated, scalable simulation data generation framework enables scaling the total training data by progressively adding non-overlapping simulation sam- ples, while keeping the real data amount fixed.

Planners for Co-training.

We Aim To Comprehensively

evaluate the effectiveness of simulation data for end-to- end planners. Modern end-to-end planning approaches can be broadly categorized into three representative paradigms: regression-based planners [12, 27], diffusion-based plan-

[60, 83], And Vocabulary Scoring-Based Planners

[34, 56]. Accordingly, we consider representative models from each paradigm in our co-training experiments. (1) Co-training with Pseudo-Expert Trajectory. Regres- sion and diffusion-based planners rely on expert demonstra- tions. The co-training process can thus be formulated as:

A,

o)\sim (\mathcal {D}\,\cup \,\mathcal {D}_{\text {sim}})} \Big [ \mathcal {L}_{\text {im}}\big (a, \pi _\theta (\hat {a}|o)\big ) \Big ], \label {eq:im}

(3)

here, Lim denotes the imitation loss; D and Dsim represent the real-world and generated simulation datasets, respec- tively; and A refers to expert trajectories, i.e., human-expert trajectories in D and pseudo-expert trajectories in Dsim. As for the vocabulary scoring-based planner, the learning ob- jective has additional prediction of reward signals r that dis-

Rg

\min _{\theta }\mathbb {E}_{(a,o,r)\sim (\mathcal {D}\cup \mathcal {D}_{\text {sim}})} \Big [ \lambda \mathcal {L}_{\text {im}} + \mathcal {L}_{\text {r}}\big (r,\, \pi _\theta (\hat {a}|o)\big ) \Big ], \end {aligned}

Rounds

Rounds

Figure 3. Simulation data statistics across multiple sampling rounds. (a) Recovery-based expert impose stronger constraints, leading to slower data accumulation than (b) Planner-based expert.

here, Lr denotes reward loss, λ is a weighting factor. (2) Co-training with Rewards Only.

For The Vocabu-

lary scoring-based planner, when the reward signal is well- aligned, the expert is theoretically unnecessary, as shown in Eq. 4. The planner can explore directions to increase the reward without being restricted to a single expert trajectory.

\ Arg

\min _{\theta } \ mat h bb {E}_{(a,o,r)\sim \mathcal {D}} [\lambda \mathcal {L}_\text {im} + \mathcal {L}_r] + \mathbb {E}_{(o,r)\sim \mathcal {D}_{\text {sim}}} [\mathcal {L}_r].

(5)

We thus investigate purely reward-driven optimization for the scoring-based planner in simulation, to assess its po- tential for fully leveraging simulation data, and assess its potential for maximizing the utility of simulated data.

3. Experiments

3.1. Setup and Protocols. Real-world Datasets.

We Use The Navtrain Split From

NAVSIM , built upon the largest publicly available an- notated driving dataset nuPlan , and comprises 100K interactive real-world scenarios.

Simulation Data Curation. We construct simulated sce- narios on navtrain split. 3DGS models of blocks with PSNR < 27 in novel-view results are removed ensure re- construction quality. For ego trajectory perturbation, we choose the plausible candidates from the vocabulary of

16,384 Trajectories From , With Epdms ≥0.8 And

relative heading |∆θ| ≤20◦, where EPDMS is detailed in Eq. 6. During spatial sampling by trajectory endpoint, the longitudinal and lateral perturbation ranges are set as rlon = 20m, rlat = 2.0m, with step sizes δlon = 5m, δlat = 0.5m. For recovery-based trajectory generation, we use all human expert trajectories in navtrain split as the retrieval vocabulary. For planner-based trajectory generation, we se-

Lect The Sota Rule-Based Planner In Nuplan , Pdm-

Closed as the expert. Only pseudo-expert trajectories with decent sub-metrics of EPDMS are considered valid. Due to computational limits, for each type of pseudo-expert, we perform up to five random samplings of valid trajecto- ries with no overlaps from navtrain split for rendering.

Through multiple samplings, expert trajectory demonstra- tions are progressively accumulated. As shown in Fig. 3, the two pseudo-expert methods exhibit different success rates, with the planner-based expert achieving higher effi- ciency. In total, we generate 147K recovery-based simula- tion scenes and 237K planner-based simulation scenes. For more details, please refer to Sec. B.1 in the supplementary Benchmark and Metrics. We utilize two benchmarks in NAVSIMv2 for end-to-end model evaluation, includ- ing navhard and navtest. navhard is the official two- stage evaluation benchmark, which contains 244 challeng- ing real-world scenarios in the first stage and corresponding 4,164 synthetic scenarios generated by 3DGS in the sec- ond stage. navtest is a one-stage evaluation benchmark, containing a large number of 12,146 real-world scenarios.

navhard focuses on assessing the model’s closed-loop per- formance in safety-critical situations, while navtest em- phasizes generalization across diverse driving conditions.

The two benchmarks share a rule-based planning metric,

Hcal {M} _\Text

{pen}} S_m \right ) }_{\text {penalties}} \cdot \underbrace {\left ( \frac { \sum _{m \in \mathcal {M}_\text {avg}} w_m S_m } { \sum _{m \in \mathcal {M}_\text {avg}} w_m }\right ) }_{\text {weighted average}}, \label {eq:epdms} (6) where Sm is the sub-metric: penalty terms set Mpen in- cludes No-at-fault Collisions (NC), Drivable Area Com- pliance (DAC), Driving Direction Compliance (DDC), and Traffic Light Compliance (TLC); weighted average terms set Mavg includes Time-to-Collision (TTC), Ego Progress (EP), Lane Keeping (LK), History Comfort (HC), and ex- tended comfort (EC). Note that EPDMS in navhard fur- ther incorporates several modifications, e.g., two-stage ag- gregation, reactive traffic simulation, and the exclusion of penalties in cases where the human expert driver also fails.

Models and Training. To thoroughly validate our model- agnostic simulation data, we select one representative open- source model for each paradigm discussed in Sec. 2.4: the regression-based LTF , the diffusion-based Diffusion-

Drive , And The Scoring-Based Gtrs-Dense . We

adopt their official implementations with two modifications: the input image resolution is unified to 2048 × 512, and the LiDAR inputs are removed to align with the navhard evaluation setting. All models are trained from scratch on NVIDIA H20-3e GPUs. To ensure fair comparison, each model follows the same training strategy, whether using only the navtrain split or augmented with simulation data.

For more details, please refer to Sec. B.2 in supplementary.

5

Table 1. Performance on the NAVSIM-v2 navhard Leaderboard. PDM-Closed uses ground-truth symbolic inputs for planning, while other methods rely on sensor data. (∗: pseudo-expert supervision; †: reward scoring; S.: per-stage EPDM score.)

Backbone

Sim.

48.0 ↑15%

Table 2. Performance on the NAVSIM-v2 navtest Leaderboard. (∗: pseudo-expert supervision; †: reward scoring.)

Backbone

Sim.

3.2. Leaderboard Results

Tab. 1 and Tab. 2 present the leaderboard results of Sim- Scale sim-real co-training for the three planner paradigms on navhard and navtest, respectively.

Navhard Leaderboard. All models exhibit significant im- provements in both Stage 1 and Stage 2. Notably, GTRS- Dense (V2-99) achieves a score of 48.0, establishing a new SOTA on navhard. These results demonstrate that incor- porating simulation data with an extended distribution sub- stantially enhances model robustness in challenging and un- seen environments. Notably, weaker baseline models, i.e., LTF and DiffusionDrive, benefit the most, with gains ex- ceeding 20%, indicating that sim-real co-training with sim- ulation data effectively enables models to exploit dataset knowledge better and unlock their latent learning potential.

Navtest Leaderboard.

All Models Show Consistent Im-

provements of up to +2.9 points, demonstrating stronger performance under large-scale and diverse conditions. The quantitative results above highlight that our simulation data is model-agnostic, and that our general sim-real co-training achieves a synergistic optimization of robustness and generalization, which is essential for reliable closed-loop deployment in the real world [9, 62].

3.3. Ablation And Data Scaling Analysis

Data Scaling Curves of Different Planners. Due to the lack of prior work studying the scaling behavior of simula- tion data under a fixed amount of real-world data, we model

Epdms

Figure 4. Scaling dynamics across different planners and pseudo-expert trajectories. We visualize how simulation data scale and supervision signals influence the driving performance of various planners, where the infection point indicates learning plateau.

Front-Right

Figure 5. Qualitative results of the simulation scenes on navtrain. Four representative simulation scenarios are shown, each mirroring a typical real-world OOD scene, with synthetic front-view and auxiliary key-view images provided. the relationship between performance and the total data size (simulation + real) using a log-quadratic function:

(7)

where S(N) denotes the planner performance with total data size N, and a, b, c are parameters fitted via nonlinear

Where S_

i is the observed performance at total data size N

I,

and M is the number of data points. If the data scaling trend exists, the quadratic coefficient a approaches zero, degenerating the model to a linear log relation; otherwise, the curve exhibits a parabolic shape with a visible satu- ration point. We evaluate four planners under 2 pseudo- experts following simulation data scaling in Fig. 3 and se- lect EPDMS in navhard as S. The scaling curves and fit- ted log-quadratic functions are shown in Fig. 4, with the residual standard deviation depicted as an error band. Ad- ditionally, we conduct extra experiments using reward scor- ing only in planner-based simulation data for GTRS-Dense, proposed in Sec. 2.4.

Some Classical Trends Can Be Ob-

served in Fig. 4. For instance, comparing planner-based and reward-scoring settings in Fig. 4 (c) and (d), larger models exhibit more pronounced data scaling trends under the same amount of data. Additional interesting and meaningful ob- servations are highlighted below.

Pseudo-Expert Should Be Exploratory. For all planners, the scaling curves under recovery-based setting converge earlier and achieve lower performance compared to planner- based. The recovery-based expert always steers to the hu- man log, limiting diversity as the simulation data grows from the same real scenarios. In contrast, the planner-based expert explores a broader range of possibilities and even produces feasible solutions in challenging situations. Con- sequently, the recovery-based expert only exhibits advan- tages in small-data regimes (Fig. 4 (d)), likely because its trajectory distribution is human-like and better aligned with real-world data, making it easier to learn. In most cases, data scaling yields diminishing returns for recovery-based methods compared to planner-based ones, and performance may even degrade as visual artifacts accumulate and begin to dominate. These observations highlight the importance of exploratory behavior for pseudo-experts, which enhances

7

Table 3. The effect of expert with simulated reward scoring on navhard using GTRS-Dense. (S1/2:per-stage EPDM scores.)

48.0

the value of simulation data under scaling. Multi-modality Modeling Sparks the Scaling. Although the regression-based LTF and diffusion-based Diffusion- Drive have comparable model sizes (56M vs 61M), they ex- hibit markedly different scaling properties for the planner- based in Fig. 4 (a) and (b). For LTF, performance satu- rates and starts to degrade when the simulation-to-real ra- tio reaches 1:1, whereas DiffusionDrive exhibits an approx- imately linear improvement. This is due to the gradually increasing diversity of demonstrations from the same real scenario, which introduces an effectively multi-modal su- pervision problem.

Single-Mode Regression Struggles To

model multi-peak distributions, leading to mode confusion and performance degradation, while diffusion models can capture multimodality, making them amenable to optimiza- tion under diverse supervision . Since real-world au- tonomous driving is inherently a multi-peak problem , our simulation-scaling results underscore the importance of multimodal modeling for scalable real-world end-to-end au- tonomous driving.

Reward is All You Need. In Fig. 4 (c) and (d), for the scoring-based GTRS-Dense planner, reward signals alone, without expert trajectories in simulation, yields even better performance. To further analyze this, we conduct reward- only training on real-world data only, which instead leads to performance degradation, as shown in Tab. 3. These results indicate that with sufficient expert supervision to stabilize the optimization direction, the reward guidance is better.

The model benefits from the feedback from rewards during its exploration and interaction within the environment . Effect of Reactive Simulation.

To Isolate The Effect Of

reactive traffic, we compare reward-scoring GTRS-Dense on navhard using non-reactive vs. reactive simulation data (Tab. 4). Two rounds of non-reactive sampling yield 141K trajectories, i.e., 21K more valid samples due to fewer col- lision rates, yet provide no performance improvement in EPDMS. When reactive simulation reaches the third round, it produces 167K samples yet delivers consistent and signif- icant EPDMS gains across both model sizes. These results Table 4. The effect between non-reactive vs. reactive data sim- ulation on navhard using GTRS-Dense, across sampling rounds.

#Round

#Sim.

47.9

indicate that reactive agent dynamics enhance the realism and diversity of traffic interactions, thereby increasing the effectiveness of simulation data. Detailed scaling curves are shown in Sec. C.3 in the supplementary material.

3.4. Qualitative Results Of The Simulation Scenes

We present qualitative visualizations from the simulation data in Fig. 5, showcasing four representative OOD scenar- ios used to train the policy. These scenarios mirror typical real-world driving challenges where learned policies tend to struggle, including (a) off-center lane drift, (b) near colli- sion, (c) departure, and (d) cutting in cases. Each scenario is illustrated with a top-down view showing the pseudo-expert trajectory as supervision, and the deviating Perturbed Tra- jectory as history actions, alongside the synthetic front-view as sensory input to the policy. For instance, scenario (b) re- quires the policy to adaptively avoid the collision in short horizon. See supplementary Sec. C.2 for extended results.

4. Conclusion

In this paper, we introduce SimScale, a sim–real learning system that reveals how scalable simulation can amplify the value of real-world datasets for end-to-end autonomy. For the simulation data pipeline, we first generate pseudo-expert demonstrations from potential OOD states by ego perturba- tion within reactive environments. Toward the real-world simulation, the associated high-fidelity multi-view observa- tions are rendered with 3DGS engine. Upon the simulated data, sim-real co-training produces synergistic improve- ments in robustness and generalization for various planners on challenging real-world benchmarks, up to +8.6 EPDMS on navhard and +2.9 navtest. Remarkably, the sim–real system scales clearly and predictably with increased simu- lation while keeping the real-world corpus fixed. We fur- ther uncover that exploration and interaction contribute to more effective simulation, and that multi-modal planners strengthen their scaling behavior. We hope SimScale will inspire the community to further explore real-world simula- tion for data scaling.

We provide additional experiments in the supplementary material, such as multi-expert ensemble and scaling with varying real data. It also provides extended discussions on related work, limitations, and broader impact.

Acknowledgments

This study is supported by the National Key R&D Program of China (2022ZD0117901), the National Natural Science the Beijing Natural Science Foundation (Grant L252033).

This work is in part supported by the JC STEM Lab of Au- tonomous Intelligent Systems funded by The Hong Kong Jockey Club Charities Trust. We also appreciate the gener- ous research sponsor from Xiaomi.

We extend our gratitude to Naiyan Wang, Yunsong Zhou, Hanxue Zhang, and the rest of the members from Open- DriveLab and Xiaomi Embodied Intelligence Team for their profound support.

Related Journal Articles & DOI Links

Selected peer-reviewed publications relevant to 12 Lead ECG Acquisition. Click the DOI to access the full paper (may require institutional access).

Why Choose Us?

Bangalore guidance for robotics, Spectre and autonomous systems projects.