Embedded Files
During the latter half of my Master's in Robotic Systems Engineering, I was drawn to generative models and their potential in robotics and simulation. I initially explored their use for traffic prediction and scene understanding. This interest led me to the Institute of Man-Machine Interaction (MMI) at RWTH Aachen, where simulation-based virtual testing is a central research theme, and where I pursued my thesis under the supervision of Jorge Luis Jiménez Aparicio and Prof. Jürgen Roßmann.
I evaluated whether generative image synthesis can serve as a computationally efficient alternative to ray-traced rendering in simulation pipelines. I trained and evaluated a GAN-based and a diffusion-based model on this task. The GAN-based model generated frames about 1.75× faster than the simulator's ray tracer at competitive image quality, while the diffusion-based model reached higher visual fidelity at significantly higher inference cost. This work has been accepted at the European Modeling & Simulation Symposium (EMSS 2026).
Key Techniques and Methods:
Simulators can produce semantic segmentation maps of a scene cheaply and reliably. The task was to turn those maps into camera-like RGB frames that look realistic and stay consistent over time. I compared three generative approaches:
SEAN, a GAN built on semantic region-adaptive normalization
DGInStyle, a diffusion-based model conditioned on the semantic layout
World-Consistent Video-to-Video Synthesis (WCVid2Vid), a GAN-based video model
Experimental Setup and Data:
Datasets
Cityscapes: real urban driving scenes
GTA V: large synthetic driving dataset from a video game
VEROSIM-Automotive: custom dataset generated using VEROSIM simulation software
VEROSIM-Construction: custom dataset generated using VEROSIM simulation software
Dataset size: DGInStyle was fine-tuned on 25,000 GTA V images and 4,003 VEROSIM images.
Compute: RWTH HPC cluster with H100 GPUs and a local computer with RTX 3090
Training: SLURM job scripts for single and multi-GPU training, Docker for reproducible experiments
Research Highlights:
Automotive Sector
Input Segmentation Map
Ground Truth
Diffusion-based model
DGInStyle
GAN-based model
SEAN
Shortcomings
SEAN
Failed multi-instance generation with GAN based model
DGInStyle
Semantic maps are not fully followed on custom dataset due to lack of sufficient training data
Construction Sector
Input segmentation map -
Cabin Camera
Generated Output - SEAN
Input segmentation map -
Arm Camera
Generated Output - SEAN
Additional Results on the Construction Sector
SEAN gave the best overall balance of image quality, semantic alignment and inference cost. It performed particularly well on construction scenes. It struggled where several vehicles were close together, as instances merged and edges blurred.
DGInStyle reached the highest visual fidelity and adapted well to the style of each domain. Object boundaries sometimes drifted from the semantic map and small structures were misplaced. Its results on GTA V were much stronger than on VEROSIM, which points to the amount of fine-tuning data as a major factor.
Inference speed: SEAN generated frames about 1.75× faster than VEROSIM's ray tracer.
Publication: Evaluating Generative Models for Accelerated Data Generation in Simulation Pipelines, EMSS 2026 (Majali, Jiménez Aparicio, Roßmann).
Conclusion:
Generative enhancement can supplement simulator rendering where the training data covers the target appearance and scene layouts. Under domain shift, and in scenes with several nearby vehicles, it is not yet reliable enough to replace traditional ray-traced rendering. SEAN performed particularly well on the construction dataset. It generated frames about 1.75× faster than VEROSIM's ray tracer at competitive image quality, which makes it a practical option for scaling up dataset generation.
Google Sites
Report abuse