Lunar hazard detection from synthetic data
You want a rover to see rocks. There are almost no labeled real images of the lunar surface. So you render the data yourself — perfect labels, zero annotation cost — and train a model on it.
That’s the whole idea. This series shows you the full loop: build a USD scene, run NVIDIA Omniverse Replicator to generate a dataset with domain randomization, fine-tune a SegFormer, measure the transfer gap to real imagery, and render a cinematic final output. Everything runs on a single NVIDIA DGX Spark.
Results — the failure, then the fix
First we made the synthetic rocks photoreal (v2). The synthetic benchmark went up (deployed rock-IoU 0.815 → 0.852) but real-world transfer got worse: the model flooded ~83% of real Apollo pixels with false rock (up from ~52%), because photoreal rocks (rough, gray, bumpy) collapsed the rock-vs-regolith boundary toward “any rough gray texture is rock” — and real lunar regolith is exactly that. That failure was a diagnosis: the fidelity was on the wrong surface.
So we moved it to the ground (v3) — a realistic cratered, dark, displaced, normal-mapped regolith floor, plus power-law rock sizes and harsh lunar lighting. With both rock and ground now rough inside the simulator, the texture shortcut no longer separates the classes, so the model had to learn shape, shadow, and scale — cues that transfer. It worked: synthetic rock-IoU rose to 0.887 and the real-photo flood dropped to 35.7%, below both prior builds. For the first time the synthetic and real arrows point the same way.
| Version | rocks | ground | synth rock-IoU | real flood (same 21 images) |
|---|---|---|---|---|
| v1 | low-poly blobs | smooth heightfield | 0.815 | 44.0% |
| v2 | photoreal basalt | smooth heightfield | 0.852 | 72.6% |
| v3 | power-law basalt | cratered dark displaced | 0.887 | 35.7% |
The flood column is an exact same-set comparison — all three dr_1500 checkpoints scored with the same hardened eval on the same 21 NASA photographs. (v1/v2’s original 7-image floods were ~52% / ~83%; re-run on the 21-image set they are 44.0% / 72.6%.)
The series closes with a 1920×1080 cinematic RTX flythrough of NASA’s VIPER rover crossing the v3 boulder field — a rover’s-eye hazard HUD: the forward hazard-cam with the deployed model’s predictions overlaid live. The overlay is clean and finally backed by a real-photo number that moved the right way.
Start here
The series
| Chapter | Topic |
|---|---|
| 01 — The lunar stage | USD scene: terrain, realistic basalt rocks, lighting |
| 02 — Domain randomization | Replicator pipeline + 2,550-frame labeled dataset |
| 03 — Training | SegFormer fine-tuning + honest size-matched DR ablation |
| 04 — Sim-to-real | The v2 flood: ~83% of real regolith called rock |
| 05 — The render | Cinematic 1920×1080 RTX flythrough with live hazard overlay |
| 06 — Rock fidelity | The v1→v2 evolution and the failure: photoreal rocks worsened real transfer |
| 07 — Realistic ground | v3: fidelity on the ground — synth 0.887, real flood 35.7%, VIPER render |
A Chaotic Curiosity project by Don Balanzat — sibling to chaotic-fine-tuning (LLM fine-tuning on the same Spark) and g1-humanoid-rl (humanoid robot RL).