Lunar hazard detection from synthetic data
You want a rover to see rocks. There are almost no labeled real images of the lunar surface. So you render the data yourself — perfect labels, zero annotation cost — and train a model on it.
That’s the whole idea. This series shows you the full loop: build a USD scene, run NVIDIA Omniverse Replicator to generate a dataset with domain randomization, fine-tune a SegFormer, measure the transfer gap to real imagery, and render a cinematic final output. Everything runs on a single NVIDIA DGX Spark.
Results — an honest surprise
We made the synthetic rocks photoreal. The synthetic benchmark went up: deployed rock-IoU 0.815 → 0.852 on an unseen synthetic domain, with domain randomization adding +0.046 size-matched. But real-world transfer got worse: on 7 real Apollo photographs the photoreal model floods ~83% of pixels with false rock (up from ~52%), because photoreal rocks (rough, gray, bumpy) collapsed the rock-vs-regolith boundary toward “any rough gray texture is rock” — and real lunar regolith is exactly that. Higher fidelity + higher synthetic score ≠ better real transfer. Synthetic metrics can mislead; you have to test on real data.
The series closes with a 1920×1080 cinematic RTX flythrough — a rover-eye dolly into a boulder field with the deployed model’s predictions overlaid live. The overlay is clean, but only because the scene is in-distribution — the same flattery the synthetic score gives.
Start here
The series
| Chapter | Topic |
|---|---|
| 01 — The lunar stage | USD scene: terrain, realistic basalt rocks, lighting |
| 02 — Domain randomization | Replicator pipeline + 2,550-frame labeled dataset |
| 03 — Training | SegFormer fine-tuning + honest DR ablation (+0.046; ceiling 0.852) |
| 04 — Sim-to-real | The flood: ~83% of real regolith called rock |
| 05 — The render | Cinematic 1920×1080 RTX flythrough with live hazard overlay |
| 06 — Rock fidelity | The v1→v2 evolution and the honest fidelity-vs-transfer tradeoff |
A Chaotic Curiosity project by Don Balanzat — sibling to chaotic-fine-tuning (LLM fine-tuning on the same Spark) and g1-humanoid-rl (humanoid robot RL).