Bridging the Sim-to-Real Gap: Robust Vision-Based Navigation for Robotic Guide Dogs

Presenter: Shiven Patel

Faculty Sponsor: Donghyun Kim

School: UMass Amherst

Research Area: Computer Science

Session: Poster Session 5, 3:15 PM - 4:00 PM, 165, D4

ABSTRACT

Over 1.1 billion people worldwide live with some form of vision loss, yet fewer than 20,000 guide dogs are in service globally, leaving approximately 98% of eligible individuals without access to this gold standard of mobility assistance. Robotic guide dogs built on quadruped platforms offer a scalable alternative, but deploying reliable vision-based navigation in diverse, real-world urban environments remains a fundamental challenge. The primary bottleneck is the sim-to-real gap, the fidelity mismatch between simulated training environments and the physical world, which causes navigation policies to fail when transferred to real hardware.

This research proposes a real-to-sim pipeline that leverages 3D Gaussian Splatting and neural scene reconstruction to create photorealistic digital replicas of real urban environments from standard RGB camera footage. These reconstructions are integrated into NVIDIA Isaac Sim, where domain randomization with synthetic assets, pedestrians, vehicles, and street furniture, generates diverse training scenarios from a single captured scene. The approach combines the visual authenticity of real-world data with the scalability, safety, and controllability of simulation. Data collection will be conducted across the University of Massachusetts Amherst campus, capturing varied lighting, weather, and pedestrian conditions representative of typical guide dog deployment. 

The research contributes a complete pipeline from video capture through Gaussian Splatting reconstruction to physics-enabled simulation, a semantic domain randomization framework that tests cooperative sidewalk navigation skills, and a benchmarking environment for systematic evaluation of vision-based navigation policies. This work aims to demonstrate that encoding real-world structure into simulation can meaningfully reduce the sim-to-real gap for assistive robotic navigation.

RELATED ABSTRACTS