Geometry-grounded audio-visual reasoning without explicit 3D reconstruction
Novel-view acoustic synthesis asks a deceptively difficult question: given observations of a scene, can a system infer how that environment should sound from a new listener position? Existing approaches often depend on expensive per-scene 3D reconstruction.
Our approach grounds spatial audio synthesis directly in feed-forward visual geometry. Given a short video, the framework combines visual information, estimated scene geometry, and acoustic representations, then uses the Geometry-Grounded Acoustic Decoder (GGAD) to synthesize listener-conditioned binaural audio without explicitly rendering the target view.
My contribution was designing the output representation extracted from the VGGT feed-forward geometry model and formulating the query/key structure of the GGAD cross-attention mechanism.
For my broader research direction, this project is an example of environment understanding across modalities: extracting a representation of physical space that supports inference beyond what is directly observed.
We present the first unified framework for novel-view acoustic synthesis that entirely bypasses explicit 3D visual rendering and costly photogrammetry by directly grounding spatial audio generation in feed-forward visual geometry. Our framework synthesizes accurate and immersive spatial audio in 3D spaces without requiring viewpoint images, dense point maps, or ground-truth poses for input video. We propose the Geometry-Grounded Acoustic Decoder (GGAD) to dynamically attend to cross-modal features embedding local and global geometries in audio and visual modalities. Extensive experiments show that our framework outperforms prior work across various benchmarks in high-quality, viewpoint-accurate spatial audio synthesis, without requiring time-consuming explicit rendering of novel-view images or dense point maps.
@inproceedings{polra2026nvas,title={Visual Geometry Grounded Novel-View Acoustic Synthesis},author={Polra, Jay and Chauhan, Dhwanil and Huang, Wenjun and Toth, Kyle and Wang, Xianhui and Ni, Yang},booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops},year={2026},url={https://dhwanil832.github.io/projects/nvas/},note={Purdue University Northwest · UC Irvine · San Diego State University}}