The recent development of 3D Gaussian Splatting (3DGS) has led to great interest in 4D dynamic spatial reconstruction from multi-view visual inputs. While existing approaches mainly rely on processing full-length multi-view videos for 4D reconstruction, there has been limited exploration of iterative online reconstruction methods that enable on-the-fly training and per-frame streaming. Current 3DGS-based streaming methods treat the Gaussian primitives uniformly and constantly renew the densified Gaussians, thereby overlooking the difference between dynamic and static features and also neglecting the temporal continuity in the scene.
To address these limitations, we propose a novel three-stage pipeline for iterative streamable 4D dynamic spatial reconstruction. Our pipeline comprises a selective inheritance stage to preserve temporal continuity, a dynamics-aware shift stage for distinguishing dynamic and static primitives and optimizing their movements, and an error-guided densification stage to accommodate emerging objects. Our method achieves state-of-the-art performance in online 4D reconstruction, demonstrating a 20% improvement in on-the-fly training speed, superior representation quality, and real-time rendering capability.
Framework. Overview of our proposed DASS framework. The selective inheritance stage (Green) exploits the temporal continuity and adaptively preserves Gaussians from the previous frame. The dynamics-aware shift stage (Blue) distinguishes the dynamic and static elements and optimizes the deformations. The error-guided densification stage (Yellow) detects and densifies the areas with weak reconstruction based on positional gradients and distortions. Variables highlighted in red represent learnable parameters in each stage, whose training is significantly lightweight compared to tuning all Gaussian parameters.
Qualitative comparisons on test view comparisons with baseline methods. We also provide visualizations of the dynamics components detected by our DASS, effectively guiding the optimization to focus on dynamic objects and complex motions.
*Encoded with H.264.