Advances in 4D Representation: Geometry, Motion and Interaction

We present a survey on 4D generation and reconstruction, a fast-evolving subfield of computer graphics whose developments have been propelled by recent advances in neural fields, geometric and motion deep learning, as well 3D generative artificial intelligence (GenAI). While our survey is not the first of its kind, we build our coverage of the domain from a unique and distinctive perspective of 4D representations, to model 3D geometry evolving over time while exhibiting motion and interaction. Instead of offering an exhaustive enumeration of many works, we take a more selective approach by focusing on representative works to highlight both the desirable properties and ensuing challenges of each representation under different computation, application, and data scenarios. The main take-away message we aim to convey is on how to select and then customize the appropriate 4D representations for a given task. Organizationally, we separate the 4D representations based on three key pillars: geometry, motion, and interaction. Our discourse encompasses the most popular representations of today, such as neural radiance fields (NeRFs) and 3D Gaussian Splatting (3DGS), and also brings attention to relatively under-explored representations in the 4D context, such as structured models and long-range motions. Throughout, we reprise the role of large language models (LLMs) and video foundational models (VFMs) in a variety of 4D applications, while steering the discussion towards their current limitations and how they can be addressed. We also provide a dedicated coverage on what 4D datasets are currently available, as well as what is lacking, in driving the subfield forward.

This website is not exhaustive and may not include all relevant 4D works. We welcome your contributions and feedback:

• Missing works — to include your work in our survey
• Corrections — if a paper is tagged incorrectly
• Updates — code availability, conference acceptance, etc.
• Feedback — suggestions for improving this resource

🚀 Submit your work via our GitHub issue template, and please give us a ⭐ star. Contact: mza143@sfu.ca.