←/→ or Space · F fullscreen · O overview · N notes
ICCV 2025 Highlight

Shape of
Motion

4D Reconstruction from a Single Video

Qianqian Wang · Vickie Ye · Hang Gao · Weijia Zeng · Jake Austin · Zhengqi Li · Angjoo Kanazawa

Explicit, persistent 3D motion from casual monocular video
开场强调这篇论文同时解决两个通常分开的任务：动态 novel-view synthesis 和 long-range 3D tracking。它不是仅仅重建每个时刻的外观，而是要恢复每个三维场景元素贯穿整个视频的运动轨迹。
01 · One-minute overview

What does this paper
actually recover?

The input is a casually captured monocular video of a dynamic scene. The method assumes camera intrinsics and poses are available or estimated, then optimizes one globally consistent 4D scene representation.

The output is a persistent set of 3D Gaussians. Their appearance and shape parameters remain attached to the same scene elements, while their positions and orientations change over time.

This single representation supports both real-time novel-view rendering and long-range 3D tracking of arbitrary visible surface points.

Core idea: model motion explicitly, constrain it to be low-dimensional, and fit it by jointly consolidating RGB, depth, camera, mask, and long-range track evidence.
InputRGB video, camera parameters, moving-object masks, monocular depth, and long-range 2D tracks.
Scene modelPersistent canonical 3D Gaussians plus time-varying motion.
Primary outputFull-sequence 3D trajectories in the world coordinate system.
Secondary outputDynamic novel-view synthesis at observed timestamps.
这一页给听众一个完整 mental model。输入并不只是 RGB，系统还使用估计的相机、动态 mask、单目深度和 TAPIR tracks。输出是同一组 persistent Gaussians 在整个时间轴上的姿态。
02 · Problem formulation

Why is monocular
4D reconstruction ill-posed?

At every timestamp, a moving scene element is observed from only one camera position and in only one pose. The system therefore cannot obtain conventional multi-view constraints for that object state.

  • Depth ambiguity: many 3D configurations can produce the same image.
  • Motion ambiguity: image displacement may come from camera movement, object movement, or both.
  • Occlusion: points can disappear for long periods but should retain a persistent 3D identity.
  • Non-rigid motion: independently optimizing every point creates severe jitter and overfitting.
Observed 2D motion
→
Camera motion + object motion + uncertain depth
→
Many plausible but incompatible 4D explanations
The paper's strategy is not to remove the ambiguity completely, but to restrict the solution space using a compact motion model and complementary learned priors.
解释为何传统 structure-from-motion 不够：动态对象在每个姿态只有一个视角。也解释为何逐点 deformation 太自由，容易产生 jitter、popping 和不稳定几何。
03 · Positioning

What is missing from
previous monocular methods?

DYNAMIC VIEW SYNTHESIS

Good rendering, implicit motion

Deformation fieldsMap between canonical and observation space, but do not naturally expose persistent world-space trajectories.
Time-dependent radiance fieldsCan reproduce appearance but often make point identity difficult to query over long time ranges.
Short-range scene flowChaining consecutive flow compounds errors and is fragile through occlusion.
TRACKING METHODS

Good correspondence, incomplete 3D scene

2D point trackersMaintain image-space trajectories but do not recover the underlying 3D geometry.
2D tracks + depthLifting each frame independently produces noisy, inconsistent world-space trajectories.
Frame-space 3D trackingMay still entangle object motion with camera motion.
Shape of Motion is designed so the same explicit representation can be evaluated as a renderer and as a long-range 3D tracker.
相关工作不需要逐篇讲。用两条主线概括：一类方法擅长 rendering 但运动是 implicit；另一类擅长 tracking 但缺少完整、可渲染的三维场景。
04 · Scene representation

One canonical Gaussian set
persists through time.

gᵢ⁰ = ( μᵢ⁰, Rᵢ⁰, sᵢ, oᵢ, cᵢ )

μᵢᵗ = R₀→ₜ μᵢ⁰ + t₀→ₜ
Rᵢᵗ = R₀→ₜ Rᵢ⁰
μᵢ⁰, Rᵢ⁰Canonical position and orientation.
sᵢ, oᵢ, cᵢScale, opacity, and color; treated as persistent properties.
T₀→ₜTime-dependent rigid transformation applied to a Gaussian.

Unlike reconstructing an independent point cloud at every frame, the method keeps one set of scene elements. This gives every Gaussian a persistent identity.

Because the representation is explicit and differentiably rasterized, it can render RGB, depth, masks, and projected trajectories. These outputs can all be compared against input priors during optimization.

Static and dynamic Gaussians are modeled separately but rendered and optimized together, allowing the method to preserve a stable background while explaining moving foreground regions.

Persistent identity is the bridge between novel-view synthesis and long-range tracking.
这里讲清楚 3DGS 在论文中承担的角色：不只是渲染 primitive，而是有身份的场景元素。static 和 dynamic Gaussians 联合 rasterize，但只有动态部分使用 motion bases。
05 · Motion representation

Shared SE(3) bases impose
a low-dimensional motion prior.

Ti(t) = Σb=1…B wi(b) T(b)(t)

The scene contains far more Gaussians than motion bases: B ≪ N. Instead of learning an independent trajectory for every Gaussian, the model learns a small set of global rigid-motion trajectories.

Each Gaussian receives a coefficient vector that blends these bases. Gaussians that move similarly are encouraged to have similar coefficients, producing a soft decomposition into approximately rigid motion groups.

In implementation, rotation is represented with a 6D rotation parameterization and combined separately from translation using the same weights.

Basis A0.75
Basis B0.35
Basis C0.15
Group A Group B Group C
Why not per-Gaussian motion?It has too many degrees of freedom. The ablation shows more popping and jitter, while shared bases improve overall tracking accuracy and visual stability.
强调 B 设置为 10，而 Gaussian 数量是数万级。这个巨大压缩比是方法的主要结构先验。PCA 可视化表明 motion coefficients 与刚性运动组有明显关联。
06 · Querying motion

How does the model produce
long-range tracks?

For a query pixel in frame t, the renderer identifies the Gaussians contributing to that pixel. Instead of rasterizing their current colors, it rasterizes their 3D positions at another target time t′.

The alpha-composited result is the expected world-space position of the corresponding surface point at time t′. Projecting this point into the target camera gives a 2D correspondence; its camera-space z-coordinate gives the target depth.

This formulation allows a point visible at one time to be queried at any other time—even when it becomes occluded—because its motion is stored in the persistent 4D representation.

X̂t→t′w(p) = Σᵢ Tᵢ αᵢ μᵢ,t′

Ût→t′(p) = Π(Kt′ Et′ X̂t→t′w(p))

D̂t→t′(p) = [Et′ X̂t→t′w(p)]z
3D queryWhere is this surface point in world coordinates at time t′?
2D queryWhere should the same point appear in the target image?
Traditional optical flow predicts local image displacement. This representation answers a stronger question: where does the corresponding physical point move in 3D over the full sequence?
这是论文容易被忽略但很关键的部分。它通过 rasterization 得到任意 query time 到 target time 的 3D world coordinate map，因此 tracking 是 representation 的直接查询，而不是额外的 tracking network。
07 · Preprocessing

What information is used
before optimization begins?

Jointly optimized
4D scene
MegaSaMinitial camera poses
Track-Anythingmoving-object masks
Depth Anythingrelative monocular depth
TAPIRlong-range 2D tracks

The relative depth maps are aligned to metric depth using a per-frame scale and shift. The authors report that relative depth retains finer local details than directly using metric-depth estimates.

Foreground 2D tracks are unprojected with the aligned depth maps to obtain noisy initial 3D trajectories. Their TAPIR visibility and uncertainty scores are retained for weighted initialization.

Moving-object masks require a few user clicks in the reported system. Static background points are initialized by directly unprojecting the aligned depth maps.

For in-the-wild sequences, MegaSaM camera poses are further refined as learnable parameters during optimization.

Important assumptionThe method is not RGB-only internally. Its contribution is the consolidation of several noisy estimates, rather than replacing those estimators.
一定要说明输入依赖。论文使用 MegaSaM、Track-Anything、Depth Anything 和 TAPIR。这个方法的价值在于将这些 noisy priors 变成全局一致表示，但性能仍然受它们影响。
08 · Initialization

A structured start matters
for a highly non-convex problem.

STEP 1Choose the canonical frame as the frame in which the largest number of 3D tracks is visible.
STEP 2Initialize dynamic Gaussians by sampling locations from the lifted noisy 3D tracks in that frame.
STEP 3Cluster vectorized track velocities with k-means to obtain candidate motion groups.
STEP 4Fit each SE(3) basis with weighted Procrustes alignment across the sequence, using track uncertainty and visibility as weights.
STEP 5Initialize per-Gaussian coefficients with exponential decay from each cluster center, then jointly fit tracks under temporal smoothness.
WHY THIS STAGE EXISTS

The final objective has many plausible local minima.

A random motion initialization can explain individual frames without finding a coherent full-length trajectory. The explicit SE(3) fitting stage gives each basis a physically meaningful starting motion before photometric optimization begins.

Ablation: removing the SE(3) initialization increases 3D tracking EPE from 0.082 to 0.111.
这页是新版里增加的重要细节。初始化不是普通随机初始化：先对 noisy 3D tracks 的速度聚类，再对每个 cluster 做 weighted Procrustes，最后初始化 basis coefficients。
09 · Optimization objectives

Four constraints tie appearance,
geometry, and motion together.

L = Lrecon + Ltrack-2D
    + Ltrack-depth + Lrigidity
L_recon

Matches rendered RGB, depth, and foreground masks independently at each frame.

L_track-2D

Matches rendered long-range correspondences to TAPIR trajectories across randomly sampled frame pairs.

L_track-depth

Requires the tracked point's predicted target depth to agree with the rendered target depth.

L_rigidity

Preserves distances between dynamic Gaussians and their k-nearest neighbors across time.

Per-frame reconstruction losses ensure the representation renders each observed frame, but they do not by themselves establish persistent point identity.

Cross-time track losses bind a Gaussian's motion to long-range correspondences, preventing each timestamp from being explained independently.

Track-depth consistency connects image-space tracking to three-dimensional geometry rather than treating them as separate signals.

Local rigidity is a physical motion prior: nearby scene elements should not arbitrarily change their pairwise distances over short neighborhoods.

The representation is the meeting point where all noisy observations must agree.
重点解释 loss 的互补性。RGB/depth/mask 是单帧约束，2D track 和 track-depth 是跨时间约束，rigidity 则限制局部三维结构。
10 · Implementation and evaluation

What does the reported
system cost?

10shared SE(3) motion bases in all experiments
40kinitial dynamic Gaussians
100kinitial static Gaussians
≈2 hoptimization for 300 frames at 960×720 on one A100
TRAINING
  • 1,000 iterations for motion initialization.
  • 500 epochs for joint scene optimization.
  • Adam optimizer and adaptive Gaussian control.
  • Camera poses are refined during in-the-wild optimization.
  • Reported rendering speed is approximately 140 FPS.
BENCHMARKS AND TASKS
  • iPhone dataset: real casual videos, 200–500 frames, tracking and validation views.
  • Kubric MOVi-F: synthetic multi-object motion with dense 3D ground truth.
  • NVIDIA dataset: additional novel-view comparison.
  • Tasks: long-range 3D tracking, long-range 2D tracking, and novel-view synthesis.
不要把这篇工作描述成实时重建。它的 rendering 是实时的，但 reconstruction 是 per-scene optimization，300 帧需要约两小时。这个 distinction 很重要。
11 · Qualitative results

Inspect input, motion,
and rendering together.

Each example is best read as a three-part diagnostic: the input defines what was observed, the trajectory view shows whether point identity remains coherent through time, and the novel view tests whether geometry and appearance are globally consistent.

Official video requires internet.
Input video
Official video requires internet.
3D tracks
Official video requires internet.
Novel view
Official video requires internet.
Input video
Official video requires internet.
3D tracks
Official video requires internet.
Novel view
Official video requires internet.
Input video
Official video requires internet.
3D tracks
Official video requires internet.
Novel view
Official qualitative media is streamed from the authors' project page.
展示时不要只说视觉效果好。让听众分别观察：轨迹是否平滑、遮挡后是否保持一致、novel view 是否出现结构退化，以及背景与动态前景是否对齐。
12 · Quantitative results

The same representation
leads across three tasks.

MethodEPE ↓δ3D@10 ↑AJ ↑OA ↑PSNR ↑LPIPS ↓
HyperNeRF0.18245.810.152.015.990.51
Deformable 3DGS0.15155.314.063.911.920.66
TAPIR + Depth Anything0.11463.227.867.4——
SpatialTracker (world)0.12563.924.973.5——
Shape of Motion0.08273.334.486.616.720.45

Selected results from the iPhone dataset. δ3D@10 is the percentage of predictions within 10 cm; AJ and OA are TAP-Vid 2D tracking metrics.

Against tracking-only baselines: the method improves over simply lifting TAPIR trajectories with Depth Anything. This shows that global 4D consolidation corrects noisy per-frame geometry and correspondence.

Against rendering-focused baselines: dynamic NeRF and deformation-based Gaussian methods can render, but their implicit motion is less accurate for long-range trajectory queries.

Why the joint task matters: tracking baselines do not synthesize novel views, while the best rendering baselines are weaker trackers. Shape of Motion is designed to perform both with one representation.

On Kubric MOVi-F, the method also reduces 3D tracking EPE to 0.16, compared with 0.19–0.20 for CoTracker/TAPIR plus depth.
表格只保留有代表性的 baseline。重点不是逐个数字读，而是说明方法同时覆盖 tracking 和 rendering；其他 baseline 通常只能在其中一类任务上输出结果。
13 · Ablation

Which design choices
actually matter?

VariantEPE ↓δ3D@5 ↑δ3D@10 ↑
Full model0.08243.073.3
Translation bases0.09342.369.9
Per-Gaussian SE(3)0.08343.670.2
Per-Gaussian translation0.08741.269.2
No SE(3) initialization0.11139.365.7
No 2D tracks0.14130.457.8
SE(3) vs translationRotation-aware bases improve tracking over translation-only bases.
Shared bases vs per-pointSimilar numeric EPE can hide visual instability; per-Gaussian SE(3) produces more popping and jitter.
InitializationThe Procrustes-based SE(3) initialization is necessary to avoid poor local minima.
2D tracksRemoving track supervision causes the largest degradation, confirming that photometric and depth losses alone cannot recover reliable long-range motion.
The strongest lesson is not simply “use Gaussians”; it is “combine a constrained motion representation with persistent cross-time supervision.”
Ablation 中 No 2D Tracks 降幅最大。Shared motion basis 的意义既体现在 tracking 指标，也体现在减少 popping/jitter。不要只看 per-Gaussian SE3 的 EPE 接近 full model。
14 · Limitations and discussion

What remains unresolved?

Official video requires internet.
Fast motion and occlusion
Official video requires internet.
Incorrect monocular depth
  • Per-scene optimization: reconstruction is not feed-forward or streamable; a 300-frame example requires about two hours on an A100.
  • Dependency on external priors: camera, depth, masks, and tracks can fail in textureless regions, under large motion, or during long occlusion.
  • Manual interaction: the reported pipeline needs a few user clicks to obtain moving-object masks.
  • Topology and disocclusion: persistent Gaussians cannot reliably reconstruct geometry that is never observed, and difficult topology changes remain challenging.
  • Linear basis blending: a small set of rigid bases is a strong regularizer, but may underfit highly complex local deformation.
Balanced interpretationThe system is powerful because it constrains noisy evidence globally. The same reliance on upstream priors and per-scene fitting is also its main practical limitation.
限制页要区分论文明确提出的限制和合理分析。明确限制包括 per-scene optimization、依赖 upstream estimates、需要用户 mask。复杂 topology 和线性 basis underfitting 可作为讨论性分析。
15 · Final takeaways

Why is this paper important?

It unifies tracking and rendering.

The same scene representation supports accurate 3D trajectory queries and dynamic novel-view synthesis.

It makes motion explicit.

Persistent world-space trajectories are easier to query, visualize, and evaluate than an implicit deformation field.

It regularizes before it optimizes.

Shared SE(3) bases dramatically reduce the freedom of monocular dynamic reconstruction.

Shape of Motion turns a monocular video into a model of not only what the 3D scene looks like, but how its physical elements move through space and time.
Persistent 3D Gaussians 10 shared SE(3) bases Depth + TAPIR supervision Long-range 3D tracking Novel-view synthesis
Paper: Wang et al., “Shape of Motion: 4D Reconstruction from a Single Video,” ICCV 2025. Media: official author project page.
结尾回到三个最重要的贡献：统一 tracking/rendering、显式 persistent motion、低维 SE3 basis regularization。不要把结论停留在“效果好”，而是强调 representation design。