Polymorphic / experiments /fourier-splatting /FOURIER_SPLATTING_REPORT.md
UncleanCode's picture
Upload via Autoresearch export_data
8e02a28 verified
|
Raw History Blame Contribute Delete
5.48 kB
# Fourier Splatting × AnySplat — Experiment Report
**Date:** 2026-10-13
**Hardware:** existing SF Compute H100 80 GB node; no new node created
**Status:** two-scene prototype, 6,000 scene-optimization iterations per scene
## Question
Can the existing AnySplat video-to-3D pipeline be improved by converting its context-only geometry into Fourier-encoded planar primitives and optimizing the scene, instead of retaining a dense Gaussian cloud?
This experiment adapts the upstream Fourier Splatting implementation (https://github.com/mbjurca/FourierSplatting) to AnySplat geometry. It is a downstream, per-scene refinement experiment—not a new feed-forward model.
## Evaluation protocol
- Apartment: 24 frames total; 12 context views for geometry initialization/training and 12 held-out target views.
- Kitchen: 32 frames total; 16 context views for geometry initialization/training and 16 held-out target views.
- All images are center-cropped and resized to 448×448.
- AnySplat context-only inference initializes the Fourier scene. All-frame AnySplat inference is used to obtain target camera poses and align them into the context reconstruction coordinate frame, matching the evaluation protocol. Held-out target pixels are not used for geometry initialization or Fourier training.
- Initial seed selection uses AnySplat opacity × maximum scale. Fourier Splatting uses K=6 frequencies with staged activation at iterations 300, 600, 900, 1,200, and 1,500.
- Each scene is optimized for 6,000 iterations. The final LOD sweep changes active Fourier frequency count while keeping primitive count fixed.
- Metrics: PSNR (higher is better), SSIM (higher is better), and VGG LPIPS (lower is better). Baseline and Fourier metrics use the same VGG LPIPS implementation.
## Main held-out results
| Scene | Method | PSNR ↑ | SSIM ↑ | VGG LPIPS ↓ | Primitives |
|---|---|---:|---:|---:|---:|
| Apartment | AnySplat | 19.1235 | 0.6612 | 0.3154 | 1,858,896 |
| Apartment | Fourier, K=6 | 20.3469 | 0.7041 | 0.3173 | 99,483 |
| Kitchen | AnySplat | 21.6155 | 0.6660 | 0.2678 | 1,983,229 |
| Kitchen | Fourier, K=6 | 25.3078 | 0.8212 | 0.2586 | 120,000 |
### Changes vs AnySplat
- **Apartment:** +1.2234 dB PSNR, +0.0429 SSIM, LPIPS worsens by 0.0019; 18.68× fewer primitives.
- **Kitchen:** +3.6923 dB PSNR, +0.1552 SSIM, LPIPS improves by 0.0092; 16.53× fewer primitives.
The result is encouraging but not uniform across all metrics: apartment LPIPS is marginally worse, while kitchen improves on all three metrics. These are two indoor scenes, not a benchmark sufficient to claim state of the art.
## Fourier level-of-detail sweep
Primitive count is held constant within each scene. Truncating Fourier frequencies reduces PLY size and trades off perceptual quality.
### Apartment (99,483 primitives)
| K | PLY size (MB) | PSNR ↑ | SSIM ↑ | VGG LPIPS ↓ |
|---:|---:|---:|---:|---:|
| 1 | 25.07 | 19.9081 | 0.7048 | 0.3831 |
| 2 | 25.87 | 20.0612 | 0.7056 | 0.3802 |
| 3 | 26.66 | 20.3420 | 0.7078 | 0.3551 |
| 4 | 27.46 | 20.3825 | 0.7071 | 0.3374 |
| 5 | 28.25 | 20.3771 | 0.7056 | 0.3236 |
| 6 | 29.05 | 20.3469 | 0.7041 | 0.3173 |
### Kitchen (120,000 primitives)
| K | PLY size (MB) | PSNR ↑ | SSIM ↑ | VGG LPIPS ↓ |
|---:|---:|---:|---:|---:|
| 1 | 30.24 | 21.9698 | 0.7638 | 0.3696 |
| 2 | 31.20 | 23.3089 | 0.7772 | 0.3599 |
| 3 | 32.16 | 24.3752 | 0.8006 | 0.3186 |
| 4 | 33.12 | 24.9584 | 0.8137 | 0.2889 |
| 5 | 34.08 | 25.1819 | 0.8190 | 0.2696 |
| 6 | 35.04 | 25.3078 | 0.8212 | 0.2586 |
**Takeaway from LOD:** K=1–3 is substantially worse on LPIPS. K=5 is a useful compromise; K=6 gives the best perceptual metric on both scenes. The K=1 to K=6 PLY-size increase is about 15–16%, while the scene representation remains 16–19× smaller in primitive count than the AnySplat output. This report does not claim an equivalent 16–19× reduction in serialized bytes; those AnySplat PLY bytes were not measured in this run.
## Interpretation and limitations
1. **Promising direction:** Fourier Splatting looks useful as an offline scene-refinement / compact-representation stage after AnySplat.
2. **Not feed-forward:** every scene requires 6,000 optimization iterations. This has not yet been integrated into a low-latency video-to-3D inference path.
3. **No SOTA claim:** only two indoor scenes have been tested. More scenes, motion, occlusion, camera-path diversity, runtime and byte-level comparisons are required.
4. **Research-code license:** the upstream Fourier Splatting repository states non-commercial research/evaluation terms. Review licensing before using its implementation in a commercial release.
## Reproduction commands
From the vendored Fourier Splatting repository, with the project environment and compiled rasterizer installed:
~~~bash
python train.py \
--source_path /path/to/fourier_scene \
--model_path /path/to/output \
--eval --iterations 6000 \
--test_iterations 3000 4500 6000 \
--save_iterations 3000 4500 6000 \
--checkpoint_iterations 3000 4500 6000 \
--densify_until_iter 0 --densification_interval 100000 \
--fourier_K 6 \
--fourier_freq_schedule 300,600,900,1200,1500 \
--no_dome --resolution 448
~~~
The bundle contains the scene transforms and images, seed point clouds, final checkpoints, Fourier PLYs, LOD variants, evaluation outputs, and the preparation/evaluation scripts. Training uses context-only geometry; target images are reserved for held-out evaluation.