View a markdown version of this page

Train 3D Gaussian Splatting from video on Deadline Cloud - Deadline Cloud

Train 3D Gaussian Splatting from video on Deadline Cloud

The gsplat_pipeline job bundle runs a 3D Gaussian Splatting pipeline. The bundle takes a video file as input and produces a Gaussian Splatting .ply file as output. After downloading the output, you can view it in any Gaussian Splatting viewer such as SuperSplat.

The pipeline runs a single task that:

  1. Extracts video frames with FFmpeg.

  2. Solves Structure-from-Motion with COLMAP and GLOMAP, saving the pinhole model and undistorted images.

  3. Trains Gaussian Splatting with NeRF Studio splatfacto, Splatfacto in the Wild, or the simple_trainer.py gsplat library example. Output is saved to the .ply file you specify.

To run this bundle, deploy the CUDA farm CloudFormation template to create a Deadline Cloud farm with a CUDA GPU fleet. Then build the NeRF Studio conda package and publish it to your S3 conda channel.

If you only need the default NERFSTUDIO (splatfacto) trainer, you don't need the CUDA farm template or the custom NeRF Studio conda package. The minimum requirements are a Deadline Cloud farm with a GPU fleet, a queue environment that includes conda-forge, and the following conda packages: ffmpeg colmap glomap nerfstudio cuda.

To capture a usable video, follow these tips:

  • Use a wide field of view so that more objects are common between image pairs for Structure-from-Motion to use.

  • Turn off video stabilization to preserve identical lens optics between frames.

  • Capture with slow and steady motion. Keep moving the camera and avoid stopping and panning from a single location.

From the job_bundles directory of the samples repository, submit the job:

deadline bundle gui-submit gsplat_pipeline

On the Job-specific settings tab, select paths for both the input video file and the output .ply file.