View a markdown version of this page

Train 3D Gaussian Splatting from video on Deadline Cloud - Deadline Cloud

Train 3D Gaussian Splatting from video on Deadline Cloud

The gsplat_pipeline job bundle on the GitHub website runs a 3D Gaussian Splatting pipeline. The bundle takes a video file as input and produces a Gaussian Splatting .ply file as output. After downloading the output, you can view it in any Gaussian Splatting viewer such as SuperSplat on the GitHub website.

The pipeline runs a single task that performs the following steps:

  1. Extracts video frames with FFmpeg on the FFmpeg website.

  2. Solves Structure-from-Motion with COLMAP on the COLMAP website and GLOMAP on the GitHub website, saving the pinhole model and undistorted images.

  3. Trains Gaussian Splatting with one of three trainers: NeRF Studio splatfacto on the Nerfstudio website, Splatfacto in the Wild on the Nerfstudio website, or the simple_trainer.py gsplat library example on the gsplat website. Output is saved to the .ply file you specify.

To run this bundle, deploy the CUDA farm CloudFormation template on the GitHub website to create a Deadline Cloud farm with a CUDA GPU fleet. Then build the NeRF Studio conda package on the GitHub website and publish it to your S3 conda channel.

If you only need the default NERFSTUDIO (splatfacto) trainer, you don't need the CUDA farm template or the custom NeRF Studio conda package. The minimum requirements are a Deadline Cloud farm with a GPU fleet, a queue environment that includes conda-forge, and the following conda packages: ffmpeg colmap glomap nerfstudio cuda.

To capture a usable video, follow these tips:

  • Use a wide field of view so that more objects are common between image pairs for Structure-from-Motion to use.

  • Turn off video stabilization to preserve identical lens optics between frames.

  • Capture with slow and steady motion. Keep moving the camera and avoid stopping and panning from a single location.

From the job_bundles directory of the samples repository, submit the job:

deadline bundle gui-submit gsplat_pipeline

On the Job-specific settings tab, select paths for both the input video file and the output .ply file.

For a complete walkthrough that covers capturing a video, submitting and monitoring the job, and viewing the trained result, see Train 3D Gaussian Splatting from your own video.