Overview

This workflow demonstrates an example of performing robust and relatively fast camera alignment (SfM) using omnidirectional images: Equirectangular (equidistant cylindrical projection) images, followed by training 3D Gaussian Splatting (3DGS).

References

DJI AVATA360 examples

• https://x.com/kotohibi_3d/status/2079907663895482456
• https://x.com/kotohibi_3d/status/2040724840504758578
• https://x.com/naribubu/status/2038881884558791088
• https://x.com/naribubu/status/2038875398717722743 ### DJI OSMO360 examples
• https://x.com/kotohibi_3d/status/2088521899160879450
• https://x.com/kotohibi_3d/status/2082426800215654725
• https://x.com/kotohibi_3d/status/2074821581948481758
• https://x.com/kotohibi_3d/status/2038179454367957106 # Requirements
• 360° Camera
  • DJI OSMO360
  • DJI AVATA360
  • Insta360
• High-end PC and NVIDIA GPU
  • Training 3DGS requires a high-performance GPU. In particular, more VRAM is better. I recommend a GPU with at least 12 GB of VRAM.

• Metashape Standard (Not supported the Professional edition)

  • Directly supports SfM with omnidirectional images; extremely fast and robust.
  • https://www.agisoft.com/features/standard-edition/

• 3D Gaussian Splatting software

  • Postshot: https://www.jawset.com/
  • LichtFeld Studio (LFS): https://github.com/MrNeRF/LichtFeld-Studio
  • Brush: https://github.com/ArthurBrussee/brush

• Still-image extraction tool from video

  • Extract Sharpest Frame (Free edition)
    • https://github.com/Kotohibi/Extract_sharpest_frame
  • 360 Extractor (paid edition)
    • https://kotohibi.f5.si/360/

• Metashape 360 SfM to COLMAP-format Cubemap conversion tool

  • Metashape 360 to COLMAP Converter (Free edition)
    • https://github.com/Kotohibi/Metashape_360_to_COLMAP_plane
  • 360 MCConverter (paid edition)
    • https://kotohibi.f5.si/360/

• (Optional) Additional license for estimating real-world scale in 3DCG

  • Metashape 360 to COLMAP Converter includes a feature that estimates the real-world scale of 3DGS using a 2D marker called AprilTag. An additional license is required to use it. The following articles explain how to use AprilTag.
  • Add-on Real Scale 3DGS with AprilTag https://kotohibi-cg.booth.pm/items/8323677
  • English version: https://x.gd/CoWJA
  • Japanese version: https://x.gd/Isahb

Video Shooting (e.g. OSMO360)

Attach the camera to a selfie stick and slowly walk through the area you want to capture.

Recommended video settings: D-Log M, 30 fps or higher.

Develop the Video

Import the captured data into DJI Studio and perform color grading (color restoration).

• Apply the settings inside the red frame in the image below. Everything else can be left at default.
• (Advanced settings) When using the seam mask implemented in Extract Sharpest Frame V1.0.0 or later, turn RockSteady off. RockSteady provides electronic image stabilization and horizon leveling, but it changes the stitch line. Turn off the equivalent feature for Insta360 cameras as well.
  • (Note) The seam mask masks misalignments along the stitch line between the front and rear fisheye cameras, allowing that area to be excluded from the camera alignment and 3DGS training described below. ### Export the video
• Export as an MP4 omnidirectional video. Example settings are shown in the image below.
• When developing multiple clips, you can develop them together using "Multiple Clips." Extract Sharpest Frame can batch-process multiple videos.

Extract Still Images from Video

There are many ways to extract still images from video. Research and choose your preferred method.

Here I introduce the tool I have published.

Extract Sharpest Frame is a tool that extracts the sharpest image at specified frame intervals.

• New features are prioritized for updates in the BOOTH edition

Main Item	Description
Video file	Select the omnidirectional video. Multiple videos can be selected and processed in a batch. Note: File paths containing multibyte characters are not supported.
Output folder	Specify the folder where still images and masks will be saved. frames and masks folders will be created under this folder. You can also choose whether images extracted from multiple videos should be collected into a single folder. Note: File paths containing multibyte characters are not supported.
Scale width	Image size used when calculating sharpness for all video frames. Larger values give more precise calculations. Note: Extracted images are always output at the original video resolution.
Chunk size	Interval for extracting still images. For a 30 fps video, setting 30 extracts images every 1 second.
Workers	Number of processes used when calculating image sharpness. Around 4 is recommended.
Start (HH:MM:SS)	Specify the time to start extraction. The format is HH:MM:SS. If left blank, processing starts from the beginning of the video.
End (HH:MM:SS)	Specify the time to end extraction. The format is HH:MM:SS. If left blank, processing continues to the end of the video.
Remove similar frames	Excludes similar frames. If Review is enabled, you can adjust the threshold during execution to control how many images are extracted.
pHash threshold	Specifies the threshold for judging similar images. Higher values remove more images. This is useful when movement speed during shooting is irregular.
Mask Generation	Generates mask images for moving objects such as people and cars. This improves SfM accuracy in later steps.
SAM3 Dual Mask	The latest version supports SAM3 masks. You can generate masks using any short sentence and preview the result with the Preview/Edit button. https://x.com/kotohibi_3d/status/2061044432837972367 (Advanced settings) Two SAM3 masks can be configured separately: one for camera alignment and one for 3DGS training. For details, see the following Google Slides: https://t.co/X0uRH959RV
YOLO Mask	[YOLO Class IDs] Specify the class IDs to detect. 0: person, 1: bicycle, 2: car, etc. Multiple IDs can be specified comma-separated. https://github.com/ultralytics/ultralytics/blob/main/ultralytics/cfg/datasets/coco.yaml [YOLO Confidence] Raising the threshold reduces false positives. Lowering it detects more objects but increases false positives. [YOLO Model] Model size and performance increase from yolo11n toward yolo11x, but so does processing load.
Seam Mask	Masks misalignments along the stitch line joining the two fisheye images. To keep the stitch line fixed, develop the video with horizon-leveling features in DJI Studio, Insta360 Studio, and similar software turned off before using this feature.
Custom Mask	Specify a fixed mask image. When used with the masks above, they are merged. This is useful for masking areas that are always visible, such as a camera rig. Note: Specify a PNG image with the same resolution as the video.
Analysis only	Perform only sharpness calculation. Calculation results (metadata) are saved in the output folder. On subsequent runs, if metadata exists in the output folder, the analysis phase is skipped and only image extraction is performed. Useful when adjusting Chunk size.
Save config	Save the above settings as a configuration file.
Load config	Load a previously saved configuration file.
Run	Execute processing

Execution Result

• Still images are extracted as shown below. Please check the SfM result in the next step and readjust the Chunk size if necessary.
• Masks are also generated automatically

Perform Camera Alignment (SfM)

Use Metashape Standard, which can directly process omnidirectional images for SfM.

Load the extracted omnidirectional images

[Workflow] → [Add Folder]

Change Camera Type to Spherical

Select [Tools] → [Camera Calibration] and set Camera type to Spherical.

Load the mask images

Select [File] → [Import] → [Import Masks].

Use the settings below and click [OK]. Then a folder selection dialog will appear; select the mask folder generated by Extract Sharpest Frame.

• Set Filename template to {filename}.png (the default is {filename}_mask.png). ### Set SfM parameters Here are two parameter examples I often use.

Example	Description
	Turn ON "Generic preselection." It first matches photos at a low-precision setting, selects overlapping pairs, and then runs the main process. This is fast, but it can fail in some cases. If that happens, try the option below. For [Apply masks to], select [Key points].
	Turn OFF "Generic preselection." This performs full matching for all image pairs. Since it takes longer, set "Key point limit" to a smaller value. Set "Tie point limit" to 0 for unlimited points. For [Apply masks to], select [Key points].

Execute

Click OK to run SfM.

Example result shown below. The spherical markers correspond to each omnidirectional image.

Clean up Tie points

• Remove low-reliability Tie points to improve SfM accuracy. This is a very important step for high-detail 3DGS. Select [Tools] → [Tie Points] → [Clean Tie points].
• Select [Reprojection error] and adjust the slider to remove about 5% of the Tie points.
• The number of Tie points is shown at the bottom left of the screen, so adjust the slider while checking how many Tie points will be removed.
• Click [Optimize Cameras] to optimize the cameras.
• Do the same for [Recostruction uncertainty], remove about 5% of the Tie points, and then run [Optimize Cameras] again.
• Do the same for [Projection accuracy], remove about 5% of the Tie points, and then run [Optimize Cameras] again.
• Repeat the above once more so that low-reliability Tie points are removed as much as possible. ### Export SfM results
  • Export Camera information [File] → [Export] → [Export Cameras] → Select Agisoft XML (*.xml) and save.
  • Export Point Cloud [File] → [Export] → [Export Point Cloud] → Select Stanford PLY (*.ply) and save.

Convert to COLMAP Cubemap

Convert the Metashape SfM results into COLMAP-format 6-direction Cubemap images.

Here I introduce the tool I have published.

Metashape 360 to COLMAP Converter

• New features are prioritized for updates in the BOOTH edition ### Settings ①

Settings ② : Mask Processing

• If you use SAM3 masks generated by Extract Sharpest Frame, turn this feature off and specify the SAM3 masks in the custom-mask settings below.
• You can generate masks for people, vehicles, and other objects. Mask generation is particularly important with 360° cameras because the operator is often captured in the frame.

Settings ③ (Advanced Settings): Custom Mask

• You can load SAM3 Dual Masks (for 3DGS training) generated by Extract Sharpest Frame, or custom masks. Specify mask images as PNG files with the same file count, resolution, and filenames as the still images.Reference: Google Slides -> https://t.co/X0uRH959RV ### Settings ④ (Advanced Settings): AprilTag
• This feature estimates the real-world scale of 3DGS using 2D markers. An additional license is required. The following articles explain how to use AprilTag.
  • English version: https://x.gd/CoWJA
  • Japanese version: https://x.gd/Isahb ### Settings ⑤ (Advanced Settings): Overlap Filter
• See the [EN]Mixed 3DGS Workflow with 360 Video and Planar Images guide.
• The ZIP file downloaded from BOOTH includes a detailed PDF operation manual. Please refer to it.
• Reference: https://x.com/kotohibi_3d/status/2078088377371681237

Settings ⑥ (Advanced Settings): Cubemap Reduction

• This feature reduces the number of Cubemaps while minimizing 3DGS quality loss. It can shorten 3DGS training time and reduce VRAM usage.
• The ZIP file downloaded from BOOTH includes a detailed PDF operation manual. Please refer to it.
• Reference: https://x.com/kotohibi_3d/status/2086408535768973659 ### Execute After processing completes successfully, the following folders and files are generated in the output folder.

(Postshot) 3D Gaussian Splatting Training

Here I explain the workflow using Postshot.

Import Cubemap

• First, drag & drop the Images folder, cameras.txt, images.txt, and points3D.txt into Postshot.

Mask Settings

• Next, drag & drop the masks folder into the Image Masks area in Postshot. Select Remove Background for Mask Mode.

Cubemap Import Result

• Once the Cubemap is successfully imported, you will see a screen like the one above.

Start 3DGS Training

• Here is an example of training parameters I use for wide-area 3DGS. Adjust parameters according to your scene.
• Turn on Photometric Compensation.
• Change Max Splat Count according to the scale of the scene (3,000-12,000 kSplats).
• Adjust Max Sph.Hrm.Degree (1-3). If VRAM is limited, I recommend 1.
• Stop Training After is calculated automatically according to the number of images.

3DGS Training Result

• As training progresses, you should start seeing the 3DGS!

(LichtFeld Studio) 3D Gaussian Splatting Training

Here I explain the workflow using LichtFeld Studio (LFS).

Import Cubemap

• Select [File] → [Import Dataset], then specify the Output Folder from Metashape 360 to COLMAP Converter.
  

• If the data is detected correctly, a dialog like the one below appears. Confirm the contents and click [Load] to continue.
  

• Once the data is loaded correctly, you will see a screen like the one below.
  

Start 3DGS Training

• Mask settings
  • Select [Training Parameters] → [Mask Mode] → [Ignore].
  • Turn off [Alpha Mask].
• Training parameters
  • Here is an example of settings I often use.
  • I recommend [Strategy] → MRNF.
  • Adjust Max Gaussians according to the scale of the scene (3,000,000-12,000,000).
  • Adjust SH Degree (1-3). If VRAM is limited, I recommend 1.
  • Iterations and Steps Scaler are calculated automatically according to the number of images.
  • With MRNF, changes to the other parameters are usually not very necessary.
  • LFS has many parameters, so please research on the web and find the best settings for your scene.
  • Recently, Bilateral Grid is often turned off because PPISP is sufficient in many cases.
• Click [Start Training] to begin 3DGS training.

3DGS Training Result

• As training progresses, you should start seeing the 3DGS!

Finally

There are many 3DGS methods, and this article is just one example. I will continue sharing the latest information on my X account.

Please research on your own and develop even better techniques. Enjoy 3DGS :)

• my X: https://x.com/kotohibi_3d
• 3DGS pipeline guide: https://github.com/Kotohibi/3DGS_pipeline_guide