Video import and possible issues

Video as an Input in RealityCapture

If you have a video taken analogically to taking images for photogrammetry, you can use it instead of photographs and/or laser scans, or as a supplement to other project inputs capturing the same object or scene. As always, you are not limited to the number of such videos. (You can even use this feature to cut any video into a sequence of pictures for purposes other than use in this software.)

RealityCapture extracts purely keyframes from the videos (defining the starting and ending points of any smooth transition) and ignores interpolated frames (synthesized by a codec, which are geometrically inaccurate) doing its best for you to get the result you want.

Without any additional codecs other than in the native Windows' pack, you will be able to import these video formats: Advanced Systems Formats (.asf), AVI (.avi), QuickTime (.mov), MPEG (.mpg, .mpeg, .mp4), and Windows Media Video (.wmv). Output frames are PNGs. 

Read more: Image and video codecs and import formats

Video import procedure:

1. To import a video or more of them, go through the Video Sequence button in the WORKFLOW tab, or simply drag and drop videos into your RealityCapture project.
2. Once selected, you will be taken to the settings dialog, where you need to choose a location for the extracted frames to be saved to, and their time span – namely Jumps length(s) – defining a pause between 2 frames. When importing data in .cmi file format, this dialog is skipped.
3. After clicking on Import, the videos will be split into frames and loaded into the application. You will be able to find them in the Images root in the 1Ds panel.
4. Now you can proceed with adding other inputs or directly aligning them and further processing like usual.

Key issues with shooting video for photogrammetry

Image resolution and image quality

The lack of image quality, resolution and dynamic range might be the first limiting factor when using a low-budget video camera, instead of a photo camera. For instance, usually, you can only get to 8K footage, and the footage might already be pixel binned when using such high resolution.

Video footage is usually cropped (not using full sensor readout)

The video can be a cropped full HD video and it may not be using the full sensor readout (it may be cropped to 16:9). The crop is an issue because the frame is then without the full information about the lens distortion model from the video frame. 

Image quality, bitrate, codec, chroma sampling

One of the possible issues might be compression/transcoding artifacts like this green line artifact from transcoding low bit-rate footage in the picture above.

Using several video processing software like Premiere, HandBreak etc. after transcoding the footage you could get a green line under each frame (in the magnified screenshot above). Any artificial changes to the footage will reduce your registration accuracy in RealityCapture.

As for the chroma sampling the higher you will go the more information you will get (of course 4:4.4: is better than 4:2:0, etc.).

If you will use modes like all-intra (raw frame by frame capture) in other compression methods you may have experience “filler frames” let’s say every 2nd frame of the sequence is just artificially created to shrink down the size of the video. If these artificially made frame will be extracted in RealityCapture this can produce a lot of accuracy problems.

Rolling shutter issues

As the drone is moving while capturing the video to the side, there can be a lot of rolling shutter introduced (vertical lines can suddenly look like squid diagonals). RealityCapture cannot correct this anyhow and actually if you would try to correct this (in other software), it would change the pixel matrix/pixel coordinates among multiple frames (create other inaccuracies in RealityCapture alignment).

Shutter speed shutter angle

Another issue is with shutter speed. Usually, when shooting a video, you are trying to get 2x shutter speed than the frame rate (for 60 fps you can use 120ss). But for photogrammetry when flying with a drone or moving while capturing a video sequence you want to have each frame shard so you need to go with higher shutter speeds like 400 and even more, so everything will be sharp.

Missing EXIF metadata for each frame

Since you will be dealing with the sequence of images, these will have no information about the used camera, lens, resolution. RealityCapture can calculate this automatically, but there may be mistakes.