Present Software of photogrammetric processing of digital images

. Four open source and three commercial software been used to create point clouds from identical data: images taken at five different locations. The investigation compares elapsed time and the size of the generated point clouds for the different software. Based on the tests, suggestions for users are provided.


Introduction
Nowadays photogrammetry and laser scanning (LiDAR) competes in generating point clouds. The present study point clouds generated with photogrammetry is discussed. The aim of the study is to provide an overview of the available commercial and open source software, and also deliver a comparative analysis on their efficiency based on certain aspects. Among the huge number of available software, seven has been selected for the test, which are considered to be most widely used ones.

Data and software
As a first step of a photogrammetric processing sequence, for generating a sparse point cloud, most software applies the Structure from Motion (abbreviated as SfM) method [1]. This algorithm presupposes that the camera is in motion while the object is still. The SfM method has primarily been developed for machine vision, though can efficiently be used for automatization of photogrammetric processing as well. The steps of the SfM method (which is not strictly followed by a software): (1) Automatic search of key points (automatically detectable, uniquely marked points) on each image, with methods such as SIFT, SURF, or ORB, delivering the position of the points (different on each images) and attribute information (independent of size and direction, thus independent on the appearance on the different images) [2].
(2) Relative orientation (optionally absolute orientation as well in case control points are available) providing the relative (or absolute) position of the images to each other and the coordinates of the reference points (a subset of key points, which can successfully be determined by the orientation).
The resulted point set is the so-called sparse point cloud, the outcome of the SfM. A sparse point cloud may visualize appropriately the texture, geometrical content is rough (e.g. edges). Even though the SfM derives a 3D model, it does not generate new corners or edges, but applies the edges of the texture [3]. After the SfM, the dense point cloud is generated. Most of the software use telemetry data or the absolute oriented point cloud to generate derived products, such as orthophoto, Digital Terrain Model (DTM) or Digital Surface Model (DSM) [4]. The source of the images can be hand-held camera or cell phone, UAV-borne camera, but also pictures available on social media (e.g. Flicker, Mapillary) can be used (c.f. Fig 1). Mapillary is a project dedicated to data acquisition from social media, which resembles to Google Street View, though Mapillary contains images of front-facing cameras only. The data is collected volunteers mostly with cell phones mounted to the windscreen of cars. The processing software may also make use of the EXIF (Exchangeable Image file Format) content of JPEG files, such as telemetry data, focal length, etc. Generally, 80% or more overlap of the images is expected. Several software is based on open source base directories (commercial also), such as OpenCV (Computer Vision) and OpenMVG (Multiple View Geometry). The OpenCV is a C++ program library, which includes methods for search of key points, referencing images, and can solve processing up to the generation of the sparse point cloud. The functions of the OpenCV library can be accessed also by using Python. OpenMVG can be used for generating the dense point cloud from the oriented images.
For most of the software, the steps of the processing is quite similar. (Steps in parentheses are sometimes skipped, usually these are required only for absolute orienting / georeferencing).
1. Start a project 2. Add images to the project 3. (Input the coordinates of control points) 4. Automatized search for key points 5. Triangulation: generation of the sparse point cloud 6. (Mark control points on images) 7. Generation of the dense point cloud 8. (Generation of derived products, e.g. orthophoto, mesh model, DTM, DSM) In case of an automatized processing sequence, all data input (images, control points) should be entered as a first step, then the processing is done completely automatically. Fig 2 shows detected key points by one of the tested software. It can be seen clearly that no adequate points were identified on the homogeneous surfaces (see the bottom-left and top-right corners of the figure). The size of the coloured circles displays the accuracy estimated while the marked radius displays the estimated direction of the key point on another image. Each image may contain ten thousands of key points.

Fig. 2.
Identified key points, with the accuracy (size of the circle) and the direction of the point on another image (marked radius of the circle) is displayed. [5] There are expectations on the Hardware capacities for most of the software, such as number of processors (CPU) and nodes; often 6-8 nodes may be demanded. The different steps of the photogrammetric processing sequence mean different workload for the processors as well. The computation time of some steps depends only on the clock speed, while others can make use of the multiple nodes. For example, generating a dense cloud is efficient with multiple nodes, while building a texture or a mesh model needs generally high clock speed but less nodes. Also, the processing power of the graphics card (GPU) may also be involved in the processing, for example the computation time of densification of the point cloud depends notably on the number (and the type) of GPUs. Since this step is the most time consuming, it is important select all the available GPUs to be used for processing. Approximately, a graphics card with an additional GPU can reduce the computational time by 20-25%, which with one more GPU can be further reduced by ~10%. Additional GPUs, however, won't affect the computation time relevantly (1-3% only), so it does not verify the additional costs.
In case of hundreds or thousands images are processed, the memory (RAM) capacity should be maximized (at least 64GB is needed). The memory demand depends on the number of images, and the demanded resolution. As the use of RAM is approximately linear, by a test run for a subset of data, the computation time can properly be estimated. Normally, generating the dense point cloud and the mesh model is the most RAMconsuming. For processing aerial images, in case of some software there are special settings for the nadir images, which enables more efficient processing. In case of oblique images, however, the demand on high memory capacity is unavoidable.
Furthermore, data storage is also a relevant issue. As primary drive, SSD is suggested to be used, which contains both the operational system and the processing software. High speed SSD enables fast loading and handling of the data.
All in all, for professional photogrammetric processing the optimal computer can be defined somewhere between the Gamer PCs and laptops. For smaller tasks and study, middle range PCs can also be sufficient. Finally, the option of online processing should also be mentioned, as cloud computing is more often supported by photogrammetric software.

Used software
For the tests, altogether 7 software has been used [6]. In the following section these are introduced, and their main parameters are summarized in Table 1.

Open source software
Due to the OpenCV and OpenMVG libraries, new photogrammetric processing software can easily and efficiently be developed. As a result, several such projects have been started from 2010, although in many cases further updating and developments are ceased. For the present study, open source software with stable background (with frequent updates, and relevant user and developer community) are considered. Note, however, that open source software may have a limited applicability, thus it is always to check carefully the software's licence before use.

VisualSfM
One of the earliest open source photogrammetric processing software, which applies SfM and MVG algorithms. The development has been started in 2011, and no new version has been published in the last two years. It has a special licence, indicating that it can only be used for private and research use [7].
For determining the key points, it requires ATI/nVidia/Intel GPU with at least 1 GB memory. For the 3D reconstruction nVidia CUDA or the CPU can also be used. For generating the dense point cloud PMVS/CMVS patch is used. No derived products (e.g. orthophoto) can be generated. Both graphical and command-line interface can be used.
Though the program can efficiently process the images, also the user interface can easliy be acquired, drawbacks are that the updates have been ceased, and also that commercial use is not supported.

Regards3D
It has been started to be developed in 2015. Very user friendly, it guides the user with clear instructions through the processing steps in an interactive manner, enabling setting for different algorithms and parameterization in dialog boxes. Within one processing sequence, plenty of variants of processing can be run in parallel. No special requirements for the hardware is defined, however, it cannot process more than some hundred images at once. It uses MIT-licence, which enables wide range of use, including its use within a commercial software as well.
No reference points can be involved into the processing; accordingly, only relative point cloud can be generated. In order to transform to an absolute coordinate system, some other software (e.g. CloudCompare) should be used. The software is frequently used in cultural heritage management for modelling buildings and sculptures [8]. It is a good starting point for beginners to get acquainted with this technology.

COLMAP
The COLMAP software has been started as a PhD research, it's first licence has been published in 2016 as a BSD-license. The main purpose of the software was pure research, development, and test of algorithms. As a result, the program has a very wide range of options for parameterization, which may be a difficulty for unexperienced users. It is rather for research purposes for experimenter users.
Large (more than 10000 images) can also be processed, although the computational time is not optimal. It can used with graphical and command-line user interfaces as well. The present version cannot be used for generating orthophoto. Even though the software is mainly developed by one person, dozens of developers contribute to his work.

OpenDroneMap (ODM) and WebODM
Among the open source software, ODM is prominent with its dynamics in development (started in 2016) and various services. The software is published under the GPL3-license. The ODM can only be used from command-line, which might be a difficulty for a regular user of present days. It is supported by the WebODM, which allows the use of ODM from a web browser with a graphical interface. With the use of the webODM, also cloud computing can be used.
Originally ODM has been developed for processing UAV-borne images. Although a plenty of parameters can be set, for beginner users also a default setting is suggested for each parameter. WebODM is not only a processing tool, it can also display nicely the results of the computations (c.f. Fig 3). Beyond the dense point cloud, the mesh model and the orthophoto, DTM and DSM can also be generated. It is recommended to be run under the Docker container of the Linux operation system. Note that the instalment is rather complex.

3D Survey
The 3D Survey has been started to be developed back in 2011 by a Sloven group of (mainly) surveyors. The software has been first published in 2014, since then its popularity is raising year by year. The company consists of 11 persons, among them 3-5 are purely working on the development. Its success is based on the user-centric and dynamic development policy and adaptive support service. Quite a few setting and parameterization option can be set by the user, which is both an advantage and a disadvantage. Nevertheless, it is an optimal choice for a beginner user.
There are a number of constantly evolving modules, such as more and more refined CAD-functions, point cloud classification, volume calculation, automatic profile-and cross section generation. A unique service of the software is the automatic reference point recognition, for which special, round-shaped markings are provided by the company. The recognition, which is based on machine vision, can notably reduce the manual workload.

Agisoft Metashape
One of the most commonly used photogrammetric processing software, originated from Russia started to be developed in 2006, first published in 2010 under the name Agisoft Photoscan. A huge range of options for parameterization and settings, which also results in an increased number of source of errors. In case a limited capacity hardware is available, it can be used by distributing the task into smaller units, so-called chunks: the processing can be done separately for each chunk, and their outcome can be merged into one. A special feature is that task lists can be defined for a certain processing sequence, yielding an automatization of the processing. Different, sometimes quite special camera calibration data can also be involved in the processing, e.g. shutter compensation. Multispectral images can also be processed by assigning for each channels a colour. This software has one of the best point cloud classification method (Fig 4).

ReCap Photo
A product of Autodesk, which has been appeared as ReCap Photo in 2016, which is a cloud computing service making use of earlier photogrammetric products such as ReMake and 123D. There are nearly no options for parameterization, with limitations on the size of a project: minimum 20, maximum 1000 images can be uploaded. Before uploading, one can choose between to options. In the first option, no map projection system and reference points can be included; this option can be used for object (e.g. building) modelling, appropriate for cultural heritage management. The other option can be used for processing nadir images, where a map projection system can be selected, output data can be defined (point cloud, texture mesh and orthophoto), and also reference points can be defined. The results of the calculations can be stored on an Autodesk cloud storage.

Tests and results
Altogether 5 worksites have been used for testing. For the tests, relatively small worksites were used as the performance of the available hardware was limited. The observations were acquired using 3 different UAVs: DJI Phantom 4, DJI Phantom 4P, UAV Birdie Geo+. The test sites and the information on the measurements is summarized in Table 2. Commercial software are in most cases optimized for Windows (possibly for IOS). Their use is definitely user friendly, and their operation is quite stable. Meanwhile less room for parameterization is enabled. Furthermore, as they work as a "black box", i.e. information on the computational method is hidden, there is very limited possibility for check / verification of the results. By time, commercial software tends to create more and more complex user interface in order to fulfil all demands and deliver a tool for wide range of applications.
According to the tests, Agisoft Metashape was found to be the most efficient providing the largest point density with the shortest computational time. Still, other software also have their speciality, in which they prove to be the best. The support, the customization of the orthophoto, and the volume calculation is outstanding for 3D Survey. ReCap Photo can create the most plastic / statuesque models; its high-level texture is a consequence of the advancement of Autodesk software, in general.
All in all, open source software have relevant potential in development, they provide a proper playground for testing unique ideas and demands. In contrary, they require a more creative and advanced user. Open source software are recommended for learning and research activity. Commercial software are better (at the time of the present investigation), due to their structural and financial background. Users can use these software with very few background in surveying or informatics, although they have very few options to change the settings of parameterization. For commercial use, in order to ensure productivity, the use of these software is still recommended over open source software.