Accurate Calibration Scheme for a Multi-Camera Mobile Mapping System

Abstract

Mobile mapping systems (MMS) are increasingly used for many photogrammetric and computer vision applications, especially encouraged by the fast and accurate geospatial data generation. The accuracy of point position in an MMS is mainly dependent on the quality of calibration, accuracy of sensor synchronization, accuracy of georeferencing and stability of geometric configuration of space intersections. In this study, we focus on multi-camera calibration (interior and relative orientation parameter estimation) and MMS calibration (mounting parameter estimation). The objective of this study was to develop a practical scheme for rigorous and accurate system calibration of a photogrammetric mapping station equipped with a multi-projective camera (MPC) and a global navigation satellite system (GNSS) and inertial measurement unit (IMU) for direct georeferencing. The proposed technique is comprised of two steps. Firstly, interior orientation parameters of each individual camera in an MPC and the relative orientation parameters of each cameras of the MPC with respect to the first camera are estimated. In the second step the offset and misalignment between MPC and GNSS/IMU are estimated. The global accuracy of the proposed method was assessed using independent check points. A correspondence map for a panorama is introduced that provides metric information. Our results highlight that the proposed calibration scheme reaches centimeter-level global accuracy for 3D point positioning. This level of global accuracy demonstrates the feasibility of the proposed technique and has the potential to fit accurate mapping purposes.

1. Introduction

Mobile mapping system (MMS) is a photogrammetric mapping agent that is usually defined as a set of navigation (global navigation satellite system (GNSS) and inertial measurement unit (IMU)) and remote sensors—such as cameras, lidar, and odometer sensors—integrated in a common moving platform [1]. The importance of an MMS has been widely highlighted based on its cost-effectiveness, high data-capturing rate, and acceptable level of accuracy [2]. System calibration is an indispensable part of the process of employing an MMS. Regarding an MMS that is based on multi-cameras and navigational systems, two calibration process could be mentioned: multi-camera calibration and MMS calibration. Multi-camera calibration aims to estimate interior and relative orientations of a multi-camera. MMS calibrations refers to estimating relative position and orientation between a multi-camera and GNSS/IMU sensors. Accurate system calibration ensures high-quality outputs for at-least a minimum period that a mapping system stays relatively still; a periodic system calibration scheme is able to guarantee the correctness of time-dependent parameters.

The first aspect of employing an optical-based MMS relates to the problem of multi-camera calibration [2,3,4,5,6,7,8,9,10,11]. Nowadays, many MMSs are equipped with multi-projective cameras (MPC) because of their sturdy design, large field of view (FOV), and promising sensor models. Sensor modeling of an MPC usually consists of interior orientation parameters (IOP) of individual cameras, relative orientation parameters (ROP) between cameras with respect to a reference camera, and a scale factor that connects ROPs to a global framework. An important aspect of integrating a camera system in an MMS relates to the problem of employing a rigorous sensor model that fits to the physics of the camera. The sensor model maps a 3D object point into its corresponding image point [12]. The next step is to employ the sensor model as the core of a statistical optimization model to find optimum values and express our uncertainties about the unknowns such as camera parameters or 3D position of object points.

Many MMSs use multiple cameras but do not necessarily generate panoramas. A panorama is a continuous presentation of an environment which is demonstrated by one photo or a series of photos that are merged together by a ‘stitching’ or a ‘geometric non-stitching’ approach. This form of photography provides viewers with an unlimited viewing possibility in all directions [12]. Because of the all-angle viewing property, panoramic photography has found a wide range of applications; few of those applications are computer vision, robotics, surveillance, virtual reality, indoor/outdoor photography, and historical heritage documentation.

Recently developed photogrammetric models for MPCs has brought a new perspective in panoramic images applications. If these cameras, which are initially designed for all-direction photography, treated with photogrammetric models, numerous potential applications will emerge for surveying and mapping purposes.

A categorization of panoramic cameras can help us to find shared properties and mathematical model for members of each class. Amiri Parian and Gruen [12] categorized panoramic imaging into four groups: stitched, mirror-based rotating-head, near 180, and scanning 360 panoramas. Stitched panoramas are mainly used for non-metric applications where accurate directions are not important; therefore, this class is outside the focus of a metric categorization. We may re-categorize most of commercially available panoramic cameras that are suitable for surveying tasks according to their internal capturing technology into four groups [13]: (1) rotating-head, (2) multi-fisheye and (3) multi-projective camera, and (4) catadioptric systems. The benefits of the latter categorization are twofold. Most of commercially available cameras that are useful in metric photogrammetry tasks fit into this categorization; moreover, it simplifies the sensor modeling since it has founded based on a similarity measure that places cameras to a category according to their common sensor model.

The category (1) rotating head cameras contains cameras based on linear array CCDs that is mounted on a vertically rotating head. Examples of this class include EYESCAN M3 that is jointly developed by DLR and KST [14] and SpheroCam which is manufactured by SpheronVR AG. Equivalently, a projective camera such as a Canon EOS 6D could be mounted on a motorized rotating head such as Phototechnik AG, Roundshot metric, piXplorer 500, or GigaPan EPIC pro. The camera and step motors are synchronously controlled by a central processing Unit (CPU) that plans the movements and takes image shots. If the system has been accurately calibrated (optical center of the projective camera’s lens should be precisely located on the center of rotation), high-precision metric information and high-resolution panoramic images can be obtained with least effort [15]. Depending on the target applications, one or two step motors are working simultaneously to rotate the central camera around a point/axe. One main configurable parameter in building this class is the option to choose between one or two rotating axes; a post-processing module is responsible to transfer and merge images that have been captured with different orientations. Subsequently, a color blending approach is required to ensure that color tunes between stitched images are consistent. Usually a full resolution shot takes few seconds to even few minutes to compile which makes this class of panoramic cameras relatively unsuitable for mobile-mapping. Some important use-case of this class is cultural heritage recording and classification [16,17], high-precision surveying [12,18,19,20,21], industrial-level visualization, indoor visualization, 3D modeling, and artistic photography.

The category (2) multi-fisheye cameras comprises a structure of a dual spherical camera module mounted on a rigid frame to construct a light-weight mobile 360 imaging system; Few consumer-grade samples of this class are Samsung Gear 360 [22], SVPro VR 360, MoFast 720, MGCOOL cam 360, SP6000 720, Ricoh Theta S [7]. Some recent works suggests multi-camera system based on a dual-fisheye design [23,24]. Fisheye camera models for photogrammetric applications were extensively studied and tested [25,26,27,28,29]. The simultaneous geometric calibration of multi-camera system based on fisheye images, aiming a 360° field of view also started to be discussed recently. For instance, a dual fisheye calibration model is proposed by [30] and [7]. For this class of cameras, a customized statistical optimization process that involves using weighted observations and initial distributions of unknowns has proved sufficiently accurate for low to medium-accuracy surveying applications. However, some limitation of these systems can be mentioned, such as non-perspective inner geometry, huge scale and illumination variations between scenes, large radial distortion and nonuniformity of spatial resolution. Therefore, the overall image quality of panoramic images that are produced by the cameras of this class are usually worse than the first or third categories, however, some desired aspects—such as data-capturing rate or simplicity—makes them ideal candidates for applications such as low accuracy mobile mapping or 3D virtualization [31,32,33,34,35].

The category (3) multi-projective cameras contains panoramic cameras with arbitrary number of projective cameras mounted fix on a rigid frame (an industrial examples is LadyBug with six cameras [v.3 and v.5; $MP{C}_6$] by FLIR; a consumer-grade example is Panono 360 with 36 cameras [$MP{C}_{36}$]). Cameras of this class (multi-projective cameras or in short MPC) are usually customized for certain imaging, navigation, or surveying tasks. A synchronous shutter mechanism is applied to take simultaneous shots (<1 msec delay). A geometric model for MPC integrated into a statistical adjustment model is proposed by many researchers, e.g., ([9,10,11]). This model ensures desirable geometric accuracies for many tasks such as 3D mapping and surveying, 3D visualization, and texturing. Panoramas that are initially generated for this class are based on stitching techniques that mainly have visualization and artistic values; it is in contrast to the geometric values of panoramic images that are taken from cameras of the first or the second categories; this class of cameras has rigorously found many applications such as surveying, robotics, visualization, cinema, and artistic photography. Similar to the second class, it has the potential to be employed in applications that needs fast data-capturing rates such as mobile mapping, or navigation.

Early attempts to employ relative orientation constraints among multiple cameras was applied to a stereo camera, e.g., ([3,36,37]). He et al. [36] developed an MMS with a stereo camera and a GPS receiver to measure global coordinates of any point through photogrammetric intersection. King [3] modified the conventional bundle block adjustment (BBA) to accept relative orientation stability constraints. Zhuang [37] employed a fixed-length moving target to calibrate a stereo camera. Later, more complex systems of cameras went under research investigations. For example, Svoboda et al. [38] calibrated an $MP{C}_4$ that were installed on a visual room. Lerma et al. [39] calibrated a $MP{C}_3$ that consisted of a stereo camera and a thermal camera by employing distance constraints. Habib et al. [40] analyzed variations in IOP/Exterior Orientation Parameter (EOP) of multi-cameras. Detchev et al. [41] presented a system calibration scheme by employing system stability analysis. Some researchers employed a calibration field for multi-camera calibration. For example, Tommaselli et al. [5] employed AURUCO coded targets [42] to design a terrestrial calibration field. They used their proposed photogrammetric field to calibrate fisheye, catadioptric, and multi-cameras. Khoramshahi and Hokovaara [10] employed a customized coded target (CT) to create a calibration room. They employed it to calibrate a complex $MP{C}_{36}$ (Panono), and an $MP{C}_3$ (LadyBug v.3). In this work, we follow the calibration model that was proposed by Khoramshahi and Honkavaara [10].

Category (4) contains catadioptric systems that employ complex optical systems. A set of spherical and aspherical lenses, shaped mirrors such as parabolic, hyperbolic, or elliptical mirror [43], and refractors are employed in catadioptric systems to cover a large FOV. An example of this class was proposed by [23] as a prototype dual catadioptric camera. A calibration model for a catadioptric camera consists of a wide-lens camera and a conic mirror was proposed by [44].

The second aspect of employing an MMS relates to the problem of direct geo-referencing. Using multi cameras of third category for mapping applications require accurate determination of orientation and position of each panorama. This orientation can be done by indirect or direct georeferencing techniques, or combining both. Direct geo-referencing is a very useful approach for real-time application such as mobile mapping, or aerial surveying. Direct geo-referencing is the process to find position and orientation of captured images (EOPs) in a global reference frame without employing any ground control point (GCP), which requires the integration of additional sensors such as GNSS receivers and IMU sensors into the camera’s mounted frame. This integration either provides initial values for positions and orientations of the camera shots as weighted observations in the BBA, or helps the system to instantly estimate position vectors and Euler angles of shots [45]. It is worth noting that a successful direct geo-referencing requires two main aspects. Firstly, GNSS and IMU sensors should be accurately synchronized to an MPC. Secondly, a robust estimation of lever-arm vector and boresight angles should be known. These later parameters are known as mounting parameters [46]. Rough estimation of the MMS calibration parameters could be performed by comparing observations of GNSS and IMU with the output of BBA. To enable a direct geo-referencing, a customized sensor model is required to be employed by considering additional parameters of lever-arm and boresight misalignments of an MPC with respect to GNSS and IMU sensors.

A variety of calibration schemes has been overly discussed in the literature; however, as far as the authors concerned, a rigorous, practical, and easy solution for direct georeferencing of an MPC does not exist that completely fit to the configuration of a multi-camera MMS. Moreover, stitching operation is required to compile a panorama from images of the third category cameras. This panoramic presentation usually has just artistic or visual values. A non-stitching panoramic creation scheme is also feasible for this class that adds geometric value to generated panoramas. This paper contributes to a rigorous, easy, and practical scheme for calibrating a multi-camera MMS equipped with an MPC, GNSS and IMU sensors in a terrestrial vehicle. We also present a novel non-stitching algorithm by introducing a panoramic correspondence map. We propose a modified BBA that comprises MPC calibration and MMS calibration. We assess quantitatively the quality of the proposed approach by checking the accuracy of the point intersection by independent checkpoints. Our results demonstrate that sub-decimeter level accuracy is achievable by this technique.

2. System Calibration

Mathematical theory behind the implementation of this work is discussed in this part. We first start with sensor model of an MPC, then we continue toward discussing a complete sensor model for the underlying MMS. Finally, a modified BBA is discussed to employ the sensor model and statistically express uncertainties about parameters.

2.1. Individual Camera Model

The following interior orientation model removes main distortions from image coordinates and outputs undistorted pinhole image coordinates as

$$\begin{gathered} {\mathrm{x}}_1=\frac{{\left(\mathrm{x}\right)}_{{\mathrm{t}}_1}-{\mathrm{PP}}_{\mathrm{x}}}{\mathrm{f}}, {\mathrm{y}}_1=\frac{{\left(\mathrm{y}\right)}_{{\mathrm{t}}_1}-{\mathrm{PP}}_{\mathrm{y}}}{\mathrm{f}},{\mathrm{r}}^2=\sqrt{{\mathrm{x}}_1^2+{\mathrm{y}}_1^2}, \mathrm{Rad}=\left(1+{\mathrm{K}}_1·{\mathrm{r}}^2+{\mathrm{K}}_2·{\mathrm{r}}^4+{\mathrm{K}}_3·{\mathrm{r}}^6\right), \\ {\left({\mathrm{x}}_{\mathrm{n}}\right)}_{{\mathrm{t}}_1}={\mathrm{x}}_1·\mathrm{Rad}+2·{\mathrm{P}}_1·{\mathrm{x}}_1·{\mathrm{y}}_1+{\mathrm{P}}_2·\left({\mathrm{r}}^2+2{\left({\mathrm{x}}_1\right)}^2\right)-\mathsf{\delta }·{\mathrm{x}}_1+\mathsf{\lambda }·{\mathrm{y}}_1 \\ {\left({\mathrm{y}}_{\mathrm{n}}\right)}_{{\mathrm{t}}_1}={\mathrm{y}}_1·\mathrm{Rad}+2·{\mathrm{P}}_2·{\mathrm{x}}_1·{\mathrm{y}}_1+{\mathrm{P}}_1·\left({\mathrm{r}}^2+2{\left({\mathrm{y}}_1\right)}^2\right)+\mathsf{\lambda }·{\mathrm{x}}_1, \end{gathered}$$

(1)

where ${\left(\mathrm{x}\right)}_{{\mathrm{t}}_1}$ and ${\left(\mathrm{y}\right)}_{{\mathrm{t}}_1}$ are distorted coordinates of a point in pixel unit,${ \mathrm{PP}}_{\mathrm{x}}$ and ${\mathrm{PP}}_{\mathrm{y}}$ are coordinates of the principal point, $K_i$ are radial distortion coefficients, $P_i$ are tangential distortion parameters, and $\delta$ and $\lambda$ are scale and shear factor respectively. It is trivial to show that the underlying model (Equation (1)) is exchangeable to Brown’s model [47,48].

An image of a projective camera is linked to the camera’s sensor model, and relates to six orientation and position parameters that uniquely determines it inside a 3D Euclidean space. Therefore, a linear relationship between an object point and its undistorted pinhole coordinates is established by the collinearity model

$$x_1=-\frac{{\mathit{M}}_{\mathbf{1}}·\left(\mathit{X}-{\mathit{X}}_{\mathbf{0}}\right)}{{\mathit{M}}_{\mathbf{3}}·\left(\mathit{X}-{\mathit{X}}_{\mathbf{0}}\right)}, y_1=-\frac{{\mathit{M}}_{\mathbf{2}}·\left(\mathit{X}-{\mathit{X}}_{\mathbf{0}}\right)}{{\mathit{M}}_{\mathbf{3}}·\left(\mathit{X}-{\mathit{X}}_{\mathbf{0}}\right)},$$

(2)

where ${\mathit{M}}_{\mathit{i}}$ is the i^th row of the rotation matrix ${\mathit{M}}_{3\times 3}$, $\mathit{X}$ is a vector of (3 × 1) of the object point’s coordinates, and ${\mathit{X}}_{\mathbf{0}}$ is a (3 × 1) vector of location of image in a 3D cartesian space (bold letters are used for vectors and matrices and normal letters are used for scalers throughout this paper).

2.2. Multi-Projective Camera Model

An MPC is presented in this work by the following calibration parameters:

• $\mathbf{I}\mathbf{O}{\mathbf{P}}_{\mathrm{i}}$ ($i=1:n$) ($\mathrm{f},{ \mathrm{PP}}_{\mathrm{x}},{ \mathrm{PP}}_{\mathrm{y}},{ \mathrm{K}}_1,{\mathrm{K}}_2,{\mathrm{K}}_3,{\mathrm{P}}_1,{\mathrm{P}}_2,\mathsf{\delta },\mathsf{\lambda }$),
• $\mathbf{R}\mathbf{O}{\mathbf{P}}_{\mathrm{i}}$ ($i=2:n$)$\{\left(\mathsf{\zeta },\mathsf{\eta },\mathsf{\psi }\right)$,(${\mathsf{\Delta }}_{\mathrm{x}},{\mathsf{\Delta }}_{\mathrm{y}} ,{\mathsf{\Delta }}_{\mathrm{z}}$)},
• ${\mathrm{\Lambda }}_{MPC}$,

where $\mathit{I}\mathit{O}{\mathit{P}}_i$ is interior orientation parameters of an individual camera of an MPC, $\mathbf{R}\mathbf{O}{\mathbf{P}}_{\mathrm{i}}$ is relative orientation of an individual camera with respect to the first camera, $\left(\mathsf{\zeta },\mathsf{\eta },\mathsf{\psi }\right)$ are Euler angles, $\left({\mathsf{\Delta }}_{\mathrm{x}},{\mathsf{\Delta }}_{\mathrm{y}} ,{\mathsf{\Delta }}_{\mathrm{z}}\right)$ are displacements, and ${\mathrm{\Lambda }}_{MPC}$ is the unknown scale factor. The last parameter (${\mathrm{\Lambda }}_{MPC}$) plays a role if scale of an MPC sensor would be different from scale of the model. It could be ignored if both scales are equal. In overall, there will be a number of $\mathrm{nCalib}.=n\times 10\left(IOP\right)+\left(n-1\right)\times 6\left(ROP\right)+1\left(scale\right)=16\times \mathrm{n}-5$ calibration parameters for an MPC with (n) projective camera; a multi projective image (MPI) is geometrically defines by its corresponding MPC sensor and six EOPs.

2.3. Space Resection of Multi-Projective Images

The goal of the space resection is to find approximate locations of a given MPI with respect to a 3D frame under the conditions that at-least (3) image-object correspondences exist and localized in a 3D Cartesian framework. Space resection helps to orient a set of MPIs with respect to a given point set (e.g., a calibration room). The orientations are finally adjusted through a BBA.

An alternative way to resect an MPI is to employ the coplanarity equation. Coplanarity condition connects two normalized pinhole coordinates and image centers through a matrix called Essential, which is a (3 × 3) matrix that is composed of three rotations and three translations as

$$\mathbf{E}={\left[{\left({\mathbf{X}}_{\mathbf{0}}\right)}_{\mathrm{t}}\right]}_{\mathrm{x}}·{\mathbf{R}}_{\mathrm{t}},$$

(3)

where ${\left[{\left({\mathbf{X}}_{\mathbf{0}}\right)}_{\mathrm{t}2}\right]}_{\mathrm{x}}$ is the matrix presentation of a cross product which is a $3\times 3$ matrix of the form

$$\left(\begin{array}{ccc}0& -{\lfloor \left({\mathbf{X}}_{\mathbf{0}}\right)}_{\mathrm{t}2}\rfloor {}_3& {\lfloor \left({\mathbf{X}}_{\mathbf{0}}\right)}_{\mathrm{t}2}\rfloor {}_2\\ {\lfloor \left({\mathbf{X}}_{\mathbf{0}}\right)}_{\mathrm{t}2}\rfloor {}_3& 0& -{\lfloor \left({\mathbf{X}}_{\mathbf{0}}\right)}_{\mathrm{t}2}\rfloor {}_1\\ -{\lfloor \left({\mathbf{X}}_{\mathbf{0}}\right)}_{\mathrm{t}2}\rfloor {}_2& {\lfloor \left({\mathbf{X}}_{\mathbf{0}}\right)}_{\mathrm{t}2}\rfloor {}_1& 0\end{array}\right)$$

(4)

Essential matrix is estimated by the following equations

$${\left({\stackrel{=}{\mathbf{x}}}_{{\mathrm{t}}_1}\right)}^{\mathrm{T}}·{\overline{\mathbf{E}}}_{\left({\mathrm{t}}_1,{\mathrm{t}}_2\right)}·\left({\stackrel{=}{\mathbf{x}}}_{{\mathrm{t}}_2}\right)=0,$$

(5)

where ${\overline{\mathbf{x}}}_{{\mathrm{t}}_1}$ is normal image point at time ($t_1$), and ${\stackrel{=}{\mathbf{x}}}_{{\mathrm{t}}_2}$ is the calculated normal location of the same 3D point in a synthesized image. Space resection of an MPI is performed for each of its projective images. At least 5–8 tie points are required based on the selected method (5-point [49], 6-point, 8-point, or normalized 8-point [50]). Then a synthesized pair is formed. The synthesized image is localized such that the resulted stereo pair becomes stable. Next, the retrieved position is translated into approximate position of the parent MPI ($X{0}_i, wp{k}_i$); finally, all positions are averaged to estimate the position and orientation of the MPI

$$X_0={{\displaystyle \sum }}_i{X}_{0i}, wpk={{\displaystyle \sum }}_{i}wp{k}_i.$$

(6)

2.4. Relative Orientation between Two Multi-Projective Images

Finding ROPs between two MPIs is essential to independently construct a scene. ROPs between two MPIs are expressed by six parameters out of which five are independent. Usually the largest element of a stereo pair’s baseline is scaled to one. Three rotations and two displacements are finally solved by relative orientation module. To estimate ROPs between two MPIs, a similar averaging approach to the space resection has been chosen. Therefore, all image pairs between MPIs are oriented. Then the resultant ROPs are transformed to the parent MPI and averaged.

In the process of MMS calibration, initial values of lever-arm and boresight misalignments are estimated by independently compiling a local network of few shots, then connecting the local network to the global frame of GNSS/IMU sensors.

2.5. Bundle Block Adjustment

BBA employs non-linear least-square adjustment method to minimize a cost function based on residuals of observational equations. It finally expresses the uncertainties about the unknowns of an MPC calibration as a covariance matrix. The MPC cost function is a combination of all observational equations. It is expressed as the vector function

$$\mathbf{F}\left({\left(\mathrm{f}, PP, K,P,\mathsf{\delta },\mathsf{\lambda } \right)}_{\mathrm{i}=1:\mathrm{n}},R_{{\left(\mathsf{\zeta },\mathsf{\eta },\mathsf{\psi }\right)}_{\mathrm{i}=2:\mathrm{n}}}, {\Delta }_{i=2:n},R_{{\left(\mathsf{\omega },\mathsf{\varphi },\mathsf{\kappa }\right)}_{\mathrm{t}={\mathrm{t}}_1:{\mathrm{t}}_{\mathrm{m}}}},{\left(X_0\right)}_{\mathrm{t}={\mathrm{t}}_1:{\mathrm{t}}_{\mathrm{m}}}, X_{\left(1:\mathrm{nO}\right)},x_{\left(1:\mathrm{nI}\right)}, \mathit{l}\mathit{a},\mathit{b}\mathit{s}\right)=\mathbf{0},$$

(7)

where, ${\left(\mathrm{f}, PP, K,P,\mathsf{\delta },\mathsf{\lambda } \right)}_{\mathrm{i}=1:\mathrm{n}}$ are IOPs for projective cameras (1:n), $R_{{\left(\mathsf{\zeta },\mathsf{\eta },\mathsf{\psi }\right)}_{\mathrm{i}=2:\mathrm{n}}}, {\Delta }_{i=2:n}$ are structural parameters regarding the $MPC$, $R_{{\left(\mathsf{\omega },\mathsf{\varphi },\mathsf{\kappa }\right)}_{\mathrm{t}={\mathrm{t}}_1:{\mathrm{t}}_{\mathrm{m}}}},{\left(X_0\right)}_{\mathrm{t}={\mathrm{t}}_1:{\mathrm{t}}_{\mathrm{m}}}$ are orientations and locations of image shots, $X_{\left(1:\mathrm{nO}\right)},x_{\left(1:\mathrm{nI}\right)}$ are position of object points and the corresponding image points, $\mathit{l}\mathit{a}$ and $\mathit{b}\mathit{s}$ are lever-arm vector and boresight misalignments respectively. To model the entire system, equations for image observations, GNSS/IMU-observations, and GCPs are introduced.

Two observational equations are considered in the MPC’s BBA for each image tie points, as

$$x_{ij}·{({\mathit{M}}_3)}_{t_{1}j} \left({\mathit{X}}_{\mathit{i}}-{({\mathit{X}}_{\mathbf{0}})}_{t_{1}j} \right)+{({\mathit{M}}_1)}_{t_{1}j}\left({\mathit{X}}_{\mathit{i}}-{({\mathit{X}}_{\mathbf{0}})}_{t_{1}j}\right)=0,$$

(8)

$$y_{ij}·{({\mathit{M}}_3)}_{t_{1}j}\left({\mathit{X}}_i-{({\mathit{X}}_{\mathbf{0}})}_{t_{1}j}\right)+{({\mathit{M}}_2)}_{t_{1}j}\left({\mathit{X}}_i-{({\mathit{X}}_{\mathbf{0}})}_{t_{1}j}\right)=0.$$

(9)

In Equations (8) and (9), $\left(x_{ij},y_{ij}\right)$ are pinhole coordinates of object point ($i$) inside image ($j$) (pinhole camera, f = 1), ${({\mathit{M}}_i)}_{t_{1}j}$ is the ($i^{th}$) row of the rotation matrix ${\mathit{M}}_{t_{1}j}$($j^{th}$ projective camera [$j:1-n$] at time $t_1$), ${({\mathit{X}}_{\mathbf{0}})}_{t_{1}j}$ is location of $j^{th}$ projective camera at time $t_1$, and ${\mathit{X}}_{\mathit{i}}$ is 3D location of the corresponding object point ($i$).

In order to combine observations of GNSS and IMU sensors to the BBA, a link between GNSS, IMU and the MPC is established (Figure 1). A number of (6) additional parameters are introduced to represent a shift (lever-arm), and orientations (boresight misalignments). The connection is formulated by six non-linear equations (Equations (10) and (11)). The observational equation for lever- arm and boresight misalignments are

$${\left(\begin{array}{c}\varphi \\ \lambda \\ \kappa \end{array}\right)}_{(I_t) }-\mathsf{ℝ}\left({\left({\mathit{R}}_C\right)}_I·\left({\mathit{R}}_{C_t}\right)\right)=0,$$

(10)

$$\left(\mathit{X}{\mathbf{0}}_{I_t}\right)-\left({\mathit{X}}_{C_t}\right)+\left({\mathit{R}}_{C_t}\right)·{\left({\mathit{R}}_C\right)}_I^T·{\left({\mathit{X}}_C\right)}_I=0,$$

(11)

where ${\left(\begin{array}{c}\varphi \\ \lambda \\ \kappa \end{array}\right)}_{(I_t) }$ are ‘observed’ Euler angles of the INS at time ($I_t$), ${\left(R_C\right)}_I$ is the rotation matrix of an MPC with respect to the INS local coordinate system, and $R_{C_t}$ is rotation matrix of the MPC at time ($t$). In Equation (11), $\mathit{X}{\mathbf{0}}_{I_t}$ is observed position of INS at time ($t$), ${\mathit{X}}_{C_t}$ is unknown position of the MPC at time ($t$), and ${\left({\mathit{X}}_C\right)}_I$ is relative position of the camera with respect to the INS in the local coordinate system of the INS. Here, $\mathsf{ℝ}$ indicates the function that maps a rotation matrix into its corresponding Euler angles.

GCPs are contributed to the BBA by three observational equations

$${\mathit{X}}_{\mathit{g}}-{\mathit{X}}_{go}=\mathbf{0},$$

(12)

where ${\mathit{X}}_{\mathit{g}}$ is unknown 3D position of a GCP, and ${\mathit{X}}_{go}$ is observed position.

2.6. Panoramic Compilation

Creating a panorama for an MPC without image stitching is possible under the condition that the MPC is calibrated (known IOPs and ROPs). Therefore, calibration data is sufficient to compile a non-stitching panorama. The algorithm for building a panorama contains looping over a hypothetical sphere or cylinder (optional projection system) that covers an MPI, then for each pixel in the final panoramic compilation the algorithm finds a corresponding camera index (0-n) and an image location (pix). For a better rendering result, an appropriate interpolation technique (bilinear or bicubic) could be considered since the projection from the sphere (or cylinder) to projective images results in subpixel positions. If the correspondence data is saved as a meta-data file (it may be called panoramic correspondence map), then the final panorama will have geometric values. It is because of the fact that collinearity condition between an object point, its corresponding image point, and perspective center of a projective camera of an MPC is established. A second usage of the correspondence map is efficiently compiling a panorama, since indexing is expected to be much quicker than direct calculation.

There are regions in the correspondence map that more than one choice of correct image is possible for every point in the final panoramic compilation. It is because of the existence of overlaps between adjacent cameras. A criterion needs to be considered to build a unique map, for example minimum incident angle could be used.

Depending on the distance of a 3D object points to a panorama, discontinuity on edges is expected to occur; it is because the projection center of different individual cameras does not match to the center of the hypothetical sphere. Discontinuity is expected to be more severe for closer objects. Finally, an important post-processing is the use of color blending on edges. It improves the quality of a compiled panorama. Without color blending, sharp steps are expected to be observed on the edges that pixel labels start to switch from one camera to another. Color blending softens those steps and improves the color consistency of the final panorama.

3. Methods and Material

3.1. Mobile Mapping System

The MMS in this works comprises an MPC, a GNSS, and an IMU that are firmly installed on a car’s roof. This section describes the experiments to assess the proposed method described in Section 2. The proposed method was implemented in C++ and MATLAB languages and evaluated using a real dataset from an MMS composed by a multi-camera LadyBug 5, and a navigation system installed on a rigid aluminum truss structure on top of a Skoda Superb (Figure 2).

Ladybug V.5 is an MPC with five side-looking cameras and one upward-looking camera. Each individual camera contains a sensor of size 2464 × 2048 pix. Its focal length is 18 mm with sensor size of (35.8 × 29.7 mm) with field of view of $89.3°$ $\pm {81}^{\prime }$. Panoramas cover an area of 360$°$ horizontally by 145$°$ vertically. Resolution of output panoramas is approximately 8k (7200 × 3600 pixels, aspect ratio 2:1).

The GNSS receiver used was a NovAtel PwrPak7, which together with IMU compiles a Synchronized Position Attitude Navigation (SPAN) system. PwrPak7 logged GPS and Glonass observations via NovAtel VEXXIS GNSS-850 satellite antenna. Sensor trajectory was computed by combining GNSS and IMU data with Waypoint Inertial Explorer software. Virtual GNSS base station reference data was acquired from commercial Trimnet service. The maximum distance from trajectory (Figure 3) to the base station was ~350 meters. The 3D RMSE for the GPS positions was 1.6 centimeters.

A NovAtel IMU-ISA-100C, manufactured by Northrop-Grumman Litef GMBH, was paired with a NovAtel SPAN GNSS receiver. The IMU-ISA-100C is a near navigation grade sensor containing GNSS-850 satellite antenna (in front of IMU) mounted on a truss structure on top of a Skoda passenger car. NovAtel PwrPak7 GNSS receiver together with operating and logging laptop were inside the car.

3.2. MPC Calibration

FGI’s calibration room is an empty space with dimension of 519 × 189 × 356 cm³ that is equipped with 215 CTs (Figure 4). 3D positions of all targets are a-priori known based on previous observations. Images taken with a Canon EOS 6D camera (sensor 20 Megapixel, image size 5472 × 3648 $\mathrm{pixels}$, lens Canon EF 24 mm, focal length $20.6 \mathrm{mm}$) were used to accurately calculate 3D coordinates of the targets [10] in the calibration room with bundle adjustment, before MPC calibration.

In the first step, FGI’s camera-calibration room was employed to calibrate the MPC. To do so, 3D positions of the CTs had been accurately estimated by employing a Canon EOS-6D camera achieving a relative 3D positioning precision of std < 1 mm. The automatic target extraction included:

• Employing adaptive thresholding using first-order statistics to roughly estimate the position of blobs (accuracy of better than 2pixel);
• Accurately fitting rotated ellipses to extracted blobs by least-square fitting (accuracy $\le$0.1 pixel),
• Clustering blobs;
• Finding CTs from extracted clusters by employing structural signature of the CTs, and
• Reading ID for each target.

The automatic CT detection was implemented in MATLAB. The standard deviation for image-point observations was set as 0.1 pixel.

The estimated focal length of the Canon was 3156.10 pixels with a standard deviation of 0.04 pixels. The principal point was estimated as (2752.37, 1509.36) pixels with standard deviation of 0.05 pixels. Regarding each of the IOPs, a significance analysis is performed assessing the amount of distortion that is corrected by freezing all the IOPs except for an underlying parameter (or set of parameters) on a grid of 10 × 10 pixels. The largest distortion on the grid was chosen as significance value for a given IOP. The significance values for radial distortions (set), tangential distortion (P1, P2, and P3), scale, and shear were 69.2, 2.4, 1.0, 0.03, and 0.06 pixel respectively. The estimated standard deviations of the CT points were 0.15, 0.03, and 0.02 mm in principal directions of 3D error ellipsoids respectively, which indicates that our target accuracy was achieved.

In the second step, the same CTs were automatically extracted in individual images of the MPC (LadyBug 5). Next, each MPI was resected (Section 2.3) using the calibration-room’s CTs, and finally, the BBA was performed to optimize the sensor parameters of the MPC. In this processing, the CTs coordinates based on the measurement by the Canon were kept fixed.

Two datasets of 26 and 53 MPIs were captured from the FGI’s calibration room to calibrate the MPC. In the first dataset, the camera is attached to a horizontal slide, therefore each few images are coplanar and at the same height level. In the second dataset, the camera is put on a tripod with two different height levels. In total, 474 individual projective images have been captured; then, CTs were automatically extracted in all individual images.

3.3. MMS Calibration

The MMS calibration was carried out using outdoor dataset from Inkoo, which is a municipality in southern Finland. A set of 8424 MPIs were captured by the LadyBug 5 camera for Inkoo harbour area and two close-by regions. Most roads have been captured by the MMS in both directions.

The GCPs were measured using the Topcon Hiper HR RTK dual band receiver with an accuracy within 5 mm + 0.5 ppm horizontally and 10 mm + 0.8 ppm vertically [1. source:TOPCON]. Nearest base station was located in Metsähovi 28 km away from Inkoo, therefore accuracy is vertically 19.0 mm and horizontally 32.4 mm, respectively.

The MMS calibration involved estimating relative parameters of the MPC with respect to the GNSS and IMU (lever-arm and boresight miss-alignments). Observational Equations (10) and (11) were added to BBA. Analytical partial derivatives of latter equations were considered for calculating the Jacobian matrix. The calibrated MPC sensor was used to construct an outdoor case. The sensor was kept fixed during the scene reconstruction except for its scale parameter. A number of six MPIs were connected together by the relative orientation module (Section 2.4) to construct a local network (initial network). Model coordinates of all 3D points were accurately determined with corresponding sub-pixel re-projection error and added as observational equations. Few GCPs were observed in individual images and added to the model to establish a link to the global coordinate system (Equation (6)). Then, GNSS and IMU observations were added to the initial network. Lever-arm and boresight was initialized from the initial network and finally optimized through the BBA. The MPC sensor kept fixed during MMS calibration to avoid a singularity in the Jacobian matrix. In the optimization process, locations, and orientations of MPIs, 3D positions of GCPs, and six parameters of lever-arm and boresight were set free.

3.4. Performance Assessment

In order to assess the accuracy of the MMS calibration and georeferencing, seven check sites approximately covering the area were selected (Figure 3). In each site, the GNSS and IMU readings of the MMS were translated into MPI orientations that were converted into individual image’s orientation and rotation by using MPC sensor calibration information from Step 1 (Section 3.2). Then image positions of check points were observed in individual images; consequently, two sets of coordinates were available for each cross-check site: 1—intersecting coordinates from image measurements and collinearity equations; and 2—observed positions of check-points from GPS. The difference was recorded along with the maximum intersecting angle of each 3D check point as a quality-control indicator. It was expected that weaker interesting geometry would result higher object-space residuals.

4. Results

4.1. MPC Calibration

Indoor calibration of the MPC resulted in sub-pixel individual image accuracies (mean image residual 0.4 pixel, std. 0.45 pixel). Moreover, internal structure of the final optimized MPC resembled very well to the physical reality of the underlying MPC. Different height levels of MPIs resembled well to the reality of the capturing configuration and played a role as a visual cross-checking. Undistorted individual image, as depicted in Figure 5 had very large radial distortions. The MPC’s significance matrix also showed a considerable value of 2183 pixel for radial distortion for camera number (1) that was caused by the lens structure. Significance values for tangential distortions were relatively small (<10 pixel). The calibrated IO parameters of individual cameras with their corresponding standard deviation are given in

Table 1. Calibrated interior orientation of individual cameras of the LadyBug camera.

Cam id.	1	2	3	4	5	6
F (px)	1247.63	1243.07	1242.20	1240.03	1245.66	1241.88
std.	0.04	0.07	0.05	0.04	0.03	0.07
$\mathit{P}{\mathit{P}}_{\mathit{x}}$ (px)	1217.37	1204.01	1219.65	1218.62	1230.57	1229.84
std.	0.01	0.08	0.07	0.04	0.04	0.10
$\mathit{P}{\mathit{P}}_{\mathit{y}}$ (px)	1036.31	1012.60	1024.03	1028.36	1036.77	1021.02
std.	0.01	0.11	0.07	0.05	0.05	0.10
${\mathit{K}}_{\mathbf{1}}$	0.383946	0.371983	0.388319	0.380270	0.384972	0.388471
std.	2.00 × 10−4	4.00 × 10−4	3.00 × 10−4	2.00 × 10−4	2.00 × 10−4	3.00 × 10−4
${\mathit{K}}_{\mathbf{2}}$	0.017540	0.035774	0.003023	0.023847	0.013512	0.003508
std.	5.00 × 10−4	8.00 × 10−4	6.00 × 10−4	4.00 × 10−4	4.00 × 10−4	6.00 × 10−4
${\mathit{K}}_{\mathbf{3}}$	0.177268	0.166916	0.191546	0.175797	0.180382	0.187534
std.	3.00 × 10−4	6.00 × 10−4	4.00 × 10−4	3.00 × 10−4	2.00 × 10−4	4.00 × 10−4
${\mathit{P}}_{\mathbf{1}}$	−4.8 × 10−4	−6.0 × 10−4	−6.3 × 10−4	1.9 × 10−4	7.8 × 10−4	−5.8 × 10−4
std.	1.0 × 10−5	1.0 × 10−5	1.0 × 10−5	1.0 × 10−5	1.0 × 10−5	2.0 × 10−5
${\mathit{P}}_{\mathbf{2}}$	−1.9 × 10−4	−9.0 × 10−4	6.2 × 10−4	5.5 × 10−4	2.6 × 10−4	1.6 × 10−4
std.	1.1 × 10−5	1.9 × 10−5	1.4 × 10−5	9.0 × 10−6	1.0 × 10−5	1.8 × 10−5
$\mathit{\delta }$	−3.4 × 10−4	−1.7 × 10−4	−5.2 × 10−4	−2.7 × 10−4	−1.7 × 10−4	−7.8 × 10−4
std.	1.0 × 10−5	3.0 × 10−5	2.0 × 10−5	1.0 × 10−5	1.0 × 10−5	3.0 × 10−5
$\mathit{\lambda }$	−3.0 × 10−5	3.3 × 10−5	7.5 × 10−5	4.7 × 10−5	−6.0 × 10−5	8.6 × 10−5
std.	7.0 × 10−6	1.0 × 10−6	9.0 × 10−6	5.0 × 10−6	5.0 × 10−6	1.0 × 10−6

. Most of the standard deviation values of camera 6 were higher than average. One reason could be the weaker distribution of CTs on the ceiling in comparison to the walls and the floors distributions. Scale and shear factors appeared meaningful in the BBA since their optimized values were relatively bigger than the standard deviation values.

Table 2. Calibrated structure of the LadyBug camera (orientations in degree and positions in (cm)).

Id.	$\mathsf{\zeta }°$	$\mathsf{\eta }°$	$\mathsf{\psi }°$	$\mathit{s}\mathit{t}\mathit{d}. \mathsf{\zeta }°$	$\mathit{s}\mathit{t}\mathit{d}. \mathsf{\eta }°$	$\mathit{s}\mathit{t}\mathit{d}. \mathsf{\psi }°$
1	0	0	0	0.0005	0.0005	0.0005
2	71.96	359.70	359.81	0.005	0.001	0.004
3	143.92	359.66	0.13	0.003	0.003	0.000006
4	216.04	359.80	359.99	0.002	0.001	0.001
5	288.00	359.97	359.97	0.002	0.0005	0.002
6	29.63	89.80	150.45	0.63	0.003	0.63
	${\Delta }_{\mathbf{x}}$	${\Delta }_{\mathbf{y}}$	${\Delta }_{\mathbf{z}}$	$\mathit{s}\mathit{t}\mathit{d}\mathbf{.} {\Delta }_{\mathbf{x}}$	$\mathit{s}\mathit{t}\mathit{d}\mathbf{.} {\Delta }_{\mathbf{y}}$	$\mathit{s}\mathit{t}\mathit{d}\mathbf{.} {\Delta }_{\mathbf{z}}$
1	0	0	0	0.002	0.002	0.002
2	0.0388	5.7022	4.1770	0.0022	0.0036	0.003
3	0.0156	3.5351	10.922	0.0021	0.0024	0.003
4	−0.0239	−3.5487	10.883	0.0016	0.0021	0.002
5	−0.0356	−5.7454	4.1450	0.0017	0.0025	0.002
6	−7.5553	−0.0293	5.9617	0.0041	0.0031	0.003

demonstrates the calibrated structure of the LadyBug camera. The order of standard deviation values demonstrated an acceptable level of precision in the calibration process. In this table, the values related to the first camera demonstrates a priori distribution of the camera 1’s structural parameters. In this table, standard deviation values that are bigger than average are highlighted with a gray color.

Figure 6 demonstrates absolute values of normalized cross-correlations between all free parameters (IOPs and ROPs (structure)) of the camera 2 (indoor calibration). In this figure off-diagonal elements are of high importance and need to be carefully treated in the adjustment to avoid singularity. Main off-diagonal elements are marked by red rectangles. In this figure, radial distortions are the first group of highly correlated parameters. This high correlation implies that the collective effect of the group is important, therefore investigating an individual parameter in this group is irrelevant. The second highly correlated group is ROPs {$\mathsf{\zeta }$ and $\mathsf{\eta }$}. This group has a strong correlation to $P{P}_x$. The last high correlation relates to ROP {$\mathsf{\psi }$} that is highly correlated to $P{P}_y$. Figure 7 demonstrates absolute values of normalized cross-correlations between all free parameters of the MPC. This figure demonstrates a slightly repeated pattern among cameras. Two diagonal parts of this figure are zero that are related to the structure of the first camera and lever-arm vector and boresight misalignment that kept fixed during the indoor calibration.

4.2. Mobile Mapping System Calibration

Table 3. Calibrated values for lever-arm vector and boresight misalignments.

$\mathrm{Boresight} (°)$	$\mathit{B}{\mathit{S}}_{\mathit{\omega }}$	$\mathit{B}{\mathit{S}}_{\mathit{\varphi }}$	$\mathit{B}{\mathit{S}}_{\mathit{\kappa }}$	$\mathit{s}\mathit{t}\mathit{d}\mathbf{.} \mathit{B}{\mathit{S}}_{\mathit{\omega }}$	$\mathit{s}\mathit{t}\mathit{d}\mathbf{.} \mathit{B}{\mathit{S}}_{\mathit{\varphi }}$	$\mathit{s}\mathit{t}\mathit{d}\mathbf{.} \mathit{B}{\mathit{S}}_{\mathit{\kappa }}$
	−105.80	−41.41	69.95	0.53	0.11	0.45
Lever−arm (m)	$L{A}_X$	$L{A}_Y$	$L{A}_Z$	$std. L{A}_X$	$std. L{A}_Y$	$std. L{A}_Z$
	−2.4189	−0.2824	0.7361	0.02	0.02	0.02

shows the calibrated values for the lever-arm vector and boresight misalignments and their standard deviation values. As expected, standard deviation values were larger than structural parameters (ROPs of the MPC). On average, 0.5-degree standard deviation were obtained for boresight misalignments, and 2 cm for each component of lever-arm vector.

In the adjustment processing, on average 9 cm residuals were obtained for the GCPs (

Table 4. Residuals for the ground control points of the outdoor calibration dataset.

Point id.					Num.
	X	Y	Z	Length	Intersect.
137	−1.6	9.1	2.2	9.5	6
138	6.2	5.0	3.1	8.6	6
139	12.6	−10.5	−2.4	16.5	6
140	−14.2	1.4	−0.6	14.3	6
141	−13.5	4.5	1.4	14.3	6
142	4.1	−7.5	0.1	8.5	6
143	3.0	−2.0	1.4	3.8	6
Mean	−0.48	0	0.74	10.78
RMSE	9.31	6.55	1.71	4.10

) on the outdoor calibration site (Figure 8). All GCPs had at-least 6 intersecting rays.

Table 5. Differences between adjusted values from BBA and direct-georeferencing for the outdoor calibration dataset.

Camera id.	$\mathbf{\Delta }\left(\mathit{\omega }\right)°$	$\mathbf{\Delta }\left(\mathit{\varphi }\right)°$	$\mathbf{\Delta }\left(\mathit{\kappa }\right)°$	$\mathbf{\Delta }{\left({\mathit{X}}_0\right)}_{\mathit{c}\mathit{m}}$	$\mathbf{\Delta }{\left({\mathit{Y}}_0\right)}_{\mathit{c}\mathit{m}}$	$\mathbf{\Delta }{\left({\mathit{Z}}_0\right)}_{\mathit{c}\mathit{m}}$
1	0.38	0.13	0.07	14.82	−4.11	−4.24
2	0.37	0.38	0.40	7.14	−1.58	−3.56
3	0.02	0.08	0.04	2.85	−0.61	−1.97
4	−0.42	−0.22	−0.15	−1.00	−1.48	−0.04
5	−0.17	−0.31	−0.32	−5.93	1.76	0.17
6	−0.19	−0.06	−0.03	−13.16	2.70	−0.09
Mean	0.26	0.20	0.17	7.48	2.04	1.68
RMSE	0.29	0.23	0.22	8.98	2.25	1.77

shows the differences between the image positions and orientations estimated by the BBA and observed by the GNSS/IMU. On average, approximately 0.2$°$ difference was observed between GNSS/IMU angles and BBA outputs. For the directly measured positions, an average difference of 11.3, 2.8, 0.6 cm in the X, Y, and Z coordinates, respectively, were observed.

4.3. Accuracy Assessment

Figure 9 and Figure 10 show the configuration of cross-check dataset (1) and (2) respectively and

Table 6. Differences between observed and intersected position of check points in check1 dataset.

Point ID	(cm)				Num.
	X	Y	Z	Length	Intersect.
187	−4.526	−2.3048	0.4974	5.1034	6
188	−4.4479	−2.2805	0.5311	5.0265	5
189	−4.599	−2.3142	0.4967	5.1723	3
190	−4.5909	−2.3126	0.5021	5.165	3
191	−4.2558	−2.3015	0.5225	4.8664	6
193	−4.9979	−0.8523	0.3865	5.0848	14
183	−6.3937	−0.5021	0.4623	6.4301	3
184	−6.3916	−0.4969	0.4592	6.4273	3
185	−6.4107	−0.4484	0.4817	6.4444	3
186	−6.4102	−0.4492	0.4771	6.4436	3
192	−4.5299	−2.2095	0.4077	5.0565	4
184	−6.3916	−0.4969	0.4592	6.4273	3
Mean	−5.32	−1.44	0.47	5.63
RMSE	0.91	0.87	0.04	0.67

and

Table 7. Differences between observed and intersected position of check points in check2 dataset.

Point ID	(cm)				Num.
	X	Y	Z	Length	Intersect.
164	0.9102	0.7129	−0.5299	1.2718	12
165	1.0354	0.6624	−0.5199	1.3346	12
166	1.1804	0.6085	−0.5237	1.4275	12
167	1.3231	0.5529	−0.5126	1.5229	12
168	1.1808	0.7859	−0.4918	1.5013	10
169	1.2843	0.879	−0.4498	1.62	9
170	0.9248	0.1996	−0.4727	1.0576	10
171	1.0858	0.3109	−0.4796	1.227	9
172	1.2463	0.4695	−0.5275	1.4325	11
173	1.4003	0.4883	−0.52	1.5715	11
Mean	1.15	0.56	−0.50	5.63
RMSE	0.02	0.03	0.0007	0.02

shows the errors for check-points, which are differences between their intersected and observed values. On average, error of 5.6 cm and 1.4 cm was observed between intersected and observed positions of check points for cross-check dataset (1) and (2), respectively. The RMSEs were 0.69 cm and 0.18 cm, respectively. Relatively comparable GPS errors for close-by observations resulted to a similarity between errors in 3D check points of a dataset.

Figure 11 plots sorted incident angles (degrees) with their corresponding errors. This plot high-lights the effect of intersection-geometry’s strength on the quality of positioning. In this plot, intersections of rays with maximum incident angle less than 20 degrees resulted in higher fluctuation in 3D-position inaccuracy. The inaccuracy in 3D positions was significantly improved when the incident angles took values over 20 degree. A logical conclusion from this figure is that a stronger intersection geometry is obtained as incident angles come closer to $\frac{\pi }{2}$.

Figure 12a shows errors for 3D check point. Figure 12b shows the corresponding intersecting angles. In Figure 12a, most errors were less than 7 cm. By comparing these two figures, a negative correlation between incident angles and accuracy was obvious which was also concluded from Figure 11. On average, 4.2 cm error was observed for all 3D check points with RMSE 3.6 cm.

4.4. Panoramic Compilation

Panoramic images were compiled using the IOPs and ROPs. Figure 13 demonstrates footprints of individual cameras on the final panoramic compilation as a contribution map of individual cameras. This map demonstrates that each individual image covers a suitable amount of area in the final panoramic compilation. It consequently confirms that the underlying MPC is well-designed.

The ‘minimum incident angle’ criterion is employed to create a unique correspondence map (Figure 14) from the contribution map (Figure 13). The correspondence map connects every pixel in the final panoramic compilation to a pixel in a camera. Therefore, a vector of 3 numbers (image id, x and y in pixel) were saved for every pixel of the final panorama as the correspondence map. Approximately 7 minutes of computational time was required to build a correspondence map of 7200 × 3600 pixels; then it was efficiently quick and straight-forward to compile any new panorama from individual images (~2.1 sec.). Non-stitching panoramic compilation was performed by employing the correspondence map and resulted into geometrically-accurate panoramas (Figure 15). Closer objects to the MPC had, as expected, larger discontinuity over the edges than far objects. An edge in this context means the boundary that pixel labels change. The discontinuity on edges has been considered by [51] as systematic errors; however, by using the correspondence map this systematic error is totally irrelevant and is completely eliminated and avoided.

5. Discussion

In this work, we followed a similar technique as [11] in employing free-network calibration inside an indoor calibration room; however, we pushed this work in three directions. Firstly, our modified BBA model contains offset and misalignment with respect to the GNSS/IMU. Secondly, a non-stitching panorama was introduced along with a correspondence map which adds geometric values to the non-stitching panorama scheme. The new scheme improves the development of [51] by taking the systematic error of edges into equations. Thirdly, we demonstrated error propagations through a statistical model as standard deviation values of the parameters under-investigation.

A complete calibration scheme for a multi-camera mobile mapping system (MMS) was presented to calibrate a multi projective camera (MPC) to GNSS and IMU. It was based on two steps: indoor MPC calibration, and outdoor MMS calibration, which calibrated an MPC with respect to the GNSS and IMU. Most of recent MPC calibration schemes are based on reducing the number of interior orientation parameters (IOP) in order to avoid singularity. In this work, the singular situation was addressed by employing a photogrammetric calibration room. Our modified bundle block adjustment model considers offset and misalignment of an MPC with respect to the GNSS/IMU, which enables 3D reconstruction using the MMS either in a direct georeferencing mode, or via integrated bundle block adjustment. Furthermore, a non-stitching panoramic compilation scheme was introduced along with a correspondence map which connects pixels of a compiled panorama to their correct position in the MPC. The new scheme takes the systematic error of discontinuous edges into equations. We finally demonstrated error propagations through a statistical model as standard deviation values of the parameters under-investigation. The statistical modeling was proved as a helpful tool to propagate the uncertainties from observations to unknowns based on a non-linear least-square model.

The proposed calibration scheme was successfully demonstrated through five crosscheck datasets. On average 4.2 cm 3D error residuals were observed for cross-check object points. This level of accuracy highlights the usability of a multi-camera MMS in many surveying tasks. Standard deviation for lever-arm vector and boresight misalignments were ~0.5 degree and ~2 cm, respectively. One reason for relatively high uncertainties for lever-arm angles and boresight misalignments in comparison to the lower uncertainties that resulted from the indoor calibration was related to the low number of observational equations in the MMS adjustment process. Those uncertainties are expected to be improved by employing more observations. Our results showed that the intersection geometry was an important factor on 3D positioning. Our rigorous approach forms the basis for developing automated processing of MMS datasets in practical applications. Our further research topics will include the assessment of image-based point clouds in reconstructing different scenes, such as urban scenes and road environments. Furthermore, autonomous object mapping based on the proposed technique is one of our future research objectives.