Abstract
Different indoor representation methods have been studied for their ability to provide indoor location-based services (LBS). Among them, omnidirectional imaging is one of the most typical and simple methods for representing an indoor space. However, a georeferenced omnidirectional image cannot be used for simple attribute searches, spatial queries, and spatial awareness analyses. To perform these functions, topological data are needed to define the features of and spatial relationships among spatial objects including indoor spaces as well as facilities like CCTV cameras considered in patrol service applications. Therefore, this study proposes an indoor space application data model for an indoor patrol service that can implement functions suited to linking indoor space data and service objects. In order to do this, the study presents a method for linking data between omnidirectional images representing indoor spaces and topological data on indoor spaces based on the concept of IndoorGML. Also, we conduct an experimental implementation of the integrated 3D indoor navigation model for patrol service using GIS data. Based on the results, we evaluate the benefits of using such a 3D data fusion method that integrates omnidirectional images with vector-based topological data models based on IndoorGML for providing indoor LBS in built environments.
1. Introduction
As time spent indoors increases, interest in different types of indoor-location-based services (indoor LBS) is also increasing. However, most indoor LBS, such as indoor navigation, are still at the level of simple viewer or attribute searches. There is demand for indoor LBS utilizing indoor spatial information that provide not only simple services but also indoor facilities management, indoor simulation, indoor monitoring, and so on. To provide services in large buildings with complex indoor structures, it is necessary to provide services based on spatial recognition and analyses of indoor spaces, rather than the simple services currently available. The fields that utilize indoor spatial information should also be considered in providing indoor LBS. The types of service and indoor spatial representation may vary depending on the field and scope of service utilization.
According to a questionnaire survey of indoor space specialists conducted by the IndoorGML Standard Working Group (SWG) in Open Geospatial Consortium (OGC) in 2015, the utilization of indoor spatial information would be most effective in disaster responses, indoor facility management, firefighting facilities, and equipment management [1]. This suggests that safety and facilities management should be considered first when utilizing indoor spatial information. Based on this, this study intends to focus on indoor patrol services for ensuring indoor safety. Such services are necessary not only for disaster prevention but also for crime reduction.
Conventional outdoor patrolling is a process of directly searching a targeted area to prevent crimes, disasters, and safety problems in advance, while indoor space patrol applications in GIS deal with not only this process but also real-time surveillance. This leads to the conclusion that indoor space patrol services conduct patrol and surveillance activities at the same time. A patrol in an indoor space is intended to record and update information, investigate a space along a certain route, and manage facilities in that space. To provide this service, it is necessary to recognize a patrol object and analyze a space that is a patrol object. In particular, in a case where not only a space but also facilities such as closed-circuit television (CCTV) cameras and fire extinguishers should be considered as service objects like patrol services, it is necessary to identify the spatial relationships between spaces as well as the relationships between objects in indoor spaces, such as spaces and facilities that are service objects. In order for users to receive various services, it should be possible to carry out integrated queries and analyses of users and service objects as well as spaces. So, indoor space modeling data should be linked so that not only the spatial relationships among spatial entities but also the relationships between indoor space and service objects can be considered simultaneously.
In order to resolve the requirements described in the above, this study proposes an indoor space application data model for an indoor patrol service that can implement functions suited to linking indoor space data and services. The study derives patrol service functions and establishes the relationships between indoor spatial information elements needed for the related functions. This study also presents a method for linking data between omnidirectional images representing indoor spaces and topological data on indoor spaces based on IndoorGML [2]. Finally, we conduct an experimental implementation of an integrated 3D indoor navigation model for a patrol service using GIS data on a building on the campus of the University of Seoul, South Korea. Based on the results, we evaluate the benefits of using such a 3D data fusion method that integrates omnidirectional images with vector-based topological data models based on IndoorGML for providing indoor LBS in built environments.
2. Previous Studies
Patrol services in indoor spaces have used one of two methods: human direct action methods or human indirect action methods. The human indirect action methods use an unmanned aerial vehicle (UAV) or robot to patrol or monitor indoor spaces that human beings cannot directly access, such as radiation-damaged areas [3]. In case where a robot or UAV is used, a patrol is carried out after determining the indoor position and computing the patrol route. Human direct patrol and surveillance using CCTV are the most widely used methods for anticrime efforts in indoor spaces such as shopping malls and public agencies. In recent years, surveillance in indoor spaces has moved beyond simply reviewing CCTV images to enhance its functioning. For more intelligent surveillance, systems have been developed to make 3D models of CCTV installation spaces [4] using Building Information Modeling (BIM) [5] and so on and to represent the location of CCTV cameras in a space so that a response can be made immediately in case of an emergency.
Indoor space models used in the implemented patrol service applications have been developed largely based on 2D representations and 3D representations of indoor spaces in built environments. The most typical methods of 2D representation include a 2D floor plan and a digital CAD drawing. 2D floor plans and digital CAD drawings represent an indoor space as a 2D object [6]. In this case, an indoor space is represented as a simple geometric object—a polygon. Although a digital CAD drawing can represent an indoor space as a solid 3D object, it cannot include topological information or various attributes, so the space is generally represented as a 2D object [6]. Although many systems still use 2D modeling data, 3D modeling data can be used to understand spatial characteristics and arrangements that cannot be identified by a 2D representation method alone; this allows for a greater variety of analyses [7]. Representing an indoor space using a 3D approach usually involves portraying it as geometric primitives or images [8]. Various studies have attempted to develop methods for representing an indoor space as geometric primitives. 3D geometric modeling uses a more detailed picture of a geometric object than the 2D representation method and enhances the sense of reality by adding texturing to the object. Technologies for indoor high-precision laser surveying or 3D vector processing and terrestrial LiDAR are used to build the higher level of data (LoD). However, they are utilized in cases where service is only possible if an indoor space is represented precisely, like military simulations and games, because of problems such as higher construction costs and larger data volumes.
Meanwhile, omnidirectional imaging is used as one of the most typical and simple methods for representing an indoor space. An omnidirectional image is created by matching and postprocessing various image data [9]. The georeferencing of a created image makes it possible to determine the location of an object in an indoor space, so a simple attribute search or query can be carried out. As this method is more realistic than a 2D representation method, it is suitable for providing services such as virtual reality, which should portray an indoor space in the most realistic way possible. Google’s Art Project [10] and Daum Store View [11] are typical examples. The Art Project is a service that makes it possible to view works in a museum, obtain information about them, and look around inside the museum. It involves creating 3D indoor spatial information using a 3D scanner. As it has both images and geometric information, services are implemented in a geotagging style so that a work can be recognized through its location. Although this method has the advantage of obtaining realistic information through images, 3D scanners such as LiDAR have a serious limitation—their high construction cost. Store View is a service that makes it possible to look inside a store, and it provides simple information about the store. It has an advantage in that its construction cost is lower because it is constructed using omnidirectional images. However, in this case, it is impossible to recognize an indoor space like in Art Project, and it is only possible to provide a viewer-level service to search for simple attribute information.
The use of 3D geometric modeling improves the understanding of an indoor space and enables the acquisition of detailed information. However, it is difficult to provide services to Web applications or smartphones because of the large data size. So, this study intends to provide services using omnidirectional images with low data construction costs and small data that represent an indoor space more realistically than a 3D modeling method. Although the georeferencing of an omnidirectional image can be used to carry out a simple property search and query, it cannot be used to analyze a space or perform various query functions like an indoor representation method utilizing 3D geometric modeling. To perform this function, topological data (showing the topological relationship) are required to define a space in the image [12]. If the relationships between indoor spaces are identified, it is possible to conduct a neighborhood analysis and derive the shortest route through an indoor space using simple-level services such as an attribute search or viewer [13]. The spatial relationships between spaces, which refers to the adjacency and connectivity between objects, are also referred to as topological relationships. Data portraying topological relationships are created using three methods: topological primitives, a matrix-based approach, and a graph-based approach [14]. In particular, the graph-based approach is the most efficient for exploring topological relationships [15].
The most typical graph-based approach is node-relation structure (NRS) [15]. NRS is based on Poincaré duality, which converts 3D objects into zero-dimensional ones, 2D objects into one-dimensional ones, one-dimensional objects into 2D ones, and 2D objects into 3D ones. So, the basic concept of NRS is representing a 3D unit space expressed as one room as a zero-dimensional vertex and the adjacency and connectivity between rooms as a one-dimensional edge representing the connection between vertices. Based on this approach, IndoorGML, which is an indoor spatial information data model, was established through the OGC, which is the international standards organization for spatial information, in December 2014 [2]. IndoorGML represents an indoor space element such as a room as a cell, the smallest unit, and defines an indoor space as a cellular space [2]. The unit space can have semantic information representing the classification and meaning of the space, geometric information representing spaces, and topological information such as the adjacency and connectivity between spaces. Geometric and topological models of a 3D space are defined through a framework, which is a structure space model. A 3D object in Euclidean space and a 3D tropology in topology space are represented with a graph using NRS [2].
Topological data created by IndoorGML and omnidirectional images provide realistic visibility and enable various analyses of an indoor space without 3D modeling. For example, it is difficult to create a patrol route using only georeferenced omnidirectional images. To realize the visualization and recognition of indoor spaces and objects needed for patrol service and use an omnidirectional image representing a space and an object efficiently at the same time, it is necessary to devise a method to link IndoorGML, which defines the indoor topological relationships, and the omnidirectional image. Also, there are cases where services can be provided only with information about CCTV cameras (such as patrol services) and facilities (such as fire hydrants). So, to apply the service in an indoor space, the representation of indoor facilities is also important. In general, the easiest way to represent service objects is to use a point of interest (POI) [16]. A POI can also represent service objects indoors but generally only represents indoor facilities. A POI is not simply a point object that only represents a location but is used for linking with various data. A POI has a great feature where it is classified and created according to the services desired by people and the functions of those services. If the service objects not provided by IndoorGML can be represented through the linking of POI and topological data such as IndoorGML, they can be applied to various services other than the patrol service in an indoor space. Besides, it is even possible to provide services for different types of spatial query other than omnidirectional-image-based services that provide viewer-level services only.
3. Development of an Indoor Space Patrol Service Application Data Model
This section presents an indoor space application data model for patrol services that integrates an omnidirectional image, a POI, and IndoorGML. Various data (either 2D, 3D, or topology) are utilized to provide a patrol service in an indoor space. We focus on image data like omnidirectional images because of their low cost and the ease of updating and maintaining them. In order to provide indoor LBS like patrol services, omnidirectional images are used instead of 3D data modeling methods through the proposed data model.
The first part of Section 3 discusses the basic concept of linking the topological data model based on IndoorGML, omnidirectional images, and a POI representing facilities for indoor patrol services. The second part of the section presents the characteristics of the reference data, which are used to link the topological data and image datasets. The next section proposes an indoor space patrol service application data model, as well as the databased schema used to implement the proposed data model.
3.1. Basic Concept of the Data Linking Method
The proposed indoor space application data model is based on the international standard IndoorGML adopted by the OGC, which defines topological relationships among spatial entities in an indoor space. Not only the topological relationships between spaces but also the topological relationships between spatial entities and facilities are described in the proposed topological data model based on IndoorGML. The data model is based on a multilayered space model presented in IndoorGML. A multilayer spatial model is a model that represents a specific space using several layers, such as a topological space layer and a sensor space layer, which are then combined [17]. In this case, the layers are spatially divided into a nonoverlapping form, and the nodes in each layer exist independently. Nodes in different layers have features that can be connected to each other. For example, in a case where a sensor is located in a room, a node representing the room and a node representing the sensor exist in a topological space layer and a sensor space layer, respectively. These nodes in different layers can have composition relationships. This is called interlayer relations in IndoorGML. However, the representation of indoor facilities is not realized in interlayer relations. So, this study intends to add the consideration of indoor facilities, which are generally service objects, based on interlayer relations. This type of data is defined as “indoor space topological data” in this study.
The indoor space topological data presented in this study represent topological relationships, as shown in Figure 1. If the existing IndoorGML data consider only the connectivity or adjacency relationships between spaces, these data consider everything from the composition relationships to the facilities in a space. As shown in the figure, in a case where there are facilities such as a bulletin board (F1) and a fire extinguisher (F2) in spaces (R1, R2, R3), the data represent not only the connectivity between the spaces but also the adjacency and composition relationships between the spaces and facilities. Based on the application of the basic concept of IndoorGML, nodes become all objects that exist in the indoor space such as spaces and facilities, and the relationship between the indoor objects is represented as an edge. Also, like a multilayer spatial model, the relationships between spaces and between spaces and facilities are represented by respective layers, and spaces and facilities are defined as independent nodes. Based on this data, the application data model proposed in this study extends the topological relationship representation method by which the existing IndoorGML represents the relationship between spaces and has a form where the subjects to which services are provided consider all relationships between the interested spaces and facilities.
However, this method is based on the NRS of IndoorGML, where one geometric object converts into one node and the topological relationships among objects are represented by edges that connect these nodes. So, this method is difficult to apply in a case where an indoor space is not represented as a geometric object. The omnidirectional image selected as the space representation method in this study is a photograph composed of a set of pixels. As it does not exist as a geometric object (i.e., a geometric model representing a room), it needs to use pixels to form images. However, the procedure for converting pixel values into nodes can be complicated. Figure 2 shows this difference. IndoorGML data derive topological relationships through one-to-one matching of geometric objects and nodes, and so the procedure is very simple. However, as the image is composed of many pixels, one-to-one matching is difficult and only one-to-many matching, where one node corresponds to several pixels, can be done.
An image can be converted into a node using a recognized object composed of several pixels. However, the data processing method becomes complicated and difficult. When one-to-one matching between indoor spatial objects is represented based on images, the topological relationships among the objects can be easily identified and applied to indoor LBS. In order to derive a topological relationship from images efficiently, this study uses “reference data” instead of directly identifying the relationships between indoor spatial objects consisting in the pixels that form the image. The reference data presented in this study, which serve as the link between images and topological data, consist of geometric objects, not pixels. One-to-one matching between the objects in an image and topological data should be made possible using the reference data. Figure 3 shows this study’s method for deriving the topological relationships among the spatial objects represented in an image through reference data. In this method, an image is converted into a graph showing the topological relationships through the reference data having geometric objects. The relationships between the objects in an image are identified through a converted graph. The relationships in this case are defined as the indoor space topological data defined above.
3.2. Reference Data for Images
As mentioned previously, this study proposes a method for identifying the topological relationships among spatial entities represented in an omnidirectional image using reference data (Figure 4). The reference data for the omnidirectional image are geometric objects that represent spatial entities and facilities in an image and serve as a link to convert a recognized object in an image into a node on IndoorGML. When the omnidirectional image is georeferenced, each pixel has 3D coordinate values. When an object is selected from a georeferenced image, the coordinates of the selected pixel can be obtained. To connect an image and the reference data, 3D image coordinates of the selected point are projected onto the floor point of the object through the floor information of the image. The image and reference data are connected by matching the coordinate values of the projected point with the reference data that have geometric data. These reference data can be easily converted into indoor space topological data representing the relations between objects in an indoor space. So, it is possible to obtain information about an object by deriving the topological relationships in the omnidirectional image and recognizing the object through this process. The performance of the function of recognizing an object through an image is determined according to the method for designing the reference data. So, it is necessary to create it using a simple method where the object in an image is recognized correctly. As shown below, this study proposes a method for creating reference data based on 2D plan layout data for spaces and facilities represented by a plane-shaped geometric object.
The reference data for indoor facilities were created based on the locations of POIs. In general, POI means an object of interest that is a service object in an indoor space, and data are created in the form of points. However, in a case where the point object is used as it is, matching may not be done correctly as mentioned above. So, it is designed as a polygonal shape larger than the size of the actual object by giving an offset of a certain size. The size of polygons in reference data is adjusted by setting the offset value larger than the actual size of the facility. When a facility is selected through this reference data, it is determined whether the coordinate value selected from an image is included in the reference data for the facility. In a case where it is included, the facility object is recognized by matching an image and the reference data.
The reference data for an indoor space are designed to have a simplified floor layout form with a 2D polygonal object. This method also identifies an object by checking the composition relationships between the coordinate values of an image and the object of the reference data, as in the case of a facility. For this, the reference data for a spatial object establish an indoor spatial plan in a modified shape according to the following three methods. First, the space in the reference data features a polygon of the simplest shape. The reference data are used to recognize objects in the omnidirectional image without being actually visualized in services. So, a room does not need to represent the actual form in detail. A room is constructed as a simplified polygon in a range where its size is kept to the maximum. Second, tolerance should be given to the reference data of the objects and space. The reference data are designed so that the coordinate value of a point selected by a cursor can be included in the geometric object of the reference data. In this study, the reference data are based on a 2D plan, and in a case where these data are used, the coordinate value of the selected point in an image may not be completely included in an object on the plan. To prevent this, an appropriate offset value is set to adjust the size of a spatial object in the plan. Finally, spaces like long corridors or wide squares may have to be subdivided. When extracting topological relationships through reference data, one polygon is converted into one node. In this method, indoor space elements such as columns and obstacles in a moving space are not taken into consideration; therefore, only simple connection information can be obtained [18]. For this reason, the photographed moving space on a 2D plan is divided based on the shooting point of the omnidirectional image.
Using the reference data, an image can be converted into an IndoorGML node, and the descriptive data of recognized objects in the images are connected to the information of the node. Although this method lacks in detail and accuracy by comparing to the direct use of 3D modeling, it enables a search for basic attribute information about a room as 3D modeling does and has the advantage in that the method of creating the reference data is also much simpler than 3D modeling. Besides, the process of deriving topological data based on an image is simplified as much as possible by setting the reference data as a 2D polygonal geometric object and matching the coordinate values of the selected point in an image with the coordinate values of the reference data. The reference data is represented as different layers for facilities and spaces and also independently for the relationships between objects. For the reference data layer proposed in this study, the interlayer relations were set and represented as an edge for the relationship between a space and a facility. The relationship between spaces is one independent layer generated from the reference data layer for spaces, and the relationship between spaces and facilities seems to be generated by the interlayer relations between two reference data layers.
3.3. Indoor Patrol Facility Data Model
Facilities in an indoor space are generally represented using POIs. A POI has a slightly different form depending on the service being utilized. In the case of car navigation, road facilities and objects that can be destinations become POI. For portal sites and map services, places such as shops and public agencies that people frequently search for are POIs. A POI in an indoor space is narrower than these services based on which utilization services are provided. The POIs in the case of providing indoor patrol service, as in this study, will be security facilities such as shops and CCTV cameras. So, this study intends to propose an indoor patrol facility data model that can be applied to various indoor LBS as well as patrol services (Figure 5).
An indoor patrol facility data model has Patrol_Fac_Basic, Patrol_Fac_Location, and Patrol_Fac_Properties classes. Patrol_Fac_Basic represents a facility object and has a self-aggregation relationship that represents a space-based hierarchical structure. A space-based hierarchical structure refers to a case where a facility can be managed through other facilities in a space and the relationship can be defined. In a case where a monitoring facility such as CCTV manages a facility such as a fire extinguisher, it can be said that it has a hierarchical structure because it can monitor and manage the fire extinguisher through CCTV. Patrol_Fac_Location represents the location of a POI object and has geometric information in the form of a 2D point. It has a 1 : 1 association relationship with Patrol_Fac_Basic. Patrol_Fac_Properties contain the properties of POI and have an aggregation relationship with Patrol_Fac_Basic. Figure 6 shows the data types and code lists of these class properties.
3.4. Indoor Space Patrol Service Application Data Model
The indoor space application data model for a patrol service proposed in this study is composed of the navigation module of IndoorGML, the indoor patrol facility data model presented in the previous chapter, the omnidirectional image feature class, and the reference data feature class. The proposed data model is proposed using a Unified Modeling Language (UML) package diagram, as shown in Figure 7.
The proposed data model is composed of three parts. The omnidirectional image class and Cell Space class corresponding to an image are elements related to images and represented in blue. The topological model is based on the navigation module of IndoorGML and represented in white in the data model. Finally, the parts related to the reference data are a facility-related class, the reference data class for a space, and a plane object class, and they are represented in pink.
Also, this data model was designed to implement two functionalities largely to provide an indoor patrol service. They are a function to recognize an indoor space object and a spatial cognition function to understand an indoor space situation based on the recognized indoor object. The objects to be served most basically in a patrol are indoor spaces and the patrol facilities in the indoor space. There is a need for a method for these service objects to be recognized as indoor spatial objects through the omnidirectional image. Also, even if the objects are recognized, a full patrol service can be provided only when a necessary patrol route is derived through the objects, or indoor spatial cognizance is conducted through various spatial analyses.
The smallest unit that represents a space in IndoorGML is a cell. A space is viewed as a set of cells and referred to as a Cell Space. In this study, as a space is represented by an omnidirectional image, a Cell Space is represented in the same way. It was found that a Cell Space represented in the UML of a data model is converted into a State represented by a node in IndoorGML. As the Cell Space presented in the existing IndoorGML is represented through a geometric object, the geometric object is directly converted into a State. However, the omnidirectional image used in this study represents Cell Space with a data structure, without the modeled geometric information of an indoor spatial object. So, the Cell Space is converted into a State by referring to the reference data as having the geometric information given in the previous chapter. An indoor patrol facility class has containment topological relationships included in Cell Space. Also, a class representing location information may have point-shaped geometric information for the actual location, but the reference data are represented by 2D plane-shaped information that can be recognized in an image.
This study uses topological information on a space to enable a spatial query and thus is based on IndoorGML, which is the standard for indoor topological information. In particular, when the navigation module of IndoorGML is used, a route can be derived, and so more different types of queries can be carried out. For example, it is also possible to derive routes that will enable close inspection of the objects in which patrols have been made most recently in a space. According to IndoorGML, when an omnidirectional image is converted into a State through the reference data, the relationship between them is represented through a Transition, represented by the edge, and this is used in creating a Route Node and Route Segment on the navigation module. A route is created through a combination of a Route Node and a Route Segment. The route class of IndoorGML is set as a superclass, and patrol and inspector routes are made to inherit. In this case, “patrol route” means a route used to carry out a patrol, and an “inspector route” is a route for inspecting whether the patrol has been performed properly.
3.5. Database Design for the Proposed Data Model
In order to actually apply an indoor space application data model to a service, it is necessary to design a database for the data model. In the case of IndoorGML (because only its conceptual model is currently presented), which is based on the proposed application data model, its application to an actual service and linkage with the omnidirectional image are designed through database schema. The designed schema is divided into three parts in the data model: an image, a topological model, and the reference data. The class of each part is composed of the schema class (Figure 7).
First, a Cell Space represents an indoor space in the image part. This Cell Space class contains information about images in the database, and it becomes data with contents about the space to be represented. In the case of database schema of the Cell Space class, the ID of the cell, the name of the image, the position of the camera, the ID of the building, and information on the floor are included as minimum attributes. When a desired point is selected from an image, not only the image information but also the position information of the camera through which images are acquired also becomes data so as to obtain 3D image coordinates. It is possible to access the omnidirectional image XML data created through image information and camera position coordinates and calculate the location on the image. So, the data on that part are needed. Also, in order to identify the height value of the selected point in the process of projecting the coordinates of the selected point on an image to a floor point, information about the building and the floor where the image was acquired become data altogether.
In this indoor spatial data model, topological data are made to be derived based on an image by designing the reference data. The reference data for a data model include four classes: one is the reference data class related to the reference data representing a space, and the other three are classes representing the facilities. As data are too complicated to create all classes as databases, the reference data for spaces and the reference data for facilities are recreated into two parts. “Class for facilities” is the reference data for facilities, and it needs at least three attributes, which represents node ID, facility type, and polygon-shaped geometric information. Also, “class for room” has a node ID, room name, polygon-shaped geometric information, building ID, and floor information. These two sets of data become the links with a class that has the information of the selected point in an image. In a case where the coordinate value of the selected point in a Cell Space is matched with polygon-shaped geometric information in the reference data, they are intermediaries that can retrieve the information of the point selected through the node ID of the matched data.
Reference data are related to the Cell Space class. The reference data on facilities are included in the Cell Space, and the reference data on the room are converted into the State class according to the duality in the process of the Cell Space class. The part showing the topological relationship of spaces represented by a Cell Space corresponds to a topology model in an indoor space application data model. The topology model is based on the relationship between the class name and class of IndoorGML but as a schema is not specified in IndoorGML; this part is designed to match the indoor patrol service that this study focuses on.
Data was created for four classes out of the five represented in the data model. These are State, Transition, RouteNode, and RouteSegement. The State class can be seen as the topological data that the objects represented by Cell Space are converted into. The node IDs of the room and facilities classes are matched with the node IDs of the State. The matched nodes have the information about objects. They also contain the information about the edge connected to the nodes and about node types. The types of nodes are divided into facilities and spaces and connected to attribute information, which is nonspatial data, according to each type. Facilities have attributes related to the indoor service. In the case of a patrol service, the attribute information, which can be accessed through the node ID, contains information on the types of facility, the number of rooms adjoining facilities, the management, and the date of inspection. Spaces also have attributes related to the indoor service, and they are also accessible through the node ID. The attribute information about spaces consists of the name of the room, the ID of the building, floors, the room’s number, and the date of inspection.
These State classes represent the mutual relationship as a Transition class, like the nodes of the topology model. The Transition class has an edge ID, a starting node, an ending node, an edge type, and a weight. The types of edge represent the connections between spaces and between spaces and facilities. A weight can be a time, distance, and so forth, depending on the situation. In this study, distance was set as a weight. The State and Transition classes were also used as elements to form the RouteNode and RouteSegment classes, respectively. RouteNode serves as a node for a route by referring to the State, and RouteSegement has information about each edge connecting routes by referring to a Transition. Given information about a starting point and a destination, a route can be derived through the combination of these two classes.
4. Experiment
The study area for the experimental implementation was the 6th floor of the 21st Century Hall of the University of Seoul, South Korea (Figure 8). For the study area, images were captured at 22 locations and acquired in 6 directions at each location. The environment for the experiment is shown in Table 1. The functions developed in this study were implemented based on a prototype developed in a study called the “Development of 3D Indoor Space Information Technology and 3D Indoor and Outdoor Integrated Platform” funded by the Korea Agency for Infrastructure Technology Advancement, South Korea.
4.1. Generating Reference Data and a Database
The reference data were generated in accordance with the three creation conditions discussed in the previous section. As the reference data were matched with the image data and used only for the purpose of retrieving information on objects, they did not need to be as detailed as the completion drawing for a building. In a case where a corridor is viewed simply as a single space and the reference data are created, the topological relationship in the omnidirectional image generated based on the reference data can only be established with the representation of the connection relations. In order to generate a network for a route search, a space needs to be divided up. Figure 9 shows the reference data generated according to this procedure.
The reference data for spaces and those for facilities were named Patrol_room and Patrol_fac, respectively, according to the database schema designed above and created through MySQL. Information was derived by selecting an object from an image and determining whether the selected object was a room or a facility; then it was retrieved from the database of Patrol_room or Patrol_fac. In this case, the attribute information used to judge whether the selected object was a room or a facility consisted of the position of Patorl_room and Patrol_fac. Position was polygon-shaped, as an attribute having information about the boundary point of the polygon. Accordingly, whether the selected object was a facility or a space could be made clear by determining whether the coordinates of the selected point were inside the polygon. Also, State and Transition had topological information based on Cell Space and IndoorGML, which contained information about images; they were also built through MySQL. Cell Space had information about images as defined in the database schema. In a case where an object was selected from images, the information about the selected screen could be obtained through the database of this Cell Space. RouteNode and RouteSegment, to which the navigation module of IndoorGML referred for route derivation, were also built as databases. A built database had attributes and a structure as designed in the database schema and contained Cell ID, image information (Scene, shooting_point), photographing position (Pos_, Pos_), building ID (Building ID), and floor information. In this case, the data on among the position values was referred to as floor information. For the process of converting the coordinates of an object selected by a screen cursor into actual image coordinates, the value was set at 1.8 m, which was the height value of the camera with which the image was captured.
4.2. Pilot Project Based on the Proposed Data Model
In this section, an image-based interest object recognition function and an image-based spatial query function, which are the functions needed to provide an indoor patrol service based on an omnidirectional image, are implemented based on an application data model. The pilot project implemented three functions: facility identification, room information identification, and patrol route creation. The functions of facility identification and room information identification enabled the recognition of service objects based on images for facility identification and room information identification. Patrol route creation functioned to carry out a spatial query based on images.
In order to recognize an object of interest through an omnidirectional image, the coordinates of the cursor should first be converted into coordinates on the image. The Krpano plugin of JavaScript was used for this. The image coordinates that were obtained using the conversion function of Krpano were processed into 3D coordinates to be matched with the reference data. Algorithm 1 shows the Pseudocode that was used to implement this.
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
The information on angles and the amount of coordinate adjustment for changing the coordinates of an image were calculated by selecting the floor point of an object from images through the aforementioned Pseudocode. If the floor point of an object was double-clicked, the conversion to find the image coordinates in screen coordinates was achieved for the floor point of the image. In this case, if the direction of the camera was due north, the heading of the camera was set to zero. The horizontal and vertical angles on the actual image were found for the point selected by the mouse, and through them, the coordinates were derived. Then, by double-clicking an object, the value of the object was found and the 3D coordinates were derived. The value was set to 0 at first, and the floor point was defined. The 3D coordinates of the selected point were derived by calculating the distance from the floor point to the point where an object was selected.
Once the coordinates were calculated, functions were performed according to the procedure shown in Figure 10. The converted coordinates were searched to see if they were included in the polygon object of the reference data and that the coordinates and reference data matched. The matched points in the reference data were located in a polygon representing a space or facility on the reference data. In this case, a search was carried out to see which polygon the coordinates found through the ST_CONTAINTS function, which is a Point in Polygon function in MySQL. The coordinates for where the cursor was clicked were converted to point geometry through a function in MySQL called GeomFromText, and after checking whether the coordinates were included in the geometry information column of the reference data class for a facility or room in the database, the ID (NodeID) of the corresponding node was selected.
The database was accessed using the ID of the node selected through this query. The information held by the State class in the database was retrieved. In this case, it was determined whether the type of node was a facility or a space. The information on the attribute was retrieved in the case of a facility or space. The information retrieved in this way was displayed using a program. If the “facility check” function was selected and a facility was selected, the information of an object was displayed through the matching process of the selected coordinates and the reference data. Likewise, if the “room information check” function was selected and a room was selected, the information of an object matching the selected coordinates was displayed.
In order to provide actual service via an indoor patrol service, it is necessary to recognize not only objects but also indoor spaces. The recognition of indoor spaces should be analyzed. This can be carried out through a spatial query of the indoor space. This study implemented a spatial query function under the assumption that patrols are arranged in the closest order for the facilities implemented before June 2016 and routes are derived for the facilities. Also, a starting point was set so that a route to the corresponding point could be derived. First, a starting point was selected on an image. The selection of a starting point was implemented in the same manner as the previously implemented indoor object recognition function. Once a desired starting point was selected, the starting point was set and the desired result was obtained by searching a facility or space according to the assumed situation. This search function was performed through spatial queries in MySQL.
The query statement selected all facilities for which the patrol date was prior to June and arranged the distances between the coordinates of the starting point on the image and the central value (centroid) of the selected point (position), calculated using the ST_Distance function of MySQL. So, when a starting point was set and a query performed, the objects corresponding to the query were derived. For derived objects, the routes from the starting point to the objects were guided. Figure 11 shows how the corresponding function was implemented. From the selected starting point, the facilities with old patrol dates were searched and arranged in the closest order at the same time. If a desired facility was selected from the list, a virtual route to it was created, and a simple route derivation function was performed on the omnidirectional image.
5. Conclusion
Various indoor representation methods have been studied for their ability to provide indoor LBS. This study proposed an indoor patrol service application data model based on an indoor representation method using omnidirectional images. This approach costs less than 3D data modeling, and it is much easier to manage and update the data. However, as an omnidirectional image lacks detailed geometric information, only the visualization function for an indoor space is generally implemented and utilized, like Store View. So, in order to derive different services through an omnidirectional image, this study proposed a linking method with topological data to identify the relationships between spaces. The most typical method for representing topological data on an indoor space is the international standard IndoorGML established by the OGC. In order to apply IndoorGML to an omnidirectional image, the object should be recognized directly on the image. However, an omnidirectional image, as simple photographic data, has no information other than a pixel value. So, this study applied the concept of reference data to recognize a room and establish the relationships between rooms. Various functions can be implemented by linking an omnidirectional image and topological data through reference data. Also, this study proposed a data model that considers indoor facilities not covered by IndoorGML. For this, this study developed an indoor space application data model that can present services to an indoor space based on the basic concept of an indoor space topological data model that considers the topological relationship between indoor spaces and between indoor spaces and facilities at the same time. In addition, we conducted an experimental implementation of the integrated 3D indoor navigation model for a patrol service using GIS data on a building on the University of Seoul campus in South Korea. From the pilot project, we demonstrated the benefit of using such a 3D data fusion method integrating omnidirectional images with vector-based topological data models based on IndoorGML for providing indoor LBS in built environments.
This article has several limitations that future research needs to address. First, the proposed data model integrated omnidirectional images to visualize geographic entities in 3D viewers, with network data being used to represent topological relationships among geographical objects in an indoor space. In other words, dual data models are required for implementing the indoor patrol service application. In order to maintain consistency in data transitions between two data models, the application requires the development of the proposed “reference data” database. Secondly, because the proposed “reference data” links the image data and vector-based topological data, the method for generating the reference data needs to be improved for the sake of detecting objects from images directly using pattern recognition analyses. Third, the proposed data model describes linkages between image data and topological data only in indoor spaces. Although discussions about this have been implemented through IndoorGML, further discussions are needed for the integration of image data like omnidirectional images with topological data linking outdoor and indoor spaces. Finally, the pilot system needs to be integrated with a 3D visualization system to allow the manipulation and exploration of georeferenced virtual environments in order to select and display spatial and attribute information on spatial entities in indoor spaces.
Conflicts of Interest
The authors declare that they have no conflicts of interest regarding the publication of this paper.
Acknowledgments
This research was supported by Civil-Military Technology Corporation Program (Grant 16-SF-SW-20) funded by Defense Acquisition Program Administration, Republic of Korea.