Next Article in Journal
HBIM Modeling from the Surface Mesh and Its Extended Capability of Knowledge Representation
Next Article in Special Issue
On the Use of Single-, Dual-, and Quad-Polarimetric SAR Observation for Landslide Detection
Previous Article in Journal
Collision Detection for UAVs Based on GeoSOT-3D Grids
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

A Convolutional Neural Network Architecture for Auto-Detection of Landslide Photographs to Assess Citizen Science and Volunteered Geographic Information Data Quality

1
Hacettepe University, Department of Geomatics Engineering, 06800 Beytepe, Ankara, Turkey
2
Hacettepe University, Department of Geological Engineering, 06800 Beytepe, Ankara, Turkey
*
Author to whom correspondence should be addressed.
ISPRS Int. J. Geo-Inf. 2019, 8(7), 300; https://0-doi-org.brum.beds.ac.uk/10.3390/ijgi8070300
Submission received: 1 June 2019 / Revised: 11 July 2019 / Accepted: 12 July 2019 / Published: 15 July 2019

Abstract

:
Several scientific processes benefit from Citizen Science (CitSci) and VGI (Volunteered Geographical Information) with the help of mobile and geospatial technologies. Studies on landslides can also take advantage of these approaches to a great extent. However, the quality of the collected data by both approaches is often questionable, and automated procedures to check the quality are needed for this purpose. In the present study, a convolutional neural network (CNN) architecture is proposed to validate landslide photos collected by citizens or nonexperts and integrated into a mobile- and web-based GIS environment designed specifically for a landslide CitSci project. The VGG16 has been used as the base model since it allows finetuning, and high performance could be achieved by selecting the best hyper-parameters. Although the training dataset was small, the proposed CNN architecture was found to be effective as it could identify the landslide photos with 94% precision. The accuracy of the results is sufficient for purpose and could even be improved further using a larger amount of training data, which is expected to be obtained with the help of volunteers.

1. Introduction

With the recent developments in mobile and geospatial technologies, it has become possible to incorporate human efforts in several scientific processes. This approach has brought the term Citizen Science (CitSci) into the sight of researchers and is especially useful for geoscience studies, where every human can act as a powerful sensor and interpreter. The term VGI (volunteered geographical information) has also been introduced, where volunteer contributions to geodata collection have brought about revolutionary changes in this field, although the contributions are often not scientific-achievement-oriented.
The paradigms in landslide research have shown drastic changes depending on the advancements in artificial intelligent algorithms during the last decade e.g., [1,2,3]. Additionally, CitSci has also become one of the most prominent scientific approaches. The trends in open data and open science mentalities strongly support these processes. In fact, these changes are not only fostering rapid scientific development, but are actually forming science-oriented societies. The Oxford English Dictionary described citizen science as: “Scientific work undertaken by members of the general public, often in collaboration with or under the direction of professional scientists and scientific institutions.” [4]. The growing public demand in CitSci [5] can be confirmed by the establishment of several regional and international citizen science associations (e.g., CSA [4], ECSA [6], etc.) in recent years. On the other hand, geodata collection and map production efforts have been increasingly carried out by nonprofessionals [7]. The term VGI, proposed by Goodchild [8], is commonly used to describe this situation and mainly emphasizes the user-generated geographic information. According to Haklay [9], CitSci stands out as a class of VGI activities that require special attention and analysis. It has been seen in the literature that the terms “CitSci” and “VGI” are used alternatively for many geographical data-related projects, and they also benefit from each other [10]. The development of data quality assessment and validation strategies both for VGI and CitSci data has been an emerging research topic and is crucial, as emphasized by several studies [11,12,13,14].
Kocaman et al. [15] have categorized the potential contributions of CitSci to disaster management by analyzing the National Climate Assessment Report [16]. The categories have later been modified for pre-disaster, during, and also post-disaster stages, and the levels of participation in landslide-CitSci (LS-CitSci) projects have been introduced [17]. The levels are (a) basic data collection; (b) data validation and interpretation; (c) advanced interpretation; and (d) project and policy development [17]. Kocaman and Gokceoglu [18] have emphasized the importance and contribution of CitSci for landslide researchers and developed an app, namely LaMA (Landslide Monitoring Application), for landslide data collection. Similarly, the landslide report website [19] by NASA started in the first quarter of 2018 and is a tool for volunteers to provide data about landslides.
LaMA, developed by Kocaman and Gokceoglu [18], is a mobile app that works on iOS and Android platforms and aims at collecting landslide data from volunteers via a simple interface that includes photos of the landslide event, time of occurrence, damage description, and other observations. Data uploaded by citizens using LaMA are stored in a spatial database together with the geolocation information on a central server and are visualized dynamically on a web GIS interface (www.geocitsci.com). Although volunteers can currently upload landslide information, the main question to be answered is the veracity of the collected data, since no quality check is being made on the user side. However, successful data validation would be beneficial for the second stage achievements of LS-CitSci and would also support the data interpretation. In addition, assessment of the landslide photos taken by ordinary people could be extremely difficult due to a lack of expertise and a manual check on the server by the experts prevents auto-run of the system. Depending on the increase in participation of citizens in LS-CitSci projects, the amount of data will be gigantic, and auto-check algorithms for such data are indispensable. Considering this necessity, it is possible to state that the auto-check algorithms for controlling the data provided by citizens are mandatory before they are processed. For this reason, the purpose of the study is to develop a convolutional neural network (CNN) algorithm to validate the landslide photos provided by citizens or nonexperts. For the purpose of the study, a CNN architecture is proposed and executed on the server. The proposed algorithm is trained using well-known image databases and the photos collected from search engines and social media platforms (Figure 1).
Although advanced machine learning algorithms have been widely used for landslide susceptibility mapping studies e.g., [1,2,3], the use of neural networks for the identification of landslide photos and as support for the data quality check of data collected by citizens or nonprofessionals is new and forms the novelty of the study. The same difficulties, described by Gu et al. [20], are encountered when using the CNN algorithm in this study, which requires big data during the training stage. The main difficulty is to provide a sufficient number of landslide images and further attempts have been carried out to avoid the overfitting problem.

2. Previous Studies on the Use of CNN in Geosciences

The popularity of CNN has increased recently and this methodology has been successfully applied to various areas [20]. Although many deep learning (DL) algorithms have been developed due to the extreme diversity of its application areas, CNN is perhaps the most widely applied method to extract reasonable information from huge datasets. Object detection, pose estimation, text recognition and detection, speech processing, and natural language processing are some application areas of CNN. However, the major drawbacks of CNN algorithms are: (i) Requirement of large-scale datasets and high computational resources for training; (ii) collecting and labelling the datasets is possible with extensive human efforts; (iii) further research on optimization methods is needed, since selecting the suitable hyper-parameters (e.g., learning rate, kernel sizes, the number of layers) is only possible with skill and experience; and (iv) solid theory of CNN is not yet established, although many successful applications based on it have been found [20].
It is possible to find a massive number of CNN studies in the international literature. Fu and Aldrich [21] applied CNN to flotation froth image analysis. Dung et al. [22] presented a robust method for crack detection by classification in joints of steel bridges and discovered that transfer learning methods improved both the accuracy and robustness of VGG16. Zhang et al. [23] applied a near-to-far learning strategy in a CNN model and improved the accuracy of terrain segmentation and increased robustness against wild environments. The study by Zhang et al. [23] concluded that traditional machine learning techniques have a low ability to generalize and to learn new environments.
Smart and effective energy management systems require reliable load forecasting. For this reason, Sadaei et al. [24] suggested combining fuzzy time series and CNN for short-term load forecasting and they obtained successful results. Bayr and Puschmann [25] applied CNN classification to landscape photographs for detecting woody regrowth vegetation and emphasized the contribution of a robust CNN algorithm in landscape monitoring together with satellite imagery and field measurements. Deng et al. [26] applied a faster region-based CNN to classify objects with multiple classes in satellite images. Coulibaly et al. [27] applied CNN to identify mildew disease in pearl and developed an approach using transfer learning and feature extraction. Mitra et al. [28] trained a CNN-based system to identify six species from microscope digital images and concluded that CNN architecture can provide the ‘brain’ for a viable robotic picking system. Imamverdiyev and Sukhostat [29] published an interesting geological application of CNN for lithofacies classification by employing different variables. Palafox et al. [30] automatically detected geological landforms on Mars and they applied CNN successfully. Digital images of Berea sandstone samples were used by Karimpouli and Tahmasebi [31] to evaluate the permeability, P- and S-wave velocities, and the formation factor. They found that CNNs demonstrate outstanding performance in predicting the physical parameters of rock if sufficient input data are available. Wu et al. [32] predicted the permeability from images when applying CNN and presented a procedure using CNN with simulated data.
One of the important topics of landslide studies is susceptibility mapping. Despite the fact that until now, several methodologies and approaches have been applied to this field, the use of CNN is rare. Wang et al. [33] produced a landslide susceptibility map of Yanshan County, China, using CNN. Following heavy precipitations and floods, the safety of transportation networks is of vital importance. For this reason, Ahmad et al. [34] proposed an approach to determine passable roads after floods in satellite images and the data from different social media platforms. According to Ahmad et al. [33], the results demonstrated that CNN performs significantly better in comparison to other state-of-the-art approaches. Liu and Wu [35] applied CNN for landslide detection on optical satellite images by performing a 2D wavelet transformation for preprocessing in order to obtain more distinct features and to reduce the image sizes.
In an extensive review on DL applications in geochemical mapping by Zuo et al. [36], it has been emphasized that this field requires more research that implements machine learning and/or DL. Another recent review on the use of machine learning techniques for river flow forecasting published by Yaseen et al. [37] underlined the need for developing more reliable and intelligent expert systems in real-time river flow predictions. According to Grekousis [38], artificial neural networks (ANNs) have a great potential for research in urban geography. Grekousis also proposed a guideline for ANN reports to ensure consistency and to ease the dissemination of the outcomes.
Huang et al. [39] developed a CNN for identifying the Time Delay of Arrival in micro-seismic events, which is essential for location determination and preventing mining disasters induced by a high-stress concentration. In addition to this micro-seismic event prediction, Derakhshani and Foruzan [40] developed DL neural networks to estimate seismic ground motion parameters. Another study on earthquake magnitude prediction has been carried out based on regression algorithms and cloud-based big data infrastructure [41]. In the mentioned study, it was shown that using big data analytics for predicting the magnitude of earthquakes opens a very promising research area and the methodology may enable simultaneous processing of massive data with a large number of variables.

3. Data Used

The training stage of an artificial intelligence algorithm is crucial and requires a sufficient amount of high-quality data. The datasets used for the training in this study have been downloaded from the freely available sources, image databases, and search engines listed in Table 1. A Python script has been implemented to download data from search engines (Google, Bing, Baidu) and Flickr, using specific keywords (i.e., Landslides, Earthslip, Rockfall, Landslip, Rockslide, and Mudslide) and their corresponding terms in 92 languages. A total of 36,984 images were downloaded from these sources and 600 of them were selected manually with visual checks by the expert to identify those which are most representative for different types of landslides. It has been observed that most landslide photos include roads, forest trails, mountains with vegetation cover, villages/settlement areas that are located in the foothills of mountains, landscapes, and other manmade objects. During the selection of the representative landslide photos, the completeness of the scene (variety of the object types obtaining the textural variation) was taken into account. For example, landslides that only showed mud were excluded and those which included mountains, forests, houses, etc., were kept. In principle, the landslide has to be one of the main themes or object types in the photo. In other words, photos where a landslide forms only a small part of the overall image were excluded. On the other hand, completely irrelevant photos (e.g., book cover pages, ads, selfies), post-disaster resilience photos without the actual landslide (e.g., with dozens of people, rescue teams, diggers), flooding photos, and images with a size smaller than 224 × 224 pixels were also disregarded while forming the training datasets.
Six other object classes (Irrelevant, Mountain-Valley, Natural, Road, Village, and Yard) have been defined for the classification of the photos with the CNN algorithm. Apart from the irrelevant class, all others include natural scenes and have the potential to be mixed with the landslide during the classification. Examples of the training images of each class are given in Figure 2. In addition to the search engine and Flickr photos, Places [46] and Natural Image Database [47] datasets, which are distributed freely, were used for the training of the above-mentioned classes other than landslides. The Places Dataset was previously provided for the Places365-Challenge 2016 and includes high-resolution training, validation, and test images with a minimum dimension of 512 pixels. This data has been used for the training of the other six classes. From this Dataset, 5000 images for each class were selected initially, and a subset of them (600 for each class) were used in this study. The Natural Image dataset was employed for the estimation of missing image information (e.g., channels or occluded parts) and super-resolution images [48].
During the selection of the training photos in the other six classes, a number of principles were followed. First of all, the variation and semantic integrity in photos for each class type were ensured. The main focus/theme of the photo was identified (e.g., mountain and village) and photos with exceptional objects were excluded (i.e., generic road photos were preferred to photos of roads with food stalls). The yard class photos were selected according to the combination of manmade objects with nature. The village and yard photos were selected with special care in order to distinguish them from the class irrelevant. Since the landslide dataset is composed of 500 training and 100 test photos (600 in total), datasets of other classes also include the same numbers of training and test photos.
Several problems were encountered during the data preparation. For example, several irrelevant photos for a given keyword were found. The number of useful landslides images among the downloaded ones was approximately 10% of the whole dataset. On the other hand, although the landslide photos were searched for in 92 languages, a significant amount of the downloaded images that were found were exactly the same. Additional processing was performed to remove the repetitive images from the same dataset. In addition, manual selection of the images by visual interpretations was one of the biggest challenges in this study.

4. Methodology

4.1. Data Preprocessing and Augmentation

An important part of the study was the derivation and preprocessing of training data, since no landslide class was available in the existing image databases. Therefore, each downloaded landslide image was assessed and classified manually. After selecting the images, two preprocessing steps were applied. First, the images were cropped and resized to 224 × 224 using the Python 3.7 programming language [49] and OpenCV computer vision library [50]. The cropping was applied to retain the aspect ratio of the pixels, as recommended by Zheng et al. [51], and thus increase the consistency of the classification. After reducing the image sizes, a data augmentation approach was applied to amplify the number of training images and thus to improve the suitability for the DL. The original training set includes 3500 images in total for all seven classes. Augmentation of the training set also helps to reduce the overfitting problem [52]. A Python Deep Learning library, Keras [53], was employed and modified to perform the CNN algorithm and tune it. The ImageDataGenerator class of Keras was used to augment the data and the best results were obtained by tuning the parameters to the following: (a) Rotation range: 15; (b) width & height shift ranges = 0.1; (c) horizontal flip: True; (d) fill mode: Nearest. In Figure 3, examples of two original landslide photos and their preprocessed and augmented versions are presented.

4.2. Photo Classification with the CNN

Girshick et al. [54] have proposed that the DL methods can be used through finetuning if the training dataset is small. Due to the small number of training images employed in this study, the VGG16 architecture proposed by Simonyan and Zisserman [55] was selected as the base model since it allows finetuning for CNN. The model was proposed by a team named VGG and placed in the top five in the localization and classification tracks of the ImageNet Challenge 2014 [56]. The VGG16 architecture was modified to increase the success rate of the classification of the landslide photos. The original VGG16 architecture and the version modified in this study are depicted in Figure 4. The modification mainly aims at finetuning of the model. The fully connected classifier part of VGG16 was removed from the model since it was trained using the ImageNet dataset [57]. The new fully connected classifier was initialized by using random values. All layers, apart from the fully connected classifier, were converted to non-trainable so that these initial random values could not update their weights during the training. In other terms, a fully connected classifier and thus, the random values, are not effective after back-propagation. This approach was defined as the first step of training and is shown in Figure 5. The r [58] (Equations (1)–(3)) and stochastic gradient descent (SGD) [59] (Equations (4) and (5)) methods were selected as the optimizers in the first and second steps of the training, respectively. The SGD method has been found as the best and RMSprop as the best adaptive method by Wilson et al. [60] for CNN. The dense layer used in the architecture is a classic fully connected neural network layer, in which each input node is connected to each output node and the number of dense layers is seven because it equals the number of classes. Additionally, the parameter of the dense layer is 256 because it is connected to the previous layer and the previous layer’s output is 256. Softmax used in the architecture is a loss function, which yields normalized class probabilities as outputs, while LeakyRelu is a variation of rectified linear units, which allows a non-zero gradient for the negative inputs [61]. One of the important components of the architecture is Dropout. Dropout, which was proposed by Srivastava et al. [62], is used for preventing overfitting. The data augmentation has been applied with a learning rate of 0.00001 and batch size of 50 using the resized dataset. The training has been stopped after 25 epochs (Figure 5) and the convolutional block 5 has been converted to trainable. This second step of training, as shown in Figure 6, activates back-propagation for the initial fully connected classifier and the fifth convolutional block.
v t =   ρ v t 1   + ( 1 ρ ) * g t 2
Δ ω t =   η ν t +   ε * g t    
ω t + 1 =   ω t +   Δ ω t    
where
  • η :   Initial   Learning   Rate
  • v t   : Exponential   Average   of   squares   of   gradients
  • g t   : Gradient   at   time   t   along   ω j
v t = β v t 1 + ( 1 β ) w L ( W , X , y )
    W = W   α v t    
where
  • w : G r a d i e n t   w i t h   r e s p e c t   t o   w e i g h t   a n d   l e a r n i n g   r a t e
  • α : L e a r n i n g   r a t e  
  • β : M o m e n t u m
The model was compiled again in this step with a learning rate of 0.00001 and momentum of 0.9. Another 50 epochs of training were carried out using these parameters. After 25 epochs, the loss/accuracy ratio, generally, does not change. For this reason, 50 epochs are selected to terminate the learning stage. Hyperparameters were selected experimentally, considering the procedure recommended by Bengio [63]. In addition to the procedure recommended by Bengio [63], Masters and Luschi [64] showed that the combination of the small batch size and stochastic gradient descent make the training process stable and give good generalization performance. Consequently, the results of the experiments showed that the best performance was obtained when the batch size was 50, and the other hyperparameters were as shown in Figure 4 and Figure 5.

5. Results and Discussions

Ten different test runs were employed with ten sets of training and test datasets for the performance assessment. For each run, 10% of the images were used for the test and 90% were used for training. The training loss and accuracy results of each test are given in Figure 7. The model performances were assessed using the metrics function in Keras. The classification reports were generated using the classification report class in Python Scikit-learn library and the results of the forth run (Train-Test Split 4) are tabulated in Table 2.
Another test dataset with 700 images (100 for each class), which was not included in the training sets, was used for the evaluation of the Train-Test Split 4 model (forth test run). Additionally, an overfitting problem is not observed because the performance indicators of train and test datasets are close (Figure 7). The closest performance results are obtained from the Train–Test Split 4 model (forth test run), and hence, this model is selected. Use of Dropout in the CNN architecture helps to prevent the problem of overfitting. The normalized confusion matrix of this classification is presented in Figure 8. Some examples of correct and false classifications are given in the Appendix A. Table A1 in the Appendix A shows correct classifications for classes, while Table A2 in the Appendix A gives some false classifications.
As can be seen from the normalized confusion matrix (Figure 8), the CNN algorithm yields a high performance. This can be especially seen in the successful classification of irrelevant images by the algorithm. Additionally, landslide images were recognized with a high performance (96%). The lowest performance was obtained for natural and village images. In fact, this result can be expected, since it is extremely difficult to separate these classes. Despite this difficulty, the classification success is 85% for the images in this class. However, if the amount of training data was increased, the performance would be higher.

6. Conclusions

Some serious paradigm changes in research methodologies have been observed during the last decade. CitSci is one of the results of these paradigm changes. In the near future, the volume of the data in almost every area of geoscience will increase drastically with the help of CitSci, since each ordinary person can act as a potential sensor. However, the quality of data provided by citizens or nonexperts should be checked automatically. With the advancements in artificial intelligence methods, the classification of photos from big datasets has become relatively easier. For specific classification purposes, suitable neural network architectures should be constructed. For the areas that utilize user-provided images, CNN provides great potential to identify the images. The amount of data is critical for the success of such architectures. An insufficient data volume results in overfitting, while a huge data volume affects the time efficiency and is a computational burden. Additionally, the complexity of the problem to be solved sometimes decreases the performance of the algorithm.
In the present study, a first attempt at classifying landslide photos using CNN was performed and promising results were obtained. The main difficulty encountered during the present study was data scarcity. To eliminate this problem, finetuning was applied to the CNN architecture, which is based on the VGG16 model. The architecture developed in the present study shows a high performance and the accuracy of the results is sufficient for the purpose. However, due to the small number of training images, the algorithm has a misclassification potential, and hence, the results should be checked manually until a sufficient amount of landslide photos is reached to train the algorithm further.
The proposed CNN architecture is part of a LS-CitSci study, where a mobile app called LaMA is employed for data collection together with a web map interface for uploading images and auxiliary data for the use of volunteers and contributors who prefer not to download a mobile app. The web map interface also allows dynamic visualization of the uploaded data (www.geocitsci.com). The future work mainly involves: (i) Integration of the proposed quality check in the developed GIS platform; (ii) improving the performance by training the proposed algorithm using new landslide photos provided by the volunteers; and (iii) utilizing the collected CitSci data for natural hazard analysis and risk assessment purposes.

Author Contributions

Conceptualization, Sultan Kocaman and Candan Gokceoglu; Methodology, Recep Can, Sultan Kocaman and Candan Gokceoglu; Software, Recep Can; Validation, Recep Can and Candan Gokceoglu; Formal Analysis, Sultan Kocaman and Recep Can; Investigation, Recep Can and Sultan Kocaman; Resources, Sultan Kocaman and Candan Gokceoglu; Data Curation, Recep Can; Writing-Original Draft Preparation, Sultan Kocaman and Candan Gokceoglu; Writing-Review & Editing, Sultan Kocaman and Candan Gokceoglu; Visualization, Recep Can; Supervision, Sultan Kocaman and Candan Gokceoglu; Project Administration, Sultan Kocaman.

Funding

This research received no external funding.

Conflicts of Interest

The authors declare no conflict of interest.

Appendix A

Table A1. Correct classifications by the CNN architecture developed in the study.
Table A1. Correct classifications by the CNN architecture developed in the study.
ImagesActual & Predicted
Ijgi 08 00300 i001
Ijgi 08 00300 i002
Landslides
Ijgi 08 00300 i003
Ijgi 08 00300 i004
Road
Ijgi 08 00300 i005Natural
Ijgi 08 00300 i006
Ijgi 08 00300 i007
Yard
Ijgi 08 00300 i008Irrelevant
Ijgi 08 00300 i009Village
Ijgi 08 00300 i010
Ijgi 08 00300 i011
Mountain/
Valley
Table A2. False classifications by the proposed CNN architecture.
Table A2. False classifications by the proposed CNN architecture.
ImagesActual vs. Predicted
Ijgi 08 00300 i012Landslides vs. Village
Ijgi 08 00300 i013Village vs. Mountain/Valley
Ijgi 08 00300 i014Yard vs. Irrelevant
Ijgi 08 00300 i015Road vs. Mountain/Valley
Ijgi 08 00300 i016Landslides vs. Irrelevant
Ijgi 08 00300 i017Natural vs. Yard
Ijgi 08 00300 i018Natural vs. Yard
Ijgi 08 00300 i019Natural vs. Road
Ijgi 08 00300 i020Village vs. Landslides

References

  1. Nefeslioglu, H.A.; Gokceoglu, C.; Sonmez, H. An assessment on the use of logistic regression and artificial neural networks with different sampling strategies for the preparation of landslide susceptibility maps. Eng. Geol. 2008, 97, 171–191. [Google Scholar] [CrossRef]
  2. Pham, B.T.; Pradhan, B.; Bui, D.T.; Prakash, I.; Dholakia, M.B. A comparative study of different machine learning methods for landslide susceptibility assessment: A case study of Uttarakhand area (India). Environ. Model. Softw. 2016, 84, 240–250. [Google Scholar] [CrossRef]
  3. Pham, B.T.; Shirzadi, A.; Bui, D.T.; Prakash, I.; Dholakia, M.B. A hybrid machine learning ensemble approach based on a Radial Basis Function neural network and Rotation Forest for landslide susceptibility modeling: A case study in the Himalayan area, India. Int. J. Sediment Res. 2018, 33, 157–170. [Google Scholar] [CrossRef]
  4. CSA. Citizen Science Association. 2019. Available online: https://www.citizenscience.org/ (accessed on 24 March 2019).
  5. Silvertown, J. A new dawn for CitSci. Trends Ecol. Evol. 2009, 24, 467–471. [Google Scholar] [CrossRef] [PubMed]
  6. ECSA. European Citizen Science Association. 2019. Available online: https://ecsa.citizen-science.net/ (accessed on 1 June 2019).
  7. See, L.; Mooney, P.; Foody, G.; Bastin, L.; Comber, A.; Estima, J.; Fritz, S.; Kerle, N.; Jiang, B.; Laakso, M.; et al. Crowdsourcing, Citizen Science or Volunteered Geographic Information? The Current State of Crowdsourced Geographic Information. ISPRS Int. J. Geo-Inf. 2016, 5, 55. [Google Scholar] [CrossRef]
  8. Goodchild, M. Citizens as sensors: The world of volunteered geography. GeoJournal 2007, 69, 211–221. [Google Scholar] [CrossRef]
  9. Haklay, M. Citizen Science and Volunteered Geographic Information: Overview and Typology of Participation. In Crowdsourcing Geographic Knowledge; Sui, D., Elwood, S., Goodchild, M., Eds.; Springer: Dordrecht, The Netherlands, 2013. [Google Scholar]
  10. Kocaman, S.; Gokceoglu, C. On the use of Citsci and VGI in Natural Hazard Assessment. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 2018, 42, 69–73. [Google Scholar] [CrossRef]
  11. Goodchild, M.F.; Li, L. Assuring the quality of volunteered geographic information. Spat. Stat. 2012, 1, 110–120. [Google Scholar] [CrossRef]
  12. Severinsen, J.; de Roiste, M.; Reitsma, F.; Hartato, E. VGTrust: Measuring trust for volunteered geographic information. Int. J. Geogr. Inf. Sci. 2019, 33, 1683–1701. [Google Scholar] [CrossRef]
  13. Moreri, K.K.; Fairbairn, D.; James, P. Volunteered geographic information quality assessment using trust and reputation modelling in land administration systems in developing countries. Int. J. Geogr. Inf. Sci. 2018, 32, 931–959. [Google Scholar] [CrossRef] [Green Version]
  14. Zhang, G.; Zhu, A.-X.; Huang, Z.-P.; Ren, G.; Qin, C.-Z.; Xiao, W. Validity of historical volunteered geographic information: Evaluating citizen data for mapping historical geographic phenomena. Trans. GIS 2018, 22, 149–164. [Google Scholar] [CrossRef]
  15. Kocaman, S.; Anbaroglu, B.; Gokceoglu, C.; Altan, O. A Review on Citizen Science (Citsci) Applications for Disaster Management. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 2018, 42, W4. [Google Scholar] [CrossRef]
  16. National Climate Assessment Report. In Climate Change Impacts and Responses; NCA Report Series; Washington, DC, USA, 2011; Volume 5c.
  17. Kocaman, S.; Gokceoglu, C. CitSci as a New Approach for Landslide Researches. In Intelligent Systems for Crisis Management; Altan, O., Chandra, M., Sunar, F., Tanzi, T., Eds.; Lecture Notes in Geoinformation and Cartography; Springer: Cham, Switzerland, 2019; pp. 161–183. [Google Scholar]
  18. Kocaman, S.; Gokceoglu, C. A CitSci app for landslide data collection. Landslides 2019, 16, 611–615. [Google Scholar] [CrossRef]
  19. Landslide Reporter. Available online: https://www.nasa.gov/solve/landslide_reporter/ (accessed on 13 April 2019).
  20. Gu, J.; Wang, Z.; Kuen, J.; Ma, L.; Shahroudy, A.; Shuai, B.; Liu, T.; Wang, X.; Wang, G.; Cai, J.; et al. Recent advances in convolutional neural networks. Pattern Recognit. 2018, 77, 354–377. [Google Scholar] [CrossRef] [Green Version]
  21. Fu, Y.; Aldrich, C. Flotation froth image recognition with convolutional neural networks. Miner. Eng. 2019, 132, 183–190. [Google Scholar] [CrossRef]
  22. Dung, C.V.; Sekiya, H.; Hirano, S.; Okatani, T.; Miki, C. A vision-based method for crack detection in gusset plate welded joints of steel bridges using deep convolutional neural networks. Autom. Constr. 2019, 102, 217–229. [Google Scholar] [CrossRef]
  23. Zhang, W.; Chen, Q.; Zhang, W.; He, X. Long-range terrain perception using convolutional neural networks. Neurocomputing 2018, 275, 781–787. [Google Scholar] [CrossRef]
  24. Sadaei, H.J.; Lima e Silva, P.C.; Guimaraes, F.G.; Lee, M.H. Short-term load forecasting by using a combined method of convolutional neural networks and fuzzy time series. Energy 2019, 175, 365–377. [Google Scholar] [CrossRef]
  25. Bayr, U.; Puschmann, O. Automatic detection of woody vegetation in repeat landscape photographs using a convolutional neural network. Ecol. Inform. 2019, 50, 220–233. [Google Scholar] [CrossRef]
  26. Deng, Z.; Sun, H.; Zhou, S.; Zhao, J.; Lei, L.; Zou, H. Multi-scale object detection in remote sensing imagery with convolutional neural networks. ISPRS J. Photogramm. Remote Sens. 2018, 145, 3–22. [Google Scholar] [CrossRef]
  27. Coulibaly, C.; Kamsu-Foguem, B.; Kamissoko, D.; Traore, D. Deep neural networks with transfer learning in millet crop images. Comput. Ind. 2019, 108, 115–120. [Google Scholar] [CrossRef] [Green Version]
  28. Mitra, R.; Marchitto, T.M.; Ge, Q.; Zhong, B.; Kanakiya, B.; Cook, M.S.; Fehrenbacher, J.S.; Ortiz, J.D.; Tripati, A.; Lobaton, E. Automated species-level identification of planktic foraminifera using convolutional neural networks, with comparison to human performance. Mar. Micropaleontol. 2019, 147, 16–24. [Google Scholar] [CrossRef]
  29. Imamverdiyev, Y.; Sukhostat, L. Lithological facies classification using deep convolutional neural network. J. Pet. Sci. Eng. 2019, 174, 216–228. [Google Scholar] [CrossRef]
  30. Palafox, L.F.; Hamilton, C.W.; Scheidt, S.P.; Alvarez, A.M. Automated detection of geological landforms on Mars using Convolutional Neural Networks. Comput. Geosci. 2017, 101, 48–56. [Google Scholar] [CrossRef] [PubMed]
  31. Karimpouli, S.; Tahmasebi, P. Image-based velocity estimation of rock using Convolutional Neural Networks. Neural Netw. 2019, 111, 89–97. [Google Scholar] [CrossRef] [PubMed]
  32. Wu, J.; Yin, X.; Xiao, H. Seeing permeability from images: Fast prediction with convolutional neural networks. Sci. Bull. 2018, 63, 1215–1222. [Google Scholar] [CrossRef]
  33. Wang, Y.; Fang, F.; Hong, F. Comparison of convolutional neural networks for landslide susceptibility mapping in Yanshan County, China. Sci. Total Environ. 2019, 666, 975–993. [Google Scholar] [CrossRef] [PubMed]
  34. Ahmad, K.; Pogorelov, K.; Riegler, M.; Ostroukhova, O.; Halvorsen, P.; Conci, N.; Dahyot, R. Automatic detection of passable roads after floods in remote sensed and social media data. Signal Process. Image Commun. 2019, 74, 110–118. [Google Scholar] [CrossRef] [Green Version]
  35. Liu, Y.; Wu, L. Geological Disaster Recognition on Optical Remote Sensing Images Using Deep Learning. Procedia Comput. Sci. 2016, 91, 566–575. [Google Scholar] [CrossRef] [Green Version]
  36. Zuo, R.; Xiong, Y.; Wang, J.; Carranza, E.J.M. Deep learning and its application in geochemical mapping. Earth Sci. Rev. 2019, 192, 1–14. [Google Scholar] [CrossRef]
  37. Yaseen, Z.M.; Sulaiman, S.O.; Deo, R.C.; Chau, K.-W. An enhanced extreme learning machine model for river flow forecasting: State-of-the-art, practical applications in water resource engineering area and future research direction. J. Hydrol. 2019, 569, 387–408. [Google Scholar] [CrossRef]
  38. Grekousis, G. Artificial neural networks and deep learning in urban geography: A systematic review and meta-analysis. Comput. Environ. Urban Syst. 2019, 74, 244–256. [Google Scholar] [CrossRef]
  39. Huang, L.; Li, J.; Hao, H.; Li, X. Micro-seismic event detection and location in underground mines by using Convolutional Neural Networks (CNN) and deep learning. Tunn. Undergr. Space Technol. 2018, 81, 265–276. [Google Scholar] [CrossRef]
  40. Derakhshani, A.; Foruzan, A.H. Predicting the principal strong ground motion parameters: A deep learning approach. Appl. Soft Comput. 2019. [Google Scholar] [CrossRef]
  41. Asencio-Cortes, G.; Morales-Esteban, A.; Shang, X.; Martínez-Alvarez, F. Earthquake prediction in California using regression algorithms and cloud-based big data infrastructure. Comput. Geosci. 2018, 115, 198–210. [Google Scholar] [CrossRef]
  42. Google. Available online: http://www.google.com (accessed on 23 June 2019).
  43. Flickr. Available online: http://www.flickr.com (accessed on 23 June 2019).
  44. Bing. Available online: http://www.bing.com (accessed on 23 June 2019).
  45. Baidu. Available online: http://www.baidu.com (accessed on 23 June 2019).
  46. Zhou, B.; Lapedriza, A.; Khosla, A.; Oliva, A.; Torralba, A. Places: A 10 Million Image Database for Scene Recognition. IEEE Trans. Pattern Anal. Mach. Intell. 2018, 40, 1452–1464. [Google Scholar] [CrossRef] [PubMed]
  47. Natural Scene Statistics in Vision Science. Available online: http://natural-scenes.cps.utexas.edu/db.shtml (accessed on 24 March 2019).
  48. Geisler, W.S.; Perry, J.S. Statistics for optimal point prediction in natural images. J. Vis. 2011, 11, 14. [Google Scholar] [CrossRef] [PubMed]
  49. Python. Available online: https://www.python.org/ (accessed on 1 June 2019).
  50. OpenCV. Available online: https://opencv.org/ (accessed on 1 June 2019).
  51. Zheng, L.; Zhao, Y.; Wang, S.; Wang, J.; Tian, Q. Good practice in CNN feature transfer. arXiv, 2016; arXiv:1604.00133. [Google Scholar]
  52. Krizhevsky, A.; Sutskever, I.; Hinton, G.E. ImageNet classification with deep convolutional neural networks. In Proceedings of the 25th Neural Information Processing Systems Conference (NIPS 2012), Lake Tahoe, NV, USA, 3–6 December 2012; pp. 1097–1105. [Google Scholar]
  53. Keras: The Python Deep Learning library. Available online: https://keras.io/ (accessed on 24 March 2019).
  54. Girshick, R.; Donahue, J.; Darrell, T.; Malik, J. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, 24–27 June 2014; pp. 580–587. [Google Scholar]
  55. Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv, 2014; arXiv:1409.1556. [Google Scholar]
  56. ILSVRC. Large Scale Visual Recognition Challenge 2014 (ILSVRC 2014). Available online: http://www.image-net.org/challenges/LSVRC/2014/results (accessed on 24 March 2019).
  57. Deng, J.; Socher, R.; Li, F.-F.; Dong, W.; Li, K.; Li, L.J. ImageNet: A large-scale hierarchical image database. In Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Miami, FL, USA, 20–25 June 2009; pp. 248–255. [Google Scholar]
  58. Tieleman, T.; Hinton, G. Lecture 6.5—RmsProp: Divide the gradient by a running average of its recent magnitude. COURSERA Neural Netw. Mach. Learn. 2012, 4, 26–31. [Google Scholar]
  59. Bushaev, V. Stochastic Gradient Descent with Momentum. Towards Data Science, 2017. Available online: https://towardsdatascience.com/stochastic-gradient-descent-with-momentum-a84097641a5d (accessed on 23 June 2019).
  60. Wilson, A.C.; Roelofs, R.; Stern, M.; Srebro, N.; Recht, B. The marginal value of adaptive gradient methods in machine learning. In Proceedings of the Advances in Neural Information Processing Systems (NIPS 2007), Long Beach, CA, USA, 4–9 December 2017; pp. 4151–4161. [Google Scholar]
  61. Xu, B.; Wang, N.; Chen, T.; Li, M. Empirical Evaluation of Rectified Activations in Convolution Network. arXiv, 2015; arXiv:1505.00853. [Google Scholar]
  62. Srivastava, N.; Hinton, G.; Krizhevsky, A.; Sutskever, I.; Salakhutdinov, R. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. J. Mach. Learn. Res. 2014, 15, 1929–1958. [Google Scholar]
  63. Bengio, Y. Practical Recommendations for Gradient-Based Training of Deep Architectures. arXiv, 2012; arXiv:1206.5533. [Google Scholar]
  64. Masters, D.; Luschi, C. Revisiting small batch training for deep neural networks. arXiv, 2018; arXiv:1804.07612. [Google Scholar]
Figure 1. General workflow of the study.
Figure 1. General workflow of the study.
Ijgi 08 00300 g001
Figure 2. Examples of the photos used for training of each class type.
Figure 2. Examples of the photos used for training of each class type.
Ijgi 08 00300 g002
Figure 3. Two examples of the preprocessing and augmentation of the images. (a,b) are the original images, and the square photos are the cropped and augmented ones.
Figure 3. Two examples of the preprocessing and augmentation of the images. (a,b) are the original images, and the square photos are the cropped and augmented ones.
Ijgi 08 00300 g003
Figure 4. The original VGG16 [55] architecture (left) and the proposed architecture of this study (right).
Figure 4. The original VGG16 [55] architecture (left) and the proposed architecture of this study (right).
Ijgi 08 00300 g004
Figure 5. The first step of training in the proposed algorithm.
Figure 5. The first step of training in the proposed algorithm.
Ijgi 08 00300 g005
Figure 6. The second (final) step of the training in the proposed algorithm.
Figure 6. The second (final) step of the training in the proposed algorithm.
Ijgi 08 00300 g006
Figure 7. Training loss and accuracy results for the ten different runs of the model.
Figure 7. Training loss and accuracy results for the ten different runs of the model.
Ijgi 08 00300 g007aIjgi 08 00300 g007b
Figure 8. Normalized confusion matrix obtained from the forth test run (Train–Test Split 4 model).
Figure 8. Normalized confusion matrix obtained from the forth test run (Train–Test Split 4 model).
Ijgi 08 00300 g008
Table 1. Photo sources for training the algorithm.
Table 1. Photo sources for training the algorithm.
Data sourceRemarks
Google Images [42]Search and download images with keywords: Landslides, Earthslip, Rockfall, Landslip, Rockslide, Mudslide (in 92 languages).
Flickr [43]
Bing Images [44]
Baidu Images [45]
Places Dataset [46]High-resolution images with a minimum dimension of 512 pixels.
Natural Image Database [47]Nature Scene Collection and Human Made Scene Collection
Table 2. The classification report of Train-Test Split 4 for all classes.
Table 2. The classification report of Train-Test Split 4 for all classes.
ClassesPrecisionRecallF1-ScoreSupport
Irrelevant1.001.001.0043
Landslides0.940.950.9462
Mountain_valley0.980.980.9849
Natural0.950.910.9345
Road0.980.930.9545
Village0.930.980.9644
Yard0.950.970.9662

Share and Cite

MDPI and ACS Style

Can, R.; Kocaman, S.; Gokceoglu, C. A Convolutional Neural Network Architecture for Auto-Detection of Landslide Photographs to Assess Citizen Science and Volunteered Geographic Information Data Quality. ISPRS Int. J. Geo-Inf. 2019, 8, 300. https://0-doi-org.brum.beds.ac.uk/10.3390/ijgi8070300

AMA Style

Can R, Kocaman S, Gokceoglu C. A Convolutional Neural Network Architecture for Auto-Detection of Landslide Photographs to Assess Citizen Science and Volunteered Geographic Information Data Quality. ISPRS International Journal of Geo-Information. 2019; 8(7):300. https://0-doi-org.brum.beds.ac.uk/10.3390/ijgi8070300

Chicago/Turabian Style

Can, Recep, Sultan Kocaman, and Candan Gokceoglu. 2019. "A Convolutional Neural Network Architecture for Auto-Detection of Landslide Photographs to Assess Citizen Science and Volunteered Geographic Information Data Quality" ISPRS International Journal of Geo-Information 8, no. 7: 300. https://0-doi-org.brum.beds.ac.uk/10.3390/ijgi8070300

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop