Spatial Prediction of Landslides Using Hybrid Integration of Artificial Intelligence Algorithms with Frequency Ratio and Index of Entropy in Nanzheng County, China

Abstract

The main object of this study is to introduce hybrid integration approaches that consist of state-of-the-art artificial intelligence algorithms (SysFor) and two bivariate models, namely the frequency ratio (FR) and index of entropy (IoE), to carry out landslide spatial prediction research. Hybrid integration approaches of these two bivariate models and logistic regression (LR) were used as benchmark models. Nanzheng County was considered as the study area. First, a landslide distribution map was produced using news reports, interpreting satellite images and a regional survey. A total of 202 landslides were identified and marked. According to the previous studies and local geological environment conditions, 16 landslide conditioning factors were chosen for landslide spatial prediction research: elevation, profile curvature, plan curvature, slope angle, slope aspect, stream power index (SPI), topographic wetness index (TWI), sediment transport index (STI), distance to roads, distance to rivers, distance to faults, lithology, rainfall, soil, normalized different vegetation index (NDVI), and land use. Then, the 202 landslides were randomly segmented into two parts with a ratio of 70:30. Seventy percent of the landslides (141) were used as the training dataset and the remaining landslides (61) were used as the validating dataset. Next, the evaluation models were built using the training dataset and compared by the receiver operating characteristics (ROC) curve. The results showed that all models performed well; the FR_SysFor model exhibited the best prediction ability (0.831), followed by the IoE_SysFor model (0.819), IoE_LR model (0.702), FR_LR model (0.696), IoE model (0.691), and FR model (0.681). Overall, these six models are practical tools for landslide spatial prediction research and the results can provide a reference for landslide prevention and control in the study area.

1. Introduction

Landslides, as one of the most frequent geological disasters, have caused casualties, property damage and a series of geological environment problems [1,2]. According to the statistics report of the China Institute of Geo-environmental Monitoring [3], a total of 2966 geological disasters occurred in 2018, including 1631 landslides, resulting in 112 deaths and a direct financial loss of CNY 1.47 billion. Hence, to minimize the damage caused by the landslides, it is necessary to carry out landslide susceptibility research in a region [1,4,5]. Generally, landslide susceptibility can be roughly defined as the probability of landslide occurrence in a certain area under the coupling action of a series of geological environmental factors and human activities [6,7,8,9]. Currently, many approaches have been applied in landslide susceptibility research and their results have been applied to landslide prevention [10,11,12]. According to previous studies [13,14,15,16], the approaches applied in landslide susceptibility assessment can be roughly divided into three types: statistical models, machine learning models and hybrid integration models. The commonly used statistical models are statistical index (SI) [17,18], index of entropy (IoE) [19,20], frequency ratio (FR) [21,22,23], weights of evidence (WoE) [24,25], certainty factor (CF) [26,27], evidential belief function (EBF) [28,29], analytical hierarchy process (AHP) [30,31,32], and logistical regression (LR) [33].

As the prediction ability of the above models has needed to be improved, machine learning methods have been introduced, such as support vector machine (SVM) [34,35,36], random forest (RF) [37,38], adaptive neuro-fuzzy inference systems (ANFIS) [39,40], artificial neural network (ANN) [41,42,43], decision tree (DT) [44,45], and classification and regression tree (CART) [46,47]. Many researchers have gradually found that the prediction ability of a single model is limited [48]. Therefore, it is necessary to develop hybrid integration approaches, such as support vector machine–evidential belief function (SVM_EBF) [49], decision tree–weights of evidence (DT_WoE) [50], adaptive neuro-fuzzy inference systems–statistical index (ANFIS_SI) [51], frequency ratio–logistic regression (FR-LR) [23,52], rotation forest–naive Bayes tree (RF_NBT) [53], and random subspace–classification and regression tree (RS_CART) [54].

In order to obtain an appropriate model that is suitable for the study area, this paper introduces an advanced artificial intelligence algorithm (SysFor) with a frequency ratio and index of entropy to carry out a landslide susceptibility assessment. A logistic regression model is used as the benchmark model to compare the performance of the hybrid models.

2. Methodology

2.1. Frequency Ratio

The frequency ratio (FR) is defined as the ratio between the percentage of landslides and the percentage of pixels within one class [20,55]. The relationship between landslide and the factors is stronger when the frequency ratio is larger than 1 [56,57]. The frequency ratio is calculated by Equation (1).

$$FR=\frac{\frac{NL{S}_{pix}}{{\displaystyle {\sum }_1^{n}NL{S}_{pix}}}}{\frac{N{C}_{pix}}{{\displaystyle {\sum }_1^{n}N{C}_{pix}}}}$$

(1)

where NLSpix is the number of landslides, and NCpix is number of pixels of a class.

2.2. Index of Entropy

The second model is the entropy index (IoE), which is based on the bivariate analysis principle [58,59]. The model can calculate the weight of each input variable, and the weight can show which variable is most relevant to the occurrence of landslides in the natural environment [20]. The weights of each variable obtained are taken as the entropy index [60]. The equations used to calculate the weight of each variable are shown as Equations (2)–(8).

$$P_{ij}=\frac{b}{a}$$

(2)

$$(P_{ij})=\frac{P_{ij}}{{\displaystyle \sum _{j=1}^{S_j}{P}_{ij}}}$$

(3)

$$H_j=-{\displaystyle \sum _{i=1}^{S_j}(P_{ij})}{\log }_2(P_{ij})\hspace{1em}j=1,2,3,\dots ,n$$

(4)

$$H_{j\max }={\log }_2{S}_j$$

(5)

$$I_j=\frac{H_{j\max }-H_j}{H_{j\max }},I=(0,1)\hspace{1em}j=1,2,3,\dots ,n$$

(6)

$$P_j=\frac{1}{S_j}{\displaystyle \sum _{i=1}^{S_j}{P}_{ij}}$$

(7)

$$W_j=I_j\times P_j$$

(8)

where a is the percentage of the defined domain; b is the landslide percentage; P_ij is the probability density; H_jmax and H_j are both entropy values; S_j is the number of classes; I_j is the information coefficient; and W_j is the weight for the variable as a whole.

2.3. SysFor

The systematically developed forest of multiple trees (SysFor) is a data mining algorithm that is based on the concept of the gain ratio and was proposed by Islam and Giggins in 2011 [61,62]. Compared with the commonly used techniques, the SysFor applies both high-dimensional and low-dimensional data sets. In addition, it also shows a higher prediction accuracy than the common techniques [63]. Generally, the SysFor is built with the following four steps: (1) According to the user-defined gain ratio and separation value, a set of good attributes and their segmentation points are identified. (2) If the number of good attributes is smaller than the number of user-defined trees, each attribute is selected as the root attribute of the tree, and the number of trees built is equal to the number of good attributes. (3) If the number of trees generated in step 2 is less than the number of user-defined trees, more trees are built. (4) All trees built in steps (2) and (3) are returned as the SysFor.

2.4. Logistic Regression

Logistic regression is a multivariable method [64,65,66]. The main goal of a logistic regression model is to obtain the most suitable method to determine whether there are landslides using particular variables [47,67]. The relationship between landslide occurrence and the variables is shown in Equation (9).

$$P=\frac{1}{1+\exp (-Z)}$$

(9)

where P is the probability of landslide occurrence, and Z is the linear sum, which is obtained by the product of independent variables and their coefficients. The calculation of Z is shown in Equation (10).

$$Z=\alpha +{\beta }_1{x}_1+{\beta }_2{x}_2+{\beta }_3{x}_3+\dots +{\beta }_n{x}_n$$

(10)

where α is a constant; β_i (i = 1, 2, 3… n) are the coefficients; and x_i (i = 1, 2, 3… n) are the independent variables.

3. Study Area and Data Used

The study area (Nanzheng County) lies in Shaanxi Province, China (Figure 1). It is located between longitude 106°30′ and 107°22′ E and latitude 32°24′ to 33°07′ N. It covers an area of 2823 square kilometers. The altitude of the study area is between 442 and 2410 m. The study area has a subtropical monsoon climate, with an annual average temperature of 14.2 °C. Rainfall is mainly concentrated from June to September and the mean annual rainfall is 909.8 mm.

A landslide inventory can give insight into landslide location, dates, type, and damage caused [68,69,70]. In this study, the landslide inventory map was prepared on the basis of historical landslide records and satellite images (Google Earth and ZY03 images). A total of 202 landslides were identified, including 190 slides and 12 rock falls [71]. The largest landslide was more than 1,000,000 m³, and the smallest landslide was nearly 160 m³ [70]. Finally, 141 landslides were randomly selected as training and validation datasets with a ratio of 70:30 (Figure 1).

There is no clear agreement about the precise cause of landslides due to their complex nature and development. According to the latest relevant research and local geological environment characteristics [72,73,74,75,76], 16 landslide conditioning factors have been compiled: elevation, profile curvature, plan curvature, slope angle, slope aspect, stream power index (SPI), topographic wetness index (TWI), sediment transport index (STI), distance to roads, distance to rivers, distance to faults, lithology, rainfall, soil, normalized different vegetation index (NDVI), and land use. The 16 landslide susceptibility conditioning factors were converted with a resolution of 30 × 30 m.

Elevation influences earth surface and topographic attributes which account for spatial variability of precipitation, soil thickness, erosion, and vegetation [38]. Profile curvature and plan curvature have significant effects on surface runoff and infiltration [77]. Slope angle is an important factor that controls the velocity of the slopes, and the slope aspect has an important effect on rainfall, wind and sunlight exposure [78]. SPI, TWI and STI are related to soil water content status, water accumulation and progress of erosion and sedimentation in a watershed, which influence landslide stability [77,78]. In the present study, the elevation map was acquired from a 30 × 30 m digital elevation model (DEM) and reclassified into 10 classes: 442–600 m, 600–800 m, 800–1000 m, 1000–1200 m, 1200–1400 m, 1400–1600 m, 1600–1800 m, 1800–2000 m, 2000–2200 m, and 2200–2410 m (Figure 2a). Similarly, the profile curvature, plan curvature, slope angle, slope aspect, SPI, TWI, and STI were also extracted from the DEM by ArcGIS. In the study, the profile curvature was grouped into three classes of −14.28 to −0.05, −0.05 to 0.05, and 0.05 to 14.77 (Figure 2b). The plan curvature was divided into three categories of −14.0 to −0.05, −0.05 to 0.05, and 0.05 to 13.07 (Figure 2c). The slope angle was grouped into seven categories: 0–10°, 10–20°, 20–30°, 30–40°, 40–50°, 50–60°, and 60–72.83° (Figure 2d). The slope aspect was reclassified into nine classes (Figure 2e). The SPI values were grouped into five classes: <20, 20–40, 40–60, 60–80, and >80 (Figure 2f). The TWI was reclassified into five categories: <4, 4–5, 5–6, 6–7, and >7 (Figure 2g). The STI values were divided into five classes: <10, 10–20, 20–30, 30–40, and >40 (Figure 2h).

Distance to roads, distance to rivers and distance to faults are three commonly used landslide conditioning factors, these factors are related to the infiltration and strength of slopes [79,80]. Distance to roads, distance to rivers and distance to faults are divided into five categories by the equal distance method, as shown in Figure 2i–k, respectively. Lithology is one of the most important restrictive factors in landslide susceptibility evaluation; areas with highly resistant rocks or highly permeable subsoil material have low drainage density [81], which are natural factors essential to determine landslide occurrence [82,83]. The geological map of this study was extracted from a geological map with 1:1,000,000 scale. The study area has 12 different lithological units, which are shown in Figure 2l. Rainfall is a widely recognized landslide-inducing factor, which not only increases the weight of the slope, but also reduces soil strength [84,85]. From Figure 2m, it can be seen that the rainfall in the middle of the study area is significantly higher than that in the north and south. The rainfall map was reclassified with an interval of 100 mm/yr. Soil is the material composition of the slope, and different soils have different physical and mechanical properties. Soil type was extracted from the geological map with a 1:1,000,000 scale, as shown in Figure 2n, and there are nine kinds of soils in the study area. NDVI is a measure of surface reflectance and gives a quantitative estimate of biomass and the vegetation growth [38,86,87,88]. The NDVI values in this paper were reclassified into five categories: −0.21 to 0.21, 0.21 to 0.36, 0.36 to 0.44, 0.44 to 0.52, and 0.52 to 0.65 (Figure 2o). Land use is considered as the direct manifestation of human activities impacting on landslide probability. Land use types control the amount of infiltration and surface runoff generation [77]. In general, landslides are concentrated in human active areas [89,90]. In this study, a land use map was extracted from regional land use maps with a 1:100,000 scale. Land use was divided into farmland, forestland, grassland, water, residential areas, and bare land (Figure 2p).

4. Results

As the LR method was employed to produce the landslide susceptibility map, it is necessary to analyze the multicollinearity of landslide conditioning factors [91]. Currently, the tolerance (TOL) and variance inflation factor (VIF) are mostly used in multicollinearity analysis [92,93]. Generally, a TOL < 0.1 or a VIF > 10 indicate multicollinearity [72]. In the study, the results indicate that there are no multicollinearities among the 16 landslide conditioning factors (

Table 1. Multicollinearity analysis.

Parameters	Collinearity Statistics
	Tolerance	VIF	Tolerance	VIF
Elevation	0.543	1.843	0.507	1.970
Profile curvature	0.897	1.115	0.897	1.115
Plan curvature	0.824	1.214	0.801	1.248
Slope angle	0.764	1.308	0.759	1.317
Slope aspect	0.912	1.096	0.904	1.106
SPI	0.524	1.907	0.464	2.157
TWI	0.830	1.205	0.834	1.200
STI	0.479	2.086	0.426	2.346
Distance to roads	0.817	1.224	0.823	1.214
Distance to rivers	0.952	1.050	0.924	1.083
Distance to faults	0.926	1.080	0.731	1.368
Lithology	0.729	1.372	0.734	1.362
Rainfall	0.741	1.349	0.802	1.247
Soil	0.805	1.242	0.687	1.456
NDVI	0.847	1.180	0.639	1.566
Land use	0.640	1.562	0.507	1.970

).

4.1. Application of FR Model

The frequency ratio (FR) is a commonly used method of univariate probability analysis in landslide susceptibility assessment [21,25]. When the frequency ratio is 1, it represents the average value; when the frequency ratio is greater than 1, it indicates that the factor has a strong correlation with the occurrence of the landslide; when the frequency ratio is less than 1, it indicates that the correlation between the factor and the occurrence of the landslide is weak. The landslide susceptibility index (LSI) calculated by the FR model can be expressed as Equation (11), as shown in

Table 2. Relationship between landslides and conditioning factors using FR and IoE models.

Factors	Class	Percentage of Domain	Percentage of Landslides	FR	Wj
Elevation (m)	442–600	13.242	19.858	1.500	0.223
	600–800	16.399	34.043	2.076
	800–1000	12.091	21.986	1.818
	1000–1200	10.450	12.057	1.154
	1200–1400	12.756	10.638	0.834
	1400–1600	12.341	1.418	0.115
	1600–1800	12.042	0.000	0.000
	1800–2000	7.920	0.000	0.000
	2000–2200	2.504	0.000	0.000
	2200–2410	0.255	0.000	0.000
Profile curvature	−14.28 to −0.05	45.753	46.809	1.023	0.083
	−0.05–0.05	5.696	11.348	1.992
	0.05–14.77	48.551	41.844	0.862
Plan curvature	−14.0 to −0.05	46.110	41.135	0.892	0.014
	−0.05–0.05	6.893	9.220	1.338
	0.05–13.07	46.996	49.645	1.056
Slope angle (°)	0–10	23.641	25.532	1.080	0.146
	10–20	29.270	44.681	1.526
	20–30	26.292	20.567	0.782
	30–40	14.987	7.801	0.521
	40–50	4.975	1.418	0.285
	50–60	0.780	0.000	0.000
	60–72.83	0.055	0.000	0.000
Slope aspect	Flat	0.028	0.000	0.000	0.075
	North	14.212	11.348	0.798
	North-east	12.976	11.348	0.875
	East	12.046	12.057	1.001
	South-east	12.505	16.312	1.304
	South	11.982	22.695	1.894
	South-west	11.044	9.220	0.835
	West	11.355	6.383	0.562
	North-west	13.853	10.638	0.768
SPI	<20	55.734	62.411	1.120	0.010
	20–40	15.930	14.184	0.890
	40–60	7.404	8.511	1.149
	60–80	4.284	3.546	0.828
	>80	16.649	11.348	0.682
TWI	<4	17.880	7.801	0.436	0.046
	4–5	32.049	35.461	1.106
	5–6	23.903	34.043	1.424
	6–7	12.599	14.184	1.126
	>7	13.569	8.511	0.627
STI	<10	55.157	63.830	1.157	0.024
	10–20	22.491	22.695	1.009
	20–30	9.447	4.255	0.450
	30–40	4.524	3.546	0.784
	>40	8.381	5.674	0.677
Distance to roads (m)	<300	11.010	23.404	2.126	0.065
	300–600	8.951	11.348	1.268
	600–900	7.852	5.674	0.723
	900–1200	7.036	10.638	1.512
	>1200	65.151	48.936	0.751
Distance to rivers (m)	<200	16.686	19.149	1.148	0.003
	200–400	14.837	15.603	1.052
	400–600	13.695	12.766	0.932
	600–800	12.002	13.475	1.123
	>800	42.781	39.007	0.912
Distance to faults (m)	<1000	21.510	22.695	1.055	0.022
	1000–2000	16.106	12.766	0.793
	2000–3000	13.198	14.894	1.129
	3000–4000	11.168	5.674	0.508
	>4000	38.019	43.972	1.157
Lithology	Group 1	11.627	19.149	1.647	0.117
	Group 2	0.054	0.000	0.000
	Group 3	4.383	1.418	0.324
	Group 4	12.756	4.255	0.334
	Group 5	0.239	0.000	0.000
	Group 6	3.453	0.000	0.000
	Group 7	7.231	3.546	0.490
	Group 8	10.228	7.092	0.693
	Group 9	8.846	6.383	0.722
	Group 10	1.254	0.709	0.566
	Group 11	13.945	22.695	1.627
	Group 12	25.982	34.752	1.338
Rainfall (mm/yr)	<900	6.069	5.674	0.935	0.111
	900–1000	18.642	20.567	1.103
	1000–1100	9.029	16.312	1.807
	1100–1200	10.544	24.823	2.354
	1200–1300	8.680	11.348	1.307
	1300–1400	20.159	12.766	0.633
	1400–1500	11.247	4.965	0.441
	1500–1600	8.670	2.128	0.245
	1600–1700	4.343	0.709	0.163
	>1700	2.618	0.709	0.271
Soil	ATc	11.538	14.184	1.229	0.326
	CMd	3.647	2.837	0.778
	CMe	7.992	21.986	2.751
	FLc	1.186	0.000	0.000
	LVh	70.808	56.738	0.801
	LVx	0.322	0.000	0.000
	PLe	0.475	2.128	4.479
	RGc	2.630	0.709	0.270
	RGe	1.403	1.418	1.011
NDVI	−0.21 to 0.21	2.161	2.837	1.313	0.128
	0.21–0.36	6.660	7.092	1.065
	0.36–0.44	20.845	44.681	2.144
	0.44–0.52	34.881	39.716	1.139
	0.52–0.65	35.454	5.674	0.160
Land use	Farmland	28.826	60.993	2.116	0.311
	Forestland	30.974	2.837	0.092
	Forestland	38.500	36.170	0.939
	Water	0.603	0.000	0.000
	Residential areas	1.075	0.000	0.000
	Bare land	0.022	0.000	0.000

.

$$\begin{array}{ll}LS{I}_{FR}=& (Elevatio{n}_{FR})+(Profile curvatur{e}_{FR})+(Plan curvatur{e}_{FR})+\\ & (Slope angl{e}_{FR})+(Slope aspec{t}_{FR})+(SP{I}_{FR})+(TW{I}_{FR})+(ST{I}_{FR})+\\ & (Distance to road{s}_{FR})+(Distance to river{s}_{FR})+(Distance to fault{s}_{FR})+\\ & (Litholog{y}_{FR})+(Rainfal{l}_{FR})+(Soi{l}_{FR})+(NDV{I}_{FR})+(Landus{e}_{FR})\end{array}$$

(11)

4.2. Application of IoE Model

The index of entropy (IoE) is based on the bivariate analysis principle. The weight of each conditioning factor in landslides can be calculated by this method. From Table 2, it can be seen that soil has the highest weight (0.326), followed by land use (0.311), elevation (0.223), slope angle (0.146), NDVI (0.128), lithology (0.117), rainfall (0.111), profile curvature (0.083), slope aspect (0.075), distance to roads (0.065), TWI (0.046), STI (0.024), distance to faults (0.022), plan curvature (0.014), SPI (0.01), and distance to rivers (0.003). The LSI calculated by the IoE model can be represented as Equation (12).

$$\begin{array}{ll}LS{I}_{IoE}=& (Elevatio{n}_{FR}\ast 0.223)+(Profile curvatur{e}_{FR}\ast 0.083)+\\ & (Plan curvatur{e}_{FR}\ast 0.014)+(Slope angl{e}_{FR}\ast 0.146)+\\ & (Slope aspec{t}_{FR}\ast 0.075)+(SP{I}_{FR}\ast 0.010)+(TW{I}_{FR}\ast 0.046)+\\ & (ST{I}_{FR}\ast 0.024)+(Distance to road{s}_{FR}\ast 0.065)+\\ & (Distance to river{s}_{FR}\ast 0.003)+(Distance to fault{s}_{FR}\ast 0.022)+\\ & (Litholog{y}_{FR}\ast 0.117)+(Rainfal{l}_{FR}\ast 0.111)+(Soi{l}_{FR}\ast 0.326)+\\ & (NDV{I}_{FR}\ast 0.128)+(Landus{e}_{FR}\ast 0.311)\end{array}$$

(12)

4.3. Application of Hybrid Models

Compared with a single model, a hybrid model has higher predictive ability when dealing with high-dimensional problems [94,95,96]. In this study, the SysFor algorithm was combined with FR and IoE models. During the modeling process, the following parameters were used for hybrid FR_SysFor and IoE_SysFor models: separation, 0.3; confidence, 0.25; goodness, 0.3; number of trees built in the forest, 500.

The hybrid integration of LR with FR and IoE was also applied as a benchmark model to build landslide susceptibility maps. A forward stepwise LR was adopted, and the analysis results are given in

Table 3. Coefficients of models.

Parameters	Coefficient of FR_LR Model	Coefficient of IoE_LR Model
Elevation	0.596	1.978
Profile curvature	0.005	−0.626
Plan curvature	1.390	72.424
Slope angle	0.182	1.042
Slope aspect	0.633	9.670
SPI	0.370	66.354
TWI	0.729	1.570
STI	0.381	7.543
Distance to roads	0.436	5.933
Distance to rivers	−0.722	——
Distance to faults	−0.060	−4.887
Lithology	−0.138	−1.090
Rainfall	0.607	5.141
Soil	−0.030	0.008
NDVI	0.275	5.007
Land use	0.399	1.260
Constant	−5.702	−6.400

. The landslide occurrence probability $P$ of FR_LR and IoE_LR models can be expressed using Equations (13)–(16), respectively.

$$\begin{array}{ll}{Z}_{FR-LR}=& (Elevatio{n}_{FR}\ast 0.596)+(Profile curvatur{e}_{FR}\ast 0.005)+\\ & (Plan curvatur{e}_{FR}\ast 1.390)+(Slope angl{e}_{FR}\ast 0.182)+\\ & (Slope aspec{t}_{FR}\ast 0.633)+(SP{I}_{FR}\ast 0.370)+(TW{I}_{FR}\ast 0.729)+\\ & (ST{I}_{FR}\ast 0.381)+(Distance to road{s}_{FR}\ast 0.436)+\\ & (Distance to river{s}_{FR}\ast (-0.722))+(Distance to fault{s}_{FR}\ast (-0.060))+\\ & (Litholog{y}_{FR}\ast (-0.138))+(Rainfal{l}_{FR}\ast 0.607)+(Soi{l}_{FR}\ast (-0.030))+\\ & (NDV{I}_{FR}\ast 0.275)+(Landus{e}_{FR}\ast 0.399)-5.702\end{array}$$

(13)

$$P_{FR-LR}=\frac{e^{Z_{FR-LR}}}{1+e^{Z_{FR-LR}}}$$

(14)

$$\begin{array}{ll}{Z}_{IoE-LR}=& (Elevatio{n}_{IoE}\ast 1.978)+(Profile curvatur{e}_{IoE}\ast (-0.626))+\\ & (Plan curvatur{e}_{IoE}\ast 72.424)+(Slope angl{e}_{IoE}\ast 1.042)+\\ & (Slope aspec{t}_{IoE}\ast 9.670)+(SP{I}_{IoE}\ast 66.354)+(TW{I}_{IoE}\ast 1.570)+\\ & (ST{I}_{IoE}\ast 7.543)+(Distance to road{s}_{IoE}\ast 5.933)+\\ & (Distance to fault{s}_{IoE}\ast (-4.887))+(Litholog{y}_{IoE}\ast (-1.090))+\\ & (Rainfal{l}_{IoE}\ast 5.141)+(Soi{l}_{IoE}\ast 0.008)+\\ & (NDV{I}_{IoE}\ast 5.007)+(Landus{e}_{IoE}\ast 1.260)-6.400\end{array}$$

(15)

$$P_{IoE-LR}=\frac{e^{Z_{IoE-LR}}}{1+e^{Z_{IoE-LR}}}$$

(16)

Finally, all the landslide susceptibility maps were reclassified into five categories using the equal area classification method [92,93]: very high (5%), high (10%), moderate (15%), low (20%), and very low (50%) (Figure 3).

4.4. Validation of Landslide Susceptibility Maps

Validating the landslide susceptibility map (LSM) and determining its accuracy is an important aspect of landslide susceptibility research. The verification of landslide susceptibility maps is very important. Without it, the rigor of the study is greatly weakened, and the results of the study have no scientific significance [97,98,99]. In consequence, three commonly used statistical parameters, the receiver operating characteristics (ROC) curve [100,101,102] and the area under the curve (AUC) [103,104,105], standard error, and 95% confidence interval are introduced to verify the landslide susceptibility maps. It can be seen that the FR_SysFor model acquired the highest AUC value (0.940) of all hybrid models in the training data, followed by the IoE_SysFor model (0.926), IoE_LR model (0.783) and FR_LR model (0.779) (Figure 4). With the exception of the AUC value, the other statistical parameters in

Table 4. Parameters of ROC curves using training data.

Variable	AUC	SE	95% CI
FR_model	0.757	0.0285	0.702 to 0.806
IoE_model	0.746	0.0292	0.691 to 0.796
FR_SysFor_model	0 940	0.0132	0.906 to 0.965
IoE_SysFor_model	0.926	0.0151	0.889 to 0.954
FR_LR_model	0.779	0.0271	0.726 to 0.826
IoE_LR_model	0.783	0.0270	0.730 to 0.829

and

Table 5. Parameters of ROC curves using validating data.

Variable	AUC	SE	95% CI
FR_model	0.681	0.0498	0.590 to 0.762
IoE_model	0.691	0.0489	0.601 to 0.771
FR_SysFor_model	0.831	0.0388	0.753 to 0.893
IoE_SysFor_model	0.819	0.0399	0.739 to 0.883
FR_LR_model	0.696	0.0485	0.606 to 0.776
IoE_LR_model	0.702	0.0483	0.612 to 0.781

show similar results; the FR_SysFor model obtained the smallest SE (0.0132) and 95% CI (0.906, 0.965), followed by the IoE_SysFor, IoE_LR and FR_LR models. The validating data showed similar results to the training data, and the FR_SysFor model achieved the highest AUC value (0.831) and the smallest SE (0.0388) and 95% CI (0.753, 0.893).

In addition, the success rate curves and prediction rate curves of the landslide susceptibility maps were also assessed (Figure 5 and Figure 6). It can be seen that the FR_SysFor model recognized a highly susceptible area with a success rate curve that contains more than 80% of landslides. Similarly, the highly susceptible area with the prediction rate curve recognized by the FR_SysFor model also includes more than 70% of landslides, which illustrates that the precision of the FR_SysFor model is the highest and that the FR_SysFor model is the best model in this study.

5. Discussion

Currently, some novel ensemble techniques have been proposed in landslide susceptibility mapping, and the excellent performance of ensemble techniques has been proven [92,106,107,108,109]. Furthermore, the hybrid integration of machine learning algorithms with bivariate statistical models can weaken the hypotheses of the conventional bivariate models and retain the merits of bivariate statistical models and machine learning models [110].

Compared with SysFor, FR, IoE and LR are three common evaluation models which have the merits of model stabilization, higher accuracy and simple calculation. Razavizadeh et al. proposed a GIS-based landslide susceptibility mapping with frequency ratio, statistical index, and weights of evidence models for a part of Mazandaran Province, Iran. Both success rate curve and prediction rate curve demonstrated that the frequency ratio (FR) is a reliable model with the highest accuracy. Moreover, the landslide susceptibility map generated by the FR model is trustworthy for hazard mitigation strategies [25]. Pourghasemi et al. introduced the index of entropy (IoE) and conditional probability models to landslide susceptibility research in Safarood Basin, Iran. The results indicated that both models have good predictive capacity, while the IoE performed slightly better than the conditional probability model in landslide susceptibility mapping [111]. Abedini et al. applied logistic regression (LR) and AHP models to landslide susceptibility assessment. The results indicated that the LR is a suitable model to classify and estimate the probability of landslide occurrence in the process of project research planning and implementation [112].

In the present study, for the class of 600–800 m of elevation (ratio value = 2.650), distance to roads <300 m (ratio value = 2.126) and 1100–1200 mm/yr of rainfall (ratio value = 2.354) facilitated the occurrence of landslides. The results based on the IoE model showed that soil (0.326), land use (0.311), elevation (0.223), and slope angle (0.143) are the most important factors, which are closely related to the occurrence and spatial distribution of landslides compared with other factors. On the contrary, distance to faults, plane curvature, SPI, and distance to rivers achieved the four lowest weight values, which were 0.022, 0.014, 0.003, and 0.014, respectively. However, the classifications of landslide conditioning factors were based on previous studies and might not be suitable for the present study. Therefore, further studies should be conducted to find an objective classification method for landslide conditioning factors [113,114]. In order to obtain more reliable landslide susceptibility maps, the significant differences of six landslide susceptibility methods were calculated and compared. According to the relevant research [58,115], the most common and effective methods include the receiver operating characteristics (ROC) curve, standard error (SE), 95% confidence interval (CI), and Wilcoxon signed-rank tests. Hence, the paper introduced three well known statistical parameters—the ROC curve, SE and 95% CI—to calculate and compare the model performance. For the training data, the results of the ROC curve, SE and 95% CI can be seen in Table 4. The FR_SysFor model performed best and acquired the highest AUC value (0.940), followed by IoE_SysFor, IoE_LR, FR_LR, FR, and IoE models with AUC values of 0.926, 0.783, 0.779, 0.757, and 0.746, respectively. Similarly, the two other statistical parameters showed the same results; the FR_SysFor model obtained the smallest SE and 95% CI. For the validating data, the parameters of the ROC curve, SE and 95% CI are shown in Table 5. The validating data showed that the FR_SysFor model had higher accuracy (AUC = 0.831) than the remaining models and performed best in the research. Meanwhile, the smallest SE and 95% CI also belonged to the FR_SysFor model.

Overall, based on the FR and IoE models, combined with SysFor and LR models, the landslide susceptibility of Nanzheng County was studied. As described above, the FR_SysFor model performed the best in this study as compared to other models. Finally, the ensembles of FR and IoE with the proposed algorithm (SysFor) can provide a reference for landslide susceptibility research in other areas.

In practice, landslide hazard managers can employ the FR_SysFor model to determine regions with high and very high susceptibility in the study area. An early warning system for landslide occurrence can transmit useful awareness and warning information for residents living in these areas [107]. Furthermore, the landslide susceptibility map of the present study can help to construct retaining wall systems and anchor systems to enhance slope stability [107].

6. Conclusions

In this case study, six landslide susceptibility evaluation methods, namely, the FR, IoE, FR_SysFor, IoE_SysFor, FR_LR, and IoE_LR models, were systematically analyzed and compared as part of landslide susceptibility research in Nanzheng Country (China). Based on previous research and the geological environmental characteristics in the study area, 16 conditioning factors were selected for the research: elevation, profile curvature, plan curvature, slope angle, slope aspect, SPI, TWI, SPI, distance to roads, distance to rivers, distance to faults, lithology, rainfall, soil, NDVI, and land use. These models were applied to the calculation of landslide occurrence probability for the first time. Finally, the model performances were compared by the statistical parameters of ROC curves, AUC values, SE and 95% CI. The results show that all the models perform well. Compared with other models, the prediction capability of the FR_SysFor model is the highest, with a success rate of 0.940 and prediction rate of 0.831. Hence, the FR_SysFor model is considered the most promising technique in this study. The results can provide a reference for land-use planning and decision-making in the study area.