2.1.

Study Area

The study area was located in the western part of Tokachi plain, Hokkaido, Japan (Fig. 1, 142°42′51″ to 143°08′47″ E, 42°43′20″ to 43°07′24″ N). Main cultivated crop types are beans, beetroots, grasses, maize, potatoes, and winter wheat. The average monthly temperatures were 8.3°C to 21.8°C and monthly precipitation was 12.0 to 94.5 mm from May to October.

Fig. 1

Study area and the distribution of croplands (background map shows Sentinel-2A data obtained on August 11, 2016, R: band 4, G: band 3, and B: band 2).

Field location and attribute data, such as crop types, were based on manual surveys and provided by Tokachi Nosai (Obihiro, Hokkaido) as a polygon-shaped file. A total of 12,639 fields [2265 beans fields, 1548 beetroot fields, 2110 grasslands (timothy and orchard grass), 1000 maize fields, 2452 potato fields, and 3264 winter wheat fields] were observed. The fields ranged from 0.05 to 18.21 ha with an averaged value of 2.54 ha. Grasslands were located on the outskirts of the built-up area.

2.2.

Remote Sensing Data

The data acquired from Sentinel-2 MSI contained blue, green, red, and near-infrared-1 bands at 10 m; red edge 1 to 3, near-infrared-2, and SWIR 1 and 2 at 20 m; and three atmospheric bands (band 1, band 9, and band 10) at 60 m. In this study, the three atmospheric bands were removed, because they were dedicated to atmospheric corrections and cloud screening.²⁸

Although Sentinel-2A imagery was gathered seven times from May to September 2016, for the whole site, all images were covered with clouds except for one acquired on 11 August. The level 1C data acquired on August 11, 2016, were downloaded from EarthExplorer.²⁹ All bands were converted to 10-m resolution with a cubic convolution resampling method and average reflectance values of each band were calculated for each field using the field polygons to compensate for spatial variability and to avoid problems related to uncertainty in georeferencing.

Some vegetation indices, such as NDVI, have been used for improving classification accuracies in previous studies.^16,22,30,31 About 82 published vegetation indices for evaluating various vegetation properties were calculated in this study (Table 1).

Table 1

Vegetation indices calculated from Sentinel-2 MSI data.

2.3.

Classification Algorithm

All samples were divided into the following three groups using a stratified random sampling approach: training data (50%) for developing classification models, validation data (25%) for hyperparameter tuning, and test data (25%) for evaluation of classification accuracies⁸⁶ and Table 2 shows the numbers of fields of each crop type.

Table 2

Crop type and number of fields.

SVM partitions data using maximum separation margins⁸⁷ and the “kernel trick” has frequently been applied instead of attempting to fit a nonlinear model in previous studies.³⁰ In this study, the Gaussian radial basis function kernel, which has mostly been used for classification purposes,³⁰ was used as a kernel and two parameters were tuned to control the flexibility of the classifier, the regularization parameter $C$, and the kernel bandwidth $\gamma$. If the $C$ value is too large, there is a high penalty for no separable points, and we may store many support vectors and overfit. If it is too small, there may be underfitting. It controls the trade-off between errors of the SVM on training data and margin maximization ($C=\infty$ leads to hard margin SVM). The $\gamma$ value defines how far the influence of a single training example reaches, with low values meaning “far” and high values meaning “close.”

RF is an ensemble learning technique composed of multiple decision trees based on random bootstrapped samples of the training data.⁸⁸ The output is determined by a majority vote of the results of decision trees. There are two user-defined hyperparameters including the number of trees (ntree) and the number of variables used to split the nodes (mtry). If ntree is made larger, the generalization error always converges, and over-training will not be a problem. On the other hand, a reduction in mtry makes each individual decision tree weaker.

The best combinations of these hyperparameters were determined using the Gaussian process, Bayesian optimization,⁸⁹ which has been widely applied for hyperparameter tuning of machine learning algorithms.¹

Ensemble machine learning methods have been used to obtain better predictive performance than from single learning algorithms, and the SL methodology has been proposed.⁹⁰ In this method, given algorithms are combined through a convex weighted combination to minimize cross-validated errors. First, classification models based on RF or SVM were trained as the base algorithms using the training data. Next, a 10-fold cross validation was performed on each and the cross-validated predicted results were obtained. $N$ is the number of rows in the training data, cross-validated predicted results were combined, and an $N$ by two matrices was obtained as the “level-one” data and meta-learning model was generated. To predict the test data, the predictions from the base learners were fed into the metalearning model to generate the ensemble prediction. The data-based sensitivity analysis (DSA),⁹¹ which performs a pure black box use of the fitted models by querying the fitted models with sensitivity samples and recording their responses, was applied for assessing the sensitivity of the classification models.

2.4.

Accuracy Assessment

Classification accuracies were evaluated based on the simple measures of quantity disagreement (QD) and allocation disagreement (AD).⁹² They provide an effective summary of confusion matrices.⁹³

The proportion of fields that are classified as crop $i$ and their actual classes are crop $j$ ($P_{ij}$) is expressed in the following

In addition, three indicators including overall accuracy [OA, Eq. (6)], producer’s accuracy [PA, Eq. (7)], and user’s accuracy [UA, Eq. (8)] were calculated because they have widely been applied for assessing classification accuracies

3.1.

Classification Accuracy

Crop classification maps are shown in Fig. 2, the maximum, minimum, and averaged accuracies of 10 repetitions and confusion matrices when all the repetitions were merged are shown in Tables 3 and 4. Averaged OAs were 89.0% for RF, 90.6% for SVM, and 91.6% for the ensemble machine learning method and the mean PAs and mean UAs derived using the machine learning algorithms were $>0.8$, excepting those of RF (mean UA for maize was 0.797). All machine learning algorithms performed well in classifying croplands. Especially, the good accuracies were confirmed for the PAs and UAs for wheat ($>93.8\%$) and beet ($>89.9\%$). However, the chi-square values based on McNemar’s tests were 12.02 to 40.60, 27.78 to 62.43, and 17.00 to 51.60 for R—SVM, RF—SL, and SVM—SL, respectively. As a result, significant differences were confirmed among the results of three machine learning algorithms ($p<0.05$).

Fig. 2

Crop classification map generated by (a) RF, (b) SVM, and (c) SL.

Table 3

Classification accuracies of each algorithm.

Table 4

Confusion matrices for (a) RF, (b) SVM, and (c) SL.

Classification results by SL had the best OA and AD + QD (8.5%) and SVM had a slightly better PA of wheat (97.1%). On the contrary, identifying maize fields was difficult due to the similarity in their reflectance. Grasses cultivation employs fewer controls and then a lot of weeds were mixed with timothy and orchard grass in grasslands. As a result, variation in reflectance features was larger than in other crop types, causing misclassifications of relatively large fields.

Figure 3 shows the relationship between field area and misclassified fields for each algorithm after 10 repetitions (i.e., the total number is 10 times of that of the test data). More than 75% of the misclassified fields were $<200$ a in area for all algorithms, and 95.1% (RF), 95.5% (SVM), and 96.1% (SL) of misclassified fields were below 450 a. Applying stacking made the model more robust for classifying smaller fields and the number of misclassified croplands decreased (813 fields for smaller than 50 a) compared with the results by RF (909 fields for smaller than 50 a) and SVM (855 fields for smaller than 50 a). It was especially useful for identifying beans fields. It was not effective for identifying small grasslands as grass cultivation employs fewer controls and many weeds were present in grasslands. However, stacking was useful for identifying grasslands more than 500 a, which had a certain homogeneity with Dactylis glomerata or Phleum pretense in the MSI image.

Fig. 3

Relationship between field area and misclassified fields (a) RF, (b) SVM, and (c) SL.

3.2.

Sensitive Factor Analysis

Reflectance values obtained from Sentinel-2A are shown in Fig. 4. Differences in reflectance were particularly clear between wheat and beans as the wheat harvest was finished on 11 August and the reflectance of wheat fields was similar to that of bare soil. Beetroot had the steepest gradient between bands 5 and 6 and some differences in the reflectance values at band 11 were confirmed between maize and potato. Differences in the reflectance patterns between grass and beans were not clear.

Fig. 4

Mean reflectance spectra and standard deviations of each crop.

To clarify which variables contributed to identifying each crop type, DSA was conducted for each algorithm and their importance values were calculated.

For identifying beans fields, Datt3 (6.0%, 6.6%, and 6.3% for RF, SVM, and SL, respectively) and REIP (6.4%, 8.2%, and 7.3% for RF, SVM, and SL, respectively) played important roles in the three algorithms. Some variables (the reflectance values at bands 2 and 3, AFRI2.1, CVI and NDSI) possessed importance values of $>5.0\%$ in the RF-based model, whereas no variables except for Datt3 and REIP had importance values of $>5.0\%$ for SVM and SL. Even though the importance values of GEMI, Maccioni, and MNSI in SVM were $<5.0\%$, they were more than five times those in RF. AFRI1.6 and SIWSI were useful for identifying beetroot fields and AFRI1.6 occupied 11.1%, 6.8%, and 9.0% and SIWSI occupied 10.6%, 7.1%, and 8.9% of the importance for RF, SVM, and SL, respectively. GEMI and NDSI also had importance values of $>10\%$ for RF, but were $<5\%$ for the others. In contrast, REIP was useful in SVM and it occupied 9.1% of the importance in SVM. AFRI1.6, REIP, and MNSI were effective for identifying grassland for all algorithms, whereas SIWSI played an important role (7.8%) for RF and the reflectance at band 6 played an important role (8.2%) for SVM. For identifying maize fields, no variable had importance values $>5.0\%$ for any algorithm, but the importance value of REIP was 25.3% for SVM (2.9% for RF). CRI550, CRI700, and MSBI were 9.1%, 12.9%, and 5.6% in RF, respectively (those in SVM were 2.4%, 2.2%, and 3.6%, respectively). REIP played the greatest role for identifying potato fields in all algorithms (12.8%, 6.9% and 9.9% for RF, SVM, and SL, respectively). The importance values of CCCI and CVI were also high in RF (9.9%) but those in SVM were $<3.0\%$. In contrast, Maccioni had an importance of 6.9% in SVM but in RF was 1.4%. REIP also played a great role for identifying wheat fields in SVM, but 1.2% of the importance value was confirmed in RF while AVI occupied 15.1% in RF (1.2% in SVM). However, the original reflectance values possessed importance values of $<1.0\%$.

In this season, the photosynthetic activities of each crop type were different; maize is a C4 plant, beans and beetroot were in their growing season, grassland was after second harvest, potato growth was inhibited by chemicals for easy harvesting, and wheat fields were cultivated. In addition to indices related to chlorophyll content, the additional use of shortwave infrared data contributed to the estimation of photosynthetic pigments, water, nitrogen, cellulose, lignin, phenols, and leaf mass per area (e.g., NDSI). As a result, vegetation indices had greater influence on the classification results than the original reflectance. However, there were differences among algorithms in which vegetation indices were more important. The importance values in SL were near the averaged values of RF and SVM. So, the differences in importance between RF and SVM were useful when stacking was applied, and the modification contributed to identifying croplands with higher accuracies.