Classification Accuracy Assessment

- Introduction -

In the previous two lab exercises, unsupervised and supervised classification were performed on the same Landsat 7 (ETM+) image of Eau Claire and Chippewa Counties captured on June 9, 2000. Qualitative confidence-building was performed on the classified LULC images and discussed in pervious blog posts. Now, statistical confidence-building will be performed on each classified LULC image through use of an error matrix. The error matrix will provide an overall accuracy, producer's accuracy and user's accuracy. Kappa statistics will also be used.

To generate an error matrix, ground reference points need to be collected. These points can be collected prior to image classification through GPS point collection or surveying, or generated after image classification by using high resolution imagery or aerial photography as a reference. The pixels corresponding to each ground reference point will be labeled with the appropriate LULC class and this value will then be compared to what the pixel was classified. This comparison is then summarized in an error matrix.

- Methods -


Figure 1: Example of how the ground reference
points were interpreted.

Accuracy assessment was performed using ERDAS IMAGINE by navigating to Raster > Supervised > Accuracy Assessment. The Landsat 7 (ETM+) imagery was opened in the Accuracy Assessment window and a high resolution aerial photograph from the National Agriculture Imagery Program (NAIP) of the United States Department of Agriculture was selected as the reference image. Ground reference points were added by navigating to Edit > Create/Add Random Points. The number of points was changed to 125 and stratified random was chosen as the distribution type to allow for an even distribution of ground reference points throughout the different LULC classes.

After the ground reference points were generated, they appeared in the Accuracy Assessment window. Each point was examined and labeled with the appropriate LULC class based on visual interpretation of the reference image. Once all the ground reference points were interpreted, accuracy assessment was performed by navigating to Report > Accuracy Assessment. Values from the resulting text file were copied into a Microsoft Excel spreadsheet for easier to understand formatting.

- Results -



 

 

 

- Discussion -

In terms of overall accuracy, the statistical confidence-building assessment confirms the conclusions of the qualitative confidence-building assessments. The unsupervised method produced a better classified LULC image because of the quality of urban/built-up training samples collected during the supervised method. As seen in Table 2, The user's accuracy for urban/built-up is only 11%. Of the 36 ground reference points that were placed within the urban/built-up class, only 4 were interpreted as urban/built-up. Forest, bare soil, and agriculture were often confused for urban/built-up by the supervised classifier. The user's accuracy for each LULC class, except for urban/built-up, is higher for the supervised method. By modifying the training samples used for the supervised classification, the user's accuracy for urban/built-up, agriculture, and bare soil could be further increased, along with its overall accuracy.

A threshold of 85% has been established as the minimum overall accuracy needed for a "good" classified image. The overall accuracy for both classified LULC images fell below this threshold indicating that neither should be used for further analysis. A revised supervised classification could potentially reach the 85% threshold or an advanced classifier could be used.

- Conclusion -

In this lab exercise, the accuracy of the classified LULC images was assessed using four different measures of accuracy (overall, producer's, user's, and kappa) obtained from interpreting error matrices. Each method used, unsupervised and supervised, has its advantages and disadvantages though neither was able to reach an appropriate level of accuracy to be used in further analysis.  Advanced classifiers like expert system/decision tree, neural networks, and object-based classifiers were developed for just this reason. In subsequent lab exercises and blog posts, these advanced classifiers will be examined.

- Sources -

Earth Resources Observation and Science Center, United States Geological Survey. Landsat 7 (ETM+).

United States Department of Agriculture (USDA) National Agriculture Imagery Program. High resolution aerial imagery.