Evaluation of Classification Accuracy

We seek to determine how likely is each classified 2m x 2m pixel to be assigned to the ground cover class which would be identified for it by an on-the-ground visit. This is a very stringent test of mapping accuracy because of a number of factors which tend toward reducing the unique spectral character and ability to identify the location of individual pixels.

  • The spectra of the different vegetation species we are seeking to identify ( Phragmites , S. alterniflora , and high tide bush) are often very similar to one another. Our image was obtained in late September after the vegetation has begun senescence after a very dry summer. Individual stands appear to show differences in canopy density, vigor and in the amount of standing necrotic material. The strong spectral return from the background wet marsh surface is an important variable which tends to mask species specific spectral characteristics and chiefly indicates differences in the density of the plant canopy.

  • Low brightness pixels which are adjacent to contrasting high brightness pixels are likely to appear brighter because light from the brighter pixel is forward scattered and combined with light directly from the lower brightness pixel.
  • Uncertainty of location is due to inherent positional uncertainty of the coordinates obtained from the GPS system used in establishing ground truthing points.
  • Classification accuracy of the upland portion of the image is expected to be quite high because features of the targeted classes (agricultural fields, forest, and developed and impervious surfaces) are easily identified through a combination of their spectral properties and geometrical shapes. The high 2m spatial resolution of the data clearly resolves the shapes of agricultural and cultural landscape features allowing their ready identification in the event of spectral ambiguity. No effort has been made to assess the classification accuracy of these upland features since they were not the primary target of this study.

    However, we are concerned to assess the accuracy of the classification of the wetland portion of the image. There are several potential reasons for difference between a ground observation and its mapped class. Classification of the observed spectrum may not correctly represent the ground cover present at the time of the image because of confusion among the reflectance spectra of different classes. There is also the possibility of difference in on the ground conditions between the times of image and ground visitation. During the time lapse between the time when the image was taken at the end of the 1999 growing season and ground visitation during and late in the year 2000 growing season, vegetation in some areas may have changed in some detail. Although tide marsh vegetation, as a rule, does not change dramatically from one year to another, some Phragmites stands may be expending. Expansion of dense stands at a rate of the order of 1-2 meters per year has been noted. Further, for those Phragmites stands which were treated with herbicide in1998 or earlier, there has been regrowth of other species within the standing canes and/or regrowth of Phragmites itself. The year 1999 saw a drought summer which stressed marsh vegetation. The summer of 2000 had ample precipitation so that growth on the marsh was vigorous. This contrast of growing conditions between the two years is likely to have had the greatest effect in the herbicide-treated marsh west of Big Stone Beach.

    Difference in tide level also affects the distinction between the two mapped water classes Shallow water/tidal flat and Open Water. There is a clear spectral distinction between open water and shallow water/tidal flat. The latter spectrum shows higher reflectance values in the spectral interval between about 740 to 800 nm than in the spectrum below 600 to 650 nm while the open water does not. This is illustrated in Figure Mudflat and Water Spectra which shows examples of haze corrected shallow water spectra taken from the eastern end of the lagoon. These spectra also reveal the presence of chlorophyll in the water. The absorption feature at about 670 nm is due to chlorophyll absorption and the peak just above 710 nm represents chlorophyll fluorescence. Reflectance values at wavelengths higher than about 740 nm probably represent reflectance from bottom. The water depth in the areas classified swtf is estimated to be 0 to 2 cm at the time of the image. Water level change in the lagoon is about 10 cm over a diurnal tide cycle and may be 20 or 30 cm in response to a storm or

    over a spring tide-neap tide cycle. A transect of water depth taken across the eastern end of the lagoon showed water depth between 2 and 16 cm. Locations which were classed in the image as shallow water/tidal flat had either been above water had water depth 1-2 cm at the time the transect was made

    Accuracy assessment has focused on the wetland lying between the vicinity of Big Stone Beach Road and Rawley Island. A set of stratified random sample points of the wetland within these bounds was generated based on a 200 m grid. One random address of UTM northing and easting coordinates was generated within each grid cell. An on-the-ground visit to each sample point was required to obtain a positive identification of land cover. A ground team used differential GPS equipment to navigate to each point to which access proved possible.

    The time required to obtain an accurate differential GPS position is variable, depending on configuration of the satellites, ranging from several minutes on station to delays of an hour or more awaiting a favorable configuration of satellites. Slow acquisition of position data can significantly impede the task of navigating to preestablished sample points and reduces the number of points which can visited during a field day.

    Access to some locations was possible on foot from Big Stone Beach Road or from the beach. Walking access is generally blocked by ditches or soft ground. While travel along Greco's canal by outboard motor is possible, consideration of tide water level is necessary. Much of the open water is too shallow for canoe or outboard motor boat. Thus, most areas of the wetland were accessible only by air boat (e. g. open water and tidal flats locations, points east of Greco's Canal and the vicinity north of Rawley Island). Fortunately, the air boat became available late in the project in time to be used for this ground sampling. Because of these impediments to travel on the marsh, multiple visits were necessary to observe the sample points and not all of the selected population could be visited with the available resources of time and funding.

    To compensate for this difficulty of access we increased the number of sample observations by altering our sample scheme to obtain multiple observations based on the location of certain of the random sample points. This was done at those locations which provided a significant diversity in ground cover. Points in the original random sample, called base sample points, were identified by an ID number (such as 437). Subsidiary sample points keyed to one of the originally selected random point were identified with a base name plus a sequentially assigned extension (such as 437e for the 5 th extension observation to point 437). Subsidiary points thus may provide a sort of local transect through cover types near the base sample point.

    The ground cover at each visited sample point was recorded and photograph taken in which a view of the ground cover and the GPS observer on position. These photographs proved to be of great value as documentation for clarifying uncertainties which frequently subsequently arose about the precise surface conditions at a point.

    The ground resolution distance (2m) of the classification in this image is of the same order with which our differential GPS equipment can locate positions on the ground. The mapping accuracy of the image is also on the order of 2 m. Additional uncertainty in the precise location of individual image pixels is introduced by the resampling involved in converting between state plane and UTM coordinate systems. Therefore, we cannot make an unambiguous association between a ground observation and its corresponding image pixel. Therefore we must define a larger area for each ground sample. To evaluate how large this area should be we evaluated the error for18 road intersections in Milford Neck between on-the ground GPS positions taken with our equipment and the corresponding image coordinate positions. The result compared with the 1 m digital orthophoto mosaic and the 2m AISA mosaic showed in both images an average offset between the intersection feature and its coordinate location of 1.5 m 1.5 m. This suggest that 96% of the time the image pixel actually corresponding to a GPS ground position lies within 4.5 m of the mapped position. Thus, we define the ground sample area to be a circle of 5 m radius. In the image this circle is centered on the coordinate location of the ground sample. On average this 5 m radius includes 29 image pixels of a 2 m ground resolution distance. If the classified image showed a uniformity of class within the sample circle, clearly, we expect the all ground cover observed during a ground visit in the vicinity of that position to be either of that or another class. This situation presents little difficulty to evaluating classification accuracy.

    However, usually all 29 image pixels within the radius of the sample point were not classified into the same class. In this situation two different methods for evaluating the agreement between ground observations and mapped class can be suggested. The first is to consider the sample area to belong in the class in which the majority of its pixels are classified. The second is to consider all of the pixels separately. Both approaches were employed.

    It is usual to obtain sample observations which avoid transitions between ground covers and are therefore uniform in a surface feature over the range of several ground resolution diameters in order to minimize confusion in associating an image pixel with the ground surface feature. However, we wanted also to evaluate how reliably we were able to identify the location of transitions between wetland ground cover classes. Therefore the ground samples which we took were of two types: those where the surface was judged to be of uniform cover class within a 5 m radius of the GPS position and those where there was a transition or change between surface types within a 5 m radius of the GPS sample point. These two types of observations were evaluated separately.

    There are 52 randomly located non-transitional sample points. Of them, in 34 cases (65%), a majority of the pixels within 5 meters of the sample point coordinate location were classified in agreement with the on the ground classification for that sample. We have previously noted that it was difficult for the two classes "shallow water" and "open water" to obtain agreement with on-the-ground conditions because of probable differences in tide level between time of ground observation and image acquisition. Also, distinguishing between Phragmites and Phragmites canes on-the-ground can be subject to confusion because of regrowth between time of image acquisition in the fall and ground visit during the following growing season. Combining these readily miss-identified classes into the two classes "water" and " Phragmites" improves agreement between ground observations and classified points to 40 out of the 52 observations or 77% .

    For these 52 sample areas there total1525 pixels. An evaluation of the classification of these individual pixels is considered in Table 3. There are 11 classes, 10 land cover categories plus one "other" class to account for the 12 wetland pixels which were classed as either impervious surface or forest. Considering all classes to obtain an overall classification accuracy note that of the 1525 cells summarized in Table 3 the classification in 982 (64%) agreed with the land cover observed on the ground. (This is obtained by summing the diagonal cell values in Table 3.) However, in understanding this result we must consider on-the-ground errors.

    There are several situations, as noted above, in which it was not possible to obtain ground observations of land cover which could be compared consistently with the image classes. For instance, the S. alterniflora classes (1) and (2) which reflect different canopy densities were not distinguished in the field. The distinction between deeper "open water" and "shallow water/tidal flat" classes could not be made reliably in the field. If we combine these two water classes into one, then the number of cells which agree with ground observations increases to 1113 or 73%. Further, if we combine the two classes Phragmites and Phragmites canes the number of cells which agree with ground observations increases to 1166 for an overall classification accuracy 76%.

    Classification accuracy can be examined from two different viewpoints. From the viewpoint of the producer of the classification one wants to know what fraction of the ground samples of a particular class were so classified in the image. From the viewpoint of the user of a classification, one wants to know for an classified image what fraction of the samples in a particular class of the image are found in that class on the ground. In Table 3 these are obtained from the column and row totals, respectively, for the individual classes and are summarized in Table 4. One can see from Table 4 that, after combining the confused classes, that the users accuracy for the individual classes is comparable with or better than the overall accuracy of the classification.


    Table 3. Evaluation of Tidal Wetland Land Cover Classes
    Land Cover Observed on Ground
    Image Land Cover Class 1 2 & 3 4 5 6 7 8 9 10 11   Row sum Users Fraction Correct
     1. Unconsolidated Shore 87               33     120 0.725
     2. Spartina alterniflora (1)   269 20     18           307 0.88
     3. Spartina alterniflora(2)   102 55 4 18 2   1       182 0.56
     4. Salt Hay   13 143 29   3           188 0.76
     5. Phragmites   21   81   4           106 0.76
     6. Phragmites canes   32 5 53 68 1           159 0.43
     7. High Tide Bush   5     1 105           111 0.95
     8. Shallow Water/Tidal Flat   44 5     8 59 44       160 0.37
     9. Open Water   25         87 43       155 0.28
    10. Vegetated Unconsolidated Shore                 25     25 1
    11. Other       8   4           12 -
    Column Sum 87 511 228 175 87 145 146 88 58 0   1525  
    Producers Fraction Correct 1 0.73 0.63 0.46 0.78 0.72 0.4 0.5 0.44 -      
                                                     Overall accuracy after combining water and Phragmites classes = 1166/1525 =76%


    Table 4. Summary of Classification Results

    Before Combining Confused Classes
    Producers Accuracy Users Accuracy
    Unconsolidated Shore 87/87 = 1.00 87/120= 0.725
    Vegetated Unconsol. Shore 25/58 = 0.44 25/25 = 1.00
    S. alterniflora(1) 269/511 = 0.53 269/307 = 0.88
    S. alterniflora(2) 102/511 = 0.20 102/182 = 0.56
    Salt Hay 143/228 = 0.63 143/188 = 0.76
    Phragmites 81/175 = 0.46 81/106 = 0.76
    Phragmites Canes 68/87 = 0.78 68/159 = 0.43
    High Tide Bush 105/145 = 0.72 105/111 = 0.95
    Shallow Water/Tidal Flat 59/146 = 0.40 59/160 = 0.37
    Open Water 44/88 = 0.50 43/155 = 0.28

    After Combining Confused Classes
    Producers Accuracy Users Accuracy
    Unconsolidated Shore 87/87 = 1.00 87/120 = 0.725
    Vegetated Unconsol. Shore 25/58 = 0.44 25/25 = 1.00
    S. alterniflora(1) and (2) 371/511 = 0.73 371/489 = 0.76
    Salt Hay 143/228 = 0.63 143/188 = 0.76
    Phragmites and Canes 202/262 = 0.77 202/265 = 0.76
    High Tide Bush 105/145 = 0.72 105/111 = 0.95
    SWTF and Open Water 233/234 = 0.99 233/315 = 0.74