Webinar by ISPRS WG IV/2
Kappa the Undead and the Inconsistent Adoption of Insightful Error Metrics
ISPRS WG IV/2 (Artificial Intelligence and Uncertainty Modeling in Spatial Analysis) Webinar
Date/Time: August 13, 2024, 3 PM [CET]
Presenter: Professor Robert Gilmore Pontius Jr
Biography: Professor Robert Gilmore Pontius Jr creates quantitative methods that contribute to Geographic Information Science (GIS), Remote Sensing, and Statistics. Pontius is an applied statistician and environmental scientist with expertise in GIS, ecological modeling, and land change science. He has derived mathematical proofs concerning measurements that are essential in various fields. Several of these methods have been incorporated into the GIS & Image Processing software TerrSet, which has over 100,000 users worldwide. His solo-authored book is entitled Metrics That Make a Difference: How to Analyze Change and Error. His collection of videos helps to disseminate his work.
Abstract: This webinar is a follow-up to the 2011 paper by Pontius and Millones entitled Death to Kappa: Birth of Quantity Disagreement and Allocation Disagreement for Accuracy Assessment. The paper’s title indicates its two recommendations: 1 do not use kappa and 2 use components of disagreement to assess errors. We analyzed 200 articles to see how authors cited the Death to Kappa paper, which has more than 2000 citations. Results show that a quarter of the articles followed both recommendations, another quarter of the articles followed only the first recommendation, another quarter of the articles followed only the second recommendation, and the last quarter of the articles followed neither recommendation. Thus, the attempt to kill kappa and to replace kappa with more enlightening metrics has been partially effective, but kappa continues to haunt our profession. This webinar discusses and illustrates persistent bad habits concerning how authors use metrics in Remote Sensing and Land Change Modeling that were not covered in the 2011 paper. These habits include the use of kappa to assess thematic map accuracy at time points when the research question concerns change during time intervals and to assess the validation of land change predictions. We speculate on some reasons why some scientists use flawed methods and cite literature erroneously. We urge all scientists to pay attention to at least the titles of articles that they cite.