Weighted Kappa Agreement

Uncategorized Apr 15

The concept of “advisor agreement” is quite simple and, for many years, the reliability of Interraters has been measured as a percentage of match among data collectors. To obtain the measurement agreement, the statistician established a matrix in which the columns represent the different advisors and the lines of the variables for which the raters had collected data (Table 1). The cells in the matrix contained the values captured by the data collectors for each variable. An example of this procedure can be made in Table 1. In this example, there are two advisors (Mark and Susan). They each record their values for variables 1 to 10. To obtain a percentage of approval, the researcher subtracted Susan`s scores from Marks Scores and counted the resulting number of zeroes. Dividing the number of zeros by the number of variables provides a measure of the agreement between advisors. In Table 1, the agreement is 80%. This means that 20% of the data collected in the study is incorrect, because only one of the advisors can be correct if there is disagreement. This statistic is directly interpreted as a percentage of correct data. The value, 1.00 – percent approval can be understood as a percentage of data that is false. That is, if the approval percentage is 82, 1.00-0.82 – 0.18 and 18% is the amount of data that the search data is wrong.