Your coded data
Which columns to compare
Results
What the numbers mean
- Percent agreement is how often the two columns match. It takes no account of the matches two coders would hit by guessing, so it always flatters.
- Cohen’s kappa corrects for that chance agreement. Report it at 0.70 or above (Landis & Koch, 1977); below it, revise and code a fresh sample.
- Krippendorff’s alpha is reported alongside kappa because a single statistic can mislead on skewed data (Lombard et al., 2002). This page computes the nominal form.
- Agreement expected by chance is the honest warning label. When one category covers nearly every unit, kappa falls even though the coders rarely differ, so report the base rates with it.
- A fresh reliability sample is what gets reported. Agreement measured on the training set the coders already discussed is not reliability.
Serves the Codebook and Coding assignments in MC 451 and MC 501, and the manual-against-model comparison in MC 427. Works on any two columns, including a human column against a model column.