Cohen's kappa: meaning, interpretation and calculation
Learn what Cohen's kappa measures, how to read agreement levels, and how to calculate the coefficient from a confusion matrix for two raters.
Contents
- What Cohen's kappa measures
- Reading the kappa value
- Landis and Koch interpretation levels
- Calculating kappa step by step
- A worked example of the calculation
- Why category balance changes kappa
- Weighted kappa for ordered categories
- Using a kappa calculator and reporting results
- Limits and cautions of Cohen's kappa
- Cohen's kappa questions answered
- What does Cohen's kappa agreement mean in plain terms?
- How do I read Cohen's kappa agreement levels?
- Can a kappa calculator use just the percentage of agreement?
- What does a negative Cohen's kappa coefficient show?
- What is the difference between kappa and weighted kappa?
- Does Cohen's kappa work with more than two raters?
Cohen's kappa is a statistic that measures how much two raters agree on categorical ratings after subtracting the agreement expected by chance. It reports a coefficient from about -1 to 1, where 0 means chance-level agreement and 1 means perfect agreement. You calculate it from the observed agreement and the chance agreement that the raters' own category totals predict.
- Cohen's kappa compares observed agreement with the agreement two raters would reach by chance.
- A kappa near 0 means the raters did no better than chance on that set of categories.
- The coefficient depends on the categories and their balance, so report those alongside it.
- Interpretation bands such as the Landis and Koch scale are conventions, not fixed thresholds of quality.
- A kappa calculator needs the full confusion matrix, not just a single percentage of agreement.
What Cohen's kappa measures
Cohen's kappa coefficient measures agreement between two raters who assign items to categories. The statistic adjusts the raw agreement for the agreement that chance alone would produce. This makes kappa a stricter measure than a simple percentage of matching ratings.
Observed agreement versus chance agreement is the core idea of the coefficient. Observed agreement is the share of items where both raters chose the same category. Chance agreement is the share expected if each rater spread ratings according to their own totals without any shared judgment.
Inter-rater agreement for categorical ratings is the setting where the statistic belongs. It applies to two raters, a fixed set of categories, and items each rater scores independently. It fits tasks such as coding text passages, labeling images, or judging whether a feature is present.
Reading the kappa value
Cohen's kappa agreement interpretation starts from the number's distance from zero. A value of 0 means the raters agreed no more than chance predicted for those categories. Positive values show agreement above chance, and 1 means the raters matched on every item.
Negative values are possible and deserve attention. A negative kappa means the raters agreed less than chance would predict. That pattern can appear when raters systematically split items in opposite directions or when categories are heavily skewed.
Kappa is not a proportion of correct ratings in the everyday sense. A kappa of 0.6 does not mean 60 percent agreement. Report the raw observed agreement next to the coefficient so a reader can see both the plain match rate and the chance-corrected figure.
Landis and Koch interpretation levels
The kappa interpretation scale by Landis and Koch sorts coefficients into labeled bands. This scale is a widely cited convention for describing agreement levels in words. It is a guide to how the number reads, not a test of whether the ratings are fit for a purpose.
The bands run from poor through slight, fair, moderate, substantial, and almost perfect agreement. A coefficient below zero falls outside these labels and signals disagreement below chance. Each band covers a range of values, so a single coefficient near a boundary could be described by either neighboring label.
Cohen's kappa agreement levels do not carry the same weight in every field. A moderate band may be acceptable for one rating task and too weak for another. Name the scale you used and the context, and avoid treating a label as a pass or fail mark.
- Below 0: less agreement than chance
- 0.00 to 0.20: slight agreement
- 0.21 to 0.40: fair agreement
- 0.41 to 0.60: moderate agreement
- 0.61 to 0.80: substantial agreement
- 0.81 to 1.00: almost perfect agreement
Calculating kappa step by step
Cohen's kappa calculation begins with a confusion matrix for two raters. The matrix counts items by the category each rater assigned. The diagonal cells hold items where both raters chose the same category, and the off-diagonal cells hold the mismatches.
Observed agreement is the diagonal total divided by the total number of items. Chance agreement is found from the row and column totals of the matrix. For each category, multiply rater A's share of that category by rater B's share, then add those products across all categories.
The formula is kappa equals observed agreement minus chance agreement, divided by one minus chance agreement. Subtract chance agreement from both the top and the bottom because kappa scales the gain above chance against the room available above chance. The result is the coefficient.
- Build the confusion matrix and total its cells
- Divide the diagonal total by the item count to get observed agreement
- Multiply each rater's category shares and sum them for chance agreement
- Subtract chance agreement from observed agreement
- Divide that result by one minus chance agreement
A worked example of the calculation
A worked example of kappa calculation makes the formula concrete, so follow a small two-category case. Suppose two raters each label a set of items as present or absent, and the confusion matrix has been counted. The diagonal holds the items both raters marked the same way.
Say the matrix totals 50 items, with 30 on the diagonal and 20 off it. Observed agreement is 30 divided by 50, which gives 0.60. Now suppose the category shares produce a chance agreement of 0.50 for those same marginals. Kappa is then 0.60 minus 0.50, divided by 1 minus 0.50, which equals 0.20.
The example shows why raw agreement can mislead. The raters matched on 60 percent of items, yet the chance-corrected coefficient is only 0.20. That gap appears when one category dominates the ratings, because chance agreement rises with that imbalance.
Why category balance changes kappa
Cohen's kappa statistic is sensitive to the balance of categories in the data. When one category fills most items, chance agreement grows and the room above chance shrinks. The same level of raw matching can then produce a much lower coefficient.
The kappa paradox describes this effect. Two studies with equal observed agreement can show different kappa values when their category distributions differ. This is not an error in the formula; the statistic is doing what it defines, measuring gain over chance.
Report the category totals with the coefficient so readers can judge the setting. A low kappa next to a high match rate often points to skewed categories rather than careless raters. Reviewing the confusion matrix shows where the mismatches fall.
Weighted kappa for ordered categories
Weighted kappa extends the coefficient to ordered categories. Ordinal scales such as mild, moderate, and severe have a natural order, and a one-step miss is less serious than a two-step miss. Weighted kappa gives partial credit for near misses instead of counting every mismatch as a full error.
Linear weights treat each step apart as equally serious. Quadratic weights penalize larger gaps more heavily and are tied to a particular way of measuring disagreement. Choose the weighting scheme before you look at the ratings and state it with the result.
The unweighted coefficient treats all mismatches alike, which suits unordered categories. For ordered scales, weighted kappa usually describes the agreement pattern more faithfully. A kappa calculator should state which weighting it uses.
Using a kappa calculator and reporting results
A Cohen's kappa statistic calculator takes the counts from the confusion matrix and returns the coefficient. Feed it the full matrix, with one row and column per category, rather than a single agreement figure. The full table is what lets the tool compute chance agreement correctly.
A Cohen's kappa coefficient calculator should also return observed agreement and the chance figure. Check those two values against your own arithmetic before trusting the coefficient. Some tools add confidence intervals, which show how precise the estimate is for the sample size.
When reporting, give the coefficient, the number of items, the number of categories, and the interpretation scale you used. State whether the kappa is weighted and which weights applied. This lets another analyst recompute the figure from the same matrix.
Limits and cautions of Cohen's kappa
Cohen's kappa covers exactly two raters. Studies with three or more raters need a different coefficient, such as Fleiss' kappa, which handles multiple raters on the same items. Using the two-rater formula on a larger panel gives a misleading result.
The statistic assumes the raters work independently and that the categories are fixed in advance. Raters who discuss items or share a reference key break the independence the formula relies on. Categories added after scoring began change the matrix and the chance calculation.
Kappa does not show which rater is correct, only how far the two agree. It also gives no information about the cause of disagreement. Review the confusion matrix and the items behind the off-diagonal cells to learn where the ratings diverge.
Cohen's kappa questions answered
What does Cohen's kappa agreement mean in plain terms?
Cohen's kappa agreement expresses how far two raters matched beyond what chance would produce for their categories. A value of 0 means chance-level matching and 1 means the raters matched on every item. The coefficient sits alongside the raw observed agreement, not in place of it.
How do I read Cohen's kappa agreement levels?
Cohen's kappa agreement levels are read through a labeled scale, such as the Landis and Koch bands from slight to almost perfect. Each band covers a range of values, so a coefficient near a boundary can be described two ways. Treat the labels as a convention and state which scale you used.
Can a kappa calculator use just the percentage of agreement?
No, a kappa calculator needs the full confusion matrix to work out chance agreement. The percentage of matching ratings gives observed agreement only. Chance agreement depends on each rater's category totals, which the matrix records.
What does a negative Cohen's kappa coefficient show?
A negative Cohen's kappa coefficient shows agreement below the chance level for those categories. Raters agreed less than independent assignment would predict. Check for systematic opposite coding or heavily skewed categories in the matrix.
What is the difference between kappa and weighted kappa?
Weighted kappa applies to ordered categories and gives partial credit when raters choose nearby categories. Plain kappa counts every mismatch as a full disagreement. Use the weighted form for ordinal scales and say which weights you applied.
Does Cohen's kappa work with more than two raters?
No, Cohen's kappa applies to exactly two raters on the same items. A panel of three or more raters needs a coefficient designed for multiple raters, such as Fleiss' kappa. Applying the two-rater formula to a larger panel gives an unreliable figure.