Cohen’s kappa measures how well two independent reviewers agree when screening records, after removing the agreement you would expect by chance. It is the standard inter-rater reliability statistic for title and abstract screening in a systematic review. This calculator takes the four counts from a two-by-two agreement table and returns kappa, the observed agreement, the expected agreement, a confidence interval, and a Landis and Koch benchmark.
Cohen's kappa calculator
Enter the agree and disagree counts from two screeners and get kappa with a 95% interval, the observed and chance-expected agreement, and a benchmark label.
Cohen's kappa
0.780
95% CI 0.689 to 0.871 | Substantial
Agreement breakdown
Observed agreement (po)90.0%
Expected agreement (pe)54.5%
Standard error of kappa0.0464
z test against kappa = 0z = 11.05, p < 0.0001
Records screened (n)200
Strength of agreementSubstantial
Prevalence and bias diagnostics
Prevalence index-0.300
Bias index-0.020
Prevalence-adjusted bias-adjusted kappa (PABAK)0.800
Maximum attainable kappa0.956
The benchmark follows Landis & Koch: below 0 poor, 0 to 0.20 slight, 0.21 to 0.40 fair, 0.41 to 0.60 moderate, 0.61 to 0.80 substantial, and 0.81 to 1.00 almost perfect.
Report-ready text
Inter-rater agreement across 200 records was substantial (Cohen's kappa = 0.78, 95% CI 0.69 to 0.87, p < 0.0001; observed agreement 90.0%). The prevalence-adjusted bias-adjusted kappa was 0.80.
Why agreement needs more than a percentage
Cohen’s kappa answers a question that raw agreement cannot: when two reviewers screen the same records, how much of their agreement is real and how much would have happened by chance alone? In a typical title and abstract screeninground most records are obvious excludes, so two reviewers will agree on the vast majority simply because almost everything is excluded. Reporting that they agreed on 95% of records sounds reassuring, yet chance alone could deliver most of that. Kappa strips out the chance-expected agreement and rescales what is left, so a value near 1 means agreement well beyond luck and a value near 0 means no better than coin-tossing. It is the standard inter-rater reliability measure for screening in a systematic review, and our guide to measuring inter-rater reliability in screening puts it in the wider context of dual screening.
Interpreting the value with the Landis and Koch benchmark
The most cited reading of kappa comes from Landis and Koch (1977): below 0 is poor, 0 to 0.20 is slight, 0.21 to 0.40 is fair, 0.41 to 0.60 is moderate, 0.61 to 0.80 is substantial, and 0.81 to 1.00 is almost perfect. For screening, most teams aim for substantial agreement (0.61 or higher) before they trust two reviewers to work in parallel. A fair or moderate value is a signal that the eligibility rules are ambiguous, not that the reviewers are careless. Treat these bands as guidance, not law: the same kappa can mean different things depending on how lopsided the include and exclude counts are, which is why you should always read kappa next to the observed agreement this calculator also reports.
A worked example and the traps to avoid
Suppose two reviewers screen 200 records. They both include 20, both exclude 168, and disagree on 12 (reviewer one includes 7 the other excludes, and excludes 5 the other includes). Observed agreement is (20 + 168) / 200 = 0.94, a comforting 94%. But because excludes dominate, the chance-expected agreement is high too, around 0.78, so kappa works out to roughly (0.94 - 0.78) / (1 - 0.78) = 0.73. That is substantial agreement on the benchmark, and a fair reflection of two well-calibrated reviewers. Drop the genuine includes to a handful and the same number of disagreements would crater kappa, even though the reviewers behaved identically. This is the well-known kappa paradox: when one category swamps the other, high observed agreement can sit next to a deflated kappa.
Common mistakes researchers make
The first mistake is reporting kappa alone. Always pair it with the observed agreement and the raw counts so a reader can see whether a low value is a real problem or a prevalence artefact. The second is measuring agreement too late, after the whole library has been screened, when a weak value is expensive to act on; pilot the criteria first and recalibrate, a workflow covered in our guide to running title and abstract screening. The third is using kappa to decide who was “right” on a disagreement; kappa measures consistency, not correctness, and every conflict still needs a documented resolution, which our walkthrough of resolving screening conflicts sets out. Finally, decide your reviewer count deliberately rather than by habit; the trade-offs are laid out in how many reviewers you need for screening. A clean agreement check and a documented audit trail are exactly what our screening and data extraction service builds into a protocol.
How it works
The four counts form a two-by-two table: a (both include), b and c (the two ways the reviewers disagree), and d (both exclude).
n = a + b + c + d
po = (a + d) / n
pe = ((a+b)/n)*((a+c)/n) + ((c+d)/n)*((b+d)/n)
kappa = (po - pe) / (1 - pe)
SE = sqrt( po*(1-po) / (n*(1-pe)^2) )
95% CI = kappa +/- 1.96 * SE (clamped to [-1, 1])
The benchmark follows Landis & Koch (1977): below 0 is poor, 0 to 0.20 slight, 0.21 to 0.40 fair, 0.41 to 0.60 moderate, 0.61 to 0.80 substantial, and 0.81 to 1.00 almost perfect.
Frequently asked questions
- What is a good Cohen's kappa for title and abstract screening?
- On the Landis and Koch benchmark, a kappa of 0.61 to 0.80 is substantial and 0.81 to 1.00 is almost perfect, and most review teams aim for substantial agreement or better at the screening stage. A lower value signals that the eligibility criteria need clarifying and the screeners need to recalibrate on a sample before continuing.
- Why is kappa lower than the percent agreement I see?
- Observed agreement counts every record the two reviewers decided the same way, including the ones they would have agreed on by chance. Kappa subtracts the agreement expected by chance and rescales, so when most records are obvious excludes the chance agreement is high and kappa is much lower than the raw percent agreement, even though the screeners rarely disagree.
- Does a high exclude count inflate agreement?
- Yes, and that is the kappa paradox. When one category dominates, such as a large block of clear excludes, the expected agreement climbs and kappa can look weak even with very high observed agreement. Report both the observed agreement and kappa together so reviewers can see the full picture rather than one number alone.
- What do the four counts mean?
- They form a two-by-two agreement table for the two screeners. The cell a is records both marked include, b is records reviewer one included but reviewer two excluded, c is records reviewer one excluded but reviewer two included, and d is records both marked exclude. The diagonal a plus d is where they agreed.
- When should I measure kappa during a review?
- Measure it on a pilot sample before the main screening run, so a weak value can be fixed by clarifying the criteria before thousands of records are decided. Many teams also report a kappa on the full screened set in the methods. A calibration check on twenty-five to fifty records is usually enough to expose a problem early, when it is cheap to correct.
- Should I use Cohen's kappa or another agreement statistic?
- Cohen's kappa is the standard for two reviewers and a binary include-or-exclude decision, which is exactly the screening case. For three or more reviewers use Fleiss' kappa, and for ordered or weighted categories use a weighted kappa. When one category overwhelmingly dominates and kappa looks paradoxically low, report observed agreement alongside it and consider a prevalence-adjusted statistic so the number is not misread.
The guides behind this tool
More review and meta-analysis tools
Meta-analysisPool study effect estimates with fixed-effect and random-effects models, with the heterogeneity statistics shown.Open tool Heterogeneity (I-squared)Quantify how much your studies disagree: Cochran's Q, I-squared, H, and the between-study variance tau-squared.Open tool Odds ratio / risk ratioTurn a 2x2 table of events into an odds ratio, risk ratio, and risk difference, each with a 95% confidence interval.Open tool