Measuring TAR Recall and Elusion With a Defensible Confidence Interval

The recall number a TAR process reports is only as defensible as the sample it was estimated from, and the most common validation failure is a recall point estimate with a confidence interval so wide it proves nothing — “recall is 80%, plus or minus 22 points” is not a defensible result. This problem lives at the validation step of Technology-Assisted Review, where an undersized or biased sample, or one that ignores the collection’s low prevalence, produces an estimate that cannot survive a challenge to the process. This guide isolates why recall estimates come out unreliable, gives the sampling mathematics that size the samples correctly, and delivers a runnable estimator that reports recall and elusion with honest confidence intervals.

Diagnostic Log Signatures

The validation report exposes the problem in its interval widths and its elusion test. Capture it verbatim — the interval, not the point, is the finding.

[INFO] TAR validation: collection=402118, prevalence_est=0.031
[WARN] recall_point=0.80 recall_ci95=[0.58, 0.94]  (width=0.36, target width<=0.10)
[ERROR] control_set responsive n=12  (too few positives for a stable recall estimate)
[WARN] elusion_point=0.009 elusion_ci95_upper=0.041  (upper bound above agreed 0.02)
[CRITICAL] Validation INCONCLUSIVE: recall interval too wide to certify target

Symptom checklist — two or more mean the estimate is not yet defensible:

• The 95% confidence interval on recall is wider than the tolerance agreed in the protocol (often ±5 points).
• The control set contains only a handful of responsive documents, so recall rests on a tiny denominator.
• The elusion estimate’s upper confidence bound exceeds the agreed threshold, even though the point estimate looks fine.
• Sampling was not random (e.g., drawn only from a keyword hit set), so the estimate is biased, not just imprecise.
• Re-drawing the sample changes the recall point estimate by more than the reported interval implies it should.

Root-Cause Breakdown

Unreliable recall estimates come from sampling that was too small, biased, or blind to prevalence.

1. Denominator too small. Recall is $\text{TP}/(\text{TP}+\text{FN})$, computed over the responsive documents in the sample. In a low-prevalence collection — 3% responsive is typical — a 400-document random sample contains only about a dozen responsive documents, so recall is estimated from ~12 data points and its interval is enormous. The sample must contain enough positives, which means either a much larger random sample or a design that targets the positive count directly.
2. Biased sampling frame. Estimating recall from a non-random frame — only keyword hits, only one custodian, only the model’s top-ranked documents — measures the model on a slice, not the collection, and the number does not generalize. Only a simple random (or properly stratified) sample of the whole collection yields an unbiased estimate.
3. Point estimate reported without an interval. A bare “recall = 80%” hides its own uncertainty. Defensibility requires the interval, because the interval is what tells the court whether the target was actually met or merely landed on by a noisy estimate.
4. Elusion judged on the point, not the bound. Elusion (responsive rate in the discard pile) is a one-sided question — you care about the upper confidence bound, because that is the worst-case amount of responsive material left behind. Judging it on the point estimate understates the risk.

Remediation Architecture

Size the samples to the precision the protocol demands, sample randomly from the whole collection, and report intervals — the upper bound for elusion.

1. Size for positives, not just documents. For a target absolute precision $E$ at confidence $z$ (1.96 for 95%), the sample size for a proportion $p$ is
   $$n = \frac{z^2\, p\,(1-p)}{E^2}.$$
   Because recall’s effective sample is the responsive subset, size the recall sample so that the expected positive count $n\cdot p$ is large enough to hit the interval width you need — in low prevalence this drives a larger total $n$.
2. Random frame over the whole collection. Draw the validation sample by simple random sampling (or stratified by custodian/date with known weights) across every document, coded or not, so the estimate generalizes.
3. Wilson interval for small counts. Use the Wilson score interval rather than the normal approximation, because with few positives the normal (“Wald”) interval misbehaves — it can even extend past 0 or 1.
4. Elusion as a one-sided bound. Draw a random sample from the discard set, code it, and report the upper 95% bound on the responsive rate; the process passes only if that upper bound is at or below the agreed threshold.

The validation design below shows the two independent samples — the control/recall sample over the whole collection and the elusion sample over the discard set — feeding one certification gate.

Recall and elusion estimator

The estimator computes recall over a coded sample, a Wilson score interval robust to small positive counts, and a one-sided upper bound for elusion. It compiles and runs with only the standard library.

import math
from dataclasses import dataclass
from typing import List, Tuple


@dataclass
class Coded:
    truly_responsive: bool
    predicted_responsive: bool


def wilson_interval(successes: int, n: int, z: float = 1.96) -> Tuple[float, float]:
    """Wilson score interval for a binomial proportion. Well-behaved for small
    n and extreme proportions, unlike the normal (Wald) approximation."""
    if n == 0:
        return (0.0, 1.0)
    p = successes / n
    denom = 1 + z * z / n
    centre = (p + z * z / (2 * n)) / denom
    half = (z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))) / denom
    return (max(0.0, centre - half), min(1.0, centre + half))


def estimate_recall(sample: List[Coded], z: float = 1.96) -> Tuple[float, Tuple[float, float]]:
    """Recall = TP / (TP + FN) over the responsive documents in the sample,
    with a Wilson interval. Raises if there are too few positives to estimate."""
    positives = [c for c in sample if c.truly_responsive]
    if len(positives) < 7:
        raise ValueError(f"only {len(positives)} responsive docs; sample too small for recall")
    tp = sum(1 for c in positives if c.predicted_responsive)
    point = tp / len(positives)
    return point, wilson_interval(tp, len(positives), z)


def elusion_upper_bound(discard_sample: List[bool], z: float = 1.645) -> Tuple[float, float]:
    """Elusion = responsive rate in the discard set. Returns (point, upper_bound);
    z=1.645 gives a one-sided 95% upper bound, the worst-case leftover rate."""
    n = len(discard_sample)
    if n == 0:
        raise ValueError("empty discard sample")
    responsive = sum(1 for r in discard_sample if r)
    point = responsive / n
    _, upper = wilson_interval(responsive, n, z)
    return point, upper


def sample_size_for_precision(p: float, e: float, z: float = 1.96) -> int:
    """Minimum n to estimate a proportion p to +/- e at confidence z."""
    if not (0 < e < 1):
        raise ValueError("precision e must be in (0, 1)")
    return math.ceil(z * z * p * (1 - p) / (e * e))


def certifies(recall_point: float, recall_ci: Tuple[float, float],
              elusion_upper: float, target_recall: float,
              max_ci_width: float, elusion_threshold: float) -> bool:
    """Pass only if recall meets target with a tight enough interval and the
    elusion worst case is within threshold."""
    width = recall_ci[1] - recall_ci[0]
    return (recall_point >= target_recall
            and width <= max_ci_width
            and elusion_upper <= elusion_threshold)

Verification Checklist

Confirm the estimate is defensible before certifying the process:

• The recall 95% interval width is within the protocol’s tolerance, and the interval’s lower bound is at or above the target.
• The recall sample contains enough responsive documents (a meaningful positive count, not a handful) — sized with the prevalence-aware formula.
• The validation sample was drawn by simple random (or documented stratified) sampling over the whole collection, not from a keyword or top-ranked slice.
• Elusion is reported as a one-sided upper bound, and that bound is at or below the agreed threshold.
• The Wilson interval (not the normal approximation) was used, and re-drawing a sample yields an estimate consistent with the reported interval.

Conclusion

A TAR recall number is only meaningful with its confidence interval, and the interval is only defensible when the sample was random, large enough in positives to be precise at the collection’s prevalence, and analyzed with a small-count-robust method. Size the recall sample for the positive count you need, draw it randomly across the whole collection, report the Wilson interval, and judge elusion on its upper bound rather than its point. Certify the process only when recall meets its target with a tight enough interval and the elusion worst case stays within threshold. Done this way, the validation package answers the one question a challenge to a TAR process always asks: how do you know you found the responsive material?

Frequently Asked Questions

Why does low prevalence make recall so hard to estimate?

Because recall is computed only over the responsive documents in the sample. At 3% prevalence, a 400-document random sample contains about twelve responsive documents, so recall rests on a twelve-item denominator and its confidence interval is very wide. Getting a tight recall interval in a low-prevalence collection requires enough positives in the sample, which means a substantially larger random sample or a design that controls the positive count directly.

Why report the upper bound for elusion but the interval for recall?

Because they answer different-shaped questions. For recall you want to know the plausible range and whether the whole interval clears the target, so you report the two-sided interval. For elusion you only care about the worst case — how much responsive material could still be in the discard pile — so the relevant quantity is the one-sided upper confidence bound. A low elusion point estimate with a high upper bound still fails, because the upper bound is the risk.

Is a Wilson interval really necessary instead of the usual formula?

Yes, when positive counts are small, which they usually are in recall estimation. The normal (Wald) approximation assumes a large sample and symmetric distribution; with a handful of positives it produces intervals that are too narrow or that spill below 0 or above 1. The Wilson score interval stays inside [0, 1] and has correct coverage at small n, so it is the appropriate default for TAR validation.

Related

• Technology-Assisted Review — the parent process whose stopping rule this validates.
• Production Compliance Frameworks — the audit rules the validation package feeds.
• Production Validation & QC — the downstream gate for documents TAR routes to production.
• Similarity Threshold Configuration — another place statistical thresholds must be validated and logged.

For the statistics, see the Wilson score interval reference and Grossman & Cormack’s work on evaluation of TAR protocols.

Up one level: Technology-Assisted Review — the subsystem that decides what gets reviewed.