Technology-Assisted Review: Defensible Predictive Coding at Scale

Technology-assisted review (TAR) is how modern eDiscovery avoids reviewing every document by hand: a classifier learns responsiveness from human-coded examples and ranks the remaining collection, so reviewers spend their hours on the documents most likely to matter. Inside the Review & Production Workflows pipeline, TAR is the subsystem that decides what gets reviewed at all, which makes its defensibility a first-order concern — a TAR process that cannot demonstrate it found the responsive material is not a cost saving, it is a spoliation risk. The engineering challenge is not the model; off-the-shelf classifiers work well on text. It is the process: reproducible training, a statistically valid stopping rule, and a validation protocol that estimates recall with a defensible confidence interval. This guide builds TAR as an auditable stage whose every coding decision, model version, and validation sample is recorded.

TAR Subsystem Flow

TAR is an iterative loop, not a one-shot classification. A control set is coded to measure against, seed documents train an initial model, the model ranks the collection, reviewers code the highest-ranked (or most-informative) documents, and those codes retrain the model — repeating until a validated stopping criterion is met and the process is measured against the control set.

Resource Constraints & Design Rationale

The dominant cost in TAR is not compute — a linear text classifier trains in seconds on hundreds of thousands of documents — it is reviewer time and the risk of an indefensible process. So the design optimizes for two things: minimizing the human coding needed to reach a target recall, and making every step reproducible. The first drives the choice between simple active learning (train once on a seed set, review a fixed cut) and continuous active learning (CAL), where the model retrains after every review batch and continually re-ranks, feeding reviewers the currently-highest-scoring documents. CAL typically reaches a target recall with less review because it adapts as it learns, which is why it has become the common default.

Reproducibility is the second constraint and the one with legal weight. Every model version is pinned to the exact set of coded documents that trained it, the feature extraction is deterministic, and the control set is frozen and never trained on, so the recall estimate is unbiased. Feature vectors are computed once from the extracted text produced upstream by PDF text extraction engines and cached, so re-ranking after each retrain is cheap and the same document always yields the same features. This determinism is what lets you show a court that the process, re-run on the same codes, produces the same ranking.

Active Learning & Validation Algorithm

The core measurement quantities are standard information-retrieval metrics computed against coded samples. With $\text{TP}$ true positives, $\text{FN}$ false negatives, and $\text{FP}$ false positives, recall and precision are

Recall is the metric that governs defensibility: it is the fraction of truly responsive documents the process surfaced, and a production is defensible when recall is high and demonstrably estimated. The elusion rate — the fraction of documents in the discard pile that are actually responsive — is the complementary check, estimated by sampling the unreviewed set:

A low elusion on a valid random sample is direct evidence that stopping review did not leave responsive material behind. The stopping rule ties these together: review continues until the estimated recall against the control set reaches the agreed target (commonly 75–80% with a stated confidence), at which point an elusion test on the discard set confirms the tail is clean. The detailed sampling mathematics — sample sizes, confidence intervals, and the elusion test — are worked through in measuring TAR recall and elusion.

Resilience & Failure Routing

TAR’s failure modes are statistical, not operational, so the guards are statistical too. The first is inconsistent coding: if reviewers code the same or similar documents differently, the model learns noise and the recall estimate is unreliable. The subsystem routes a stream of overlap documents — items coded by more than one reviewer — and flags families whose codes disagree, feeding a conflict queue that a senior reviewer resolves before those codes retrain the model. The second is control-set contamination: if a control-set document is ever coded as a training example, the recall estimate becomes optimistically biased. The design hard-partitions the control set from the trainable pool and asserts, at every retrain, that no control document entered training.

The third is model instability: early in the loop the ranking shifts substantially between retrains, and stopping too early on a still-moving estimate risks missing responsive material. The subsystem tracks the recall estimate’s trajectory and requires it to stabilize — successive retrains changing it by less than a threshold — before the stopping rule can fire. Every one of these guards writes to the same audit trail as the rest of the production compliance frameworks, so the statistical validity of the process is itself part of the record.

Production Python Implementation

The module implements a continuous-active-learning loop with a frozen control set, versioned models, deterministic features, and a stabilization-aware stopping rule. The classifier and vectorizer calls use scikit-learn shapes; the loop, control-set partitioning, and stopping logic are complete and runnable.

import logging
from dataclasses import dataclass, field
from typing import Callable, Dict, List, Sequence, Tuple

logger = logging.getLogger("ediscovery.tar")
logger.setLevel(logging.INFO)


class TARError(Exception):
    pass


@dataclass
class Doc:
    doc_id: str
    features: Sequence[float]     # deterministic, cached text features
    code: str = ""               # "responsive" | "not_responsive" | "" (uncoded)


@dataclass
class Round:
    version: int
    reviewed: int
    recall_estimate: float


def estimate_recall(control: List[Doc], predict: Callable[[Doc], str]) -> float:
    """Unbiased recall against the frozen, never-trained control set."""
    tp = sum(1 for d in control if d.code == "responsive" and predict(d) == "responsive")
    fn = sum(1 for d in control if d.code == "responsive" and predict(d) != "responsive")
    denom = tp + fn
    if denom == 0:
        raise TARError("control set has no responsive documents; cannot estimate recall")
    return tp / denom


def cal_loop(
    pool: List[Doc],
    control: List[Doc],
    train_fn: Callable[[List[Doc]], Callable[[Doc], str]],
    batch_size: int,
    target_recall: float,
    stability_eps: float = 0.02,
    max_rounds: int = 100,
) -> Tuple[List[Round], Callable[[Doc], str]]:
    """Continuous active learning with a stabilization-aware stopping rule.
    `train_fn` returns a predict callable; `pool` documents get coded by review."""
    control_ids = {d.doc_id for d in control}
    if any(d.doc_id in control_ids for d in pool):
        raise TARError("control-set contamination: a control doc is in the trainable pool")

    history: List[Round] = []
    prev_recall = -1.0
    predict = train_fn([d for d in pool if d.code])  # seed model on any seeds

    for version in range(1, max_rounds + 1):
        # Rank uncoded pool by responsiveness; review the top batch.
        uncoded = [d for d in pool if not d.code]
        if not uncoded:
            logger.info("Pool exhausted at round %d", version)
            break
        ranked = sorted(uncoded, key=lambda d: _score(d, predict), reverse=True)
        for d in ranked[:batch_size]:
            d.code = _human_review(d)          # authoritative human coding

        # Retrain on all coded pool docs (never on the control set).
        predict = train_fn([d for d in pool if d.code])
        recall = estimate_recall(control, predict)
        history.append(Round(version, sum(1 for d in pool if d.code), recall))
        logger.info("round=%d reviewed=%d recall=%.3f", version, history[-1].reviewed, recall)

        # Stop only when the target is met AND the estimate has stabilized.
        stabilized = abs(recall - prev_recall) < stability_eps
        if recall >= target_recall and stabilized:
            logger.info("Stopping rule met at round %d (recall=%.3f)", version, recall)
            break
        prev_recall = recall

    return history, predict


def _score(doc: Doc, predict: Callable[[Doc], str]) -> float:
    # Placeholder deterministic score hook; real impl returns model probability.
    return float(sum(doc.features))


def _human_review(doc: Doc) -> str:
    # Authoritative coding is supplied by the review platform, not the model.
    raise NotImplementedError("wired to the review platform in production")

Observability & Compliance Metrics

Three KPIs make TAR defensible. Estimated recall against the control set is the headline metric and the one the stopping rule gates on; it is reported with its confidence interval, never as a bare point. Review efficiency (responsive documents found per thousand reviewed) measures how well the active-learning loop is prioritizing and should stay high until the responsive material is exhausted. Coding consistency (agreement rate on overlap documents) is the reliability signal — falling consistency means the model is being trained on noise and the recall estimate is suspect. Together with a versioned record of every model and its training set, these metrics constitute the statistical validation package a court expects.

from prometheus_client import Gauge, Counter

TAR_RECALL = Gauge("tar_recall_estimate", "Recall against the control set")
TAR_REVIEWED = Counter("tar_reviewed_total", "Documents coded by review")
TAR_CONSISTENCY = Gauge("tar_coding_consistency", "Agreement rate on overlap docs")


def record_round(rnd: "Round", consistency: float) -> None:
    TAR_RECALL.set(rnd.recall_estimate)
    TAR_REVIEWED.inc(rnd.reviewed)
    TAR_CONSISTENCY.set(consistency)

Conclusion

TAR earns its cost savings only if the process is defensible, and defensibility is an engineering property, not a model property. Freeze a control set and never train on it, version every model against the exact codes that produced it, retrain continuously so reviewers always see the most informative documents, and gate the stopping rule on a recall estimate that has both reached its target and stabilized. Guard against inconsistent coding, control-set contamination, and premature stopping with statistical checks that write to the audit trail. Built this way, TAR does not just review fewer documents — it produces a measured, reproducible recall estimate and an elusion test that together demonstrate the responsive material was found.

Frequently Asked Questions

What is the difference between simple active learning and continuous active learning?

Simple active learning (sometimes TAR 1.0) trains a model once on a seed set, ranks the collection, and reviews a fixed cutoff. Continuous active learning (CAL, or TAR 2.0) retrains after every review batch and continually re-ranks, always feeding reviewers the currently highest-scoring uncoded documents. CAL usually reaches a target recall with less total review because it adapts as it learns, and it removes the need to pick a seed set that perfectly represents the collection up front.

Why must the control set never be used for training?

Because the control set is the unbiased yardstick for recall. If any control document is also a training example, the model has seen the answer, and its performance on that document overstates how well it generalizes — the recall estimate becomes optimistically biased and indefensible. Hard-partitioning the control set from the trainable pool, and asserting no leakage at every retrain, is what keeps the recall estimate trustworthy.

How do you decide when to stop reviewing?

With a statistical stopping rule, not a fixed page count. Review continues until the estimated recall against the control set reaches the agreed target — often 75–80% at a stated confidence — and the estimate has stabilized across successive retrains. An elusion test on a random sample of the discard set then confirms the unreviewed tail contains few responsive documents. Stopping on a still-moving estimate risks leaving responsive material behind, so stability is a required condition, not just the target.

Is TAR defensible in court?

TAR is widely accepted when the process is documented and statistically validated. Courts have endorsed predictive coding since Da Silva Moore (2012), but acceptance turns on demonstrating a defensible protocol: a valid control set, a documented stopping rule, a recall estimate with a confidence interval, and an elusion test. The model choice matters far less than the reproducibility and the validation, which is why the engineering emphasis is on versioning, audit trails, and sampling rigor rather than on a sophisticated classifier.

Related

• Measuring TAR Recall and Elusion — the sampling math behind the recall estimate and elusion test.
• PDF Text Extraction Engines — the text source the classifier’s features are built from.
• Production Compliance Frameworks — the audit and reproducibility rules TAR’s validation feeds.
• Production Validation & QC — the downstream gate for the documents TAR routes to production.

Up one level: Review & Production Workflows — the pipeline that turns reviewed documents into court-ready productions.