Production Validation & QC: Gating a Volume Before It Ships

Production validation is the last automated checkpoint between a review-approved collection and a delivery that cannot be recalled. It exists because the production stage deliberately fabricates new artifacts — endorsed images, extracted text, a delimited load file — and every one of those is a chance for the volume to become internally inconsistent: an image with no manifest row, a Bates gap that hints at a dropped page, a redacted document whose native slipped into the export, a load file whose text column points at the wrong document. Inside the broader Review & Production Workflows pipeline, QC is the stage that refuses to seal a volume until it has proven, mechanically, that the images, text, natives, metadata, and privilege designations all agree. This guide builds that gate as a streaming, fail-closed validator engineered so that the same volume checked twice yields the same verdict, and every rejection points at the exact document that caused it.

QC Subsystem Flow

Validation is not a single assertion; it is a sequence of independent checks, each owning a distinct defect class and a distinct remediation path. Manifest reconciliation confirms the image set and the manifest are bijective, Bates auditing confirms the numbering is gap-free and collision-free, parity checking confirms every produced document has the artifacts its disposition requires, and redaction verification confirms nothing protected survived into the deliverable. A failure at any gate diverts the offending document to a rejection queue rather than failing the whole run silently.

Production validation and QC subsystem flow Six stages run left to right. Manifest reconciliation checks that every image has a manifest row and every row an image. Bates continuity auditing confirms the numbering is gap-free and has no collisions. Image-text-native parity confirms each document carries the artifacts its disposition requires. Redaction verification confirms no protected text or native survived. Load-file integrity checks delimiters, encoding, and link resolution. The final gate seals the volume or routes failing documents to a rejection queue keyed by defect class. Fail closed: no gate passes on doubt 1 Manifest reconcile image ↔ row, bijective 2 Bates continuity no gaps, no collisions 3 Parity check image · text · native 4 Redaction verify no protected text left 5 Load-file integrity delimiter · encoding 6 Seal or reject defect-class queue A failing document is diverted to a rejection queue keyed by defect class — the volume names the culprit, it never fails blind.

Memory & Resource Constraints

A production volume can hold hundreds of thousands of documents and millions of image pages, so a validator that materializes the whole manifest, the whole image list, and the whole Bates set in memory at once will exhaust a worker long before it finishes. The naive implementation — load the load file into a list of dicts, glob the image directory into a list, build a Python set of every Bates number — carries three unbounded collections whose combined footprint scales with volume size, and it is exactly the pattern that dies at the 80th percentile of real productions.

The validator instead streams. The manifest is read line by line, the image tree is walked lazily, and reconciliation is performed against bounded, disk-backed structures rather than in-memory sets. Bates continuity is checked by sorting the claimed ranges once and scanning for gaps and overlaps in a single linear pass, which needs only the previous range in memory, not the whole set. Parity is checked per document as its manifest row is read, so the working set is one document’s artifacts at a time. This keeps memory a function of the widest single document, not of the volume, and it means a 50,000-document volume and a 500,000-document volume validate with the same resident footprint — only the wall-clock time grows.

Validation Algorithm Deep-Dive

The core of the validator is the Bates continuity scan, because Bates defects are both the most common and the most consequential. A production’s Bates numbers must form a monotonic, gap-free sequence within a volume, and every document must occupy a contiguous range whose length equals its page count. Two failure shapes matter: a gap (a missing number, which suggests a dropped page or a document that failed to render) and a collision (a number claimed by two documents, which is fatal because it gives two different pages the same citation). Both are detected by claiming that the ranges, once sorted by start value, must tile the number line with no gap and no overlap.

Family integrity is the second algorithm. A production must not split a family across a volume boundary or a numbering discontinuity — a parent email and its attachments have to travel together, in order, so a reviewer on the receiving side sees the family intact. The validator groups produced documents by their family_id, carried down from deduplication and family grouping, and asserts that each family occupies a contiguous Bates block with the parent first. When either the continuity scan or the family scan fails, the specific documents are routed for remediation using the diagnostic procedure in diagnosing Bates gaps and family breaks.

Resilience & Failure Routing

QC is a fail-closed stage: any check that cannot be completed with certainty is treated as a failure, never as a pass-by-default. If an image file is unreadable, the parity check for that document fails rather than skipping it; if the load file has a malformed row, the whole file is quarantined rather than partially trusted. Each failure is written to a rejection queue as a structured record — document id, defect class, the two values that disagreed, and the volume position — so remediation is targeted and the rest of the volume continues to validate.

The rejection queue is also the audit artifact. A production that shipped with zero rejections and a production that shipped after fixing forty rejections are equally defensible only if the forty rejections and their resolutions are on record. The queue therefore never deletes; a resolved rejection is marked resolved with the corrective action and the re-validation result, so the QC history of a volume is a complete, append-only account of every defect found and how it was cured. This is the record that answers a completeness challenge months later, when opposing counsel asks how you know the production was internally consistent.

Production Python Implementation

The module streams a production manifest and its image tree, runs the reconciliation, Bates-continuity, parity, and family checks, and emits a structured rejection record for every defect. It compiles and runs against a manifest of JSON-lines rows like those written by the production engine. The redaction and load-file checks are represented by focused hooks; the failure-routing and audit behavior is complete.

python
import json
import logging
from dataclasses import dataclass, field
from pathlib import Path
from typing import Dict, Iterable, Iterator, List, Optional

logger = logging.getLogger("ediscovery.qc")
logger.setLevel(logging.INFO)


@dataclass
class ManifestRow:
    doc_id: str
    family_id: str
    bates_start: int
    bates_end: int
    image_path: Optional[str]
    text_path: Optional[str]
    native_path: Optional[str]
    withheld: bool
    redacted: bool


@dataclass
class Rejection:
    doc_id: str
    defect: str
    expected: str
    found: str


@dataclass
class QCReport:
    checked: int = 0
    rejections: List[Rejection] = field(default_factory=list)

    @property
    def sealed(self) -> bool:
        return not self.rejections


def read_manifest(path: Path) -> Iterator[ManifestRow]:
    """Stream the manifest one row at a time so memory stays bounded."""
    with open(path, "r", encoding="utf-8") as fh:
        for lineno, line in enumerate(fh, 1):
            line = line.strip()
            if not line:
                continue
            try:
                d = json.loads(line)
            except json.JSONDecodeError as exc:
                raise ValueError(f"Malformed manifest row at line {lineno}: {exc}") from exc
            yield ManifestRow(
                doc_id=d["doc_id"],
                family_id=d["family_id"],
                bates_start=int(d["bates_start"]),
                bates_end=int(d["bates_end"]),
                image_path=d.get("image_path"),
                text_path=d.get("text_path"),
                native_path=d.get("native_path"),
                withheld=bool(d.get("withheld", False)),
                redacted=bool(d.get("redacted", False)),
            )


def check_bates_continuity(rows: List[ManifestRow]) -> List[Rejection]:
    """Sorted single pass: ranges must tile with no gap and no overlap."""
    produced = sorted((r for r in rows if not r.withheld), key=lambda r: r.bates_start)
    out: List[Rejection] = []
    prev: Optional[ManifestRow] = None
    for r in produced:
        if r.bates_end < r.bates_start:
            out.append(Rejection(r.doc_id, "bates_inverted",
                                 f">= {r.bates_start}", str(r.bates_end)))
        if prev is not None:
            if r.bates_start <= prev.bates_end:
                out.append(Rejection(r.doc_id, "bates_collision",
                                     f"> {prev.bates_end}", str(r.bates_start)))
            elif r.bates_start != prev.bates_end + 1:
                out.append(Rejection(r.doc_id, "bates_gap",
                                     str(prev.bates_end + 1), str(r.bates_start)))
        prev = r
    return out


def check_parity(row: ManifestRow, root: Path) -> List[Rejection]:
    """Each produced document must carry the artifacts its disposition requires."""
    out: List[Rejection] = []
    if row.withheld:
        return out  # withheld items are validated against the privilege log, not here
    if not row.image_path or not (root / row.image_path).exists():
        out.append(Rejection(row.doc_id, "missing_image", "image on disk", "absent"))
    if not row.text_path or not (root / row.text_path).exists():
        out.append(Rejection(row.doc_id, "missing_text", "text on disk", "absent"))
    # A redacted document must NOT carry its native into the production set.
    if row.redacted and row.native_path:
        out.append(Rejection(row.doc_id, "redacted_native_leak", "no native", row.native_path))
    return out


def check_family_contiguity(rows: List[ManifestRow]) -> List[Rejection]:
    """Every family must occupy one contiguous Bates block, parent first."""
    out: List[Rejection] = []
    by_family: Dict[str, List[ManifestRow]] = {}
    for r in rows:
        if not r.withheld:
            by_family.setdefault(r.family_id, []).append(r)
    for family_id, members in by_family.items():
        members.sort(key=lambda r: r.bates_start)
        span = members[-1].bates_end - members[0].bates_start + 1
        pages = sum(r.bates_end - r.bates_start + 1 for r in members)
        if span != pages:
            out.append(Rejection(members[0].doc_id, "family_split",
                                 f"contiguous block of {pages}", f"span of {span}"))
    return out


def validate_volume(manifest_path: Path, image_root: Path) -> QCReport:
    report = QCReport()
    rows = list(read_manifest(manifest_path))
    report.checked = len(rows)
    for row in rows:
        report.rejections.extend(check_parity(row, image_root))
    report.rejections.extend(check_bates_continuity(rows))
    report.rejections.extend(check_family_contiguity(rows))
    if report.sealed:
        logger.info("Volume sealed: %d documents, 0 defects", report.checked)
    else:
        for rej in report.rejections:
            logger.error("QC defect %s on %s: expected %s, found %s",
                         rej.defect, rej.doc_id, rej.expected, rej.found)
    return report

Observability & Compliance Metrics

Three KPIs make the QC stage legible at scale. Validation throughput (documents per second) tells you whether QC finishes inside the production window. Defect rate (rejections per thousand documents, by defect class) is the quality signal for the upstream production engine — a rising bates_gap rate points at a failing renderer, a rising redacted_native_leak rate points at a regression in redaction automation. Seal latency (time from validation start to a clean seal, including remediation loops) is the operational signal that predicts whether the volume ships on time.

python
from prometheus_client import Counter, Histogram

QC_DOCS = Counter("qc_documents_total", "Documents validated")
QC_DEFECTS = Counter("qc_defects_total", "QC defects", ["defect"])
QC_SECONDS = Histogram("qc_volume_seconds", "Seconds to validate one volume")


def instrumented_validate(manifest_path, image_root) -> "QCReport":
    with QC_SECONDS.time():
        report = validate_volume(manifest_path, image_root)
    QC_DOCS.inc(report.checked)
    for rej in report.rejections:
        QC_DEFECTS.labels(defect=rej.defect).inc()
    return report

Each metric is also a compliance signal: the defect-class breakdown is the evidence that the production was validated against a defined standard, and the append-only rejection history is the record that every defect found before shipping was resolved and re-validated.

Conclusion

Production validation is the gate that converts “we exported the volume” into “we can prove the volume is internally consistent.” Streaming the manifest keeps memory flat, a sorted single-pass scan catches every Bates gap and collision, per-document parity confirms each disposition got the artifacts it requires, and a family-contiguity check keeps parents and attachments together. Route every failure to an append-only rejection queue keyed by defect class, refuse to seal on any doubt, and the QC stage becomes both a quality gate and the durable record that answers a completeness or consistency challenge long after the production has shipped.

Frequently Asked Questions

What is the difference between a Bates gap and a Bates collision?

A gap is a missing number in the sequence — the ranges leave a hole — which usually means a page or document failed to render and dropped out of the volume. A collision is the opposite: two documents claim overlapping numbers, so a single Bates citation resolves to two different pages. Gaps are recoverable (re-render the missing item into the hole or reissue), but collisions are fatal to a production’s integrity because the citation is no longer unique, and they must be fixed before the volume ships.

Should QC ever skip a document it cannot read?

No. QC is fail-closed: a document whose image or text cannot be read fails the parity check rather than being skipped. Skipping-on-error is how defective documents reach the deliverable — the check that could have caught the defect silently passed. An unreadable artifact is a defect in itself and belongs in the rejection queue with everything else.

How does QC confirm a redaction actually held?

Parity alone confirms the redacted document carries no native into the production set, but full redaction verification also extracts text from the produced image and asserts that none of the protected terms survive. Because a correct burn-in rasterizes the protected span into the image, the extracted text should contain no recoverable protected content; if it does, the redaction was applied to a layer that the renderer preserved rather than to the pixels, and the document is rejected.

Why keep resolved rejections instead of deleting them?

Because defensibility rests on the record, not just the outcome. A volume that shipped clean and a volume that shipped after curing forty defects are equally defensible only if the forty defects and their resolutions are documented. The append-only rejection history is the artifact that answers “how do you know this production was consistent?” — it shows every defect found, the corrective action, and the passing re-validation.

Up one level: Review & Production Workflows — the pipeline that assembles the volume this stage certifies.