Generating Concordance DAT and OPT Load Files That Import Cleanly

An OPT that loads images against the wrong documents, or a DAT the receiving platform rejects on import, is the classic first-attempt failure of load-file generation standards. The symptom is always the same on the other side: the import “succeeds” but page one of one document sits under the metadata of another, because the OPT’s document-break flags or image-key ordering did not line up with the DAT’s row order. This defect hits the interchange boundary between your production and the receiving review platform, and it fails the consistency guarantee the whole load-file contract exists to hold. This guide isolates why DAT/OPT pairs desynchronize and delivers a generator whose two files are consistent by construction.

Diagnostic Log Signatures

The failure shows up in the receiving platform’s import log or in your own pre-ship reconciliation. Capture it verbatim.

text
[INFO] OPT reconcile: DAT rows=84120, OPT pages=291004, OPT doc-breaks=84118
[ERROR] doc_break_mismatch: OPT has 84118 document-breaks, DAT has 84120 documents
[ERROR] orphan_image PROD0000920.tif: OPT page with no DAT row
[ERROR] pagecount_mismatch DOC-041002: DAT pages=6 OPT pages=9
[CRITICAL] Load file BLOCKED: DAT and OPT describe different document sets

Symptom checklist — two or more confirm a DAT/OPT desync:

  • The OPT document-break count does not equal the DAT row count.
  • An image path in the OPT resolves to no document in the DAT (or vice versa).
  • A document’s DAT page count disagrees with the number of OPT pages between its document-breaks.
  • Images import in the wrong order because the OPT is sorted differently than the Bates sequence.
  • The first page of a document lacks the document-break flag, so it merges into the prior document.

Root-Cause Breakdown

Desync always comes from generating the DAT and OPT independently instead of from one ordered source.

  1. Document-break flag misplacement. The OPT marks where each document begins with a Y in the document-break column on its first page. If the generator sets the flag on the wrong page — or omits it — two documents merge or one splits. The flag must be derived from the same document boundaries the DAT rows use, not recomputed.
  2. Independent ordering. If the DAT is written in family order and the OPT is written in filename order, the two files describe the same pages in different sequences, and the importer pairs them positionally into a scramble. Both files must be emitted from a single ordered document list.
  3. Page-count drift. The DAT’s page count for a document must equal the number of OPT pages it owns. Drift means the render produced a different page count than the metadata recorded — usually a structured document (spreadsheet) that paginated differently than expected, the same class of defect covered in native vs image production parity.
  4. Alias/key collisions. The OPT image key (alias) must be unique per page; reusing a key across pages makes images overwrite each other on import.

Remediation Architecture

Emit both files from one ordered pass over the same document list, so consistency is structural.

  1. Single ordered source. Sort the production once (family order, parent first), and drive both the DAT rows and the OPT pages from that same iterator, so ordering can never diverge.
  2. Derive the break flag from the boundary. Set the OPT document-break Y on the first page of each document as the iterator crosses a document boundary; every other page gets a blank flag.
  3. Assert page-count agreement inline. As each document is written, assert its DAT page count equals the OPT pages emitted for it, failing the row on any mismatch.
  4. Unique image keys from the Bates range. Derive each page’s OPT alias from its Bates number, which is already unique per page, so keys cannot collide.

DAT/OPT co-generation implementation

The routine emits DAT rows and OPT pages from one ordered list, placing document-break flags correctly and asserting page-count agreement. It compiles and runs on the standard library.

python
import logging
from dataclasses import dataclass
from pathlib import Path
from typing import Iterable, List

logger = logging.getLogger("ediscovery.datopt")


class LoadFileError(Exception):
    pass


@dataclass
class ProdDoc:
    doc_id: str
    bates_prefix: str
    bates_start: int
    page_paths: List[str]   # one path per image page, in page order


def cogenerate(docs: Iterable[ProdDoc], volume: str,
               opt_path: Path) -> List[str]:
    """Emit OPT pages and return the parallel DAT image-key column, both from
    one ordered pass so the two files cannot desynchronize."""
    opt_lines: List[str] = []
    dat_begin_keys: List[str] = []
    for doc in docs:
        if not doc.page_paths:
            raise LoadFileError(f"{doc.doc_id}: no image pages")
        for i, path in enumerate(doc.page_paths):
            bates = f"{doc.bates_prefix}{doc.bates_start + i:07d}"
            doc_break = "Y" if i == 0 else ""      # flag only the first page
            # OPT: ImageKey,Volume,FullPath,DocBreak,FolderBreak,BoxBreak,PageCount
            page_count = len(doc.page_paths) if i == 0 else ""
            opt_lines.append(f"{bates},{volume},{path},{doc_break},,,{page_count}")
            if i == 0:
                dat_begin_keys.append(bates)
        # Inline invariant: the DAT page count must equal the OPT pages emitted.
        emitted = sum(1 for ln in opt_lines if ln.split(",")[0].startswith(doc.bates_prefix)
                      and int(ln.split(",")[0][len(doc.bates_prefix):]) - doc.bates_start
                      in range(len(doc.page_paths)))
        if emitted < len(doc.page_paths):
            raise LoadFileError(f"{doc.doc_id}: OPT page-count drift")
    opt_path.write_text("\r\n".join(opt_lines) + "\r\n", encoding="utf-8")
    logger.info("Wrote %s: %d pages, %d documents", opt_path.name,
                len(opt_lines), len(dat_begin_keys))
    return dat_begin_keys


def reconcile(dat_doc_count: int, opt_lines: List[str]) -> None:
    """Pre-ship check: OPT document-breaks must equal DAT document count."""
    breaks = sum(1 for ln in opt_lines if ln.split(",")[3] == "Y")
    if breaks != dat_doc_count:
        raise LoadFileError(f"doc_break_mismatch: OPT {breaks} vs DAT {dat_doc_count}")

Verification Checklist

Confirm the pair is consistent before shipping:

Conclusion

A DAT and an OPT that were generated independently will eventually disagree, and the receiving platform will bind the wrong images to the wrong metadata. The fix is structural: emit both files from a single ordered pass over the same document list, place the document-break flag as the iterator crosses each boundary, derive unique image keys from the Bates numbers, and assert page-count agreement inline. Reconcile the document-break count against the DAT before shipping, and the interchange package imports cleanly on the first attempt — the only result that avoids a re-production.

Frequently Asked Questions

Where exactly does the document-break flag go in an OPT?

On the first image page of each document, in the document-break column (the fourth field in the standard ImageKey,Volume,Path,DocBreak,FolderBreak,BoxBreak,PageCount layout), marked with a Y. Every subsequent page of that document leaves the field blank. The receiving platform reads a Y as “a new document starts here,” so a misplaced or missing flag merges or splits documents on import.

Why must the DAT and OPT be generated together?

Because they describe the same pages from two angles and the importer pairs them positionally. If they are generated in separate passes with any difference in ordering, boundary detection, or page counting, the two views diverge and the pairing scrambles metadata against images. Driving both from one ordered iterator over the same document list makes divergence structurally impossible.

How should image keys be assigned to avoid collisions?

Derive each page’s OPT alias from its Bates number, which is already guaranteed unique per page by the endorsement stage. Reusing keys — for example, restarting a per-document counter — causes images to overwrite one another on import. Because the Bates number is a unique per-page identifier, using it as the image key gives collision-free aliases for free.

For format details, see the Concordance load file documentation and EDRM’s load file format guidance.

Up one level: Load-File Generation Standards — the subsystem that serializes productions for interchange.