Load-File Generation Standards: Concordance DAT/OPT and Relativity Interchange

A load file is the contract between two review platforms: it is the delimited index that tells the receiving system how to import a production — which metadata belongs to which document, where each document’s image pages and extracted text live, and how the natives are linked. Inside the Core Architecture & eDiscovery Taxonomy that governs the platform’s data interchange, load-file generation is the subsystem that serializes a finished production into the Concordance DAT/OPT and Relativity-compatible formats every vendor expects. It is unglamorous and unforgiving: the receiving party trusts the load file completely, so a delimiter that appears unescaped in a field, an encoding mismatch, or a field-count drift silently binds the wrong text to the wrong document on their system. This guide builds a load-file generator whose delimiters are correct by construction, whose encoding is declared and verified, and whose every row is validated before the volume ships.

Load-File Structure at a Glance

A production travels as a small family of coordinated files: a DAT carrying one delimited row of metadata per document, an OPT cross-referencing every image page to its file on disk with document-break markers, and the extracted-text and native files the DAT’s path fields point at. The generator emits them together and asserts they agree.

Concordance DAT/OPT load-file family structure A DAT metadata file holds one delimited row per document with fields for the Bates range, metadata, a text path, and a native path. An OPT image cross-reference lists every image page with its full path and a document-break flag. The DAT's path fields resolve to extracted-text files and native files on disk. A validator reconciles the DAT rows, the OPT page list, and the referenced files, confirming they all agree before the volume ships. One production, coordinated files that must agree DAT metadata þBEGþ ¶ þdateþ ¶ þtextþ one row per document OPT image x-ref IMG,VOL,path.tif,Y,,,3 page break flags Text files one .txt per doc Native files linked, not redacted Validator DAT ↔ OPT ↔ files agree The DAT's path fields resolve to the text and native files; the OPT resolves to the images — the validator proves every reference lands.

The Delimiter Problem and Design Rationale

The reason load files use exotic delimiters is that ordinary ones appear in the data. A comma-and-double-quote CSV breaks the moment a document’s subject line contains a comma inside an unescaped field or a quote character, and email metadata is full of both. The Concordance standard solves this by choosing delimiters that essentially never occur in document text: the field separator is ASCII 20 (the pilcrow ¶), the text qualifier is ASCII 254 (the thorn þ), and multi-value fields use ASCII 174 (the registered mark ®). Because these code points do not appear in normal metadata, the format sidesteps escaping entirely — but only if the generator guarantees they are absent from field values, which means scanning and stripping (or entity-encoding) any stray occurrence before serialization.

That guarantee is the whole design. A load-file generator that concatenates fields with the delimiter characters but never checks whether a field contains one has simply moved the CSV injection problem to a rarer character. The correct generator treats the delimiters as reserved bytes: every field value is validated to contain none of them, and any that slips through is a hard error, not a silent pass. Field selection and normalization draw on the routing rules in ESI format mapping standards, which define which metadata fields exist for each document type and how they are normalized before they reach the serializer.

Serialization Algorithm & Encoding

Two decisions determine whether the receiving platform reads the file correctly: the delimiter discipline above, and the character encoding. Modern productions are Unicode — document metadata contains names, subjects, and paths in every script — so the DAT is written as UTF-8 or UTF-16LE with a byte-order mark, and the encoding is declared, not left for the importer to guess. An unmarked UTF-16 file read as UTF-8 (or vice versa) mangles every non-ASCII character and, worse, can misalign the delimiters themselves, shifting entire columns. The generator therefore writes a BOM, records the encoding in the production’s transmittal, and verifies on read-back that the declared encoding decodes the whole file.

The header row is the field contract. Its column order and names define what each subsequent row means, and the receiving platform maps its own fields from that header. The generator emits a stable, documented header, holds every data row to exactly the header’s field count, and pads or omits per the agreed schema rather than letting a missing value drop a field and shift the row. The Relativity-oriented mapping — how these DAT fields correspond to Relativity’s expected field names and its BEGDOC/ENDDOC/BEGATTACH/ENDATTACH family fields — is handled in mapping metadata fields to Relativity load files, and the concrete DAT/OPT emission in generating Concordance DAT and OPT load files.

Resilience & Failure Routing

Load-file generation fails closed at the row level. A document whose field value contains a reserved delimiter byte, whose text or native path does not resolve on disk, or whose field count does not match the header is not written with a best-effort fixup — it is diverted to a defect queue with the specific violation, and the volume does not ship until the queue is empty. This is deliberate: a single malformed row can desynchronize a naive importer for every row after it, so the safe failure is to withhold the row, not to emit a guess.

Cross-file consistency is the second guard. The DAT and the OPT describe the same documents from different angles — the DAT one row per document, the OPT one row per image page — and they must agree on the document set and the Bates ranges. Before shipping, the generator reconciles them: every DAT document must have OPT pages, every OPT document-break must correspond to a DAT row, and the page counts must match. This reconciliation is the same discipline the downstream production validation and QC stage applies, run here at generation time so defects are caught before, not after, the load file is built.

Production Python Implementation

The module serializes a DAT with the Concordance standard delimiters, enforces the reserved-byte guarantee, writes a declared encoding with a BOM, and holds every row to the header’s field count. It compiles and runs on the standard library.

python
import logging
from dataclasses import dataclass
from pathlib import Path
from typing import Dict, List, Sequence

logger = logging.getLogger("ediscovery.loadfile")
logger.setLevel(logging.INFO)

# Concordance standard delimiters (as single characters / code points).
FIELD_SEP = "\x14"      # ¶  ASCII 20, separates fields
QUOTE = "\xfe"          # þ  ASCII 254, wraps each field value
MULTI_SEP = "\xae"      # ®  ASCII 174, separates values inside a multi-value field
RESERVED = {FIELD_SEP, QUOTE, MULTI_SEP}


class LoadFileError(Exception):
    pass


@dataclass
class DatRow:
    doc_id: str
    values: Dict[str, str]


def _guard_reserved(doc_id: str, field: str, value: str) -> None:
    """A field value may never contain a reserved delimiter byte. The exotic
    delimiters only work because they are absent from the data, so a stray
    occurrence is a hard error, not something to silently strip."""
    bad = RESERVED.intersection(value)
    if bad:
        names = ", ".join(f"U+{ord(c):04X}" for c in bad)
        raise LoadFileError(f"{doc_id}: field '{field}' contains reserved delimiter(s) {names}")


def serialize_row(row: DatRow, header: Sequence[str]) -> str:
    """Serialize one DAT row: þvalueþ¶þvalueþ… held to the header's field count."""
    cells: List[str] = []
    for field in header:
        value = row.values.get(field, "")
        _guard_reserved(row.doc_id, field, value)
        cells.append(f"{QUOTE}{value}{QUOTE}")
    if len(cells) != len(header):
        raise LoadFileError(f"{row.doc_id}: {len(cells)} fields, header has {len(header)}")
    return FIELD_SEP.join(cells)


def write_dat(path: Path, header: Sequence[str], rows: List[DatRow],
              encoding: str = "utf-16-le") -> int:
    """Write a DAT with a BOM and a declared encoding. Returns rows written.
    Fails the whole file if any row violates the reserved-byte guarantee."""
    if encoding.lower() not in {"utf-8", "utf-16-le"}:
        raise LoadFileError(f"unsupported encoding {encoding!r}")
    lines = [serialize_row(DatRow("__header__", {h: h for h in header}), header)]
    for row in rows:
        lines.append(serialize_row(row, header))
    data = "\r\n".join(lines) + "\r\n"
    # Prepend the BOM so the importer detects the encoding rather than guessing.
    bom = ""
    path.write_bytes((bom + data).encode(encoding))
    logger.info("Wrote %s: %d data row(s), %s", path.name, len(rows), encoding)
    return len(rows)


def verify_readback(path: Path, expected_rows: int, encoding: str = "utf-16-le") -> None:
    """Re-read the file under its declared encoding and confirm it decodes and
    has the expected row count — the fast check that catches an encoding lie."""
    text = path.read_bytes().decode(encoding)
    n = text.count("\r\n") - 1  # minus the header
    if n != expected_rows:
        raise LoadFileError(f"{path.name}: read back {n} rows, expected {expected_rows}")

Observability & Compliance Metrics

Three KPIs keep load-file generation honest. Row reject rate (rows diverted to the defect queue per thousand) surfaces upstream metadata problems — a spike usually means a normalization step let reserved bytes or over-long fields through. Cross-file discrepancy count (DAT-OPT mismatches per volume) must be zero before shipping and is the direct measure of the two files’ agreement. Encoding verification pass rate confirms every emitted file decodes under its declared encoding on read-back. Because the receiving party’s import success depends entirely on these, the metrics double as the compliance record that the interchange files were validated before transmittal.

python
from prometheus_client import Counter, Gauge

LF_ROWS = Counter("loadfile_rows_total", "DAT rows written")
LF_REJECTS = Counter("loadfile_rejects_total", "Rows diverted to the defect queue")
LF_XFILE_DISCREPANCIES = Gauge("loadfile_xfile_discrepancies", "DAT/OPT mismatches per volume")


def record_generation(written: int, rejected: int, discrepancies: int) -> None:
    LF_ROWS.inc(written)
    LF_REJECTS.inc(rejected)
    LF_XFILE_DISCREPANCIES.set(discrepancies)

Conclusion

A load file is trusted absolutely by the platform that reads it, so its correctness has to be guaranteed by construction, not hoped for. The Concordance delimiters work only because they are absent from the data, which makes the reserved-byte guarantee the generator’s central obligation; the encoding must be declared with a BOM and verified on read-back so columns never shift; and the DAT, the OPT, and the referenced files must be reconciled before shipping so every path resolves. Fail closed at the row level, keep a defect queue that must empty before transmittal, and instrument the reject and discrepancy rates. A generator built to these standards produces an interchange package the receiving party can import cleanly on the first attempt — which is the only outcome that does not generate a dispute.

Frequently Asked Questions

Why do Concordance load files use characters like ¶, þ, and ® as delimiters?

Because ordinary delimiters appear in the data. Commas and quotes are everywhere in email metadata, so a CSV needs constant escaping and breaks when the escaping is wrong. The Concordance standard picks code points — ASCII 20, 254, and 174 — that essentially never occur in document text, so the format can avoid escaping entirely. The trade-off is that the generator must guarantee those bytes are truly absent from every field value, treating them as reserved and erroring on any occurrence.

What encoding should a DAT file use?

Unicode with a declared byte-order mark — UTF-16LE (the traditional Concordance/Relativity default) or UTF-8, both with a BOM. The critical point is that the encoding is declared, not guessed: an unmarked file read as the wrong encoding mangles non-ASCII characters and can misalign the delimiters, shifting entire columns. Writing a BOM and verifying on read-back that the declared encoding decodes the whole file prevents the most common import failure.

What is the difference between the DAT and the OPT file?

The DAT is the metadata load file: one delimited row per document, carrying the Bates range, the metadata fields, and the paths to the extracted text and native. The OPT (Opticon) is the image cross-reference: one row per image page, giving each page’s alias, volume, and full path, with a document-break flag marking where each document begins. The DAT tells the platform about each document; the OPT tells it where the pages are. They must describe the same document set and agree on page counts.

How does a single bad row corrupt an entire production import?

Because delimited importers are positional. If one row has the wrong field count — a value that dropped out, or a reserved byte that split a field — the importer can misalign that row’s columns, and depending on the parser, the misalignment can cascade to following rows. The result is metadata bound to the wrong documents across a swath of the production. That is why generation fails closed at the row level: a malformed row is withheld to a defect queue rather than written, so it can never desynchronize the file.

Up: Core Architecture & eDiscovery Taxonomy — the domain that governs the platform’s data interchange contracts.