Validating Load-File Encoding and Delimiters Before a Production Ships

Garbled accented characters, shifted metadata columns, and an importer that rejects the whole DAT are the visible symptoms of an encoding or delimiter defect — the failure mode that turns an otherwise-correct production into an unusable one at the receiving platform. This problem sits in the validation step of load-file generation standards, and it is insidious because the file looks fine in a text editor that guesses the encoding correctly while failing on the importer that does not. It violates the interchange contract’s most basic promise: that the bytes you send decode to the values you meant. This guide isolates the encoding and delimiter defects that break imports and delivers a validator that reads the file back exactly as the receiving platform will.

Diagnostic Log Signatures

The validator reads the DAT as bytes and reports what a strict importer would see. Capture it verbatim.

[INFO] DAT validate: prod012.dat, declared=utf-16-le, bom=present
[ERROR] encoding_mismatch: declared utf-16-le but byte 0 is 0xEF (looks like UTF-8 BOM)
[ERROR] field_count_drift row=40218: 24 fields, header has 25
[ERROR] reserved_in_field row=51120 field='Subject': contains U+00FE (text qualifier)
[WARN] unqualified_field row=9001 field='Custodian': value not wrapped in text qualifier
[CRITICAL] DAT BLOCKED: 3 structural defects, encoding declaration is wrong

Symptom checklist — two or more confirm an encoding/delimiter defect:

• The declared encoding does not match the actual byte pattern (BOM says one thing, bytes say another).
• Some rows have a different field count than the header when split on the field separator.
• A field value contains a raw field-separator, text-qualifier, or multi-value byte.
• Non-ASCII characters render as mojibake (é, ’) when decoded with the declared encoding.
• The importer reports a hard parse error on a specific row rather than a data warning.

Root-Cause Breakdown

Encoding and delimiter defects reduce to a mismatch between what the file declares and what it contains.

1. Encoding declared but not applied (or vice versa). The transmittal says UTF-16LE but the file was written UTF-8, or a BOM for one encoding sits in front of bytes in another. The importer trusts the declaration, decodes with the wrong codec, and every non-ASCII character is corrupted — and worse, in UTF-16 a wrong-codec read can misalign the two-byte units so the delimiters themselves land mid-character.
2. Reserved byte inside a field. A metadata value legitimately contains the code point chosen as a delimiter — rare, but it happens with certain document subjects or paths. If the generator did not enforce the reserved-byte guarantee, that byte splits the field, and the row now has too many columns.
3. Field-count drift from a dropped value. A missing value that was omitted rather than emitted as an empty qualified field leaves the row one column short, shifting every field after it and binding the wrong values to the wrong columns.
4. Inconsistent qualification. Some fields wrapped in the text qualifier and some not; a strict importer that expects every field qualified fails on the unqualified ones.

Remediation Architecture

Validate the file the way the importer will: as bytes, under the declared encoding, split on the exact delimiter bytes.

1. Verify the encoding declaration against the bytes. Inspect the BOM and attempt a strict decode under the declared encoding; a decode error or a BOM/declaration mismatch is a hard fail.
2. Split on the reserved bytes and count. Decode each row, split on the field separator, and assert the field count equals the header’s — the direct test for drift and for reserved bytes hiding in values.
3. Scan values for reserved bytes. After stripping the qualifiers, confirm no field value contains any reserved delimiter, catching the injection at its source.
4. Round-trip a sample. Decode, re-encode under the declared encoding, and confirm the bytes are stable, proving the encoding is self-consistent.

The validation flow reads the produced DAT as bytes and runs the four checks before the file is cleared to ship.

Byte-level validator implementation

The validator inspects the BOM, decodes strictly under the declared encoding, splits on the reserved separator, and scans values. It compiles and runs on the standard library.

import logging
from dataclasses import dataclass, field
from pathlib import Path
from typing import List, Sequence

logger = logging.getLogger("ediscovery.loadfile.validate")

FIELD_SEP = "\x14"
QUOTE = "\xfe"
MULTI_SEP = "\xae"
RESERVED_IN_VALUE = {FIELD_SEP, MULTI_SEP}   # QUOTE is stripped before the scan

BOMS = {"utf-8": b"\xef\xbb\xbf", "utf-16-le": b"\xff\xfe"}


@dataclass
class Defect:
    row: int
    kind: str
    detail: str


def validate_dat(path: Path, declared_encoding: str,
                 header: Sequence[str]) -> List[Defect]:
    raw = path.read_bytes()
    defects: List[Defect] = []

    # 1. BOM must match the declared encoding.
    expected_bom = BOMS.get(declared_encoding.lower())
    if expected_bom and not raw.startswith(expected_bom):
        defects.append(Defect(0, "encoding_mismatch",
                              f"declared {declared_encoding} but BOM absent/wrong"))
        return defects  # decoding under a lie is pointless

    # 2. Strict decode: any error is a hard fail, not a replaced character.
    try:
        text = raw.decode(declared_encoding)
    except UnicodeDecodeError as exc:
        defects.append(Defect(0, "decode_error", str(exc)))
        return defects

    rows = [ln for ln in text.split("\r\n") if ln]
    for i, row in enumerate(rows):
        cells = row.split(FIELD_SEP)
        # 3. Field count must equal the header.
        if len(cells) != len(header):
            defects.append(Defect(i, "field_count_drift",
                                  f"{len(cells)} fields, header has {len(header)}"))
        # 4. No reserved separator may survive inside a value.
        for h, cell in zip(header, cells):
            value = cell.strip(QUOTE)
            bad = RESERVED_IN_VALUE.intersection(value)
            if bad:
                names = ", ".join(f"U+{ord(c):04X}" for c in bad)
                defects.append(Defect(i, "reserved_in_field", f"{h}: {names}"))
    if not defects:
        logger.info("%s: clean, %d rows under %s", path.name, len(rows) - 1, declared_encoding)
    return defects

Verification Checklist

Confirm the file decodes as the importer will before shipping:

• The BOM matches the declared encoding, and a strict decode of the whole file succeeds with no replaced characters.
• Every row splits into exactly the header’s field count on the field separator.
• No field value contains a raw field-separator or multi-value delimiter after qualifiers are stripped.
• A sample of non-ASCII values (accented names, non-Latin subjects) round-trips through decode and re-encode unchanged.
• The declared encoding is recorded in the production transmittal so the receiving party imports with the correct codec.

Conclusion

Encoding and delimiter defects are a mismatch between what a load file declares and what it contains, and they are invisible to a forgiving text editor while fatal to a strict importer. Validate the file the way the receiving platform will — as bytes, under the declared encoding, split on the exact reserved separators — and check the four things that break imports: the BOM/encoding declaration, the per-row field count, reserved bytes hiding in values, and round-trip stability. Block on any defect and record the declared encoding in the transmittal. A load file validated this way decodes to exactly the values you produced, on the first import, on the receiving party’s system.

Frequently Asked Questions

Why does a file that opens fine in my editor still fail the import?

Because your editor guesses the encoding and usually guesses right, while a strict importer uses the declared encoding and does not guess. If the declaration is wrong, the editor’s correct guess masks the defect that the importer will hit. Validating as bytes under the declared encoding — not opening it in a lenient editor — is what reproduces the importer’s view and surfaces the problem before you ship.

How can a delimiter end up inside a field if the delimiters are so rare?

Rarely, but it happens: a document subject, a file path, or a pasted snippet can legitimately contain the code point chosen as the text qualifier or multi-value separator. If the generator did not enforce the reserved-byte guarantee, that byte passes into the serialized value and splits the field on import. The validator’s reserved-byte scan is the backstop that catches an occurrence the generator missed.

Should the DAT be UTF-8 or UTF-16?

Either works if it is declared and applied consistently. UTF-16LE with a BOM is the traditional Concordance/Relativity default and many importers assume it; UTF-8 with a BOM is increasingly accepted and more compact. The failure is never the choice itself — it is a mismatch between the declaration and the bytes. Pick one, write the matching BOM, verify on read-back, and record it in the transmittal.

Related

• Load-File Generation Standards — the reserved-delimiter and encoding contract this validates.
• Generating Concordance DAT and OPT Load Files — the emission step whose output this checks.
• Mapping Metadata Fields to Relativity Load Files — the field schema the count check is measured against.
• Production Validation & QC — the volume-level gate the load file feeds into.

For encoding fundamentals, see the Unicode BOM FAQ and Python’s codecs documentation.

Up one level: Load-File Generation Standards — the subsystem that serializes productions for interchange.