Resolving libmagic MIME Misclassification and octet-stream Fallbacks in eDiscovery Ingestion

An extension-spoofed .docx that is really a ZIP archive, or a truncated PST that resolves to application/octet-stream, silently derails the classification step at the front of every Native File Ingestion Pipeline. MIME resolution is the primary routing decision in the EDRM Processing stage: it selects the extraction engine, the text-normalization path, and the hashing sequence for each artifact. When libmagic returns the wrong type, PDF parsers choke on misidentified OLE2 containers, OCR workers exhaust memory on binary blobs, and production manifests emit hash mismatches that break the reproducibility guarantee opposing counsel probes first. This page isolates why libmagic misclassifies native files in litigation datasets, gives the verbatim log signatures that expose it, and provides a deterministic, auditable detector that routes every file to a handler, a secondary scan, or quarantine.

Diagnostic Log Signatures

The failure is deterministic, not stochastic: it reproduces whenever a custodian-renamed container, a header-damaged archive, or a multi-signature compound file passes through a detector that trusts the first matched rule. Worker logs carry a recognizable signature well before a manifest rejection surfaces:

WARN  esi.mime_scanner: unrecognized_mime application/octet-stream file=/vol/ingest/L0007/CUST-0442.docx size=2088441
DEBUG esi.router: routed application/zip -> office_open_xml handler; xml_namespace_parse returned 0 parts
ERROR esi.mime_scanner: libmagic signature conflict at /vol/ingest/L0007/dwg_export.pdf: could not find any valid magic files
ERROR esi.manifest: hash_mismatch id=CUST-0442 handler=octet_stream_fallback expected=sha256:9f2c… got=sha256:00e1…
INFO  esi.mime_scanner: secondary_scan header_ambiguous -> full_file_scan file=/vol/ingest/L0007/archive.pst

Two lines are diagnostic. unrecognized_mime application/octet-stream on a file with a valid extension means the magic signature fell outside the header read buffer — a truncated or padded header, not an unknown format. routed application/zip -> office_open_xml handler ... returned 0 parts means libmagic was correct (the file is a bare ZIP) but downstream routing assumed Office Open XML structure and silently extracted nothing. Symptom checklist:

• A file with a known-good extension resolves to application/octet-stream and drops into the generic binary queue.
• A .docx/.xlsx is detected as application/zip and the OOXML handler returns zero document parts.
• libmagic signature conflict or could not find any valid magic files appears — a stale or missing compiled magic.mgc.
• Manifest hash mismatches correlate exactly with octet_stream_fallback handler tags, never with cleanly routed files.
• Reprocessing the same file yields the same wrong type — reproducible, which rules out transient I/O corruption.

Root-Cause Breakdown

libmagic ignores file extensions entirely, reading a configurable header buffer (default 64 KB) and matching raw byte sequences against a compiled signature database under strict offset constraints. Four contributing factors turn that design into misclassification at ESI scale:

1. Truncated or padded headers falling outside the read buffer. Email archives (.pst, .ost) and forensic disk images routinely arrive with damaged initial sectors or null-byte padding at offset 0x00/0x08. When the distinguishing signature lands beyond the header buffer, libmagic defaults to application/octet-stream, halting specialized parsers and obscuring chain-of-custody metadata.
2. Container masquerading and extension spoofing. Custodians rename .zip archives as .docx or .xlsx. libmagic correctly reports the underlying ZIP signature (PK\x03\x04), but routing logic that keys off the extension rather than the detected type feeds a bare archive into an Office Open XML parser, producing a silent zero-part extraction.
3. Conflicting multi-signature files with no priority weighting. Modern native files embed multiple valid signatures — a PDF containing an embedded ZIP, or an OLE2 compound file with a JPEG thumbnail at a non-standard offset. Without MAGIC_CONTINUE and explicit first-match priority, libmagic returns whichever rule matched first, which may not be the primary document type the pipeline must route on.
4. A stale or environment-mismatched magic.mgc. Outdated distributions predate modern container formats and misclassify them (for example, treating newer OOXML variants as generic ZIP). If the compiled database shipped with the base image differs from the one the code loads, detection is non-reproducible across nodes — the same defensibility problem that surfaces in cryptographic hash generation when byte streams are read non-deterministically.

Remediation Architecture

Deterministic MIME resolution needs four controls: MAGIC_CONTINUE with explicit first-match priority, a header-buffer read with a full-file secondary scan on ambiguity, a routing allowlist that quarantines anything unrecognized instead of guessing, and an audit record for every decision. The flowchart below traces how a magic-byte match resolves to a handler, a full-file rescan, or quarantine.

1. Route on the detected type, never the extension

Before the detector runs, fix the defect that turns a correct ZIP detection into a broken OOXML parse: a bare PK\x03\x04 archive and a genuine .docx share a container signature but not a routing target. The table below is the authoritative signature-to-handler map the router consults on the detected MIME type, not the filename.

2. Deterministic detector with secondary scan and allowlist gate

The detector loads MAGIC_CONTINUE | MAGIC_ERROR | MAGIC_MIME_TYPE, retains the highest-priority (first) match, falls back to a full-file scan when the header is ambiguous, and validates the result against a routing allowlist. It offloads the blocking libmagic call to a thread so it is safe to dispatch from an async batch processing worker pool.

import asyncio
import logging
from pathlib import Path
from typing import Any, Optional

import magic

logger = logging.getLogger("esi.mime_scanner")

# Detected-type prefixes cleared for downstream extraction routing.
ALLOWED_MIME_PREFIXES: frozenset[str] = frozenset({
    "application/pdf",
    "application/msword",
    "application/vnd.openxmlformats-officedocument",
    "application/vnd.ms-excel",
    "application/vnd.ms-outlook",
    "application/x-ole-storage",
    "application/zip",
    "message/rfc822",
    "text/plain",
    "text/html",
    "image/jpeg",
    "image/png",
})


class DefensibleMimeDetector:
    """Thread-safe, audit-friendly MIME detection for native ESI ingestion."""

    def __init__(self, magic_db_path: Optional[str] = None, buffer_size: int = 65536):
        # MAGIC_CONTINUE returns every matching rule, newline-separated; we keep
        # the first (highest-priority) line. MAGIC_ERROR surfaces a missing or
        # stale magic.mgc as an exception instead of a silent misclassification.
        flags = magic.MAGIC_MIME_TYPE | magic.MAGIC_ERROR | magic.MAGIC_CONTINUE
        if magic_db_path:
            self._handle = magic.Magic(magic_file=magic_db_path, flags=flags)
        else:
            self._handle = magic.Magic(flags=flags)
        self._buffer_size = buffer_size

    @staticmethod
    def _primary_mime(raw: str) -> str:
        # Candidates are newline-separated; the first line is the highest-priority
        # match. A valid MIME type is "type/subtype", so never split on "/".
        return raw.splitlines()[0].strip() if raw else ""

    async def detect(self, file_path: str) -> dict[str, Any]:
        path = Path(file_path)
        if not path.is_file():
            raise FileNotFoundError(f"Target file not found: {file_path}")

        file_size = path.stat().st_size
        if file_size == 0:
            return {"mime": "inode/x-empty", "status": "resolved", "path": str(path)}

        try:
            with open(path, "rb") as f:
                header = f.read(self._buffer_size)

            # Offload the blocking libmagic computation to the thread pool.
            mime_result = await asyncio.to_thread(self._handle.from_buffer, header)
            primary_mime = self._primary_mime(mime_result)

            # Header was truncated or ambiguous: rescan the whole file so a
            # distinguishing signature past the buffer window is not missed.
            if not primary_mime or primary_mime == "application/octet-stream":
                full_scan = await asyncio.to_thread(self._handle.from_file, str(path))
                primary_mime = self._primary_mime(full_scan)

            if not any(primary_mime.startswith(p) for p in ALLOWED_MIME_PREFIXES):
                logger.warning("unrecognized_mime %s file=%s", primary_mime, file_path)
                return {
                    "mime": primary_mime,
                    "status": "quarantine_required",
                    "path": str(path),
                    "size_bytes": file_size,
                }

            logger.info("mime_ok %s file=%s bytes=%d", primary_mime, file_path, file_size)
            return {
                "mime": primary_mime,
                "status": "resolved",
                "path": str(path),
                "size_bytes": file_size,
            }
        except magic.MagicException as e:
            # Missing/stale magic.mgc or an internal signature conflict.
            logger.error("libmagic signature conflict at %s: %s", file_path, e)
            return {
                "mime": "application/octet-stream",
                "status": "fallback_required",
                "path": str(path),
                "error": str(e),
            }

3. Isolate a live misclassification

When manifests already report mismatches, confirm the root cause on the suspect file rather than guessing:

1. Hexdump the header. Run xxd -l 512 <file> | head -n 10 and compare the raw signature against the ESI format mapping standards reference. Null-byte padding at offset 0x00 or 0x08 confirms header corruption, not an unknown format.
2. Audit the compiled database. Verify the active magic.mgc matches the deployment image; recompile custom forensic signatures with file -C -m /path/to/custom/magic. A version drift between nodes is the usual cause of non-reproducible detection.
3. Cross-reference the registry. Validate ambiguous multi-signature offsets against the UK National Archives PRONOM registry, which publishes authoritative container hierarchies that resolve MAGIC_CONTINUE priority conflicts.

Verification Checklist

• The detector loads MAGIC_CONTINUE | MAGIC_ERROR | MAGIC_MIME_TYPE and keeps only the first matched line as the primary type.
• Routing keys off the detected MIME type, not the file extension; a spoofed .docx that is really a ZIP routes to archive_expand, not the OOXML handler.
• Any header that resolves to application/octet-stream triggers a full-file secondary scan before quarantine.
• Unrecognized types are quarantined with audit metadata, never silently routed to the generic binary queue.
• The active magic.mgc version is identical across every worker image and is logged with each decision.
• Every detection event records the resolved MIME type, buffer size, file size, and SHA-256 to the immutable audit ledger.
• A full re-run shows no unrecognized_mime, signature conflict, or octet_stream_fallback hash mismatches.

Conclusion

MIME misclassification in ingestion is almost never a broken library — it is a router that trusted a filename, a header that fell outside the read buffer, or a compiled database that drifted between nodes. Loading libmagic with MAGIC_CONTINUE and a fixed first-match priority makes the classification deterministic, the full-file secondary scan rescues truncated headers instead of dumping them into application/octet-stream, and the allowlist gate quarantines the genuinely unknown rather than guessing. With every decision keyed to detected bytes and written to an audit ledger alongside its SHA-256 anchor, each artifact routes to the correct extraction engine and the classification step regains the reproducibility the Processing stage exists to guarantee.

Frequently Asked Questions

Why does a valid .docx resolve to application/zip instead of an Office type?

Because Office Open XML files are ZIP containers — libmagic reads the PK\x03\x04 signature and, if the compiled database is older than the format, stops at the generic archive rule. A current magic.mgc inspects the [Content_Types].xml member and returns the specific OOXML type. Either way, route on the detected type: when the result is bare application/zip, hand the file to an archive-expand handler that inspects the members, not to an OOXML parser that assumes document structure and silently extracts nothing.

How do I stop truncated PST files from defaulting to application/octet-stream?

The distinguishing signature has fallen outside the header read buffer or is masked by null-byte padding. First, let the detector fall back to a full-file scan on any octet-stream header — the secondary scan reaches signatures past the 64 KB window. If the header itself is damaged, hexdump the first 512 bytes to confirm corruption, then re-extract the store from the original source media rather than certifying a guess. Never let a truncated container route to the generic binary queue without an audit entry recording the fallback.

Does trusting libmagic over the file extension satisfy forensic defensibility?

Content-based detection is the defensible choice precisely because extensions are custodian-controlled metadata that spoofing trivially defeats. To make it audit-ready, pin one magic.mgc version across every node, log the resolved type and database version with each file’s SHA-256, and record every quarantine decision with its justification. That record demonstrates a deterministic, reproducible classification process — the standard a Rule 34 production and a Daubert challenge both scrutinize.

Related

• Native File Ingestion Pipelines — the ingestion subsystem whose routing gate this detector feeds.
• Cryptographic Hash Generation — the SHA-256 anchor logged alongside every MIME decision for chain of custody.
• ESI Format Mapping Standards — the signature-to-format reference this router consults for detected types.
• Async Batch Processing Design — the worker pool that dispatches the thread-safe detector at scale.
• Production Compliance Frameworks — the reproducibility and audit obligations content-based detection must satisfy.

Up one level: Native File Ingestion Pipelines — the ingestion architecture whose first routing decision depends on accurate MIME resolution.