Cross-Matter Deduplication: Global Hash Indexing With Case Isolation

Large legal organizations process the same custodians and the same documents across many matters, and re-reviewing an identical file in every matter is pure waste — but sharing deduplication state across matters collides head-on with the strictest boundary in eDiscovery: case isolation. A document’s presence in one matter, and any review decision made about it, is often privileged or confidential to that matter, so a naive global hash index that lets Matter B see “this file was already reviewed and marked privileged in Matter A” leaks work-product and violates ethical walls. Inside the Deduplication & Family Grouping pipeline, cross-matter deduplication is the subsystem that captures the efficiency of a shared index while enforcing that no matter can learn anything about another. This guide builds that subsystem: a global content index that suppresses redundant processing without ever sharing decisions across a case boundary.

Cross-Matter Subsystem Flow

The design separates two things that a single-matter pipeline conflates: the content-addressed processing cache (safe to share, because a digest reveals nothing) and the matter-scoped review state (never shared). A document’s digest checks the global index for cached processing artifacts; its review disposition is always resolved within its own matter.

Resource Constraints & Design Rationale

The efficiency prize is real: text extraction, OCR, and image rendering are the expensive stages of a pipeline, and they are pure functions of a document’s bytes. If a 40-page scanned PDF was OCR’d for Matter A, re-OCRing the identical bytes for Matter B is wasted compute. A content-addressed cache keyed on the document’s digest captures that saving safely, because the key — a SHA-256 — and the cached artifact — extracted text — reveal nothing about which matters contain the document or what anyone decided about it. Sharing the processing cache is therefore both valuable and, done correctly, disclosure-free.

Review state is the opposite. Whether a document was marked responsive, privileged, or hot, which custodians it came from in a given matter, and how it was grouped into a family are all matter-specific and frequently protected. The design rule is a hard partition: the global index stores only content-derived artifacts that are safe to share, and every matter keeps its own isolated store of dispositions that no query from another matter can reach. This is the same append-only, boundary-enforced discipline the single-matter hash-based deduplication strategies apply, extended with an access-control boundary between matters.

Isolation Algorithm & Access Control

Two mechanisms enforce isolation. The first is artifact classification: every piece of state is tagged as content-derived (shareable) or matter-derived (isolated), and only content-derived artifacts may be written to or read from the global index. A digest, extracted text, a page count, a MIME type — content-derived. A responsiveness code, a privilege tag, a custodian, a family assignment — matter-derived. The classification is enforced at the storage boundary, so a bug that tries to cache a review decision globally fails rather than leaks.

The second is matter-scoped access control on every read. A query into the global index carries the requesting matter’s identity, and the index returns only content-derived artifacts — it has no matter-derived state to return, by construction. Crucially, the index does not even reveal which other matters contain a digest, because a mere count of matters can itself be sensitive (it signals that a document is significant across an organization’s litigation portfolio). The index answers “is there a cached extraction for this digest?” but never “which matters have this document?” Review suppression — the actual “don’t re-review this” decision — happens only within a matter, using that matter’s own prior dispositions, never another’s. When a document must be handled specially because it crosses a family boundary or an ethical wall, it routes to the same tiered review the similarity threshold configuration layer uses.

Resilience & Failure Routing

The subsystem fails toward isolation: any ambiguity about whether an artifact is shareable is resolved by treating it as matter-derived and keeping it private. If the classification of a new artifact type is undefined, it is not cached globally until explicitly classified as content-derived. This “private by default” posture means a misconfiguration can cost efficiency (a cache miss) but can never cause a leak — the safe direction for a boundary whose violation is an ethics problem, not a performance one.

Cache integrity is the second guard. A content-addressed cache is only safe if the digest truly determines the artifact, so the index verifies on read that the cached artifact’s recorded source digest matches the requested digest, and treats a mismatch as a poisoned entry to be evicted and recomputed rather than served. Every cross-matter cache hit and every isolation decision is logged within the requesting matter’s audit trail — never in a shared log, which would itself be a cross-matter disclosure — so each matter can independently demonstrate that its use of the shared index disclosed nothing about, and learned nothing improper from, any other.

Production Python Implementation

The module implements a content-addressed global index that stores only classified content-derived artifacts, enforces matter-scoped access, and never reveals cross-matter membership. Review suppression is resolved within a matter. It compiles and runs on the standard library.

import logging
from dataclasses import dataclass, field
from enum import Enum
from typing import Dict, Optional, Set

logger = logging.getLogger("ediscovery.crossmatter")


class Scope(str, Enum):
    CONTENT = "content_derived"   # safe to share across matters
    MATTER = "matter_derived"     # isolated to one matter


class IsolationError(Exception):
    pass


# Whitelist of artifact types allowed in the shared index. Anything not listed
# is treated as matter-derived and stays private (fail toward isolation).
SHAREABLE = {"extracted_text", "ocr_text", "page_count", "mime_type", "rendered_image"}


@dataclass
class Artifact:
    source_sha256: str
    kind: str
    payload: object


class GlobalContentIndex:
    """Content-addressed cache of shareable processing artifacts only. Keyed by
    (digest, kind); never stores matter identity or review state."""

    def __init__(self):
        self._store: Dict[tuple, Artifact] = {}

    def put(self, artifact: Artifact) -> None:
        if artifact.kind not in SHAREABLE:
            raise IsolationError(f"refusing to globally cache non-shareable '{artifact.kind}'")
        self._store[(artifact.source_sha256, artifact.kind)] = artifact

    def get(self, requesting_matter: str, digest: str, kind: str) -> Optional[Artifact]:
        """Return a cached content-derived artifact, or None. Never reveals which
        other matters hold the digest — there is no such state to reveal."""
        if kind not in SHAREABLE:
            raise IsolationError(f"'{kind}' is matter-derived; resolve it within the matter")
        art = self._store.get((digest, kind))
        if art is None:
            return None
        # Integrity: the cached artifact must belong to the requested digest.
        if art.source_sha256 != digest:
            logger.error("poisoned cache entry for %s; evicting", digest)
            del self._store[(digest, kind)]
            return None
        logger.info("cross-matter cache hit kind=%s for matter=%s", kind, requesting_matter)
        return art


class MatterReviewStore:
    """Per-matter, isolated review state. No cross-matter query can reach it."""

    def __init__(self, matter_id: str):
        self.matter_id = matter_id
        self._dispositions: Dict[str, str] = {}   # digest -> disposition, this matter only

    def suppress_within_matter(self, digest: str) -> Optional[str]:
        """Reuse a prior review decision — but only one made in THIS matter."""
        return self._dispositions.get(digest)

    def record(self, digest: str, disposition: str) -> None:
        self._dispositions[digest] = disposition


def process_document(digest: str, kind: str, matter: MatterReviewStore,
                     index: GlobalContentIndex) -> Dict[str, object]:
    """Reuse cached processing across matters; resolve review within the matter."""
    cached = index.get(matter.matter_id, digest, kind)
    prior = matter.suppress_within_matter(digest)   # same-matter suppression only
    return {
        "processing_reused": cached is not None,
        "review_suppressed": prior is not None,
        "disposition": prior,   # None => must be reviewed in this matter
    }

Observability & Compliance Metrics

Three KPIs make cross-matter deduplication both efficient and auditable. Processing cache-hit rate (cross-matter artifact reuses ÷ processing requests) quantifies the compute saved by sharing content-derived artifacts. Isolation-violation attempts (blocked attempts to cache or read matter-derived state globally) must be zero and is the primary safety signal — any nonzero value is a design or configuration bug caught at the boundary. Same-matter suppression rate (documents suppressed using this matter’s own prior dispositions) measures the review reduction achieved without crossing any boundary. Because each metric is logged within the matter that generated it, the metrics themselves respect the isolation they measure.

from prometheus_client import Counter

XM_CACHE_HITS = Counter("xmatter_cache_hits_total", "Content artifacts reused across matters")
XM_ISOLATION_BLOCKS = Counter("xmatter_isolation_blocks_total", "Blocked matter-derived shares")
XM_SUPPRESSED = Counter("xmatter_same_matter_suppressed_total", "Docs suppressed within a matter")


def record_processing(matter_id: str, reused: bool, suppressed: bool) -> None:
    if reused:
        XM_CACHE_HITS.inc()
    if suppressed:
        XM_SUPPRESSED.inc()

Conclusion

Cross-matter deduplication is worth building only if it captures the compute saving of a shared index without ever leaking a decision across a case boundary, and that requires separating two kinds of state that single-matter pipelines conflate. Content-derived artifacts — text, OCR, renders — are pure functions of a document’s bytes and safe to share through a content-addressed index that reveals nothing about matter membership. Matter-derived state — dispositions, privilege, families — stays hard-partitioned per matter, and review suppression uses only a matter’s own prior decisions. Classify every artifact, fail toward isolation on any ambiguity, verify cache integrity, and log within each matter. Built this way, the organization stops re-processing identical bytes across its portfolio while every ethical wall stays intact.

Frequently Asked Questions

Isn’t sharing any deduplication state across matters an ethical-wall violation?

Not if you share the right state. The ethical wall protects a matter’s decisions and confidences — who reviewed a document, what they concluded, which custodians and families it belongs to. It does not protect the raw content-derived facts that any processing of the bytes would produce, like extracted text or a page count. Sharing a content-addressed cache of those artifacts discloses nothing about any matter, because a digest and its extracted text reveal neither matter membership nor any review decision. The violation only occurs when matter-derived state — a privilege call, a responsiveness code — crosses the boundary, which the design forbids by construction.

Why can’t the global index reveal which matters contain a document?

Because that membership is itself sensitive. Knowing that a specific document appears in five of an organization’s matters signals that it is significant across a litigation portfolio, which can imply strategy, exposure, or the existence of related matters that a given team is walled off from. The index therefore stores no matter identity at all — it answers only “is there a cached artifact for this digest?” — so there is no membership information to leak even under a compromised query.

How is review suppression handled without crossing matters?

Suppression is resolved entirely within a matter, using that matter’s own prior dispositions. If a document identical to one already reviewed in this matter arrives, the matter’s isolated review store can suppress the re-review. A document reviewed only in a different matter carries no reusable decision across the boundary — it must be reviewed independently in the new matter. The shared index accelerates the expensive processing (extraction, OCR) but never the review decision, which is the part that carries privilege.

Related

• Cross-Matter Dedup With Case Isolation — diagnosing and fixing an isolation leak in a shared index.
• Hash-Based Deduplication Strategies — the single-matter exact-match foundation this extends.
• Near-Duplicate Detection — similarity indexing that can also be shared as content-derived state.
• Security Boundary Configuration — the zero-trust boundaries that enforce matter partitioning.
• Production Compliance Frameworks — the audit and segregation rules this inherits.

Up one level: Deduplication & Family Grouping — the pipeline stage that reduces review volume defensibly.