Forwarded Emails and Nested Attachments: Similarity Edge Cases That Break Family Grouping
Two document shapes reliably defeat a similarity threshold that works fine on ordinary files: the forwarded email, which is textually near-identical to the original but belongs to a different conversation and custodian, and the nested attachment that appears under two different parents. Both surface in similarity threshold configuration as decisions the threshold gets wrong — a forwarded email suppressed as a duplicate erases a distinct custodial fact, and a shared attachment suppressed everywhere orphans it from one of its families. These edge cases violate the family-integrity guarantee the deduplication layer must hold, because a threshold decision made without family context silently discards relationships a reviewer needs. This guide isolates the two edge cases and delivers routing rules that keep similarity and family grouping consistent.
Diagnostic Log Signatures
The edge cases surface when family integrity is audited after similarity clustering. Capture the signatures verbatim.
[INFO] Family audit after near-dup: 512044 docs, 88210 families
[ERROR] forwarded_suppressed DOC-07711: sim=0.97 to DOC-04120 but different custodian/thread
[ERROR] shared_attachment_orphaned DOC-09980: attachment of FAM-100 and FAM-205, kept only in FAM-100
[WARN] thread_collapse FAM-330: 12 forwarded variants merged into 1, custodial context lost
[CRITICAL] Family integrity broken: 3 similarity decisions discarded relational context
Symptom checklist — two or more indicate a family-edge-case defect:
- A forwarded email is suppressed as a near-duplicate of the original, though it has a different custodian, timestamp, or recipient set.
- A near-duplicate cluster collapses many forwarded variants into one, erasing who forwarded what to whom.
- An attachment shared by two parents is retained under one family and missing from the other.
- Family membership counts drop after near-duplicate clustering runs.
- A reviewer cannot see the custodial path by which a document reached the collection.
Root-Cause Breakdown
Both edge cases come from applying a content-similarity decision without family and custodial context.
- Forwarded email textual overlap. A forward reproduces the original’s body almost verbatim, so a pure text-similarity score is very high — often above any reasonable threshold. But the forward is a distinct communication: different sender, recipients, timestamp, and often a new custodian. Suppressing it as a duplicate destroys the fact that a specific person forwarded the message, which can be the single most important fact in the matter. Similarity alone cannot make this call; it needs the routing header context that email threading algorithms reconstruct.
- Shared attachment under multiple parents. The identical attachment binary can be attached to two different parent emails. Exact dedup correctly recognizes one binary, but family grouping needs the attachment represented in both families. Suppressing the redundant binary everywhere orphans it from one parent; the fix is to keep one canonical binary but a logical reference under each family, as attachment and parent-child mapping requires.
- Threshold blind to metadata. A threshold that scores only body text ignores the metadata that distinguishes a forward from its original — so the same score is produced for “identical document” and “same body, different transmission.”
- Suppression before family resolution. Running similarity suppression before family grouping lets a suppression decision remove a document the family graph still needs.
Remediation Architecture
Make similarity decisions family-aware: never suppress a distinct communication, and represent shared members in every family.
- Distinguish content identity from communication identity. For emails, combine body similarity with transmission metadata (sender, recipients, sent time). High body similarity plus different transmission means a distinct communication — cluster it as related, never suppress it.
- Keep forwarded variants as a related set. Group forwarded copies under a “near-duplicate family” that preserves each variant’s custodial context rather than collapsing them to one record.
- One binary, a reference per family. For a shared attachment, retain a single canonical binary and attach a logical reference under each parent family, so no family loses its member.
- Order the stages. Resolve families first (or jointly), and let similarity operate on the family graph, so suppression can never orphan a family member or erase a custodial path.
The routing below shows how a high-similarity pair is disambiguated by transmission metadata and family membership.
Family-aware disambiguation implementation
The routine disambiguates a high-similarity email pair by transmission metadata and represents a shared attachment under every parent family. It compiles and runs on the standard library.
import logging
from dataclasses import dataclass, field
from typing import Dict, List, Sequence, Set, Tuple
logger = logging.getLogger("ediscovery.family.edgecases")
@dataclass(frozen=True)
class Transmission:
sender: str
recipients: Tuple[str, ...]
sent_time: str
@dataclass
class EmailDoc:
doc_id: str
body_similarity_key: str # signature bucket for near-dup body match
transmission: Transmission
custodian: str
def is_true_duplicate(a: EmailDoc, b: EmailDoc) -> bool:
"""Same body AND same transmission => a genuine duplicate. Same body but a
different sender/recipients/time is a distinct forwarded communication."""
if a.body_similarity_key != b.body_similarity_key:
return False
return a.transmission == b.transmission
def route_high_similarity(a: EmailDoc, b: EmailDoc) -> str:
"""Return the routing decision for a high body-similarity pair."""
if is_true_duplicate(a, b):
logger.info("suppress %s as duplicate of %s", b.doc_id, a.doc_id)
return "suppress_duplicate"
logger.info("keep %s as forwarded variant of %s (custodian %s)",
b.doc_id, a.doc_id, b.custodian)
return "keep_related_variant"
@dataclass
class Attachment:
binary_sha256: str
parent_families: Set[str] = field(default_factory=set)
def represent_shared_attachment(att: Attachment) -> List[dict]:
"""One canonical binary, one logical reference per parent family, so a
shared attachment is never orphaned from any of its families."""
if not att.parent_families:
raise ValueError("attachment has no parent family")
canonical = min(att.parent_families) # deterministic canonical holder
refs = [{"family_id": fam, "binary_sha256": att.binary_sha256,
"role": "canonical" if fam == canonical else "reference"}
for fam in sorted(att.parent_families)]
return refs
Verification Checklist
Confirm family integrity survives similarity clustering:
Conclusion
High textual similarity is not the same as duplicate identity, and the forwarded email and the shared attachment are the two cases where that distinction matters most. A forward reproduces the body but is a distinct communication with its own custodial context; a shared attachment is one binary that must appear in every family it belongs to. Make similarity decisions family-aware — disambiguate emails by transmission metadata, keep forwarded variants as related sets, and represent shared attachments by reference under each parent — and run similarity on the resolved family graph so suppression can never orphan a member. Done this way, the deduplication layer reduces redundancy without ever discarding the relationships and custodial facts a reviewer depends on.
Frequently Asked Questions
Why shouldn’t a forwarded email be suppressed even though its body is identical?
Because the forward is a different communication with independent evidentiary value. The body may be verbatim, but the sender, recipients, timestamp, and often the custodian are new, and the fact that a specific person forwarded the message — to whom and when — can be the pivotal fact in a matter. Suppressing it as a duplicate erases that fact. Deduplication should collapse identical communications, not merely identical text, which is why the decision needs transmission metadata, not just a body-similarity score.
How can one attachment belong to two families without being duplicated?
By storing one canonical binary and placing a logical reference under each parent family. Exact deduplication correctly recognizes there is a single binary, so it is stored once; family grouping then attaches a reference to that binary under every parent email that carried it. Each family shows the attachment as its member, the reviewer sees complete families, and the storage holds the binary only once — the redundancy is removed without orphaning any family.
Should similarity run before or after family grouping?
Family grouping first, or jointly, so similarity operates on the resolved family graph. If similarity suppression runs first, it can remove a document the family graph still needs — a shared attachment or a distinct forward — before the relationships are known. Resolving families first means every suppression decision is made with full knowledge of which families would be affected, so it can preserve members and custodial paths rather than silently discarding them.
Related
- Similarity Threshold Configuration — the governance layer these edge cases stress.
- Email Threading Algorithms — the header reconstruction that distinguishes a forward from its original.
- Attachment & Parent-Child Mapping — the reference-per-family model for shared attachments.
- Near-Duplicate Detection — the similarity engine whose decisions these rules constrain.
For family-handling guidance, see the EDRM processing standards and Sedona Conference commentary on email threading and families.
Up one level: Similarity Threshold Configuration — the subsystem that governs near-duplicate decision gates.