Part III · Collection · Chapter 10
Mirrors & Clones
Commands only. What each step does, why it is built this way, and the judgment behind it are in the book.
LAB 10.1
Where exact hashing fails
Meet the clone lab
$ ls labs/artifacts/dedup/corpus/ $ ./lab dedup selftest # the whole detector self-tests offline — no Docker, no Tor
Expected
selftest: shingle Jaccard separates near-duplicates from unrelated pages,
and MinHash estimates it from a signature -> PASS
selftest: structural SimHash matches a reworded clone by template — and also a
different site on the same template (needs 10.4) -> PASS
selftest: shared assets separate a keyless clone from a different site on the
same template, and payment swap flags impersonation -> PASS
selftest: naive groups mirror 0.33, clone 0.50;
full groups mirror 1.00, clone 1.00, no false merge -> PASS
dedup self-tests passed (shingle, structure, signals, cluster+scorer)
Score the naive detector
# the Chapter 9 detector, graded on the clone-lab ground truth $ ./lab dedup score --naive
Expected
scored dedup against clone-lab ground truth mirrors grouped 1 / 3 recall 0.33 (missed: forum-mirror.html, market-mirror-banner.html) clones caught 1 / 2 recall 0.50 (missed: market-clone-keyless.html) clone precision 1 flagged precision 1.00 false merges 0 (no unrelated services merged)
Verify in the Docker host
$ ./lab check 10.1
- The clone-lab corpus is present and every page carries the synthetic watermark
./lab dedup selftestpasses offline, shingle, structure, signals, cluster, and the scorer- The naive detector scores mirror recall 0.33 and clone recall 0.50, and you can name the pages it missed
LAB 10.2
Shingling and MinHash: near-duplicates
Shingle the pages and compare
$ python3 labs/artifacts/dedup/shingle.py --selftest
Expected
selftest: shingle Jaccard separates near-duplicates from unrelated pages,
and MinHash estimates it from a signature -> PASS
Why MinHash
Verify in the Docker host
$ ./lab check 10.2
- Shingling scores the banner mirror above the near-duplicate threshold and the unrelated pages below it
- The MinHash signature estimate tracks the exact Jaccard on the same pairs
- The reworded clone scores low on shingling, establishing the gap that structure must close
LAB 10.3
Structural fingerprinting
Fingerprint the skeleton
$ python3 labs/artifacts/dedup/structure.py --selftest
Expected
selftest: structural SimHash matches a reworded clone by template — and also a
different site on the same template (needs 10.4) -> PASS
Verify in the Docker host
$ ./lab check 10.3
- The structural fingerprint scores the reworded clone as a template match despite its low shingle score
- A different service on the same template also scores as a match, the false positive is reproducible
- A genuinely different template scores well below the structural threshold
LAB 10.4
Identity without keys: secondary signals
Read the evidence a copy leaves
$ python3 labs/artifacts/dedup/signals.py --selftest
Expected
selftest: shared assets separate a keyless clone from a different site on the
same template, and payment swap flags impersonation -> PASS
Verify in the Docker host
$ ./lab check 10.4
- Shared asset detection links the keyless clone to the market and correctly finds no overlap with the decoy
- A swapped payment identity is flagged as impersonation
- The intent rule labels the keyless clone a clone and the decoy neither a mirror nor a clone of the market
LAB 10.5
Clustering: one operator, many addresses
Collapse the edges into clusters
# cluster the corpus with the full engine and print the result $ ./lab dedup run
Expected (abridged)
clustering the clone-lab corpus (full engine):
{
"clusters": [
{ "canonical": "forum.html", "members": [
{ "address": "forum.html", "role": "canonical" },
{ "address": "forum-mirror.html", "role": "mirror", "reason": "benign_copy" } ] },
{ "canonical": "market.html", "members": [
{ "address": "market.html", "role": "canonical" },
{ "address": "market-mirror-exact.html", "role": "mirror", "reason": "shared_payment" },
{ "address": "market-mirror-banner.html", "role": "mirror", "reason": "shared_payment" },
{ "address": "market-clone-keyswap.html", "role": "clone", "reason": "payment_swap" },
{ "address": "market-clone-keyless.html", "role": "clone", "reason": "keyless_copy" } ] },
{ "canonical": "other-market.html", "members": [
{ "address": "other-market.html", "role": "canonical" } ] },
{ "canonical": "paste.html", "members": [
{ "address": "paste.html", "role": "canonical" } ] }
]
}
Verify in the Docker host
$ ./lab check 10.5
- The full engine produces four clusters with the market's five look-alikes in one and the decoy in its own
- Each cluster names a canonical, and every non-canonical member carries a role and a reason
- No unrelated services are merged, the decoy and the forum stay out of the market cluster
LAB 10.6
Mirror vs clone: redundancy vs impersonation
Redundancy versus impersonation
What this closes from Chapter 9
Verify in the Docker host
$ ./lab check 10.6
- Cluster members sharing the canonical's payment identity are labelled mirrors; those swapping it are labelled clones
- The keyless clone is flagged without any key comparison, closing the Chapter 9 gap
- You can state the operational rule: act on a mirror's addresses, flag a clone's before acting
LAB 10.7
Scoring the detector and closing the loop
Score the full engine against the baseline
$ ./lab dedup score # full engine $ ./lab dedup score --naive # the Chapter 9 detector, for comparison
Expected, full engine
scored dedup against clone-lab ground truth mirrors grouped 3 / 3 recall 1.00 clones caught 2 / 2 recall 1.00 clone precision 2 flagged precision 1.00 false merges 0 (no unrelated services merged)
Expected, naive baseline
scored dedup against clone-lab ground truth mirrors grouped 1 / 3 recall 0.33 (missed: forum-mirror.html, market-mirror-banner.html) clones caught 1 / 2 recall 0.50 (missed: market-clone-keyless.html) clone precision 1 flagged precision 1.00 false merges 0 (no unrelated services merged)
What the detector still cannot do
Verify in the Docker host
$ ./lab check 10.7
- The full engine scores mirror recall 1.00, clone recall 1.00, clone precision 1.00, and zero false merges
- The full engine outscores the naive baseline on both recalls while matching it on precision and false merges
- You can name the three residual gaps, stored page bodies, the asset arms race, LSH at scale, and which chapter owns each