Part III · Collection · Chapter 10

Mirrors & Clones

Commands only. What each step does, why it is built this way, and the judgment behind it are in the book.

Infrastructure tier 7 labs ≈ 5–7 h Pure Python · no Docker Windows · macOS · Linux

Labs in this chapter

What you'll be able to do

Explain why exact content hashing collapses a mirror correctly and then fails on a one-line banner, and name the two Chapter 9 gaps this chapter closes.
Detect near-duplicate pages with k-shingles and Jaccard, and estimate that similarity at scale with a MinHash signature.
Fingerprint a page's structure with SimHash to catch a clone that reworded its text, and say why structure alone over-merges.
Separate a keyless clone from a different site on the same template using shared assets, and flag impersonation from a swapped payment identity.
Cluster look-alikes into per-operator sites, classify each as a mirror or a clone, and grade the whole detector against a shipped answer key, full engine versus the naive baseline.

LAB 10.1

Where exact hashing fails

0.5–1 h the deficit clone lab naive score

Meet the clone lab

Ubuntu

$ ls labs/artifacts/dedup/corpus/
$ ./lab dedup selftest     # the whole detector self-tests offline — no Docker, no Tor

Expected

selftest: shingle Jaccard separates near-duplicates from unrelated pages,
          and MinHash estimates it from a signature  -> PASS
selftest: structural SimHash matches a reworded clone by template — and also a
          different site on the same template (needs 10.4)  -> PASS
selftest: shared assets separate a keyless clone from a different site on the
          same template, and payment swap flags impersonation  -> PASS
selftest: naive groups mirror 0.33, clone 0.50;
          full groups mirror 1.00, clone 1.00, no false merge  -> PASS
  dedup self-tests passed (shingle, structure, signals, cluster+scorer)

Score the naive detector

Ubuntu

# the Chapter 9 detector, graded on the clone-lab ground truth
$ ./lab dedup score --naive

Expected

scored dedup against clone-lab ground truth
  mirrors grouped   1 / 3     recall    0.33   (missed: forum-mirror.html, market-mirror-banner.html)
  clones caught     1 / 2     recall    0.50   (missed: market-clone-keyless.html)
  clone precision   1 flagged      precision 1.00
  false merges      0     (no unrelated services merged)

Verify in the Docker host

Ubuntu

$ ./lab check 10.1

The clone-lab corpus is present and every page carries the synthetic watermark
./lab dedup selftest passes offline, shingle, structure, signals, cluster, and the scorer
The naive detector scores mirror recall 0.33 and clone recall 0.50, and you can name the pages it missed

What this lab looks like

Terminal showing ./lab check 10.1 and its output

Terminal: ./lab check 10.1

Firefox rendering an HTML view for lab 10.1

Firefox on this lab's HTML view

Terminal output captured during lab 10.1

Terminal output for this lab

LAB 10.2

Shingling and MinHash: near-duplicates

0.5–1 h shingle.py Jaccard MinHash

Shingle the pages and compare

Ubuntu

$ python3 labs/artifacts/dedup/shingle.py --selftest

Expected

selftest: shingle Jaccard separates near-duplicates from unrelated pages,
          and MinHash estimates it from a signature  -> PASS

Why MinHash

Verify in the Docker host

Ubuntu

$ ./lab check 10.2

Shingling scores the banner mirror above the near-duplicate threshold and the unrelated pages below it
The MinHash signature estimate tracks the exact Jaccard on the same pairs
The reworded clone scores low on shingling, establishing the gap that structure must close

What this lab looks like

Terminal showing ./lab check 10.2 and its output

Terminal: ./lab check 10.2

Firefox rendering an HTML view for lab 10.2

Firefox on this lab's HTML view

Mousepad showing an artifact dump from lab 10.2

Artifact dump in Mousepad

LAB 10.3

Structural fingerprinting

0.5–1 h structure.py SimHash the shape

Fingerprint the skeleton

Ubuntu

$ python3 labs/artifacts/dedup/structure.py --selftest

Expected

selftest: structural SimHash matches a reworded clone by template — and also a
          different site on the same template (needs 10.4)  -> PASS

Verify in the Docker host

Ubuntu

$ ./lab check 10.3

The structural fingerprint scores the reworded clone as a template match despite its low shingle score
A different service on the same template also scores as a match, the false positive is reproducible
A genuinely different template scores well below the structural threshold

What this lab looks like

Terminal showing ./lab check 10.3 and its output

Terminal: ./lab check 10.3

Firefox rendering an HTML view for lab 10.3

Firefox on this lab's HTML view

Terminal output captured during lab 10.3

Terminal output for this lab

LAB 10.4

Identity without keys: secondary signals

1 h signals.py shared assets payment identity

Read the evidence a copy leaves

Ubuntu

$ python3 labs/artifacts/dedup/signals.py --selftest

Expected

selftest: shared assets separate a keyless clone from a different site on the
          same template, and payment swap flags impersonation  -> PASS

Verify in the Docker host

Ubuntu

$ ./lab check 10.4

Shared asset detection links the keyless clone to the market and correctly finds no overlap with the decoy
A swapped payment identity is flagged as impersonation
The intent rule labels the keyless clone a clone and the decoy neither a mirror nor a clone of the market

What this lab looks like

Terminal showing ./lab check 10.4 and its output

Terminal: ./lab check 10.4

Terminal output captured during lab 10.4

Terminal output for this lab

Firefox rendering an HTML view for lab 10.4

Firefox on this lab's HTML view

LAB 10.5

Clustering: one operator, many addresses

1 h cluster.py union-find sites, not pages

Collapse the edges into clusters

Ubuntu

# cluster the corpus with the full engine and print the result
$ ./lab dedup run

Expected (abridged)

  clustering the clone-lab corpus (full engine):
{
  "clusters": [
    { "canonical": "forum.html", "members": [
        { "address": "forum.html", "role": "canonical" },
        { "address": "forum-mirror.html", "role": "mirror", "reason": "benign_copy" } ] },
    { "canonical": "market.html", "members": [
        { "address": "market.html", "role": "canonical" },
        { "address": "market-mirror-exact.html",  "role": "mirror", "reason": "shared_payment" },
        { "address": "market-mirror-banner.html", "role": "mirror", "reason": "shared_payment" },
        { "address": "market-clone-keyswap.html", "role": "clone",  "reason": "payment_swap" },
        { "address": "market-clone-keyless.html", "role": "clone",  "reason": "keyless_copy" } ] },
    { "canonical": "other-market.html", "members": [
        { "address": "other-market.html", "role": "canonical" } ] },
    { "canonical": "paste.html", "members": [
        { "address": "paste.html", "role": "canonical" } ] }
  ]
}

Verify in the Docker host

Ubuntu

$ ./lab check 10.5

The full engine produces four clusters with the market's five look-alikes in one and the decoy in its own
Each cluster names a canonical, and every non-canonical member carries a role and a reason
No unrelated services are merged, the decoy and the forum stay out of the market cluster

What this lab looks like

Terminal showing ./lab check 10.5 and its output

Terminal: ./lab check 10.5

Mousepad showing an artifact dump from lab 10.5

Artifact dump in Mousepad

Firefox rendering an HTML view for lab 10.5

Firefox on this lab's HTML view

LAB 10.6

Mirror vs clone: redundancy vs impersonation

0.5–1 h the judgment money is the tell closes 9's gap

Redundancy versus impersonation

What this closes from Chapter 9

Verify in the Docker host

Ubuntu

$ ./lab check 10.6

Cluster members sharing the canonical's payment identity are labelled mirrors; those swapping it are labelled clones
The keyless clone is flagged without any key comparison, closing the Chapter 9 gap
You can state the operational rule: act on a mirror's addresses, flag a clone's before acting

What this lab looks like

Terminal showing ./lab check 10.6 and its output

Terminal: ./lab check 10.6

Firefox rendering an HTML view for lab 10.6

Firefox on this lab's HTML view

Mousepad showing an artifact dump from lab 10.6

Artifact dump in Mousepad

LAB 10.7

Scoring the detector and closing the loop

1 h the payoff full vs naive residual gaps

Score the full engine against the baseline

Ubuntu

$ ./lab dedup score            # full engine
$ ./lab dedup score --naive    # the Chapter 9 detector, for comparison

Expected, full engine

scored dedup against clone-lab ground truth
  mirrors grouped   3 / 3     recall    1.00
  clones caught     2 / 2     recall    1.00
  clone precision   2 flagged      precision 1.00
  false merges      0     (no unrelated services merged)

Expected, naive baseline

scored dedup against clone-lab ground truth
  mirrors grouped   1 / 3     recall    0.33   (missed: forum-mirror.html, market-mirror-banner.html)
  clones caught     1 / 2     recall    0.50   (missed: market-clone-keyless.html)
  clone precision   1 flagged      precision 1.00
  false merges      0     (no unrelated services merged)

What the detector still cannot do

Verify in the Docker host

Ubuntu

$ ./lab check 10.7

The full engine scores mirror recall 1.00, clone recall 1.00, clone precision 1.00, and zero false merges
The full engine outscores the naive baseline on both recalls while matching it on precision and false merges
You can name the three residual gaps, stored page bodies, the asset arms race, LSH at scale, and which chapter owns each

What this lab looks like

Terminal showing ./lab check 10.7 and its output

Terminal: ./lab check 10.7

Firefox rendering an HTML view for lab 10.7

Firefox on this lab's HTML view

Terminal output captured during lab 10.7

Terminal output for this lab