Part III · Collection · Chapter 8
The Simulated Darknet
Commands only. What each step does, why it is built this way, and the judgment behind it are in the book.
Infrastructure tier
7 labs
≈ 6.5–7 h
8 GB RAM
Windows · macOS · Linux
Labs in this chapter
What you'll be able to do
- Stand up a self-contained darknet range and explain why collection is tested against it, never the real thing (Lab 8.1 - Why simulate, and the tier shift)
- Explain why a service directory is always partial and how seeds and cross-links drive discovery (Lab 8.2 - A directory and the seed problem)
- Recognise dark web services as stateful applications, logins, sessions, pagination, not static pages (Lab 8.3 - A market and a forum: services have state)
- Explain why point-in-time collection fails against content that flaps and reappears (Lab 8.4 - A leak site: persistence and flicker)
- Distinguish a legitimate mirror from a malicious clone, and say why both must be detected (Lab 8.5 - Mirrors, clones, and phishing)
- Follow references across Tor, I2P, and Hyphanet in a single collection scope (Lab 8.6 - Cross-network links)
- Score a crawl's recall and precision against ground truth and count the clones it was fooled by (Lab 8.7 - Ground truth and the scoring harness)
LAB 8.1
Why simulate, and the tier shift
1 h
8 GB
Key concept
Step 1, Bring up the range
# generate the synthetic content, then publish it as a set of onion services $ ./lab up range $ ./lab stop range # stop — containers kept; ./lab up range resumes them $ ./lab down range # delete — removes containers (add --volumes to drop the content)
Expected, a small darknet is published (addresses are ephemeral)
range-web started (synthetic content generated, watermarked)
range-web-clone started (altered content for the clone)
range-tor started publishing onion services...
range: 7 services published — directory, market, forum, leak, paste,
market-mirror (= market), market-clone (look-alike, altered)
note: onion keys are regenerated each bring-up; these are not real addresses
Step 2, List the services and read the ground truth
# the published services, with their (ephemeral) onion addresses $ ./lab range list # the answer key: what exists, what mirrors what, what is a clone $ ./lab range truth
Step 3, Why this, and not the real dark web
Verify in the Docker host
$ ./lab check 8.1
- The range services are running and publishing onion addresses
- The generated content is watermarked synthetic on every page (the safety control holds)
- You can explain the three reasons collection targets the range, not the real dark web
What this lab looks like
Terminal:
./lab check 8.1
Firefox on this lab's HTML view
Terminal output for this lab
LAB 8.2
A directory and the seed problem
0.5–1 h
8 GB
Key concept
Step 1, Read the directory
# the directory service lists links to (some of) the range $ ./lab range list --directory
Step 2, Where the rest of the list comes from
Verify in the Docker host
$ ./lab check 8.2
- The directory service is reachable and lists a strict subset of the ground-truth services
- At least one ground-truth service is absent from the directory (discovery is incomplete)
- You can name three seed sources beyond a directory that drive service discovery
What this lab looks like
Terminal:
./lab check 8.2
Firefox on this lab's HTML view
Artifact dump in Mousepad
LAB 8.3
A market and a forum: services have state
1 h
8 GB
All platforms
Step 1, Hit the login wall
Step 2, Why this defeats the naive approach
# fetch the market's listing page the naive way — no session $ ./lab range fetch market /listings
Expected; you get the wall, not the listings
HTTP 302 -> /login (no session cookie; listings not served) naive fetch sees the login wall, not the catalogue behind it
Verify in the Docker host
$ ./lab check 8.3
- The market and forum are reachable and gate their content behind a login
- A no-session fetch of a protected path is redirected to the wall, not served the content
- You can list what a crawler must do to collect a stateful, paginated service
What this lab looks like
Terminal:
./lab check 8.3
Terminal output for this lab
Firefox on this lab's HTML view
LAB 8.4
A leak site: persistence and flicker
1 h
8 GB
Key concept
Step 1, Read the leak site
Step 2, Make it flap
# simulate the seizure/rebrand cycle: the leak onion goes down, a new one comes up $ ./lab range flap leak
Expected, the address you had is dead; the content is elsewhere
leak: primary onion taken down ("seized")
leak: content republished at a new onion (the "rebrand")
a crawler holding only the old address now collects nothing
Verify in the Docker host
$ ./lab check 8.4
- The leak site is reachable and its content is watermarked, empty "proof" only
- A flap takes the primary address down and republishes the content at a new one
- You can explain why point-in-time collection fails and what continuous collection must do
What this lab looks like
Terminal:
./lab check 8.4
Artifact dump in Mousepad
Firefox on this lab's HTML view
LAB 8.5
Mirrors, clones, and phishing
1 h
8 GB
Key concept
Step 1, Compare the market, its mirror, and its clone
# fetch the three market addresses and diff their content + key material $ ./lab range compare market market-mirror market-clone
Expected, mirror is identical; clone differs where it matters
market onion A content-hash H1 pgp FP:9A3F.. btc bc1q..k7 market-mirror onion B content-hash H1 pgp FP:9A3F.. btc bc1q..k7 (identical -> mirror) market-clone onion C content-hash H1' pgp FP:2E77.. btc bc1q..z9 (altered pgp+btc -> clone)
Step 2, What "detect" means for each
Verify in the Docker host
$ ./lab check 8.5
- The mirror serves content identical to the market at a different address
- The clone serves look-alike content with altered PGP and coin-address fields
- You can state the different test that detects a mirror versus a clone
What this lab looks like
Terminal:
./lab check 8.5
Firefox on this lab's HTML view
Terminal output for this lab
LAB 8.6
Cross-network links
1 h
8 GB
All platforms
Step 1, Bring up an overlay alongside the range
# the range's directory links off to an eepsite and a freesite; bring those up $ ./lab up i2p # and/or: ./lab up hyphanet $ ./lab range list --cross $ ./lab stop overlays # stop — both overlays kept; ./lab up resumes them $ ./lab down overlays # delete — removes both (add --volumes to drop seeds)
Expected, links that leave the Tor network
cross-network references in the range directory: market -> (i2p) ...b32.i2p an eepsite mirror of a listing archive -> (hyphanet) USK@.../archive/ a freesite holding "leaked" docs following these requires the I2P / Hyphanet transports, not Tor
Step 2, What this means for a collection pipeline
Verify in the Docker host
$ ./lab check 8.6
- The range directory contains references to an I2P and a Hyphanet address
- The ground truth records those cross-network links and their required transport
- You can explain how a multi-transport pipeline routes a fetch by an address's network
What this lab looks like
Terminal:
./lab check 8.6
Firefox on this lab's HTML view
Artifact dump in Mousepad
LAB 8.7
Ground truth and the scoring harness
1 h
8 GB
Synthesis
Step 1, Score a sample crawl
# score a crawl output file against the range ground truth $ ./lab range score sample-crawl.json
Expected, recall, precision, and the failures that matter
scored crawl against range ground truth services found 4 / 5 recall 0.80 (missed: paste) reported real 4 / 7 precision 0.57 (3 off-range / invented) mirrors collapsed 1 / 1 (market-mirror recognised as market) clones caught 0 / 1 FAIL — market-clone reported as genuine (swapped btc trusted) overall: good coverage; the clone was believed. fix before trusting output.
Step 2, Why the range is an instrument, not just a target
Step 3, The handoff to Chapter 9
Verify in the Docker host
$ ./lab check 8.7
- The ground-truth manifest ships and the scorer self-tests (recall/precision + clone detection)
- Scoring a sample crawl reports coverage and whether the clone was believed
- You can explain why these metrics are computable against the range but not the live dark web
What this lab looks like
Terminal:
./lab check 8.7
Firefox on this lab's HTML view
Terminal output for this lab