Part III · Collection · Chapter 8
The Simulated Darknet
Commands only. What each step does, why it is built this way, and the judgment behind it are in the book.
LAB 8.1
Why simulate, and the tier shift
Step 1, Bring up the range
# generate the synthetic content, then publish it as a set of onion services $ ./lab up range $ ./lab stop range # stop — containers kept; ./lab up range resumes them $ ./lab down range # delete — removes containers (add --volumes to drop the content)
Expected, a small darknet is published (addresses are ephemeral)
range-web started (synthetic content generated, watermarked)
range-web-clone started (altered content for the clone)
range-tor started publishing onion services...
range: 7 services published — directory, market, forum, leak, paste,
market-mirror (= market), market-clone (look-alike, altered)
note: onion keys are regenerated each bring-up; these are not real addresses
Step 2, List the services and read the ground truth
# the published services, with their (ephemeral) onion addresses $ ./lab range list # the answer key: what exists, what mirrors what, what is a clone $ ./lab range truth
Step 3, Why this, and not the real dark web
Verify in the Docker host
$ ./lab check 8.1
- The range services are running and publishing onion addresses
- The generated content is watermarked synthetic on every page (the safety control holds)
- You can explain the three reasons collection targets the range, not the real dark web
LAB 8.2
A directory and the seed problem
Step 1, Read the directory
# the directory service lists links to (some of) the range $ ./lab range list --directory
Step 2, Where the rest of the list comes from
Verify in the Docker host
$ ./lab check 8.2
- The directory service is reachable and lists a strict subset of the ground-truth services
- At least one ground-truth service is absent from the directory (discovery is incomplete)
- You can name three seed sources beyond a directory that drive service discovery
LAB 8.3
A market and a forum: services have state
Step 1, Hit the login wall
Step 2, Why this defeats the naive approach
# fetch the market's listing page the naive way — no session $ ./lab range fetch market /listings
Expected; you get the wall, not the listings
HTTP 302 -> /login (no session cookie; listings not served) naive fetch sees the login wall, not the catalogue behind it
Verify in the Docker host
$ ./lab check 8.3
- The market and forum are reachable and gate their content behind a login
- A no-session fetch of a protected path is redirected to the wall, not served the content
- You can list what a crawler must do to collect a stateful, paginated service
LAB 8.4
A leak site: persistence and flicker
Step 1, Read the leak site
Step 2, Make it flap
# simulate the seizure/rebrand cycle: the leak onion goes down, a new one comes up $ ./lab range flap leak
Expected, the address you had is dead; the content is elsewhere
leak: primary onion taken down ("seized")
leak: content republished at a new onion (the "rebrand")
a crawler holding only the old address now collects nothing
Verify in the Docker host
$ ./lab check 8.4
- The leak site is reachable and its content is watermarked, empty "proof" only
- A flap takes the primary address down and republishes the content at a new one
- You can explain why point-in-time collection fails and what continuous collection must do
LAB 8.5
Mirrors, clones, and phishing
Step 1, Compare the market, its mirror, and its clone
# fetch the three market addresses and diff their content + key material $ ./lab range compare market market-mirror market-clone
Expected, mirror is identical; clone differs where it matters
market onion A content-hash H1 pgp FP:9A3F.. btc bc1q..k7 market-mirror onion B content-hash H1 pgp FP:9A3F.. btc bc1q..k7 (identical -> mirror) market-clone onion C content-hash H1' pgp FP:2E77.. btc bc1q..z9 (altered pgp+btc -> clone)
Step 2, What "detect" means for each
Verify in the Docker host
$ ./lab check 8.5
- The mirror serves content identical to the market at a different address
- The clone serves look-alike content with altered PGP and coin-address fields
- You can state the different test that detects a mirror versus a clone
LAB 8.6
Cross-network links
Step 1, Bring up an overlay alongside the range
# the range's directory links off to an eepsite and a freesite; bring those up $ ./lab up i2p # and/or: ./lab up hyphanet $ ./lab range list --cross $ ./lab stop overlays # stop — both overlays kept; ./lab up resumes them $ ./lab down overlays # delete — removes both (add --volumes to drop seeds)
Expected, links that leave the Tor network
cross-network references in the range directory: market -> (i2p) ...b32.i2p an eepsite mirror of a listing archive -> (hyphanet) USK@.../archive/ a freesite holding "leaked" docs following these requires the I2P / Hyphanet transports, not Tor
Step 2, What this means for a collection pipeline
Verify in the Docker host
$ ./lab check 8.6
- The range directory contains references to an I2P and a Hyphanet address
- The ground truth records those cross-network links and their required transport
- You can explain how a multi-transport pipeline routes a fetch by an address's network
LAB 8.7
Ground truth and the scoring harness
Step 1, Score a sample crawl
# score a crawl output file against the range ground truth $ ./lab range score sample-crawl.json
Expected, recall, precision, and the failures that matter
scored crawl against range ground truth services found 4 / 5 recall 0.80 (missed: paste) reported real 4 / 7 precision 0.57 (3 off-range / invented) mirrors collapsed 1 / 1 (market-mirror recognised as market) clones caught 0 / 1 FAIL — market-clone reported as genuine (swapped btc trusted) overall: good coverage; the clone was believed. fix before trusting output.
Step 2, Why the range is an instrument, not just a target
Step 3, The handoff to Chapter 9
Verify in the Docker host
$ ./lab check 8.7
- The ground-truth manifest ships and the scorer self-tests (recall/precision + clone detection)
- Scoring a sample crawl reports coverage and whether the clone was believed
- You can explain why these metrics are computable against the range but not the live dark web