Part III · Collection · Chapter 9

Crawling Hidden Services

Commands only. What each step does, why it is built this way, and the judgment behind it are in the book.

LAB 9.1

The frontier: crawling without an index

Stand up the range

Ubuntu 
$ ./lab up range
$ ./lab stop range     # stop — containers kept; ./lab up range resumes them
$ ./lab down range     # delete — removes containers (add --volumes to drop the content)

Read the frontier before you run it

Ubuntu 
# the frontier's brain runs offline — no Docker, no Tor
$ ./lab crawl selftest
# classify an address the way the parser will
$ python3 labs/artifacts/crawler/frontier.py --classify http://example.b32.i2p/
Expected
selftest: classification, normalization, the scope guard (clearnet and
          off-range refused), dedup, and politeness all hold  -> PASS
...
i2p

Run the naive crawl and score it

Ubuntu 
# a first crawl: seeds and links only, then grade it against ground truth
$ ./lab crawl score --naive
Expected, illustrative; the scorer prints your run's real numbers
scored crawl against range ground truth
  services found     3 / 5     recall    0.60    (missed: leak, paste)
  reported real      4 / 5     precision 0.80    (1 off-range / invented)
  mirrors collapsed  0 / 1
  clones caught      0 / 1     FAIL — clone reported as genuine (swapped payment trusted)
Verify in the Docker host
Ubuntu 
$ ./lab check 9.1
  • The frontier self-tests pass: normalization, the seen-set, and politeness all hold
  • The scope guard refuses a clearnet URL and an off-range onion, and admits a range service
  • A naive crawl scores against the range and its recall reflects the unreached services
LAB 9.2

Fetching through Tor: slow, unreliable, and you must be polite

Fetch through the workstation's Tor

Ubuntu 
# fetch one range service through Tor, to confirm the path works
$ ./lab range fetch market /
# the crawler uses the same path; a run reports slow and failed fetches, not just successes
$ ./lab crawl range --naive
Expected, a fetch layer narrates failure as routine
  crawling the range through the workstation's Tor (naive crawl)...
wrote /evidence/crawl-output.json: 4 services (naive crawl)
Verify in the Docker host
Ubuntu 
$ ./lab check 9.2
  • A single fetch through the workstation's Tor reaches a range service
  • The fetch layer applies a timeout and a bounded retry, and treats failure as routine
  • A per-host politeness delay is enforced, so the crawler paces itself
LAB 9.3

Parsing and link extraction across networks

Extract links the dark web actually uses

Ubuntu 
# the extractor self-tests on fixture pages: href + bare links, key material, hashing
$ python3 labs/artifacts/crawler/extract.py --selftest
# classification is form-based; try each network
$ python3 labs/artifacts/crawler/frontier.py --classify "USK@cafe/archive/0/"
Expected
selftest: href + bare-text links are extracted and classified, key material
          is pulled, and content hashing is stable  -> PASS
hyphanet

Let the guard decide what is followable

Verify in the Docker host
Ubuntu 
$ ./lab check 9.3
  • The extractor pulls both href links and bare dark-web addresses from a page
  • Addresses classify correctly into Tor, I2P, Hyphanet, and clearnet by form
  • The scope guard refuses clearnet and off-range links; cross-network links are recognised, not fetched
LAB 9.4

Sessions and state: getting past the wall

Walk the login wall

Ubuntu 
# the full engine holds a session, clears the wall, and reaches what lies past it
$ ./lab crawl score
Expected, illustrative; recall rises once the wall is cleared
scored crawl against range ground truth
  services found     4 / 5     recall    0.80    (missed: leak)
  reported real      4 / 4     precision 1.00
  mirrors collapsed  1 / 1
  clones caught      1 / 1
Verify in the Docker host
Ubuntu 
$ ./lab check 9.4
  • The crawler holds a session, walks the login redirect, and retries the request authenticated
  • The market catalogue is collected instead of the login wall, and its onward links are followed
  • Recall rises relative to the naive crawl, reflecting the service reached only through the session
LAB 9.5

Continuous collection: revisiting a moving target

Crawl, wait, crawl again

Ubuntu 
# crawl once, then flap a service so it republishes at a NEW onion
$ ./lab crawl range
$ ./lab range flap leak
# re-read the directory and crawl again — the moved service is re-found at its new address
$ ./lab range list
$ ./lab crawl range
Expected, the crawler recovers a moved target
  range: simulating seizure/rebrand of 'leak' — dropping its onion key
  range: 'leak' will republish at a NEW onion; a crawler holding the old address now collects nothing
  crawling the range through the workstation's Tor (full engine)...
wrote /evidence/crawl-output.json: 4 services (full engine)
Verify in the Docker host
Ubuntu 
$ ./lab check 9.5
  • Services carry a next-revisit time and the scheduler recrawls on it
  • A content hash from the prior visit detects change and suppresses re-storing identical pages
  • After a flap, a fresh crawl re-finds the moved service at its new address
LAB 9.6

Storage, provenance, and dedup

Keep provenance, then resolve identity

Ubuntu 
# the range holds a mirror and a clone of the market; compare the three by hash + keys
$ ./lab range compare market market-mirror market-clone
# the full engine records these decisions in its crawl-output
$ ./lab crawl range
Expected, identity by content, not address
  market         hash 1f3c…  pgp 9A3F…  btc bc1q_market_k7
  market-mirror  hash 1f3c…  pgp 9A3F…  btc bc1q_market_k7   → identical content → MIRROR
  market-clone   hash 7b90…  pgp 2E77…  btc bc1q_clone_z9    → same layout, swapped keys → CLONE
Verify in the Docker host
Ubuntu 
$ ./lab check 9.6
  • Stored pages carry provenance: source address, fetch time, and the crawl that found them
  • Mirrors are collapsed by content hash and recorded as mirrors of a canonical service
  • A clone with swapped key material is flagged by structural match, and precision improves as a result
LAB 9.7

Scoring the crawler and closing the loop

Grade the full engine, then compare

Ubuntu 
# the finished engine, graded against the range's ground truth
$ ./lab crawl score
# the same run with every capability off, for comparison
$ ./lab crawl score --naive
Expected, full engine, then naive; illustrative but reconciled to the tool
# full engine
scored crawl against range ground truth
  services found     4 / 5     recall    0.80    (missed: leak)
  reported real      4 / 4     precision 1.00
  mirrors collapsed  1 / 1
  clones caught      1 / 1

# naive crawl
scored crawl against range ground truth
  services found     3 / 5     recall    0.60    (missed: leak, paste)
  reported real      4 / 5     precision 0.80    (1 off-range / invented)
  mirrors collapsed  0 / 1
  clones caught      0 / 1     FAIL — clone reported as genuine (swapped payment trusted)
Verify in the Docker host
Ubuntu 
$ ./lab check 9.7
  • The full crawler runs against the range and emits a crawl-output the Lab 8.7 scorer grades
  • The full engine outscores the naive crawl on recall, precision, mirror-collapse, and clone-catching
  • You can name the remaining gaps, the seed problem, shallow clone detection, cross-network fetch, and which chapter owns each