Part III · Collection · Chapter 9

Crawling Hidden Services

Commands only. What each step does, why it is built this way, and the judgment behind it are in the book.

Infrastructure tier 7 labs ≈ 4.5–7 h 8 GB RAM Windows · macOS · Linux

Labs in this chapter

What you'll be able to do

  • Explain why a dark-web crawler's architecture follows from the clearnet assumptions that break, and name which lab handles each.
  • Run a seed-driven frontier with URL normalization, a seen-set, and a scope guard that refuses everything off-range.
  • Fetch through Tor the way collection actually works: with timeouts, retries, rate-limiting, and failure treated as the normal case.
  • Extract and classify links across Tor, I2P, and Hyphanet, and get past a login wall to collect a catalogue instead of a 302.
  • Run a crawler continuously against a moving target, keep provenance, collapse mirrors, tag clones, and grade the whole thing on the Lab 8.7 harness.
LAB 9.1

The frontier: crawling without an index

1 h foundation frontier.py first score

Stand up the range

Ubuntu 
$ ./lab up range
$ ./lab stop range     # stop — containers kept; ./lab up range resumes them
$ ./lab down range     # delete — removes containers (add --volumes to drop the content)

Read the frontier before you run it

Ubuntu 
# the frontier's brain runs offline — no Docker, no Tor
$ ./lab crawl selftest
# classify an address the way the parser will
$ python3 labs/artifacts/crawler/frontier.py --classify http://example.b32.i2p/
Expected
selftest: classification, normalization, the scope guard (clearnet and
          off-range refused), dedup, and politeness all hold  -> PASS
...
i2p

Run the naive crawl and score it

Ubuntu 
# a first crawl: seeds and links only, then grade it against ground truth
$ ./lab crawl score --naive
Expected, illustrative; the scorer prints your run's real numbers
scored crawl against range ground truth
  services found     3 / 5     recall    0.60    (missed: leak, paste)
  reported real      4 / 5     precision 0.80    (1 off-range / invented)
  mirrors collapsed  0 / 1
  clones caught      0 / 1     FAIL — clone reported as genuine (swapped payment trusted)
Verify in the Docker host
Ubuntu 
$ ./lab check 9.1
  • The frontier self-tests pass: normalization, the seen-set, and politeness all hold
  • The scope guard refuses a clearnet URL and an off-range onion, and admits a range service
  • A naive crawl scores against the range and its recall reflects the unreached services
What this lab looks like
Terminal showing ./lab check 9.1 and its output
Terminal: ./lab check 9.1
Terminal output captured during lab 9.1
Terminal output for this lab
Firefox rendering an HTML view for lab 9.1
Firefox on this lab's HTML view
LAB 9.2

Fetching through Tor: slow, unreliable, and you must be polite

0.5–1 h the fetch layer timeouts & retries politeness

Fetch through the workstation's Tor

Ubuntu 
# fetch one range service through Tor, to confirm the path works
$ ./lab range fetch market /
# the crawler uses the same path; a run reports slow and failed fetches, not just successes
$ ./lab crawl range --naive
Expected, a fetch layer narrates failure as routine
  crawling the range through the workstation's Tor (naive crawl)...
wrote /evidence/crawl-output.json: 4 services (naive crawl)
Verify in the Docker host
Ubuntu 
$ ./lab check 9.2
  • A single fetch through the workstation's Tor reaches a range service
  • The fetch layer applies a timeout and a bounded retry, and treats failure as routine
  • A per-host politeness delay is enforced, so the crawler paces itself
What this lab looks like
Terminal showing ./lab check 9.2 and its output
Terminal: ./lab check 9.2
Mousepad showing an artifact dump from lab 9.2
Artifact dump in Mousepad
Firefox rendering an HTML view for lab 9.2
Firefox on this lab's HTML view
LAB 9.3

Parsing and link extraction across networks

0.5–1 h extract.py classify by network scope guard

Extract links the dark web actually uses

Ubuntu 
# the extractor self-tests on fixture pages: href + bare links, key material, hashing
$ python3 labs/artifacts/crawler/extract.py --selftest
# classification is form-based; try each network
$ python3 labs/artifacts/crawler/frontier.py --classify "USK@cafe/archive/0/"
Expected
selftest: href + bare-text links are extracted and classified, key material
          is pulled, and content hashing is stable  -> PASS
hyphanet

Let the guard decide what is followable

Verify in the Docker host
Ubuntu 
$ ./lab check 9.3
  • The extractor pulls both href links and bare dark-web addresses from a page
  • Addresses classify correctly into Tor, I2P, Hyphanet, and clearnet by form
  • The scope guard refuses clearnet and off-range links; cross-network links are recognised, not fetched
What this lab looks like
Terminal showing ./lab check 9.3 and its output
Terminal: ./lab check 9.3
Firefox rendering an HTML view for lab 9.3
Firefox on this lab's HTML view
Mousepad showing an artifact dump from lab 9.3
Artifact dump in Mousepad
LAB 9.4

Sessions and state: getting past the wall

0.5–1 h cookies & login pagination recall 0.60 → 0.80

Walk the login wall

Ubuntu 
# the full engine holds a session, clears the wall, and reaches what lies past it
$ ./lab crawl score
Expected, illustrative; recall rises once the wall is cleared
scored crawl against range ground truth
  services found     4 / 5     recall    0.80    (missed: leak)
  reported real      4 / 4     precision 1.00
  mirrors collapsed  1 / 1
  clones caught      1 / 1
Verify in the Docker host
Ubuntu 
$ ./lab check 9.4
  • The crawler holds a session, walks the login redirect, and retries the request authenticated
  • The market catalogue is collected instead of the login wall, and its onward links are followed
  • Recall rises relative to the naive crawl, reflecting the service reached only through the session
What this lab looks like
Terminal showing ./lab check 9.4 and its output
Terminal: ./lab check 9.4
Firefox rendering an HTML view for lab 9.4
Firefox on this lab's HTML view
Terminal output captured during lab 9.4
Terminal output for this lab
LAB 9.5

Continuous collection: revisiting a moving target

0.5–1 h recrawl schedule change detection run across a flap

Crawl, wait, crawl again

Ubuntu 
# crawl once, then flap a service so it republishes at a NEW onion
$ ./lab crawl range
$ ./lab range flap leak
# re-read the directory and crawl again — the moved service is re-found at its new address
$ ./lab range list
$ ./lab crawl range
Expected, the crawler recovers a moved target
  range: simulating seizure/rebrand of 'leak' — dropping its onion key
  range: 'leak' will republish at a NEW onion; a crawler holding the old address now collects nothing
  crawling the range through the workstation's Tor (full engine)...
wrote /evidence/crawl-output.json: 4 services (full engine)
Verify in the Docker host
Ubuntu 
$ ./lab check 9.5
  • Services carry a next-revisit time and the scheduler recrawls on it
  • A content hash from the prior visit detects change and suppresses re-storing identical pages
  • After a flap, a fresh crawl re-finds the moved service at its new address
What this lab looks like
Terminal showing ./lab check 9.5 and its output
Terminal: ./lab check 9.5
Firefox rendering an HTML view for lab 9.5
Firefox on this lab's HTML view
Mousepad showing an artifact dump from lab 9.5
Artifact dump in Mousepad
LAB 9.6

Storage, provenance, and dedup

0.5–1 h provenance mirror-collapse clones tagged

Keep provenance, then resolve identity

Ubuntu 
# the range holds a mirror and a clone of the market; compare the three by hash + keys
$ ./lab range compare market market-mirror market-clone
# the full engine records these decisions in its crawl-output
$ ./lab crawl range
Expected, identity by content, not address
  market         hash 1f3c…  pgp 9A3F…  btc bc1q_market_k7
  market-mirror  hash 1f3c…  pgp 9A3F…  btc bc1q_market_k7   → identical content → MIRROR
  market-clone   hash 7b90…  pgp 2E77…  btc bc1q_clone_z9    → same layout, swapped keys → CLONE
Verify in the Docker host
Ubuntu 
$ ./lab check 9.6
  • Stored pages carry provenance: source address, fetch time, and the crawl that found them
  • Mirrors are collapsed by content hash and recorded as mirrors of a canonical service
  • A clone with swapped key material is flagged by structural match, and precision improves as a result
What this lab looks like
Terminal showing ./lab check 9.6 and its output
Terminal: ./lab check 9.6
Terminal output captured during lab 9.6
Terminal output for this lab
Firefox rendering an HTML view for lab 9.6
Firefox on this lab's HTML view
LAB 9.7

Scoring the crawler and closing the loop

1 h the payoff naive vs full gaps → Ch 10 · 11

Grade the full engine, then compare

Ubuntu 
# the finished engine, graded against the range's ground truth
$ ./lab crawl score
# the same run with every capability off, for comparison
$ ./lab crawl score --naive
Expected, full engine, then naive; illustrative but reconciled to the tool
# full engine
scored crawl against range ground truth
  services found     4 / 5     recall    0.80    (missed: leak)
  reported real      4 / 4     precision 1.00
  mirrors collapsed  1 / 1
  clones caught      1 / 1

# naive crawl
scored crawl against range ground truth
  services found     3 / 5     recall    0.60    (missed: leak, paste)
  reported real      4 / 5     precision 0.80    (1 off-range / invented)
  mirrors collapsed  0 / 1
  clones caught      0 / 1     FAIL — clone reported as genuine (swapped payment trusted)
Verify in the Docker host
Ubuntu 
$ ./lab check 9.7
  • The full crawler runs against the range and emits a crawl-output the Lab 8.7 scorer grades
  • The full engine outscores the naive crawl on recall, precision, mirror-collapse, and clone-catching
  • You can name the remaining gaps, the seed problem, shallow clone detection, cross-network fetch, and which chapter owns each
What this lab looks like
Terminal showing ./lab check 9.7 and its output
Terminal: ./lab check 9.7
Mousepad showing an artifact dump from lab 9.7
Artifact dump in Mousepad
Firefox rendering an HTML view for lab 9.7
Firefox on this lab's HTML view