Part III · Collection · Chapter 11

Markets & Forums

Commands only. What each step does, why it is built this way, and the judgment behind it are in the book.

Infrastructure tier 8 labs ≈ 4–8 h Pure Python · no Docker Windows · macOS · Linux

Labs in this chapter

What you'll be able to do

  • Model a market as an entity schema and stand up a content-addressed page store that collapses byte-identical mirrors at storage and feeds real bodies to the Chapter 10 detector.
  • Parse listing and vendor pages into typed records that survive markup drift, and measure the field recall a class-only scraper loses when the market restyles.
  • Detect a market's countermeasures, CAPTCHA, rate limit, honeypot, poisoned catalogue, and route each to the correct restrained response, queuing a wall for a human rather than solving it.
  • Build a vendor and reputation graph that reuses Chapter 10's clustering and signals to surface resale rings, borrowed keys, gamed reputation, and bait prices.
  • Grade a full extractor against a brittle baseline on a shipped answer key, and name the residual gaps that Part IV inherits.
LAB 11.1

The market as a database, and a page store

0.5–1 h the schema store.py content-addressed

Meet the market lab

Ubuntu 
$ ls labs/artifacts/market-extract/corpus/
$ ./lab market selftest     # the whole extractor self-tests offline — no Docker, no Tor
Expected
selftest: byte-identical mirrors collapse to one stored object, provenance keeps
          every sighting, and the store exports bodies for dedup  -> PASS
selftest: listings parse to typed records, and label fallback recovers a drifted
          page the class-only parser drops  -> PASS
selftest: vendor profiles parse to typed records, carrying the fingerprint and
          join-vs-feedback history the graph will judge  -> PASS
selftest: CAPTCHA is queued for a human (never solved), rate limits back off,
          honeypots are skipped, poisoned catalogues refused  -> PASS
selftest: the graph flags the resale ring, the borrowed key, the gamed reputation,
          and the bait price — the market lying in its own data  -> PASS
selftest: naive completeness 0.92, flags 0.00;
          full completeness 1.00, flags 1.00, clean collection  -> PASS
  market self-tests passed (store, listings, vendors, defenses, graph, pipeline+scorer)

Build the store, and collapse a mirror

Ubuntu 
$ ./lab market store
Expected
  ingested 19 fetched pages -> 18 distinct objects
  1 byte-identical mirror collapsed at storage (content addressing = Ch10 exact-mirror, at storage time)
  provenance kept every sighting; exported 18 bodies for: ./lab dedup run --dir <store>/bodies
Verify in the Docker host
Ubuntu 
$ ./lab check 11.1
  • The market-lab corpus is present and every page carries the synthetic watermark
  • ./lab market selftest passes offline, store, listings, vendors, defenses, graph, and the scorer
  • ./lab market store ingests 19 pages into 18 objects, collapsing the byte-identical mirror at storage
What this lab looks like
Terminal showing ./lab check 11.1 and its output
Terminal: ./lab check 11.1
Firefox rendering an HTML view for lab 11.1
Firefox on this lab's HTML view
Mousepad showing an artifact dump from lab 11.1
Artifact dump in Mousepad
LAB 11.2

Structured extraction: pages to records

0.5–1 h listings.py vendors.py markup drift

From a page to a typed record

Ubuntu 
# the full pipeline: resilient parsing, defense detection, adversarial flags
$ ./lab market extract
Expected
  extracting the market-lab corpus (full pipeline):
    listings parsed: 7 (7 complete)   vendors parsed: 5
    flags: rings=[[1001, 1006]] borrowed_keys=[['Mimic', 'NightHawk']] gamed=['SaltMine'] scam=[1005]
    poisoned extracted: 0   honeypots skipped: 1

What the brittle scraper loses

Ubuntu 
$ ./lab market extract --naive
Expected
  extracting the market-lab corpus (naive scraper):
    listings parsed: 7 (6 complete)   vendors parsed: 5
    flags: rings=[] borrowed_keys=[] gamed=[] scam=[]
    poisoned extracted: 1   honeypots skipped: 0
Verify in the Docker host
Ubuntu 
$ ./lab check 11.2
  • The listing and vendor parsers self-test, producing typed records from the corpus
  • The drift variant is complete under the resilient parser and incomplete under the class-only parser
  • The full extraction reports 7 complete listings where the naive scraper reports 6
What this lab looks like
Terminal showing ./lab check 11.2 and its output
Terminal: ./lab check 11.2
Terminal output captured during lab 11.2
Terminal output for this lab
Firefox rendering an HTML view for lab 11.2
Firefox on this lab's HTML view
LAB 11.3

Anti-crawling: detect, never defeat

0.5–1 h defenses.py CAPTCHA → queue no solver

The four walls, and the restrained answers

Ubuntu 
$ ./lab market defenses
Expected
  classifying the market's countermeasures (detect-only):
  wall-captcha.html            -> captcha      action: queue
  wall-429.html                -> rate_limited action: backoff
  catalogue-poisoned.html      -> poisoned     action: skip
  honeypot links detected: ['/trap/9f2a'] (skipped)
  CAPTCHA is queued for a human — never solved (no solver in this module)
Verify in the Docker host
Ubuntu 
$ ./lab check 11.3
  • The defense detector self-tests: CAPTCHA, rate limit, poisoned catalogue, and honeypot are each classified
  • A CAPTCHA routes to queue and never to a solver, the module exposes no solve path
  • The honeypot link in the hardware category is detected and marked skipped
What this lab looks like
Terminal showing ./lab check 11.3 and its output
Terminal: ./lab check 11.3
Mousepad showing an artifact dump from lab 11.3
Artifact dump in Mousepad
Firefox rendering an HTML view for lab 11.3
Firefox on this lab's HTML view
LAB 11.4

Sessions at scale, and the shadow-ban

0.5–1 h session pool shadow-ban identity is content

Managing a pool, and detecting degradation

Ubuntu 
# a brittle scraper on what looks like a normal session but is a flagged one
$ ./lab market score --naive
Expected
  extracted the market-lab corpus (naive scraper); grading against ground truth:
scored extraction against market-lab ground truth
  field recall        81 / 88     0.92
  record completeness 11 / 12       0.92
  adversarial flags   0 / 4         recall 0.00
  defenses detected   0 / 3         (captcha queued, never solved)
  poisoned extracted  1           (FAIL — adversarial content taken as real)
  honeypots skipped   0
Verify in the Docker host
Ubuntu 
$ ./lab check 11.4
  • The poisoned catalogue is classified as degraded content and routed to skip, not extracted
  • The naive scraper extracts the poisoned catalogue; its poison count is one
  • The distinction rests on the catalogue's content, not its HTTP status
What this lab looks like
Terminal showing ./lab check 11.4 and its output
Terminal: ./lab check 11.4
Firefox rendering an HTML view for lab 11.4
Firefox on this lab's HTML view
Terminal output captured during lab 11.4
Terminal output for this lab
LAB 11.5

The vendor and reputation graph

0.5–1 h graph.py resale rings reuses Ch10

Rings, on a content edge

Ubuntu 
$ ./lab market graph
Expected
  building the vendor/reputation graph:
    vendors: 5   listings: 6
    prolific: [('NightHawk', 2), ('GreyOwl', 1), ('Mimic', 1), ('PaperTrail', 1), ('SaltMine', 1)]
    resale rings:     [[1001, 1006]]
    borrowed keys:    [['Mimic', 'NightHawk']]
    gamed reputation: ['SaltMine']
    bait prices:      [1005]
Verify in the Docker host
Ubuntu 
$ ./lab check 11.5
  • The graph self-tests: it identifies NightHawk as the prolific vendor with two listings
  • The resale ring is exactly listings 1001 and 1006; one item under two handles
  • Rings key on content similarity, not the template-and-asset edge that clustered whole sites
What this lab looks like
Terminal showing ./lab check 11.5 and its output
Terminal: ./lab check 11.5
Firefox rendering an HTML view for lab 11.5
Firefox on this lab's HTML view
Mousepad showing an artifact dump from lab 11.5
Artifact dump in Mousepad
LAB 11.6

The data is adversarial

0.5–1 h bait price borrowed key gamed reputation

Three lies, three checks

Ubuntu 
# the same graph, read for the market's lies about itself
$ ./lab market graph
Expected
    resale rings:     [[1001, 1006]]
    borrowed keys:    [['Mimic', 'NightHawk']]
    gamed reputation: ['SaltMine']
    bait prices:      [1005]
Verify in the Docker host
Ubuntu 
$ ./lab check 11.6
  • The bait price is listing 1005, flagged against the hardware category median
  • The borrowed key is Mimic advertising NightHawk's fingerprint, caught by the shared-key signal
  • The gamed reputation is SaltMine, caught by feedback velocity against its join date
What this lab looks like
Terminal showing ./lab check 11.6 and its output
Terminal: ./lab check 11.6
Firefox rendering an HTML view for lab 11.6
Firefox on this lab's HTML view
Terminal output captured during lab 11.6
Terminal output for this lab
LAB 11.7

Scoring extraction, and closing the loop

0.5–1 h market-scorer full vs naive the number

The full extractor

Ubuntu 
$ ./lab market score
Expected
  extracted the market-lab corpus (full pipeline); grading against ground truth:
scored extraction against market-lab ground truth
  field recall        88 / 88     1.00
  record completeness 12 / 12       1.00
  adversarial flags   4 / 4         recall 1.00
  defenses detected   3 / 3         (captcha queued, never solved)
  poisoned extracted  0           (clean)
  honeypots skipped   1

The brittle baseline

Ubuntu 
$ ./lab market score --naive
Expected
  extracted the market-lab corpus (naive scraper); grading against ground truth:
scored extraction against market-lab ground truth
  field recall        81 / 88     0.92
  record completeness 11 / 12       0.92
  adversarial flags   0 / 4         recall 0.00
  defenses detected   0 / 3         (captcha queued, never solved)
  poisoned extracted  1           (FAIL — adversarial content taken as real)
  honeypots skipped   0

Reflection

Verify in the Docker host
Ubuntu 
$ ./lab check 11.7
  • The full extractor scores field recall 1.00, completeness 1.00, flag recall 1.00, and refuses the poison
  • The full extractor outscores the naive baseline on completeness, flags, defenses, and the poison count
  • You can name the three residual gaps, live collection through the store, the consistency-check arms race, cross-market linkage, and which part owns them
What this lab looks like
Terminal showing ./lab check 11.7 and its output
Terminal: ./lab check 11.7
Terminal output captured during lab 11.7
Terminal output for this lab
Firefox rendering an HTML view for lab 11.7
Firefox on this lab's HTML view
LAB 11.8

Markets and forums over a real, scrubbed archive

Runs the extraction engine against a public archived slice of a marketplace that has been offline since 2015, so you can hold what the synthetic range taught you against real listings.

Written up in the book, commands and all.