Part III · Collection · Chapter 11

Markets & Forums

Commands only. What each step does, why it is built this way, and the judgment behind it are in the book.

LAB 11.1

The market as a database, and a page store

Meet the market lab

Ubuntu 
$ ls labs/artifacts/market-extract/corpus/
$ ./lab market selftest     # the whole extractor self-tests offline — no Docker, no Tor
Expected
selftest: byte-identical mirrors collapse to one stored object, provenance keeps
          every sighting, and the store exports bodies for dedup  -> PASS
selftest: listings parse to typed records, and label fallback recovers a drifted
          page the class-only parser drops  -> PASS
selftest: vendor profiles parse to typed records, carrying the fingerprint and
          join-vs-feedback history the graph will judge  -> PASS
selftest: CAPTCHA is queued for a human (never solved), rate limits back off,
          honeypots are skipped, poisoned catalogues refused  -> PASS
selftest: the graph flags the resale ring, the borrowed key, the gamed reputation,
          and the bait price — the market lying in its own data  -> PASS
selftest: naive completeness 0.92, flags 0.00;
          full completeness 1.00, flags 1.00, clean collection  -> PASS
  market self-tests passed (store, listings, vendors, defenses, graph, pipeline+scorer)

Build the store, and collapse a mirror

Ubuntu 
$ ./lab market store
Expected
  ingested 19 fetched pages -> 18 distinct objects
  1 byte-identical mirror collapsed at storage (content addressing = Ch10 exact-mirror, at storage time)
  provenance kept every sighting; exported 18 bodies for: ./lab dedup run --dir <store>/bodies
Verify in the Docker host
Ubuntu 
$ ./lab check 11.1
  • The market-lab corpus is present and every page carries the synthetic watermark
  • ./lab market selftest passes offline, store, listings, vendors, defenses, graph, and the scorer
  • ./lab market store ingests 19 pages into 18 objects, collapsing the byte-identical mirror at storage
LAB 11.2

Structured extraction: pages to records

From a page to a typed record

Ubuntu 
# the full pipeline: resilient parsing, defense detection, adversarial flags
$ ./lab market extract
Expected
  extracting the market-lab corpus (full pipeline):
    listings parsed: 7 (7 complete)   vendors parsed: 5
    flags: rings=[[1001, 1006]] borrowed_keys=[['Mimic', 'NightHawk']] gamed=['SaltMine'] scam=[1005]
    poisoned extracted: 0   honeypots skipped: 1

What the brittle scraper loses

Ubuntu 
$ ./lab market extract --naive
Expected
  extracting the market-lab corpus (naive scraper):
    listings parsed: 7 (6 complete)   vendors parsed: 5
    flags: rings=[] borrowed_keys=[] gamed=[] scam=[]
    poisoned extracted: 1   honeypots skipped: 0
Verify in the Docker host
Ubuntu 
$ ./lab check 11.2
  • The listing and vendor parsers self-test, producing typed records from the corpus
  • The drift variant is complete under the resilient parser and incomplete under the class-only parser
  • The full extraction reports 7 complete listings where the naive scraper reports 6
LAB 11.3

Anti-crawling: detect, never defeat

The four walls, and the restrained answers

Ubuntu 
$ ./lab market defenses
Expected
  classifying the market's countermeasures (detect-only):
  wall-captcha.html            -> captcha      action: queue
  wall-429.html                -> rate_limited action: backoff
  catalogue-poisoned.html      -> poisoned     action: skip
  honeypot links detected: ['/trap/9f2a'] (skipped)
  CAPTCHA is queued for a human — never solved (no solver in this module)
Verify in the Docker host
Ubuntu 
$ ./lab check 11.3
  • The defense detector self-tests: CAPTCHA, rate limit, poisoned catalogue, and honeypot are each classified
  • A CAPTCHA routes to queue and never to a solver, the module exposes no solve path
  • The honeypot link in the hardware category is detected and marked skipped
LAB 11.4

Sessions at scale, and the shadow-ban

Managing a pool, and detecting degradation

Ubuntu 
# a brittle scraper on what looks like a normal session but is a flagged one
$ ./lab market score --naive
Expected
  extracted the market-lab corpus (naive scraper); grading against ground truth:
scored extraction against market-lab ground truth
  field recall        81 / 88     0.92
  record completeness 11 / 12       0.92
  adversarial flags   0 / 4         recall 0.00
  defenses detected   0 / 3         (captcha queued, never solved)
  poisoned extracted  1           (FAIL — adversarial content taken as real)
  honeypots skipped   0
Verify in the Docker host
Ubuntu 
$ ./lab check 11.4
  • The poisoned catalogue is classified as degraded content and routed to skip, not extracted
  • The naive scraper extracts the poisoned catalogue; its poison count is one
  • The distinction rests on the catalogue's content, not its HTTP status
LAB 11.5

The vendor and reputation graph

Rings, on a content edge

Ubuntu 
$ ./lab market graph
Expected
  building the vendor/reputation graph:
    vendors: 5   listings: 6
    prolific: [('NightHawk', 2), ('GreyOwl', 1), ('Mimic', 1), ('PaperTrail', 1), ('SaltMine', 1)]
    resale rings:     [[1001, 1006]]
    borrowed keys:    [['Mimic', 'NightHawk']]
    gamed reputation: ['SaltMine']
    bait prices:      [1005]
Verify in the Docker host
Ubuntu 
$ ./lab check 11.5
  • The graph self-tests: it identifies NightHawk as the prolific vendor with two listings
  • The resale ring is exactly listings 1001 and 1006; one item under two handles
  • Rings key on content similarity, not the template-and-asset edge that clustered whole sites
LAB 11.6

The data is adversarial

Three lies, three checks

Ubuntu 
# the same graph, read for the market's lies about itself
$ ./lab market graph
Expected
    resale rings:     [[1001, 1006]]
    borrowed keys:    [['Mimic', 'NightHawk']]
    gamed reputation: ['SaltMine']
    bait prices:      [1005]
Verify in the Docker host
Ubuntu 
$ ./lab check 11.6
  • The bait price is listing 1005, flagged against the hardware category median
  • The borrowed key is Mimic advertising NightHawk's fingerprint, caught by the shared-key signal
  • The gamed reputation is SaltMine, caught by feedback velocity against its join date
LAB 11.7

Scoring extraction, and closing the loop

The full extractor

Ubuntu 
$ ./lab market score
Expected
  extracted the market-lab corpus (full pipeline); grading against ground truth:
scored extraction against market-lab ground truth
  field recall        88 / 88     1.00
  record completeness 12 / 12       1.00
  adversarial flags   4 / 4         recall 1.00
  defenses detected   3 / 3         (captcha queued, never solved)
  poisoned extracted  0           (clean)
  honeypots skipped   1

The brittle baseline

Ubuntu 
$ ./lab market score --naive
Expected
  extracted the market-lab corpus (naive scraper); grading against ground truth:
scored extraction against market-lab ground truth
  field recall        81 / 88     0.92
  record completeness 11 / 12       0.92
  adversarial flags   0 / 4         recall 0.00
  defenses detected   0 / 3         (captcha queued, never solved)
  poisoned extracted  1           (FAIL — adversarial content taken as real)
  honeypots skipped   0

Reflection

Verify in the Docker host
Ubuntu 
$ ./lab check 11.7
  • The full extractor scores field recall 1.00, completeness 1.00, flag recall 1.00, and refuses the poison
  • The full extractor outscores the naive baseline on completeness, flags, defenses, and the poison count
  • You can name the three residual gaps, live collection through the store, the consistency-check arms race, cross-market linkage, and which part owns them
LAB 11.8

Markets and forums over a real, scrubbed archive

Runs the extraction engine against a public archived slice of a marketplace that has been offline since 2015, so you can hold what the synthetic range taught you against real listings.

Written up in the book, commands and all.