Wander Around — A Walkable Cartography of the Forgotten Web

State paper, 2026-05-04

Abstract

Wander Around is a single-server substrate that turns large public corpora into walkable 3D environments. Eleven walkable surfaces run on one Hetzner CPX42 node (89.167.7.54), spatializing roughly 720,000 items across nine distinct corpora — long-tail Common Crawl (6.83M useful cards in DB, 197K placed in /world), all of English Wikipedia (~6.8M articles, 200K placed in /wiki), Project Gutenberg (~22.9K books, 17.9K placed), Internet Archive Books (33.7K placed), arXiv papers (30.7K placed), DOAB open-access books (6.6K placed), plus an interactive 3D sphere view (/4d) and a thumb-only walkable subset (/rendered, ~33K cards with real WebP screenshots).

The thesis is simple: the virtual realm is a dimension. We just never had a projection of it. SEO turned the open web into a flat ranked list optimized for clickthrough — but the web is actually a vast city of hand-built homesites, weird hobbyist pages, irregular blogs, raw HTML. Most of it is invisible to the algorithms that decide what you see. Wander Around is a different projection: a cartographic instrument that lets you walk through what the algorithms threw away.

What’s running, end of 2026-05-04

Surface	Corpus	Items placed	Status
`/world`	long-tail web (Common Crawl)	197,512 / 6,830,239 useful	full chrome, lazy thumbs, gold real-thumb markers
`/game`	2D mandala over web	200,000 (categories_flat)	8 graph paradigms, Set A/B/C panels
`/rendered`	web subset with real thumbs	33,811 placed	tighter pinwheel, populates as thumbs render
`/wiki`	English Wikipedia	200,000 / 6,841,288	walkable, slow thumb pipeline (0.02/s)
`/4d`	sphere of any corpus	configurable	12 longitudinal continents, OrbitControls
`/feed`	swipeable card feed	4M	unchanged baseline
`/library/gutenberg`	Project Gutenberg	17,959 / 22,876	covers fixed, 99.6% have real thumbs
`/library/ia_books`	Internet Archive Books	33,711 placed	covers partial
`/library/arxiv`	arXiv papers	30,674 placed	procedural facades only (papers, no covers)
`/library/doab`	Open-Access Books	6,650 placed	covers untested
`/library`	hub default → gutenberg	redirect	dropdown switches corpora

Background services on Box C

wander-api — FastAPI + nginx, the public entry point
wander-warc — Common Crawl parser, currently paused at 6.83M useful cards
wander-thumbs — Playwright thumb renderer, 86% success at 0.80 thumbs/sec
wander-wkhtml — wkhtmltoimage parallel fast lane, 82% success at 0.73/sec
wander-wiki-thumbs — Wikipedia thumb renderer (slower, ~0.02/s)
wander-refresh.timer — regenerates 4 layout JSONs every 30 min

Combined web-thumb throughput: ~1.5 successful renders/sec ≈ 130,000/day. At this rate the top 200K cards by score will all have real WebP screenshots within 36 hours.

Architecture

The substrate is corpus-agnostic. The same code path serves Common Crawl, Wikipedia, Gutenberg, arXiv, DOAB, and IA Books — the only difference is the source database and a small per-corpus adapter (scripts/library/library_layout.py).

The shared pipeline:

Ingest — corpus-specific scripts pull metadata into a SQLite DB. Each row gets at minimum id, host/title, category, score.
Layout — scripts/10_layout_ring.py produces a 12-sector pinwheel. Per-category radial arms are sized proportional to card count (personal/community arms reach 25-60K, sparse cats stop at 5-10K). Within each sector, era band widths are proportional to per-cat per-era distribution. Within each leaf cell, polar wedge fill places buildings as concentric rings with walkable streets between rows and angular alleys every 4 buildings. Slot size 28 ensures buildings (max width 16 in WORLD_JS) never overlap inside a row.
Thumb pipeline — Playwright + wkhtmltoimage in parallel. Score-prioritized queue. Real WebP saved per card; procedural SVG facade as fallback.
Serve — FastAPI streams chunked card data to the client based on player position (/api/game/region?cx=&cy=&r=).
Render — Three.js scene loads only the ~9 chunks around the player at any time. The ~10 nearest in-frustum buildings get individual Mesh objects with real thumb textures + EdgesGeometry outlines + a gold cone marker if the card has a real screenshot. The bulk InstancedMesh holds the rest.

What changed in the past 72 hours

2026-05-01 evening session (the layout fight)

Variable-width pinwheel: per-cat R_OUTER, per-cat quartile thresholds, per-cat era band widths
Plaza expanded: R_CENTER_PLAZA 700 → 1500, R_PLAZA_INNER 350, R_INNER 1700
Buildings: w 8-16, h 15-45 (player height 4 = real city skyline)
Slot size MIN bumped 16 → 28 to defeat overlap from score-derived widths
Defensive tail loop extends inward at proper spacing instead of stacking
Real overlap audit at end of layout regen
Lazy thumbs: rich procedural facade SVG (host as masthead, era-typed font, category-suggested layout) + gold cone marker over buildings with real WebP thumbs
/world identity: psiloceyeben (was Ben Horn). Central pillar body expanded to 6 paragraphs covering the thesis
$HW commerce pylon correctly framed as Hermes Webkit token
Pillar/pylon preview placards floating above each landmark
Speed slider, bottom-left legend
Minimap pans with player; expand-map button opens fullscreen 2D dot view

2026-05-01 → 2026-05-02 session (the surface explosion)

/game massively upgraded: drag-pan, zoom 0.005-20×, curved arc labels, flat-categories layout, info modal
Set A (display): score threshold, sort, color-by, tooltip detail
Set B (filters): chip multi-select on category/language/era/site_type/score/TLD with AND-stacking
Set C (graph types): mandala, heatmap, treemap, sankey, histograms, sunburst, bubble, stream
/rendered surface: walkable subset of cards with real thumbs
/4d sphere with strip projection (12 longitudinal continents)
Phantom players over WebSocket — see other walkers as ghost capsules with proximity radius
Novel-marks ★ system — anonymous community bookmarks, future /novel route
Library wing live: Gutenberg, IA Books, arXiv, DOAB ingested + walkable
Substrate-template repo at wander-substrate/ ready to push to GitHub
Thumb pipeline claim_batch SQL bug fixed: empty-string sentinel was being re-claimed forever; success jumped from 3% to 87-94%
pgrep shows ingest and cover_fetch processes still running in background

2026-05-04 (this session)

Verified Gutenberg covers at 99.6% (was 1% three days ago — fix script worked, plus the corrected URL pattern derived from book IDs directly)
Trees, lamp posts, benches added to /world plaza ring (18 trees at radius 320-900, 8 glowing lamps at radius 190, 6 benches scattered)
Cartoon procedural facade overhaul: striped awning, scalloped bottom edge, rounded shop sign with cartoon font — Toon Town shop-front aesthetic instead of the previous web-2-formal layout
Edge outlines were already on detail meshes from prior session; bulk InstancedMesh stays untextured (cleaner at distance, GPU-cheap)

The Toon Town palette

The world’s permanent visual rule: floor and sky must never be the same color. Currently:

Sky: #7cc8ff (bright Pixar sky-blue, with #ffd890 golden-horizon fog)
Ground: #88c850 (saturated cartoon grass)
Plaza: #fff5d0 (warm cream paving)
Pillar: #4a6a90 cylinder + #e06850 cone top, 100 units tall
Pylons: 12 colored boxes, one per category, at radius 140
Trees: green cones on brown trunks, MeshToonMaterial with gradient ramp
Clouds: 28 sprites drifting between altitude 280 and 520
Lamps: dark posts with #ffe4a0 glowing sphere tops
Benches: dark brown plain rects facing inward

The MeshToonMaterial + 4-step gradient ramp gives all built geometry a banded cel-shaded look. Add new props with the same material to keep the visual language consistent.

The corpus numbers

Long-tail web: 6,830,239 useful cards in DB. 197K placed in /world (top by composite_score). 33K of those have real WebP screenshots — populates /rendered.
Wikipedia: 6,841,288 articles parsed from a 25GB enwiki bz2 dump. 200K placed in /wiki. Categorization uses Wikipedia’s category graph projected onto a 12-bucket scheme.
Project Gutenberg: 22,876 books pulled from Gutendex API. 17,959 placed (top by download count). 22,794 have real cover images (99.6% coverage). The fix that got us here was deriving cover URLs directly from book IDs (https://www.gutenberg.org/cache/epub/{id}/pg{id}.cover.medium.jpg) instead of trusting the Gutendex API’s broken cover field.
Internet Archive Books: 33,711 placed via IA scrape API.
arXiv: 30,674 placed via Atom API. No covers (papers don’t have them) — procedural SVG is the permanent solution.
DOAB: 6,650 placed via REST API. Covers field needs expand= parameter to populate; current state ~zero coverage.
Total walkable items: ~720K across 11 surfaces.

What’s pending and why

The user has flagged three polish priorities for the current 3-day window:

Covers / lazy thumbs: largely DONE for Gutenberg (99.6%). IA Books cover URL pattern still needs investigation (__ia_thumb.jpg vs services/img/). DOAB needs the expand= parameter. arXiv stays procedural.
Toon Town aesthetic: largely DONE. Palette overhauled to bright cartoon, MeshToonMaterial throughout, clouds drifting, trees/lamps/benches added this session, cartoon shop-front facade replacing the web-2-formal look. Optional next: cel-shading outline pass on the bulk building InstancedMesh (technically harder due to Three.js InstancedMesh limitations; detail meshes already get EdgesGeometry).
Gutenberg completeness: 99.6% (17,959 / 22,876). The remaining 4,917 are likely below the score threshold; could be addressed by lowering --max-cards floor in library_layout.py if user wants 100%.

Beyond the three priorities, deferred work includes:

Wikisource ingestion (~5M public-domain primary sources, architecturally identical to /wiki, 4-6h estimated)
Standard Ebooks (~600 hand-curated books, blocked on OPDS auth issue)
/novel walkable subset of community-marked items
arXiv classifier improvement (currently collapses everything to “science”; should map cs→tech, math→tech, physics→physics, etc.)
Per-corpus info modals (each library page still reads as if you’re in /world)
WARC resume when polish is done (corpus toward 10M+)
Domain registration — wanderaround.io at Porkbun (~$32/yr) + Cloudflare DNS + certbot. User action required.
GitHub publish of the substrate template repo

Phantom players + novel marks

The presence layer makes Wander Around inhabited rather than just navigable:

WebSocket at /ws/presence — clients send {type:'pos', x, y, z, facing, mode}; server broadcasts to peers in the same mode within 12000-unit radius, capped at 80 nearest. Sessions are anonymous pseudonyms (amber-rabbit-7421-style). Stale sessions auto-drop after 6s.
Novel-marks — markings table in wander.db with (mode, item_id, session_id, kind) PK. kind='novel' for now; supports future kinds without schema migration. Anonymous session IDs in wsid cookie. ★ button in every item modal.

These are operational but lightly used; visible polish (walking animation, distance fade, name-tag toggle) is deferred until traffic warrants it.

The substrate-template repo

C:\Users\BenHo\Desktop\ClaudeCode\wander-substrate\ is staged for public release. v0.1.0 tag prepared. Pending user actions:

Create the GitHub repo
git config user.email "..." (replace placeholder)
git commit --amend --reset-author --no-edit
git remote add origin <url>; git push -u origin main; git push origin v0.1.0
Cut a Release from v0.1.0 with CHANGELOG content

Known issues / debt

The category_edges precompute is heavy SQL (9.5M edges × 4M cards). Timer disabled to save load. Run manually with systemctl start wander-category-edges.service (~2 min) when load is low; chord graph type in /game then works.
Inter-cell smearing creates ~1.87% pairwise card-overlap in the layout audit. User has explicitly OK’d this — smearing > overlap stacking.
/api/stats rendered_last_5min returns 0 because the thumb scripts update thumb_path but not rendered_at. Add rendered_at = strftime('%s', 'now') to both scripts for accurate stats.
cards.id namespaces: each corpus has its own ID space. Don’t try to merge them; the JSON mode field disambiguates.
CSP script-src 'self' blocks inline scripts and importmaps. Use full /static/ URLs for all imports.
Route ordering matters in FastAPI: /library/{corpus} must be registered AFTER /library/main.js or the latter gets captured as corpus="main.js".

Box C state at this writing

uptime              4 days, 1:23, load avg 17.46 / 18.12 / 18.73
disk free           116 GB of 301 GB (60% used)
wander-api          active
wander-thumbs       active (86% success, 0.80/s)
wander-wkhtml       active (82% success, 0.73/s)
wander-wiki-thumbs  active (slow, 0.02/s)
wander-warc         stopped (paused 2026-05-01 20:40 UTC)
wander-refresh      active timer

The load average is high but stable — running for 4 days continuously with all pipelines under demand. RAM has not OOMed. The system is comfortable.

Why this matters

The open web has not been mapped meaningfully since the early 2000s. The page-rank graph is mapped, but the content of pages — what they look like, what era they’re from, who made them, how hand-crafted they are — has been invisible to the tools that decide what you see. Wander Around makes that visible at city scale. Walking through 197K hand-made sites in /world feels like walking through a city of vanished neighborhoods. Walking through 17K Project Gutenberg books in /library/gutenberg feels like a public-domain library you can browse with your feet. The same substrate generalizes to any corpus with (id, category, score).

If wanderaround.io launches with all four library corpora at 80%+ thumb coverage and the Toon Town aesthetic settled, the surface area for going viral is substantial. The thumbnail-on-every-building moment (“wait, this is a real site I’ve never heard of”) is the conversion vector. We’re 36 hours from full thumb coverage at current pipeline rate, three days from Toon Town being polished, and one user-action from the domain being live.

Author: psiloceyeben (Prometheus7 Institute) with Claude Code. State of the project as of 2026-05-04 08:15 UTC. All deployment lives at 89.167.7.54. The substrate template lives at wander-substrate/ ready for public release.