Skip to content

feat: era data pipeline and generated cata/mop data (2/3) - #815

Merged
Xian55 merged 10 commits into
devfrom
feature/era-data-pipeline
Jul 26, 2026
Merged

Xian55 merged 10 commits into
devfrom
feature/era-data-pipeline

Conversation

@Xian55

@Xian55 Xian55 commented Jul 26, 2026

Copy link
Copy Markdown
Owner

Summary

The data half of multi-era support: the tools that generate per-client data, and the generated data itself for cata, legacy_cata, mop and legacy_mop.

PR 2 of 3 in a stack — base is feature/multi-era-navmesh-leaflet, not dev. Review PR 1 first; this branch contains its commits.

PR Branch Base
1 feature/multi-era-navmesh-leaflet dev
2 (this) feature/era-data-pipeline feature/multi-era-navmesh-leaflet
3 feature/cata-mop-addon-support feature/era-data-pipeline

~500 files, but the overwhelming majority is generated JSON. The reviewable surface is scripts/, Utilities/ and the four data commits' rationale.

Related issues

  • MOP classic support #702 — MoP classic support. Supplies the Pandaria data: dbc, area/POI, spawn and mailbox data, area grids and the navmesh/leaflet assets for mop and legacy_mop.
  • 3.3.5 / 4.3.4 - 3.4.3 / 4.4.2 backport #704 — 3.3.5 / 4.3.4 backport. Generates the legacy_cata (4.3.4) data set. Does not address the 3.3.5 frame-detection problem in that issue.

Tools

BakeTool (new)

Cross-platform navmesh and area-grid bake CLI — the only bake entry point that is not net10.0-windows, so tiles can be baked on macOS and Linux. --root points it at a directory whose MPQ is some other client's Data, which is what makes it usable against a client other than the one Json/MPQ holds.

Not added to MasterOfPuppets.sln, in keeping with the rest of Utilities/. Note the consequence: dotnet run --no-build silently runs a stale binary. That already cost one wasted run where a new flag appeared to do nothing.

--area-grid is per client rather than per era, because the subzone output path is keyed by Exp.

extract-*-sqldump.py (new)

Parse a mysqldump directly — no MySQL server needed. Emulator schemas drift, so each column is resolved by trying candidate names rather than assuming a layout: SkyFire 5.4.8 splits faction into faction_A/faction_H and renames rank to npc_rank, and a query written for 3.3.5 TrinityCore does not error on those, it selects the wrong column.

extract-area-sqldump.py replaces the Wowhead scrape, and the reason is worth reading carefully:

Wowhead's node coordinates are percentages of whichever map its zone page uses, and that map is not always the zone whose id names the file. Cataclysm split Stranglethorn Vale (5339) into Northern Stranglethorn (33) and The Cape of Stranglethorn (5287). Wowhead's zone=5339 data is plotted against the Cape, but the frontend rescales it with 5339's bounds — which still span both halves — so every node landed roughly a thousand yards out to sea. There is no correction factor available, because a single file mixes both maps.

The replacement assigns each spawn a zone from the baked area grid, then converts world → map using that same zone's bounds, so the value written and the value the frontend reads back are the same rectangle by construction.

Validated against pre-Cata data that has not changed in years — Innkeeper Farley in Elwynn:

source coords
som / tbc / wrath (existing Wowhead data) [43.8, 65.8]
regenerated [43.77, 65.8]

extract-tele-sqldump.py merges a world DB's game_tele — the table behind the .tele GM command, so every row is a named, reachable landmark already in world coordinates.

CDN distribution

Both uploaders GET and merge index.json before writing it. That index describes the whole bucket, not one run: publishing --era mop with a wholesale rewrite would strip the precata and cata entries and break every existing user's download. --skip-unchanged avoids re-zipping gigabytes when adding a continent to an already published era; --index-only rebuilds a lost index from what is already in the bucket instead of re-pushing every bundle.

extract-minimap.py now indexes listfiles before chaining patches — attaching a PTCH patch also patches (listfile) and hides base entries, which silently dropped whole continents.

WowheadDB_Extractor fixes

Every request now sends full browser headers; a bare User-Agent gets a CloudFront 403. PerZoneGatherable had its own bare HttpClient issuing four of the five requests per zone, so most of a run failed regardless. Adds jittered delays, resume-by-skipping, and abort on 403.

Generated data

Json/dbc — creature data was wrong for three clients

cata, legacy_cata and mop all shared one file that was a Wrath dump: ids 1..43282, zero MoP-era entries. Every Cataclysm and Pandaria NPC was nameless.

The failure mode is indirect and worth knowing: area/<id>.json loads fine (HTTP 200) and the zone still renders nothing, because a skinnable id resolves to no creature entry. Diagnosing it looks like broken area data when the area file is perfect.

All four are now the same 56,661-entry SkyFire build, and a client and its legacy_ twin are byte-identical.

Json/area — rebuilt, 24 stale files deleted

Every coordinate is now a percentage of the bounds of the zone that names the file. The 24 deletions are zones the dump does not populate; leaving them would keep serving mis-scaled Wowhead coordinates, and AreaDB treats an absent file as normal.

Content is not lost where it exists:

  • The area grid is a top-down 2D projection with no cells for interiors, so Ironforge and Undercity file under Dun Morogh and Tirisfal.
  • Vashj'ir's subzones roll up into Vashj'ir.

teleport_locations.txt — 1003 → 1534

The committed list came from a WotLK DB exported with LIMIT 1000, which is why Northrend had two entries. Adds 531 rows, including the first POIs for Pandaria, Deepholm, the Lost Isles and the Maelstrom. Purely additive — existing rows kept verbatim, since they are hand-augmented and pre-Shattering. Both tracked copies stay byte-identical.

Known limits

  • City interiors file under their enclosing outdoor zone (2D grid limitation, not fixable here). Preferring the smallest containing WorldMapArea rect was tried and reverted — Northern Stranglethorn's and the Cape's rects overlap, so it recreates the exact bug this replaces.
  • Un'Goro, Silithus and Mount Hyjal are genuinely near-empty in SkyFire 5.4.8 (2–5 spawns each). Confirmed by resolving through the grid rather than a bounding box; zone rects overlap, so a bbox query is not a zone query.
  • Two known navmesh tile holes: Kalimdor (68,159) and Deepholm (126,125).

Verification

dotnet build MasterOfPuppets.sln — 0 warnings, 0 errors. Coordinate round-trip checked across all four clients: 17,691 coords (cata) and 12,341 (mop), 4 outside 0..100 in each, all consumer paths exercised live in the browser against a running PathingAPI.

Xian55 and others added 10 commits July 27, 2026 00:40
…e CLI

The only bake entry point that is not net10.0-windows, so tiles can be baked on
macOS and Linux. --root points it at a directory whose MPQ is some other
client's Data, which is what makes it usable against a client other than the one
Json/MPQ holds.

--area-grid is per client rather than per era, because the subzone output path
is keyed by Exp; it has to be run for each client whose worldmap extractor will
read that directory.

Not in MasterOfPuppets.sln, in keeping with the rest of Utilities.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fixes the per-extractor copy lists so talent, faction and worldmap output
actually lands in the client folder.

Legacy_Cata stays pinned to 8.1.0.27826 deliberately. Moving it to Cata Classic
4.4.2.60895 was tried and reverted: measured against that build the old area ids
map to only 133 of 444 UiMapIds, against 439 of 444 on the pinned build, and the
addon's WorldMapAreaIDToUiMapID table is written against the latter.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every request now sends full browser headers: a bare User-Agent gets a
CloudFront 403, and PerZoneGatherable had its own bare HttpClient issuing four
of the five requests per zone, so most of a run failed. It shares the configured
client now.

Adds a jittered per-request delay, resume by skipping zones already written, and
an abort on 403 rather than grinding through the remaining zones once blocked.
Expansion is a runtime argument instead of a compile-time constant.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… dump

Parses a mysqldump directly, so no MySQL server is needed. Emulator schemas
drift, so each column is resolved by trying candidate names rather than assuming
a layout: SkyFire 5.4.8 splits faction into faction_A/faction_H and renames rank
to npc_rank, and a query written for 3.3.5 TrinityCore does not error on those,
it selects the wrong column.

extract-area-sqldump replaces the Wowhead scrape. Wowhead's node coordinates are
percentages of whichever map its zone page uses, and that map is not always the
zone whose id names the file: Cataclysm split Stranglethorn Vale (5339) into
Northern Stranglethorn (33) and the Cape (5287), and the zone=5339 data is
plotted against the Cape while the frontend rescales it with 5339's bounds,
putting every node a thousand yards out to sea. There is no correction factor,
because one file mixes two maps.

This assigns each spawn a zone from the baked area grid and converts with that
same zone's bounds, so the value written and the value the frontend reads back
are the same rectangle by construction. Verified against pre-Cata data that has
not changed in years: Innkeeper Farley reads [43.8, 65.8] in som/tbc/wrath and
[43.77, 65.8] here.

extract-tele-sqldump merges a world DB's game_tele - the table behind the .tele
GM command, so every row is a named reachable landmark - into
teleport_locations.txt, keeping existing rows on a name collision because those
coordinates are pre-Shattering and that is what pre-Cata users need.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Uploaders GET and merge index.json before writing it, because that index
describes the whole bucket rather than one run - publishing --era mop with a
wholesale rewrite would strip the precata and cata entries and break every
existing user's download.

--skip-unchanged avoids re-zipping gigabytes when adding a continent to an
already published era, and --index-only rebuilds a lost index from what is
already in the bucket instead of re-pushing every bundle.

extract-minimap indexes listfiles before chaining patches: attaching a PTCH
patch also patches (listfile) and hides base entries, which silently dropped
whole continents.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… clients

Json/dbc/cata, legacy_cata and mop all shared one file that was a Wrath dump -
ids 1..43282, zero MoP-era entries - so every Cataclysm and Pandaria NPC was
nameless. All four are now the same 56661-entry SkyFire build, and a client and
its legacy_ twin are byte-identical.

The symptom this fixes is indirect: area/<id>.json loaded fine and the zone still
rendered nothing, because a skinnable id resolved to no creature entry.

WorldMapArea gains subzone rows so zone names resolve without AreaTable.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…owhead files

Every coordinate is now a percentage of the bounds of the zone that names the
file, which is what the frontend rescales with, so the Stranglethorn-style
cross-map displacement cannot occur.

Deletes 24 files for zones the dump does not populate rather than leaving them
carrying mis-scaled Wowhead coordinates. AreaDB treats an absent file as normal,
so those zones now return no data instead of wrong data. Their content is not
lost where it exists: the area grid is a top-down 2D projection with no cells for
interiors, so Ironforge and Undercity file under Dun Morogh and Tirisfal, and
Vashj'ir's subzones roll up into Vashj'ir.

Adds the baked area grids the extractor resolves spawns against.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The committed list came from a WotLK DB exported with LIMIT 1000, which is why
Northrend had two entries. Adds 531 rows, including the first POIs for Pandaria,
Deepholm, the Lost Isles and the Maelstrom.

Purely additive: existing rows are kept verbatim because the file is
hand-augmented and its coordinates are pre-Shattering. Both tracked copies stay
byte-identical.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Xian55 Xian55 added documentation Improvements or additions to documentation refactor This ticket concerns the possible simplification of code/data. enhancement This pull request implements a new feature. ai Artificial Intelligence tool has been contributed. feature request This is a request for a new feature, or an expansion of an existing feature. labels Jul 26, 2026
@Xian55
Xian55 changed the base branch from feature/multi-era-navmesh-leaflet to dev July 26, 2026 22:54
@Xian55
Xian55 merged commit db319da into dev Jul 26, 2026
1 check passed
@Xian55
Xian55 deleted the feature/era-data-pipeline branch July 26, 2026 23:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai Artificial Intelligence tool has been contributed. documentation Improvements or additions to documentation enhancement This pull request implements a new feature. feature request This is a request for a new feature, or an expansion of an existing feature. refactor This ticket concerns the possible simplification of code/data.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant