Repository navigation
feat: era data pipeline and generated cata/mop data (2/3) - #815
Merged
Merged
Conversation
…e CLI The only bake entry point that is not net10.0-windows, so tiles can be baked on macOS and Linux. --root points it at a directory whose MPQ is some other client's Data, which is what makes it usable against a client other than the one Json/MPQ holds. --area-grid is per client rather than per era, because the subzone output path is keyed by Exp; it has to be run for each client whose worldmap extractor will read that directory. Not in MasterOfPuppets.sln, in keeping with the rest of Utilities. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fixes the per-extractor copy lists so talent, faction and worldmap output actually lands in the client folder. Legacy_Cata stays pinned to 8.1.0.27826 deliberately. Moving it to Cata Classic 4.4.2.60895 was tried and reverted: measured against that build the old area ids map to only 133 of 444 UiMapIds, against 439 of 444 on the pinned build, and the addon's WorldMapAreaIDToUiMapID table is written against the latter. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every request now sends full browser headers: a bare User-Agent gets a CloudFront 403, and PerZoneGatherable had its own bare HttpClient issuing four of the five requests per zone, so most of a run failed. It shares the configured client now. Adds a jittered per-request delay, resume by skipping zones already written, and an abort on 403 rather than grinding through the remaining zones once blocked. Expansion is a runtime argument instead of a compile-time constant. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… dump Parses a mysqldump directly, so no MySQL server is needed. Emulator schemas drift, so each column is resolved by trying candidate names rather than assuming a layout: SkyFire 5.4.8 splits faction into faction_A/faction_H and renames rank to npc_rank, and a query written for 3.3.5 TrinityCore does not error on those, it selects the wrong column. extract-area-sqldump replaces the Wowhead scrape. Wowhead's node coordinates are percentages of whichever map its zone page uses, and that map is not always the zone whose id names the file: Cataclysm split Stranglethorn Vale (5339) into Northern Stranglethorn (33) and the Cape (5287), and the zone=5339 data is plotted against the Cape while the frontend rescales it with 5339's bounds, putting every node a thousand yards out to sea. There is no correction factor, because one file mixes two maps. This assigns each spawn a zone from the baked area grid and converts with that same zone's bounds, so the value written and the value the frontend reads back are the same rectangle by construction. Verified against pre-Cata data that has not changed in years: Innkeeper Farley reads [43.8, 65.8] in som/tbc/wrath and [43.77, 65.8] here. extract-tele-sqldump merges a world DB's game_tele - the table behind the .tele GM command, so every row is a named reachable landmark - into teleport_locations.txt, keeping existing rows on a name collision because those coordinates are pre-Shattering and that is what pre-Cata users need. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Uploaders GET and merge index.json before writing it, because that index describes the whole bucket rather than one run - publishing --era mop with a wholesale rewrite would strip the precata and cata entries and break every existing user's download. --skip-unchanged avoids re-zipping gigabytes when adding a continent to an already published era, and --index-only rebuilds a lost index from what is already in the bucket instead of re-pushing every bundle. extract-minimap indexes listfiles before chaining patches: attaching a PTCH patch also patches (listfile) and hides base entries, which silently dropped whole continents. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… clients Json/dbc/cata, legacy_cata and mop all shared one file that was a Wrath dump - ids 1..43282, zero MoP-era entries - so every Cataclysm and Pandaria NPC was nameless. All four are now the same 56661-entry SkyFire build, and a client and its legacy_ twin are byte-identical. The symptom this fixes is indirect: area/<id>.json loaded fine and the zone still rendered nothing, because a skinnable id resolved to no creature entry. WorldMapArea gains subzone rows so zone names resolve without AreaTable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…owhead files Every coordinate is now a percentage of the bounds of the zone that names the file, which is what the frontend rescales with, so the Stranglethorn-style cross-map displacement cannot occur. Deletes 24 files for zones the dump does not populate rather than leaving them carrying mis-scaled Wowhead coordinates. AreaDB treats an absent file as normal, so those zones now return no data instead of wrong data. Their content is not lost where it exists: the area grid is a top-down 2D projection with no cells for interiors, so Ironforge and Undercity file under Dun Morogh and Tirisfal, and Vashj'ir's subzones roll up into Vashj'ir. Adds the baked area grids the extractor resolves spawns against. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The committed list came from a WotLK DB exported with LIMIT 1000, which is why Northrend had two entries. Adds 531 rows, including the first POIs for Pandaria, Deepholm, the Lost Isles and the Maelstrom. Purely additive: existing rows are kept verbatim because the file is hand-augmented and its coordinates are pre-Shattering. Both tracked copies stay byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Jul 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The data half of multi-era support: the tools that generate per-client data, and the generated data itself for
cata,legacy_cata,mopandlegacy_mop.PR 2 of 3 in a stack — base is
feature/multi-era-navmesh-leaflet, notdev. Review PR 1 first; this branch contains its commits.feature/multi-era-navmesh-leafletdevfeature/era-data-pipelinefeature/multi-era-navmesh-leafletfeature/cata-mop-addon-supportfeature/era-data-pipeline~500 files, but the overwhelming majority is generated JSON. The reviewable surface is
scripts/,Utilities/and the four data commits' rationale.Related issues
mopandlegacy_mop.legacy_cata(4.3.4) data set. Does not address the 3.3.5 frame-detection problem in that issue.Tools
BakeTool(new)Cross-platform navmesh and area-grid bake CLI — the only bake entry point that is not
net10.0-windows, so tiles can be baked on macOS and Linux.--rootpoints it at a directory whoseMPQis some other client'sData, which is what makes it usable against a client other than the oneJson/MPQholds.Not added to
MasterOfPuppets.sln, in keeping with the rest ofUtilities/. Note the consequence:dotnet run --no-buildsilently runs a stale binary. That already cost one wasted run where a new flag appeared to do nothing.--area-gridis per client rather than per era, because the subzone output path is keyed byExp.extract-*-sqldump.py(new)Parse a mysqldump directly — no MySQL server needed. Emulator schemas drift, so each column is resolved by trying candidate names rather than assuming a layout: SkyFire 5.4.8 splits
factionintofaction_A/faction_Hand renamesranktonpc_rank, and a query written for 3.3.5 TrinityCore does not error on those, it selects the wrong column.extract-area-sqldump.pyreplaces the Wowhead scrape, and the reason is worth reading carefully:The replacement assigns each spawn a zone from the baked area grid, then converts world → map using that same zone's bounds, so the value written and the value the frontend reads back are the same rectangle by construction.
Validated against pre-Cata data that has not changed in years — Innkeeper Farley in Elwynn:
[43.8, 65.8][43.77, 65.8]extract-tele-sqldump.pymerges a world DB'sgame_tele— the table behind the.teleGM command, so every row is a named, reachable landmark already in world coordinates.CDN distribution
Both uploaders GET and merge
index.jsonbefore writing it. That index describes the whole bucket, not one run: publishing--era mopwith a wholesale rewrite would strip the precata and cata entries and break every existing user's download.--skip-unchangedavoids re-zipping gigabytes when adding a continent to an already published era;--index-onlyrebuilds a lost index from what is already in the bucket instead of re-pushing every bundle.extract-minimap.pynow indexes listfiles before chaining patches — attaching a PTCH patch also patches(listfile)and hides base entries, which silently dropped whole continents.WowheadDB_ExtractorfixesEvery request now sends full browser headers; a bare User-Agent gets a CloudFront 403.
PerZoneGatherablehad its own bareHttpClientissuing four of the five requests per zone, so most of a run failed regardless. Adds jittered delays, resume-by-skipping, and abort on 403.Generated data
Json/dbc— creature data was wrong for three clientscata,legacy_cataandmopall shared one file that was a Wrath dump: ids 1..43282, zero MoP-era entries. Every Cataclysm and Pandaria NPC was nameless.The failure mode is indirect and worth knowing:
area/<id>.jsonloads fine (HTTP 200) and the zone still renders nothing, because a skinnable id resolves to no creature entry. Diagnosing it looks like broken area data when the area file is perfect.All four are now the same 56,661-entry SkyFire build, and a client and its
legacy_twin are byte-identical.Json/area— rebuilt, 24 stale files deletedEvery coordinate is now a percentage of the bounds of the zone that names the file. The 24 deletions are zones the dump does not populate; leaving them would keep serving mis-scaled Wowhead coordinates, and
AreaDBtreats an absent file as normal.Content is not lost where it exists:
teleport_locations.txt— 1003 → 1534The committed list came from a WotLK DB exported with
LIMIT 1000, which is why Northrend had two entries. Adds 531 rows, including the first POIs for Pandaria, Deepholm, the Lost Isles and the Maelstrom. Purely additive — existing rows kept verbatim, since they are hand-augmented and pre-Shattering. Both tracked copies stay byte-identical.Known limits
WorldMapArearect was tried and reverted — Northern Stranglethorn's and the Cape's rects overlap, so it recreates the exact bug this replaces.(68,159)and Deepholm(126,125).Verification
dotnet build MasterOfPuppets.sln— 0 warnings, 0 errors. Coordinate round-trip checked across all four clients: 17,691 coords (cata) and 12,341 (mop), 4 outside 0..100 in each, all consumer paths exercised live in the browser against a running PathingAPI.