From 6622afe4bd05c837af6d9125a04febddab628810 Mon Sep 17 00:00:00 2001 From: wtclaude Date: Tue, 28 Jul 2026 09:28:04 -0500 Subject: [PATCH 01/19] docs(link): add the Protocol 3.0 design Surveys the live ServUO tree against everything the bridge already surfaces and records the full gap list (17 items), then specs the four features scoped for 3.0. 3.0 has three scope areas: - A: the visibility framework. Admin-configurable, per-feature and per-field audience control over all ten shard-derived surfaces (the four new ones plus the six that already ship), on an anonymous -> logged_in -> player -> staff -> admin ladder. Every default reproduces today's behavior, so the retrofit is a no-op until an admin changes something. Two rules an admin cannot override: acct and webId are admin-only always, and an unmapped event kind is never broadcast below admin. This also fixes a verified leak - guild leader acct/webId are readable today on the anonymous /public/shard/guilds. - B: three new wire streams - world.ruleset, points.board, and vendor.listing/vendor.listing.remove. - C: the spawn atlas, built from static ServUO data files with no wire involvement. Visibility lives entirely on the website; the sidecar stays a dumb forwarder that defines no access parameters and advertises no capabilities. PROTOCOL_VERSION goes 2 -> 3 once, at the end: every part PRs into an edge branch per repo, and the coordinated edge -> main merge is the cutover. A schema migration moves uo_link_config.protocol so operators don't have to. Co-Authored-By: Claude --- README.md | 1 + link/v3.md | 577 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 578 insertions(+) create mode 100644 link/v3.md diff --git a/README.md b/README.md index 9ceffd5..2ec481c 100644 --- a/README.md +++ b/README.md @@ -27,6 +27,7 @@ ci/ cross-cutting CI/quality notes |---|---| | [INTEGRATION.md](link/INTEGRATION.md) | How the website integrates with the uo-link sidecar | | [PROTOCOL_2.md](link/PROTOCOL_2.md) | Protocol 2.0 / 2.1 design | +| [v3.md](link/v3.md) | Protocol 3.0 design — shard content/standings streams + the visibility framework | | [ADMIN_CONTROLS.md](link/ADMIN_CONTROLS.md) | Staff write-plane (kick/ban/broadcast, page queue) | | [SHARD_PREREQS.md](link/SHARD_PREREQS.md) | Shard-side prerequisites for the bridge | | [PLAN.md](link/PLAN.md) | uo-link build plan | diff --git a/link/v3.md b/link/v3.md new file mode 100644 index 0000000..bd6e92c --- /dev/null +++ b/link/v3.md @@ -0,0 +1,577 @@ +# Protocol 3.0 — Shard content, standings & the visibility framework + +**Status:** Planned, approved 2026-07-28. Not yet built. All work lands on an `edge` branch in each repo; `edge` → `main` is the v3 cutover. +**Date:** 2026-07-28 +**Codebase:** ServUO 57.4, ``, net48 / x64, Expansion **EJ**. +**Companion to** [`PLAN.md`](PLAN.md) (1.0 read/event plane), [`PROTOCOL_2.md`](PROTOCOL_2.md) (2.0 provisioning + world-state streams), [`ADMIN_CONTROLS.md`](ADMIN_CONTROLS.md) (staff write plane), [`INTEGRATION.md`](INTEGRATION.md) (website API). + +--- + +## 1. Why 3.0 + +A survey of the live ServUO tree against everything the bridge already surfaces end-to-end found that +**the bridge covers live *activity* well and covers shard *content and standings* almost not at all.** + +Covered by 1.0 + 2.0: presence/online, region transitions, char vitals + profile + roster, house +registry + IDOC decay, champion spawns, guild board, city governors + term history, help-page queue, +total gold supply, player-vendor sales log, deaths/murders/kills, skill gains, fame/karma, quest +completes, staff/cheat audit, account linking + creation, town crier + news. + +Not covered by anything: every leaderboard, every ruleset fact, every "where do I find X", and the +entire player economy outside a player's own vendors. + +3.0 has **three scope areas**: + +- **A — The visibility framework (§3).** Admin-configurable, per-feature and per-field audience + control over every shard-derived surface on the website. Ships first; the rest depends on it. +- **B — Three new wire streams (§5, §7, §8).** `world.ruleset`, `points.board`, + `vendor.listing`/`vendor.listing.remove`. +- **C — One website-only feature (§6).** The spawn atlas, built from static ServUO data files with no + wire involvement at all. + +--- + +## 2. Survey: the full gap list + +Recorded so the items *not* scoped for 3.0 aren't re-derived later. + +| # | Gap | Source on the shard | Value | Cost | Status | +|---|---|---|---|---|---| +| 1 | **Points/loyalty leaderboards** — 25 point currencies | `Scripts/Services/PointsSystems/PointsSystem.cs` → `static List Systems`, each `List{Player,Points}` | Very high | Low | **3.0 §7** | +| 2 | **Shard ruleset page** | `Config/*.cfg` via `Server.Config.Get` | High | Very low | **3.0 §5** | +| 3 | **Shard-wide marketplace** | `PlayerVendor.PlayerVendors` + `VendorSearch.cs` | Very high | High | **3.0 §8** | +| 4 | **Spawn atlas / bestiary** | `Spawns/*.xml` (6,455 spawners), `RevampedSpawns/*.xml` (333), `Data/Regions.xml`, `Data/Locations/*.xml`, `Config/ChampionSpawns.xml`, `Data/teleporters.csv`, `Data/HarvestLocs/*` | High | Medium | **3.0 §6** | +| 5 | VvV standings + battle status | `Services/ViceVsVirtue/{ViceVsVirtueSystem,GuildStats,VvVBattle}.cs` | High | Medium | `PROTOCOL_2.md` §10.4 deferred this pending "which PvP system does this shard run?" — **now answered: `VvV.cfg Enabled=True`, `Factions.cfg` off.** Unblocked, not scoped here | +| 6 | Skill leaderboards + shard census | `Services/Reports/Reports.cs` → `GetSkillDistribution()`, `CompileGeneralStats()`, `StaffHistory` | High | Low | Spec'd `PROTOCOL_2.md` §14 (Part B phase 6), unbuilt. Shares §7's UI — fold in after | +| 7 | Custom mounts/pets codex — ~35 across 4 tiers | `Scripts/Custom/{Companions,Legendary,Named,New Legacy}` | Medium-high | Very low | Pure wiki/CMS content, zero bridge work. The shard's most distinctive content, with zero site presence | +| 8 | Community Collections progress | `Services/CommunityCollections/CollectionsSystem.cs` | Medium | Low | Natural public "community goal" widget | +| 9 | Seasonal/holiday event calendar | `Services/Seasonal Events/SeasonalEventSystem.cs`, Krampus, Forsaken Foes | Medium | Low | "What's live now / what's next" | +| 10 | Crafting / taming / harvesting feeds | `EventSink.CraftSuccess` / `TameCreature` / `ResourceHarvestSuccess` | Medium | Low | Spec'd `PROTOCOL_2.md` §11 #3/#4/#5, unbuilt | +| 11 | Virtue progression | `EventSink.VirtueLevelChange`, `Services/Ethics/` | Medium | Low | Spec'd §11 #6, unbuilt | +| 12 | Bulk Order Deeds + reward tables | `Services/BulkOrders/`, `Data/Bulk Orders/*` | Medium | Low | Feed spec'd §11 #7; the static reward tables are a free wiki page | +| 13 | Guild wars | war state on `Guild` | Low-medium | Low | Spec'd §11 #8, unbuilt | +| 14 | Astronomy discovery log | `Services/Astronomy/AstronomySystem.cs` (104 KB save) | Low | Low | Niche completion leaderboard | +| 15 | In-game chat relay | `Services/Chat/`, `Logs/Chat/{General,Help,Trade,LFG}` | Low | Medium | Privacy-sensitive; staff-only at most | +| 16 | Shard health telemetry | Crash logs, `LayerConflict.log`, `throttle.log`, `world.save.after` counts, AutoSave/AutoRestart schedule | Low-medium | Low | `world.save.after` is already ingested but never charted — world-size-over-time is nearly free | +| 17 | Ultima Store / Sovereigns balance | `Store.cfg Enabled=True, CurrencyName=Sovereigns`; `UltimaStore.GetCurrency` | ? | Medium | Only worth it if sovereigns are actually sold | + +**Excluded permanently** — see [`ADMIN_CONTROLS.md`](ADMIN_CONTROLS.md) Tier H/N: firewall/IP-block, +kill/resurrect, jail, item/gold grants, set-access-level, arbitrary `[set`/`[add`. + +**Noticed during the survey, out of scope:** `Scripts/Custom/PerryOwnerFix.cs` hardcodes an +`EventSink.Login` hook granting `AccessLevel.Owner` to account `"ShardOnwerPerry"`. Worth reviewing +independently of this work. + +--- + +## 3. Part A — The visibility framework + +### 3.1 The leak this replaces (verified 2026-07-28) + +`BridgeJson.Actor()` (`BridgeJson.cs:85-117`) writes `serial`, `name`, **`acct`**, **`webId`**, +`player`. `shardState.model.js:346 shapeGuild()` returns `r.payload` verbatim, and +`GET /api/v1/public/shard/guilds` (anonymous, `shard.controller.js:131`) serves it. **A guild +leader's game account name and website user id are readable on an anonymous public endpoint today.** +The same path exists for `shapeGovernor` → `/public/shard/governors`. `Actor` also feeds +`guild.join`, `city.update` and `region.enter`, all three in `PUBLIC_KINDS` on the anonymous SSE +stream. + +The framework below is the vehicle for the fix, and the reason it ships before anything else. + +### 3.2 Where visibility lives + +**On the website, never in the sidecar.** The sidecar's job for 3.0 is unchanged in character: accept +frames, persist them to its SQLite store, forward them verbatim over WS, and serve store-backed reads +that survive a shard outage. It defines no access parameters, no audiences, no field projection, and +advertises no capabilities. + +### 3.3 The audience ladder + +`anonymous → logged_in → player → staff → admin`, each rung implying the ones below it. + +`viewerLevel(req)` resolves: no session ⇒ `anonymous`; authenticated ⇒ `logged_in`; authenticated +with a linked shard account ⇒ `player`; moderator/admin role ⇒ `staff`/`admin`. **Staff always +satisfy the `player` rung** even without a linked game account, consistent with the existing rule +that `/player/*` is role-agnostic self-service. + +### 3.4 Two limits an admin cannot override + +1. **`acct` and `webId` are admin-only, always.** They are not in-game-visible and are not exposed as + configurable fields. +2. **A kind absent from the kind→feature map is never broadcast below `admin`.** Fail closed. This + preserves the property that today's static `PUBLIC_KINDS` allowlist is a security boundary rather + than a convenience filter. + +### 3.5 Configuration + +```sql +CREATE TABLE IF NOT EXISTS shard_feature_visibility ( + feature VARCHAR(48) NOT NULL PRIMARY KEY, + enabled TINYINT(1) NOT NULL DEFAULT 1, + audience VARCHAR(20) NOT NULL DEFAULT 'anonymous', + field_rules JSON NULL, -- {"": ""} for sensitive fields only + updated_by INT NULL, + updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4; +``` + +Seeded on boot in `server.js`, one row per feature. **All ten shard features are covered — the four +new ones and the six that already ship — and every default reproduces today's behavior, so the +retrofit is a no-op until an admin changes something.** + +| Feature | Default audience | Sensitive fields (default rung) | +|---|---|---| +| `status`, `activity`, `champs`, `guilds`, `governors` | `anonymous` | guilds/governors: `leaderAcct` / `leaderWebId` → **admin (locked)** | +| `houses` | `anonymous` | `owner` → `staff`, `price` → `staff` (matches today's IDOC-only public view) | +| `presence` | `anonymous` | `location` → `staff` (matches today's staff-only, location-gated `/online`) | +| `ruleset` (new) | `anonymous` | `connect` → `anonymous` | +| `atlas` (new) | `anonymous` | — | +| `leaderboards` (new) | `anonymous` | `characterName` → `anonymous` | +| `market` (new) | `anonymous` | `ownerName` → `anonymous`, `location` → `anonymous` | + +### 3.6 Enforcement — three points, one config + +New `website/server/src/utils/shardVisibility.js`: + +- `LADDER = ['anonymous','logged_in','player','staff','admin']`, `rank()`, `meets(viewer, required)` +- `viewerLevel(req)` (§3.3) +- `KIND_FEATURE` — every event kind → its feature; unmapped ⇒ admin-only (§3.4) +- `getConfig()` — DB-backed, cached ~5 s like `uoLinkClient`'s config cache, busted on admin `PUT` +- `requireFeature(name)` — **404 when disabled** (don't leak existence), **403 when enabled but the + viewer is below the audience** +- `projectFeature(name, payload, viewerLevel)` — strips fields whose rung the viewer doesn't meet; + `acct`/`webId` always stripped below `admin` + +Applied at: + +1. **Routes** — `requireFeature(…)` on every `/public/shard/*`, `/public/atlas/*` and the + shard-derived player routes; `projectFeature` in the controllers, replacing the ad-hoc + `shapeGuild`-returns-payload-verbatim path. +2. **SSE** — `shardBroadcast.js` moves from *"one public channel with a static `PUBLIC_KINDS` + allowlist plus one admin channel"* to **per-connection filtering**: each subscriber carries its + `viewerLevel`; each frame is mapped kind→feature, gated on `enabled && meets(...)`, then passed + through `projectFeature` before write. `PUBLIC_KINDS` becomes the seed data for `KIND_FEATURE` + rather than a hardcoded gate. **This is the largest single change in Part A and where the security + boundary now lives.** +3. **Nav** — `GET /api/v1/public/shard/features` returns only the features the calling viewer can + see, so the SPA hides nav entries rather than rendering links that 403. + +### 3.7 Admin surface + +`GET` / `PUT /api/v1/admin/shard/visibility` (admin-only). Validate feature names against the known +set and rungs against the ladder; reject any attempt to set `acct`/`webId` below `admin`. Writes an +`admin.audit`-style row so visibility changes are traceable. New client panel +`routes/admin/ShardVisibility.jsx` at `/admin/shard-visibility`, linked from `ShardAdmin.jsx`. + +--- + +## 4. The version bump and the rollout + +`PROTOCOL_VERSION` **2 → 3** in `link/sidecar/src/main.rs:27`. v3 is defined as *"adds +`world.ruleset`, `points.board`, `vendor.listing` / `vendor.listing.remove`"*. + +A bump is an operator-visible hard cutover — `web.rs::gate` returns 409 on every protected route on +mismatch, `uoLinkSocket.js::handleHello` closes the WS, and the website's declared version is the +admin-set `uo_link_config.protocol` column — so it happens **exactly once**, at the end: + +- Cut an **`edge`** branch from `main` in each of `website/`, `link/`, `servuo-plugins/`, `docs/`. +- Every phase PRs into `edge`, never `main`. Feature branches are cut from `edge`. +- Part A lands first, alone. +- When all parts are built and tested, one `edge` → `main` PR per repo, merged together. **That merge + is the v3 cutover.** +- A schema migration sets `uo_link_config.protocol` (the existing row **and** the column default) + from 2 to 3, so the cutover doesn't require a manual admin edit. `UOLINK_PROTOCOL` still overrides. +- No feature-negotiation array anywhere — v3 implies all three kinds. + +--- + +## 5. Part B/1 — `world.ruleset` + +`PROTOCOL_2.md` §10.4 sketches a `world.systems` capability frame that was never implemented +(`grep` returns nothing across all four repos). **`world.ruleset` subsumes it**, carrying a `systems` +sub-object with the `cityLoyalty` / `vvv` / `factions` booleans §10.4 asked for. §10.4 is marked +superseded; no orphan kind is left behind. + +### 5.1 Plugin + +NEW `servuo-plugins/overlay/Scripts/Custom/Bridge/BridgeRuleset.cs`, modelled on +`BridgeBoot.EmitHello` — **not** a sweep. Subscribes `BridgeLink.Connected_Core += Emit` so a sidecar +that comes up second still learns the ruleset. + +Built from an **explicit allowlist** of `Server.Config.Get` calls. **Never enumerate +`Config.Entries`** (`Server/Config.cs:162`) — it would sweep in secrets. An FNV-1a `rev` over the body +makes an unchanged reconnect a site-side no-op (`String.GetHashCode()` is not stable across runs and +must not be used). + +`BridgeConfig.cs` + `overlay/Config/Bridge.cfg`: `RulesetEnabled=true`, `PublicConnectAddress=""`, +`RulesetIncludeSchedule=true`. `BridgeBoot.cs`: `reload` → re-emit, `status` → rev/bytes. Not wired +to `sweepnow`; it isn't a sweep. + +### 5.2 Payload + +Every block optional, omitted when its system is off: + +`shard`, `expansion`, `connect` (only from `PublicConnectAddress`), +`systems{cityLoyalty,vvv,factions,siege,chat,store,dailyRares,honesty,shadowguard,treasureMaps,vetRewards,testCenter}`, +`caps{skill:1000,totalSkill:7000,stat:225,str/dex/int:125,strMax/dexMax/intMax:150}`, +`housing{accountHouseLimit:1}`, `accounts{perIp:3,charSlots:7,autoCreate}`, +`vetRewards{enabled,rewardIntervalDays:30}`, +`loot{feluccaLuckBonus:1000,feluccaBudgetBonus:100,feluccaMaxProps:11}`, +`vendors{restockDelayMinutes,maxSell,economyStockAmount}`, +`champions{powerScrolls:6,statScrolls:16,scrollChance,transcendenceChance,rankThresholds}`, +`treasureMaps`, `vvv{enabled,startSilver:2000,enhancedRules}`, `store{enabled,currencyName}`, +`schedule{autoSaveFrequencyMinutes,autoRestart*}`. + +**Excluded by name — in a code comment and here:** `Server.cfg` (Address/Listen/Port; only +`PublicConnectAddress` is published), `Staff.cfg`, `Email.cfg`, `DataPath.cfg`, `Bridge.cfg`, +`Compiler.cfg`, `Reports.cfg`, `Client.cfg`. + +### 5.3 Sidecar and website + +Sidecar — `store.rs`: singleton `ruleset(id CHECK(id=1), rev, json, updated_t)` + upsert/get; +`main.rs`: new arm in the board-projection match; `web.rs`: `GET /ruleset` served from the store, so +it answers during a shard outage (`PROTOCOL_2.md` §12.2). + +Website — `uoLinkClient.getRuleset()`; `uoLinkSocket.backfill()` (object-shaped, so follow the +`getPresence()` block's explicit form, not the array-only `snapshot()` helper); `shardIngest.js` → +`shardState.setRuleset`, **not** in `LOGGED_KINDS` (it re-arrives every reconnect and `server.hello` +already marks those); `KIND_FEATURE['world.ruleset'] = 'ruleset'`; `shard_ruleset` singleton table +(`rev`, `expansion`, `payload JSON`, `t`); `GET /public/shard/ruleset` behind +`requireFeature('ruleset')`, returning `null` ⇒ "not published yet". + +Client — NEW `routes/public/Rules.jsx` at `/site/rules`, alongside +`/site/champs|guilds|governors|houses`; live via `useShardFeed({ filter: new Set(['world.ruleset']) })`. + +### 5.4 Risk + +Perf is nil (~3 KB per connect). The only real risk is publishing a secret, mitigated by the explicit +allowlist, the no-`Config.Entries` rule, the named exclusion list, and a manual eyeball of the emitted +frame during verification. + +--- + +## 6. Part C — Spawn atlas / bestiary (website-only) + +**No plugin, no sidecar, no `Bridge.cfg` knob, no new kinds.** Not part of the v3 wire change. + +**Decision: committed generated artifact + idempotent DB import**, split in two because the build +needs the ServUO tree (which the website container does not have) and the import does not. Not +runtime import (10.5 MB of XML per boot), not a browser-served blob. + +New in `website/server/`: + +- `src/utils/spawnAtlasParse.js` — **pure functions, no fs**, so they are unit-testable in CI without + a ServUO tree: `parseObjects2()`, `parsePoints()`, `parseRegions()`, `parseLocations()`, + `resolveRegion()`. +- `scripts/buildSpawnAtlas.js` (`--servuo --out db/data/`) and `scripts/importSpawnAtlas.js` + (TRUNCATE + batched INSERT in one transaction); `package.json` scripts `atlas:build`, `atlas:import`. +- `db/data/spawnAtlas..json` ×13 + `spawnAtlas.index.json` (creatures, champions, regions, + landmarks, meta with per-source-file hashes). +- `src/model/shardAtlas/{shardAtlas.db.js,shardAtlas.model.js}` following the `shardState` split. +- `src/router/v1/public/atlas.{router,controller}.js`; `test/spawnAtlas.parse.test.js`. + +Two parsing notes that matter: + +- `` is `Type:MX=n:SB=…` segments joined by `:OBJ=` — verified against `trammel.xml`, where + a single point carries six types. Split on `:OBJ=`; the token before the first `:` is the type. +- **The high-value transform:** point-in-rect each spawn against the facet's `Regions.xml` rects + (highest `priority` wins), falling back to the nearest `Data/Locations` landmark, else + `"Wilderness"`. This is what turns *"lizardman at 5411,1234"* into ***"Despise, Felucca"*** and is + the entire reason the page is worth building. `Regions.xml` is genuinely nested and needs a ~120-line + recursive tokenizer **or** one devDependency (`fast-xml-parser`) — the server has zero XML deps + today, so that is an explicit call to make at implementation time. The flat `` files need + only regex/streaming; **do not** put 10.5 MB through a DOM parser. + +Tables: `shard_spawn_creatures` (slug PK, name, total, facets JSON), `shard_spawn_points` (slug, +facet, x, y, region, landmark, max_count, tod_*), `shard_regions`, `shard_landmarks`, +`shard_champion_spawns`, `shard_atlas_meta`. Plain `INDEX` on name, **not `FULLTEXT`** — ~1,500 +creature rows makes a `LIKE` scan free, and FULLTEXT brings min-token-length trouble for names like +"orc". No FKs, consistent with every existing `shard_*` table. + +Routes at `/api/v1/public/atlas`, **not** under `/shard` — the atlas is static shard *content*, not +live shard *state*; it must not look sidecar-dependent, and unlike `/shard/*` it *should* be +`siteMode`-gated like `/posts` and `/wiki`. `GET /creatures?q=&facet=`, `/creatures/:slug`, +`/regions`, `/landmarks`, `/champions`, `/meta`, all behind `requireFeature('atlas')`. Admin: +`GET /admin/shard/atlas/status` (artifact-vs-DB drift) and `POST /admin/shard/atlas/import`. **Build +stays CLI-only.** + +Client: `routes/public/Atlas.jsx` (`/site/atlas`) and `AtlasCreature.jsx` (`/site/atlas/:slug`). + +**Payload risk** — a monolithic artifact would be 2–3 MB of committed JSON. Shard per facet and drop +every `` field the site cannot use (`UniqueId`, all trigger/refractory/proximity/sequential +fields, sound ids), keeping Name/Map/X/Y/W/H/Range/MaxCount/MinDelay/MaxDelay/TOD*/types — well under +1 MB. The artifact never reaches the browser; the browser sees only paginated API responses. + +**Operator re-run story** — spawns changed → `npm run atlas:build -- --servuo ` on a machine +with the tree → commit the regenerated `db/data/spawnAtlas.*.json` → deploy → `npm run atlas:import` +(or `POST /admin/shard/atlas/import`). `shard_atlas_meta.source` holds per-file hashes, so +`GET /admin/shard/atlas/status` reports when the DB is behind the artifact. Full detail in +`docs/website/SPAWN_ATLAS.md`. + +--- + +## 7. Part B/2 — `points.board` + +Two deliverables: a diff sweep for the boards, and a `points` block folded into `char.profile` — +the `PROTOCOL_2.md` §10.3 `titles` precedent (read-model enrichment, no new request kind). + +### 7.1 Plugin + +NEW `BridgePoints.cs`, copying the `BridgeHousing.cs` diff-sweep shape (`Initialize` → +`ServerStarted`, `Connected_Core += OnConnected` clearing `_last` + `Rearm()`, `SweepOnce()`, +`Status()`, skip when `!BridgeLink.Connected`, try/catch throughout). + +Which systems: default to `PointsSystem.Systems` filtered to `ShowOnLoyaltyGump == true` — reuse the +shard's own "this is player-facing" signal rather than inventing one. `Bridge.cfg PointsSystems=` +overrides. Null-guard `PointsSystem.Systems`; it is a mutable static populated by 25 separate +subsystem constructors. + +**The perf trap.** `PlayerTable` is a plain `List`, and `QueensLoyalty` has `AutoAdd`, so +it can hold an entry for every `PlayerMobile` that ever existed. A naive +`.OrderByDescending().Take(N)` across 25 systems is 25 full sorts — at 20,000 historical characters, +~7.5 M comparisons, tens of ms on the Core thread. `BRIDGE_PLUGIN_PLAN.md` §1 found that nothing +except bulk profile generation comes close to a frame budget; this would be the second thing that +does. + +**Mitigation — single-pass bounded selection** into a fixed N-element sorted array (N=10): O(n·N) with +tiny constants and one allocation. Skip `Player == null || Deleted` and `Points <= 0`. ~500 k cheap +iterations at a 300 s interval. + +Diff signature per system: `concat(serial + ":" + (long)points)` over the top N, plus the entry count. +**No `points.remove`** — the system set is fixed, the same argument `city.update` already uses. + +### 7.2 Payload — one frame per system + +25 × ~600 B rather than one 12 KB frame, matching `champ.update` / `guild.update`: + +```jsonc +{"t":…,"kind":"points.board","system":"QueensLoyalty", + "nameString":"Queen's Loyalty","nameNumber":1114938, + "maxPoints":30000,"showOnGump":true,"players":842, + "top":[{"rank":1,"serial":"0x1A2B","name":"Darrow","points":29500}, …]} +``` + +`nameString` **and** `nameNumber` are both emitted (a `TextDefinition` may be a cliloc), resolved +website-side — the contract `titles.reward` already documents at `BridgeProfile.cs:107-110`. + +**Entries are written inline as `{serial, name}` — never via `BridgeJson.Actor`.** Deliberate even +though the website can now reveal fields by rung: `acct`/`webId` are not needed here, because the +website resolves serial→user from its own `shard_account_links` mirror for staff views. Keep the wire +minimal. + +### 7.3 `char.profile` enrichment + +`BridgeProfile.cs` gains `WritePoints(sb, m)` alongside `WriteTitles`: +`"points":[{system,nameString,points,maxPoints}]`, omitting systems with no entry or 0 points. + +**Deliberately no `rank`** — computing it means scanning each system's `PlayerTable` once per profile +(25 × n), which would dominate the measured 0.069 ms/profile budget. The website derives rank from +the board when the character appears in the top N. Gate behind `PointsProfileRank=false` if it is +ever wanted. + +### 7.4 Config, sidecar, website + +`Bridge.cfg`: `PointsSweepSeconds=300`, `PointsLeaderboardEnabled=true`, `PointsTopN=10`, +`PointsSystems=` (blank ⇒ auto), `PointsProfileEnabled=true`, `PointsProfileRank=false`. +`BridgeBoot.cs`: `Rearm()` in `reload`, `SweepOnce()` in `sweepnow`, `Status()` in both. + +Sidecar — `points_boards(system PK, name, json, updated_t)`; `main.rs` arm keyed on `system`; +`GET /points` and `GET /points/:system`. + +Website — `shard_points_boards(system PK, name, name_cliloc, max_points, players, show_on_gump, +payload JSON, t)`. **The top-N list stays in `payload`** — a fixed-size list read whole, exactly like +`shard_governors.candidates`. Do not normalize into a `shard_points_entries` table until a +per-character reverse lookup is actually needed. `shardIngest.js` → `upsertPointsBoard`, **not** in +`LOGGED_KINDS` (board state, like `guild.update`). `KIND_FEATURE['points.board'] = 'leaderboards'`, +with `characterName` as its per-field rule. `GET /public/shard/points` and `/points/:system` behind +`requireFeature('leaderboards')`; validate `system` ≤ 48 chars. + +**No new player route** — per-character points ride inside `char.profile`, already served by +`GET /player/shard/char/:serial` with its `shardLinks.ownsAccount` check. + +Client — NEW `routes/public/Leaderboards.jsx` at `/site/leaderboards`; a "Loyalty & Points" section +added to `components/CharacterSheet.jsx`, one edit serving both `PlayerCharacter.jsx` and +`AdminCharacter.jsx`. + +--- + +## 8. Part B/3 — `vendor.listing` + +### 8.1 It cannot be an RPC, and this is load-bearing + +`rpc.rs::try_route` correlates on the **first** frame carrying a matching `reqId` and resolves a +single `oneshot`. A chunked reply sharing one `reqId` would deliver chunk 1 to the HTTP caller and +**leak chunks 2..N onto the broadcast feed**. `REPLY_TIMEOUT` is 10 s (the client waits 12 s), so a +whole-world snapshot could not fit regardless. + +⇒ **a per-vendor diff sweep on the broadcast stream**, like `champ.update` / `house.update`. The +existing per-account `vendor.snapshot` RPC is untouched; the player portal keeps using it. + +Kinds: `vendor.listing` (one frame per vendor, authoritative for that vendor) and +`vendor.listing.remove`. Payload: `serial, shopName, owner:{serial,name}, map, x, y, region, house, +count, truncated, items:[{serial,itemId,hue,amount,price,name,cliloc,child}]`. + +### 8.2 Two perf traps + +Measured baseline (`BRIDGE_PLUGIN_PLAN.md` §1): 30 vendors / 1,200 listings = 0.343 ms via +`pack.Items` + `GetVendorItem`; extrapolated to 500 vendors / 40,000 listings ≈ 12 ms per full pass. +Except: + +1. **`VendorSearch.GetItemName(Item)` is a packet builder, not a field read.** It constructs an + `ObjectPropertyList`, calls `GetProperties`, serialises, then byte-parses the packet + (`VendorSearch.cs:681-789`) — per item. Across 40,000 items in one tick that is a + multi-hundred-millisecond stall. **Mandatory: never call it in the sweep.** Emit `itemId`, `hue`, + `amount`, `price`, `item.Name` (the plain field, null for most) and `item.LabelNumber`, resolving + display names website-side — exactly what `char.profile.equipment` already does + (`BridgeProfile.cs:173`). +2. **`VendorSearch.GetItems(PlayerVendor)` is private** (`:791`). The reusable public API is + `GetItems(Container, List)` (`:807`), which recurses into sub-containers, so real item counts + run above the top-level `pack.Items` the 0.343 ms measurement used. Budget accordingly. + +### 8.3 Mitigations + +- **Amortized round-robin sweep** — `MarketSweepSeconds=60`, at most `MarketSweepBatch=25` vendors per + tick, with a persistent cursor over `PlayerVendor.PlayerVendors`. Full coverage in + `ceil(vendors/25) × 60 s`, with **per-tick cost bounded independent of world size**. This is the one + genuinely new pattern versus the existing sweeps and should be flagged in review. +- **Per-vendor signature diff** (`count | Σ(serial ^ price) | x | y | shopName`), as `BridgeHousing` + does — most vendors are static, so steady-state emission is near zero. +- **`MarketMaxListings=250`**, then `"truncated":true`. `BridgeJson.Parse` caps *inbound* at 1 MB; + outbound is uncapped and `shard.rs::read_line` will allocate whatever arrives. +- On `Connected_Core`, clear `_last` **and reset the cursor**; the re-emit is self-throttled by the + round-robin window. + +### 8.4 Player opt-out and privacy + +**Honour `pv.VendorSearch`** — ServUO's own per-vendor opt-out, which `DoSearch` filters on (`:62`). +Skip opted-out vendors entirely; the seen-set removal then drops them from the board, so **a player +who hid their vendor in game is hidden on the website too.** Also skip `Map == null || Map.Internal` +and `Backpack == null`, matching `DoSearch`. + +A vendor's shop name, owner character name and location are **already globally visible in-game** — the +stock Vendor Search gump surfaces exactly this set to any player — which is why they default to +`anonymous`. They remain per-field configurable (`ownerName`, `location`) so an admin can tighten +them. Account name and website user id never go on the wire. + +### 8.5 Sidecar and website + +Sidecar — one table `vendors(serial PK, shop_name, owner_name, map, x, y, region, count, json, +updated_t)` storing the whole-vendor blob. **No `vendor_items` table** — the sidecar's job here is +outage resilience (`PROTOCOL_2.md` §12.2), not search; search lives in MariaDB. Endpoint is +**`GET /market`**, not `/vendors` — axum would route the latter fine, but the collision with the +per-account RPC is a readability trap. + +Website — `shard_vendors` + `shard_vendor_items` (indexes on `vendor_serial`, `price`, `item_id`, +`display_name`; delete-then-insert per vendor in one transaction; no FKs). `shardIngest.js` handles +both kinds; **not** in `LOGGED_KINDS`. + +`KIND_FEATURE['vendor.listing'] = 'market'`, but the market feature's **SSE mapping is disabled by +default**: a live firehose of full vendor inventories would be the site's single biggest bandwidth +consumer, and no page needs it live. The page is a paginated DB query with a staleness stamp; an +admin can turn the stream on. `uoLinkSocket` paginates `/market` on reconnect, bounded by +`MARKET_SNAPSHOT_MAX = 5000` vendors so a pathological world cannot hang startup. + +`GET /public/shard/market?q=&minPrice=&maxPrice=&itemId=&map=®ion=&sort=&limit=&offset=` +(limit 1..100, default 50; `q` ≤ 60 chars; `sort ∈ {price_asc, price_desc, recent}`) and +`/market/vendors/:serial`, behind `requireFeature('market')`. **Rate-limit it** — this is the first +genuinely expensive public endpoint; `express-rate-limit` is already a dependency. + +### 8.6 The open dependency — cliloc names + +`CharacterSheet.jsx:14-15` already documents the gap ("without a cliloc table on the site we can only +show literals") and renders equipment as `id {itemId}`. Search-by-name needs that table. + +- **Recommended:** `scripts/buildClilocs.js` reads the UO client's `Cliloc.enu` → committed + `db/data/clilocs.json`; ingest denormalizes into `shard_vendor_items.display_name`. Same + build-artifact pattern as §6, and it **also fixes the character sheet**. +- **Fallback:** ship with item-art + price + region filters, and name search only over renamed items. + +This decision is the reason §8 is sequenced last. + +### 8.7 Client + +`routes/public/Market.jsx` at `/site/market`, with a *"prices last refreshed N minutes ago"* banner +driven by `staleAt` (the oldest `shard_vendors.updated_at`). The round-robin sweep means data is +inherently up to one full cycle old, and the UI must say so. + +--- + +## 9. Sequencing + +| Order | Part | Repos touched | Wire change | +|---|---|---|---| +| 1 | **A** — visibility framework + actor-leak fix | website, docs | none | +| 2 | **B/1** — `world.ruleset` (§5) | all four | new kind | +| 3 | **C** — spawn atlas (§6) | website, docs | none | +| 4 | **B/2** — `points.board` (§7) | all four | new kind + `char.profile` field | +| 5 | **B/3** — `vendor.listing` (§8) | all four | new kinds | +| 6 | **Cutover** — `PROTOCOL_VERSION` 2→3, `edge` → `main` | all four | the bump | + +--- + +## 10. Documentation obligations + +- This file (`link/v3.md`) is the canonical 3.0 design. +- `PROTOCOL_2.md` §10.4 gains a note that `world.systems` is superseded by `world.ruleset`, and that + the deferred VvV question is answered (`VvV.cfg Enabled=True`, Factions off). +- `INTEGRATION.md` — catalog entries and §6 consumer sections for each new kind, plus the v2→v3 + upgrade note for operators. +- `PLAN.md` — phasing. +- `website/BACKEND_DESIGN.md` — every new table and route, and **the visibility framework as a + security contract**: the audience ladder, the two locked rules, and the fail-closed kind map belong + in the security section. +- NEW `website/SHARD_VISIBILITY.md` — admin-facing: what each feature exposes, what each rung means, + what cannot be loosened. +- NEW `website/SPAWN_ATLAS.md`, NEW `website/MARKETPLACE.md`. +- `PROJECT_TREE.md` in each touched repo. +- `npm run swagger` **and** `npm run routes:manifest` on every route-touching PR — both are committed + artifacts, and `test/routeManifest.test.js` fails on drift. + +**Follow-up, not scoped for 3.0:** the Android app consumes the same public/player shard API and will +need `/public/shard/features` to hide its own nav. Track separately against `android-app/`. + +--- + +## 11. Verification + +**Plugin** — `servuo-plugins\deploy.ps1 -ServerPath -Verify`, inspect the ADD/CHANGE list, +then re-run without `-Verify` (ServUO must be stopped). Boot with `tools/stub_sidecar.ps1` listening +and **confirm the compile banner in the console, not merely the absence of errors** — +`BRIDGE_PLUGIN_PLAN.md` §1 warns that a failing build is silently ignored and the previous +`Scripts.dll` reloads. Then `[bridge status`, `[bridge sweepnow`, `[bridge reload`. + +- §5: eyeball the emitted `world.ruleset` frame for anything sourced from `Server.cfg`, `Staff.cfg`, + `Email.cfg`, `DataPath.cfg` or `Bridge.cfg`. +- §8: with a seeded world, time one sweep tick and confirm the batch cap holds it under ~1 ms. + +**Sidecar** — `cargo build && cargo clippy`; `curl -H "Authorization: Bearer " +localhost:8080/ruleset` (and `/points`, `/market`); confirm `X-UOLink-Version: 3` and that a client +declaring 2 receives a 409. + +**Website server** — `DB_HOST=127.0.0.1 DB_PORT=59999 node --test`. New tests, each modelled on an +existing sibling: `test/shardVisibility.test.js`, `test/shardBroadcast.visibility.test.js`, +`test/shardIngest.{ruleset,points,market}.test.js` (after `shardIngest.protocol2.test.js` — stubbed +deps, asserting routing and `logged` flags), `test/spawnAtlas.parse.test.js` (pure functions, inline +fixtures). Then `npm run routes:manifest` and `npm run swagger`, committing both. + +**Full stack** — against a local MariaDB: apply `db/schema.sql` (idempotent), start the server, +confirm `uoLinkSocket` backfill logs the new snapshot lines and that `uo_link_config.protocol` +migrated to 3, then load `/site/rules`, `/site/atlas`, `/site/leaderboards`, `/site/market`. + +**Visibility smoke test** — for each of the five rungs, walk every shard page and confirm gating and +field projection match the configured matrix. Same shape as the 200-routes × 5-access-levels sweep +already run for the domain split. + +--- + +## 12. Critical files + +| File | Why | +|---|---| +| `website/server/src/utils/shardBroadcast.js` | The security boundary; reworked from a static allowlist to per-connection audience filtering. **The highest-risk file in 3.0.** | +| `website/server/src/utils/shardVisibility.js` (new) | Ladder, kind→feature map, projection | +| `website/server/src/utils/shardIngest.js` | The dispatcher every new kind routes through | +| `website/server/src/model/shardState/shardState.model.js` | The `shape*` projections, including the `shapeGuild` leak §3.1 fixes | +| `servuo-plugins/overlay/Scripts/Custom/Bridge/BridgeHousing.cs` | Cleanest copy of the diff-sweep pattern; template for `BridgePoints.cs` and `BridgeMarket.cs` | +| `link/sidecar/src/main.rs` | `PROTOCOL_VERSION` 2→3 and the board-projection match | +| `website/server/db/schema.sql` | All new `shard_*` tables plus the `uo_link_config.protocol` migration | -- 2.49.1 From bf41105ec0c584cc342c5b768ef8614c1bb8f92a Mon Sep 17 00:00:00 2001 From: wtclaude Date: Tue, 28 Jul 2026 10:04:29 -0500 Subject: [PATCH 02/19] docs(website): record the shard visibility framework Protocol 3.0 Part A. Admin-configurable, per-feature and per-field audience control over every shard-derived surface, replacing the static PUBLIC_KINDS allowlist that used to be the whole boundary. - SHARD_VISIBILITY.md (new): the admin-facing guide - the ladder, what each of the ten features exposes, the defaults, the two rules that are code rather than configuration, and worked examples. - BACKEND_DESIGN.md 6.5 (new): the same thing as a security contract - the ladder and how viewerLevel resolves it, the locked acct/webId rule, the fail-closed kind map, the asymmetric ladder fallbacks, and the three enforcement points. Plus the shard_feature_visibility schema, the /public/shard/features route, and the adminOnly tier on /admin/shard/visibility. Defaults reproduce pre-3.0 behavior everywhere, with one deliberate exception which is the leak Part A was written to close: guilds and governors previously returned the raw stored payload, whose leader and governor actors carry acct and webId, to anonymous callers. Co-Authored-By: Claude --- README.md | 1 + website/BACKEND_DESIGN.md | 70 +++++++++++++++++++- website/SHARD_VISIBILITY.md | 124 ++++++++++++++++++++++++++++++++++++ 3 files changed, 194 insertions(+), 1 deletion(-) create mode 100644 website/SHARD_VISIBILITY.md diff --git a/README.md b/README.md index 9ceffd5..132d42d 100644 --- a/README.md +++ b/README.md @@ -19,6 +19,7 @@ ci/ cross-cutting CI/quality notes | [BACKEND_DESIGN.md](website/BACKEND_DESIGN.md) | API contract, DB schema, security model | | [HERO_EDITOR.md](website/HERO_EDITOR.md) | Hero canvas editor feature spec | | [WIKI_UPGRADE.md](website/WIKI_UPGRADE.md) | Wiki subsystem upgrade notes | +| [SHARD_VISIBILITY.md](website/SHARD_VISIBILITY.md) | Who sees which shard data — the admin-configurable audience framework | | [website-README.md](website/website-README.md) | Snapshot of the website repo's README (setup/run reference) | | [PROJECT_TREE.md](website/PROJECT_TREE.md) | Auto-generated snapshot of the repo's tracked file layout | diff --git a/website/BACKEND_DESIGN.md b/website/BACKEND_DESIGN.md index ccb4f0d..c160e74 100644 --- a/website/BACKEND_DESIGN.md +++ b/website/BACKEND_DESIGN.md @@ -99,7 +99,7 @@ server/ pages.router.js (2) /public/pages — the draft-preview route precedes /:slug and is deliberately not site-mode gated - shard.router.js (12) /public/shard/* incl. the anonymous + shard.router.js (13) /public/shard/* incl. the anonymous SSE stream; never site-mode gated site.router.js (4) /settings /status /version /contact — the group-root singletons; declares no @@ -358,6 +358,19 @@ analogue to a password — and there is no hash-lookup constraint (verification unused rows and `bcrypt.compare`s each, like password verification). `used_at` is the single-use marker. Cleared wholesale on TOTP disable / password change / password reset. +### shard_feature_visibility — per-feature audience config (Protocol 3.0) + +One row per shard feature: `feature` (PK), `enabled`, `audience` (a rung on the ladder in §6.5), +`stream` (whether the feature's kinds fan out over SSE at all), `field_rules` JSON (`{field: rung}` +for the sensitive fields only), `updated_by`, `updated_at`. + +**An absent row means "use the compiled default", and the compiled defaults reproduce pre-3.0 +behavior — so an empty table is a no-op and there is nothing to seed.** Stored rows are merged over +the defaults on read, which is also where the invariants are re-applied: a row naming an unknown +feature is ignored (a stale row must not resurrect a removed feature), an invalid rung falls back to +the default rather than failing open, and a rule touching a locked field (`acct` / `webId`) is +discarded. See §6.5. + --- ## 4. API contract @@ -556,6 +569,7 @@ from the per-route **siteMode** middleware (§5), never from an auth gate. | GET | `/wiki` | list of pages (slug + title) | | GET | `/wiki/:slug` | single page | | POST | `/contact` | (rate-limited) send mail via SMTP; if unconfigured, respond `{fallback:"mailto", email}` | +| GET | `/shard/features` | the shard features **this caller** may reach plus the audience rung they resolved to (§6.5), so a client hides nav it can't follow. Reports only what the caller can see — the list itself never discloses a gated feature. Consumed by the SPA header and (pending) the Android nav. | Public content GETs pass through the **siteMode** gate (§5). @@ -570,6 +584,10 @@ the whole gate. The ops/config capabilities — `uo-link`, `email`, `discord-bot linking carries no extra gate and the in-game staff operations carry `modAccess`. There is no residual file: every admin route is declared in a capability router. +`GET`/`PUT /admin/shard/visibility` are the third tier on that mixed prefix: **`adminOnly`**, because +they decide what *anonymous* visitors can see (§6.5). They sit above `modAccess` deliberately — a +moderator can ban a player but cannot decide what the public internet reads. + `GET /dashboard` and `PUT /site-mode` are the one place where a **single screen spans two tiers**: the dashboard is staff-wide, but the site-mode toggle on it is `adminOnly`. The client must therefore gate that control on its own (`Dashboard.jsx` renders it only for `role === 'admin'`) rather than relying on @@ -672,6 +690,56 @@ who"; `activity_log` provides the history feed. - **`app.set('trust proxy', 1)`** so secure cookies, `req.ip`, and rate-limiting work behind Pangolin. - **CORS**: same-origin in prod (SPA served by Express). Dev only: allow `CLIENT_ORIGIN` (Vite, `http://localhost:5173`) with `credentials:true`. +### 6.5 Shard visibility — the audience boundary (Protocol 3.0) + +Every shard-derived surface is gated by an **admin-configurable, per-feature and per-field** audience +setting. This **replaces** the static `PUBLIC_KINDS` allowlist that used to be the whole boundary. +Policy lives in `utils/shardVisibility.js`; rows live in `shard_feature_visibility`; the admin surface +is `GET`/`PUT /admin/shard/visibility` (`adminOnly`). Admin-facing guide: +[`SHARD_VISIBILITY.md`](SHARD_VISIBILITY.md). Design: [`../link/v3.md`](../link/v3.md) §3. + +**The ladder.** `anonymous < logged_in < player < staff < admin`, each rung implying the ones below. +`viewerLevel(req)` resolves it: no session ⇒ `anonymous`; authenticated ⇒ `logged_in`; authenticated +with a linked game account ⇒ `player`; moderator ⇒ `staff`; admin ⇒ `admin`. **Staff satisfy the +`player` rung without a linked account** (consistent with `/player/*` being role-agnostic). +**`editor` gets no shard privilege** — it is a content role, and mapping it to `staff` would silently +widen what editors see. + +**Two invariants that are code, not configuration.** Both are enforced server-side and both reject +rather than silently ignore: + +1. **`acct` and `webId` are admin-only, always.** They are not exposed as configurable fields, and a + stored row attempting to loosen them is discarded on read as well as rejected on write. A character + name is visible in game; the account behind it and the website user it links to are not. +2. **A kind absent from `KIND_FEATURE` is never broadcast below `admin`.** Fail closed. This is what + keeps the kind map a security boundary rather than a convenience filter, and it means a shard that + starts emitting an unknown event degrades to staff-only, never to public. + +**Fail-closed everywhere else too.** An unreadable visibility config withholds every public frame; a +DB failure falls back to the compiled defaults (pre-3.0 behavior), not to open; an unresolvable viewer +subscribes as `anonymous`. The ladder comparison uses **asymmetric** fallbacks by design — an unknown +*viewer* level floors to the bottom rung and an unknown *requirement* ceils to admin, so an +unrecognised value loses on both sides. (A single shared fallback cannot do that: whichever direction +it picks, it fails open on one side.) + +**Three enforcement points, one config:** + +| Where | Mechanism | +|---|---| +| Routes | `requireFeature(name)` — **404** when the feature is disabled (don't leak that it exists), **403** when the caller is below its audience. `projectFeature` then strips out-of-rung fields from the body. | +| SSE (`utils/shardBroadcast.js`) | Per-connection filtering. A subscriber's rung is resolved **once at subscribe time and frozen** for that connection, so a long-lived stream can't gain privilege; each frame is then mapped kind→feature, gated, and field-projected per viewer. Two subscribers can legitimately receive different versions of one event, or one of them nothing. | +| Nav | `GET /public/shard/features` returns only what the caller may reach, so the SPA never renders a link that would 403. Presentation only. | + +Config reads are cached ~5s, so admin changes take effect within seconds **including on already-open +streams**. `PUBLIC_KINDS` still exists and is still exported (`/feed` filtering, `notificationStreams.js`) +but is now **derived** from the kind map rather than hand-maintained, so the two cannot drift. + +**Defaults reproduce pre-3.0 behavior exactly**, so installing the framework is a no-op until an admin +changes something — with one deliberate exception, which is the leak it was written to close: +`/public/shard/guilds` and `/public/shard/governors` previously returned the raw stored payload, whose +`leader` / `governor` actors carry `acct` and `webId`. Those fields are now stripped for every caller +below admin. + --- ## 7. Email diff --git a/website/SHARD_VISIBILITY.md b/website/SHARD_VISIBILITY.md new file mode 100644 index 0000000..4f092f8 --- /dev/null +++ b/website/SHARD_VISIBILITY.md @@ -0,0 +1,124 @@ +# Shard visibility — who sees which shard data + +**Status:** Built (Protocol 3.0 Part A). Admin → Shard Visibility. +**Audience:** shard owners and admins. +**Companion to** [`../link/v3.md`](../link/v3.md) §3 (the design) and +[`BACKEND_DESIGN.md`](BACKEND_DESIGN.md) §6 (the security contract). + +The website surfaces a lot of live shard data. What your players, your staff and the anonymous +internet may each see is **yours to decide**, per feature, from Admin → Shard Visibility. + +Nothing changes until you change it: every setting ships at the value that reproduces how the site +behaved before this panel existed. + +--- + +## 1. The audience ladder + +Five rungs. Each one includes everyone below it. + +| Rung | In the UI | Who that is | +|---|---|---| +| `anonymous` | **Everyone** | Anyone at all, signed in or not. | +| `logged_in` | **Signed in** | Any registered account, whether or not they've linked a game account. | +| `player` | **Linked players** | Accounts with a linked in-game account. **Staff always qualify**, linked or not. | +| `staff` | **Staff** | Admins and moderators. | +| `admin` | **Admins only** | Admins. | + +Two notes that surprise people: + +- **`editor` is a content role, not a shard role.** Editors write news and wiki pages; they get no + shard privilege from that. An editor is treated by link status like any other member. This matches + the rest of the site, where shard staff powers are admin-or-moderator. +- **Staff satisfy `player` without linking.** Otherwise an admin would be locked out of surfaces + they'd gated to players, which is how the `/player/*` routes already behave. + +## 2. What you can set per feature + +**Enabled.** Off means gone. The feature's pages return “not found”, not “forbidden” — a disabled +feature doesn't advertise that it exists. + +**Who can see it.** The minimum rung, from the ladder above. + +**Live updates.** Whether this feature pushes changes to open pages in real time. Turning it off +doesn't break the page; it just refreshes on load instead of updating in place. + +**Sensitive fields.** Some features expose a field that deserves its own rung — you can publish the +board while holding back one column. See the table in §3. + +## 3. The features, and their defaults + +| Feature | What it exposes | Default | Sensitive fields | +|---|---|---|---| +| **Shard status** | Connection state, online count, gold-supply series | Everyone | — | +| **Activity feed** | Deaths, kills, skill gains, quests, logins | Everyone | — | +| **Champion spawns** | The live champion / mini-champ / sea-boss board | Everyone | — | +| **Guilds** | Guild rosters, alliances, leaders | Everyone | — | +| **Town governors** | City Loyalty governors, elections, term history | Everyone | — | +| **Houses / IDOC** | Houses in danger | Everyone | House owner → Staff · House price → Staff | +| **Players online** | Population aggregate, staff-online widget | Everyone | In-game location → Staff | +| **Shard rules** | Skill/stat caps, house limits, vet rewards, the ruleset | Everyone | Connect address → Everyone | +| **Spawn atlas** | Bestiary and spawn locations (static content) | Everyone | — | +| **Leaderboards** | Point and loyalty standings | Everyone | Character names → Everyone | +| **Marketplace** | The shard-wide player-vendor index | Everyone, **live updates off** | Vendor owner name → Everyone · Location → Everyone | + +**Why the marketplace ships with live updates off.** A live feed of every vendor's full inventory +would be the single largest thing the site sends. No page needs it — the marketplace is a search over +stored data with a “prices last refreshed N minutes ago” stamp. Turn it on only if you want it. + +**Why house owner/price default to Staff.** The public Houses page has always been a "where are the +falling houses" board — location only. Owner and price are the staff view. That split is preserved. + +## 4. What you cannot change + +Two rules are enforced in code and are not settings. Attempting to set them returns an error rather +than silently ignoring you. + +**1. Game account names and website user ids are admin-only, always.** +`acct` and `webId` never appear below the admin rung on any surface. A character *name* is visible in +game to anyone standing next to them; the **account** behind it is not, and neither is the website +user it's linked to. Publishing those would disclose something the shard itself doesn't, and would +tie a player's in-game identity to their forum identity without their consent. + +**2. Unknown event kinds are never broadcast below admin.** +The live stream maps each event kind to a feature. A kind with no mapping — a new event from a shard +plugin the site doesn't know yet, say — goes to admins only. It fails closed. This is what keeps the +stream safe by default when the shard starts sending something new: the worst case is that staff see +it and players don't, never the reverse. + +## 5. How it's enforced + +Three places, one config: + +- **Page and API requests** are checked before the handler runs, and the response is then stripped of + any field above the caller's rung. +- **The live stream** resolves a viewer's rung once, when they connect, and freezes it for that + connection — a long-open page can't gain privilege because something changed underneath it. Each + event is then gated and stripped per viewer, so two people watching the same page can legitimately + receive different versions of the same event, or one of them nothing. +- **Navigation** hides links a viewer can't follow, so they don't hit a wall. This is presentation + only — the gate is server-side either way. + +Changes take effect within about five seconds, **including on streams that are already open**. You +don't need to restart anything. + +If the database is briefly unreachable, the site falls back to the built-in defaults — the pre-v3 +behavior — rather than to "everything is public". + +## 6. Worked examples + +**"I want a private shard — nothing public until people register."** +Set every feature to **Signed in**. Anonymous visitors still get the site itself; the shard data +disappears from the nav. + +**"Publish the market, but don't tie vendors to players."** +Marketplace → Everyone, with **Vendor owner name** → Staff. Prices, items and locations stay public; +who owns each vendor doesn't. + +**"Leaderboards for members only."** +Leaderboards → **Linked players**. Anyone who's linked a game account sees the standings; drive-by +visitors don't. + +**"Let players see house owners."** +Houses → Everyone, **House owner** → Linked players. Note this is a real disclosure: house ownership +is visible in game, but the website makes it searchable in a way the game doesn't. -- 2.49.1 From 35ad440bad24c8d6c5416e2f1fec6d0191174c1e Mon Sep 17 00:00:00 2001 From: wtclaude Date: Tue, 28 Jul 2026 10:52:13 -0500 Subject: [PATCH 03/19] docs(shard): record the REST projection gap the Part A smoke test found The live five-rung smoke test of the visibility framework found that Part A enforced it on the SSE path and on /guilds + /governors, but not on the remaining public REST reads - so one event was projected live and served verbatim from stored history. link/v3.md gains 3.6.1 with the full list (the anonymous acct/webId leak on /feed, the flattened ownerAcct on /idoc, the dead `houses` field rules, /feed ignoring live config, the empty-allowlist fall-through, and the Date-to-{} projection bug), plus the rule it leaves behind: a read path that returns shard data and does not project is a bug, and every new Part B/C surface must gate its kind set on live config rather than on PUBLIC_KINDS. 3.5 also corrected: the table is NOT seeded on boot. An absent row means "use the compiled default", which keeps the defaults in one place instead of duplicating them into a seeder that could drift. BACKEND_DESIGN.md 6.5 records the same as a security contract: rule 1 locks a field by meaning rather than spelling; PUBLIC_KINDS is a module-load constant and must not answer per-caller questions; projectFeature walks arrays and plain objects only. SHARD_VISIBILITY.md gets the admin-facing version - that stored history answers the same way the live stream does, and that turning live updates off stops the push, not the reading. Co-Authored-By: Claude --- android/TRUSTED_DEVICES_APP_HANDOFF.md | 126 +++++++++++++++++++++++++ link/v3.md | 44 ++++++++- website/BACKEND_DESIGN.md | 33 +++++-- website/SHARD_VISIBILITY.md | 14 +++ 4 files changed, 207 insertions(+), 10 deletions(-) create mode 100644 android/TRUSTED_DEVICES_APP_HANDOFF.md diff --git a/android/TRUSTED_DEVICES_APP_HANDOFF.md b/android/TRUSTED_DEVICES_APP_HANDOFF.md new file mode 100644 index 0000000..81ef644 --- /dev/null +++ b/android/TRUSTED_DEVICES_APP_HANDOFF.md @@ -0,0 +1,126 @@ +# Handoff — Trusted Devices & MFA: remaining work + +> Written 2026-07-22 at the end of the backend/web implementation session, for a +> fresh session to continue. Canonical design: `docs/website/TRUSTED_DEVICES_MFA.md`. +> App-side plan: `docs/android/PLAN.md §4.1.1`. This doc is the "what's left + how". + +## 1. Status at handoff + +**Done, in review, and live-smoke-tested** (real MariaDB + AVD): +- Backend (schema, session service, web + mobile login, self-service + admin + endpoints, invalidation), web client UI, admin front-end UI, OpenAPI spec, + 33 server tests — all in **website PR #93** (branch `feature/trusted-devices-mfa`). +- Docs (BACKEND_DESIGN §3/§4/§6, TRUSTED_DEVICES_MFA.md, PLAN §4.1.1) — **docs PR #32** + (branch `docs/trusted-devices-mfa`). +- Live smoke test passed end-to-end: trusted-device TOTP-skip (web + mobile + `X-Trust-Token`), recovery-code login (single-use), cap 409, admin MFA reset, + audit logging, and a real 2FA login through the Android app on an emulator. + +**Not done — the two remaining items below.** + +## 2. Remaining item A — merge gate (no code) + +- **website#93** and **docs#32** need CI green + review, then merge to `main`. +- CI (`.gitea/workflows/pr-checks.yml`) runs server tests, client build, bot install. +- Nothing to build here; just get them reviewed/merged. The Android work should land + **after** #93 merges so the app builds against the merged contract. + +## 3. Remaining item B — Android app implementation (the real work) + +The backend is fully ready and additive; the existing app is unaffected (verified). +The app just needs to *consume* the new endpoints. Own PR in the **`android-app`** +repo, branch `feature/trusted-devices-mfa`. Package id `com.runicgateway.app`. + +### 3.1 Scope (from PLAN §4.1.1) +1. **Login/TOTP screen:** on the existing `401 { totpRequired }` step, add + - a **"Trust this device"** checkbox, and + - a **"use a recovery code instead"** toggle (send `recoveryCode` instead of `code`). +2. **Trust token storage:** when a login response carries `trustToken`, store it in + **EncryptedSharedPreferences** (same secure store as the bearer tokens — never + plain prefs/logs). On subsequent logins send it as the **`X-Trust-Token`** header + so the server skips the TOTP prompt. +3. **Cap handling:** a login response with `{ trustLimitReached: true, devices }` + means show the device list and prompt the user to revoke one + (`DELETE /auth/me/trusted-devices/:id`) then retry trusting. +4. **Account screens:** + - **Trusted Devices**: list (`GET /auth/me/trusted-devices`), revoke one, untrust + all; "trust this device" (`POST /auth/me/trusted-devices` → returns `trustToken` + for native — store it). + - **Recovery Codes**: show the one-time batch returned by TOTP enable; remaining + count (`GET …/recovery-codes/status`); regenerate (`POST …/recovery-codes/generate`, + password step-up) with a show-once display + copy/share. +5. **Invalidation:** on logout / dead-refresh sign-out / Settings→Server switch, + **clear the stored `trustToken`** alongside the bearer tokens. (A server-side + password change/reset or TOTP disable already revokes it.) +6. **Tests:** JVM `:app:testDebugUnitTest` — DTO decode for the new fields, and + repository logic (trust-token persist/clear, recoveryCode vs code branch). + +### 3.2 Exact API contract the app consumes + +`POST /auth/mobile/login` — body `{ username, password, code?, recoveryCode?, trustDevice?, device_name? }`, optional header `X-Trust-Token: ` +- `200` → `{ accessToken, refreshToken, expiresIn, user:{id,username,role}, trustToken?, trustLimitReached?, devices? }` + - `trustToken` present only when `trustDevice:true` was accepted (store it). + - `trustLimitReached:true` + `devices[]` when at the cap (login still succeeded). +- `401` → `{ totpRequired:true, message }` (missing/invalid 2nd factor) — reveal the + code field (existing behavior); or generic `{ message }` for bad credentials. +- A valid `X-Trust-Token` bound to the user makes a code unnecessary → straight `200`. + +Self-service (Bearer access token): +- `GET /auth/me/trusted-devices` → `[{ id, platform, deviceName, userAgent, createdAt, lastUsedAt, expiresAt }]` +- `POST /auth/me/trusted-devices` body `{ deviceName? }` → `{ trusted:true, trustToken }` (native) | `409 { error:'trusted_device_limit', devices }` +- `DELETE /auth/me/trusted-devices/:id` → `{ revoked:boolean }` +- `DELETE /auth/me/trusted-devices` → `{ revoked:number }` +- `GET /auth/me/account/recovery-codes/status` → `{ remaining:number }` +- `POST /auth/me/account/recovery-codes/generate` body `{ currentPassword? }` → `{ recoveryCodes:[string] }` +- `POST /auth/me/account/totp/enable` body `{ code }` → `{ totp_enabled:true, recoveryCodes:[string] }` (codes shown once) + +The OpenAPI spec (`website/server/swagger/swagger-output.json`, schemas `TrustedDevice`, +`TrustDeviceResult`, `TrustedDeviceLimit`, `RecoveryCodes`) is the source of truth. + +### 3.3 Where it likely goes in the app +The app already handles `totpRequired` (reveals a code field — verified live), so the +auth surface exists. Extend the existing auth Retrofit API + repository + login +ViewModel/screen, add a secure `trustToken` accessor to the encrypted token store, +and add two account screens. Explore `android-app/app/src/main/java/com/runicgateway/app/` +(auth/data/core modules) at the start — don't assume file names. + +## 4. Environment & how-to (verified this session) + +- **DB:** Docker container `uomm-db`, MariaDB on host port **3307**, db `uomysticmoon`, + user `uomm` (password: `docker exec uomm-db printenv MARIADB_PASSWORD`). +- **Run the server:** `cd website/server && node src/server.js` (uses `.env`, already + points at 127.0.0.1:3307). It ensures schema + seeds on boot. For cap testing set + `MAX_TRUSTED_DEVICES=2`; recovery count via `RECOVERY_CODE_COUNT`. + - Pre-existing noise: `uo-link-socket … Unsupported state` decrypt errors are an old + encrypted `uo_link_config` row, unrelated — ignore. +- **Server tests:** `cd website/server && DB_HOST=127.0.0.1 DB_PORT=59999 node --test` + (dead port by design; models stubbed). Client: `cd website/client && npm test`. +- **TOTP codes for manual testing:** from `website/server`, + `node -e "process.stdout.write(require('speakeasy').totp({secret:'',encoding:'base32'}))"`. +- **Android build:** JDK 21; `cd android-app && ./gradlew :app:assembleDebug -Pksp.incremental=false`. + Unit tests: `./gradlew :app:testDebugUnitTest -Pksp.incremental=false`. +- **Emulator:** SDK at `~/AppData/Local/Android/Sdk`; AVDs `Medium_Phone_API_36.1`, + `s22_ultra`. `adb install -r -g app/build/outputs/apk/debug/app-debug.apk`. +- **⚠ Cleartext HTTP gotcha (dev only):** the app blocks plain HTTP to `10.0.2.2` + (`CLEARTEXT communication … not permitted`). Cleartext to `127.0.0.1` **is** allowed, + so for local testing run `adb reverse tcp:3000 tcp:3000` and set the app's server URL + to `http://127.0.0.1:3000` (its default). Production uses HTTPS via the proxy — not a + code issue. (Consider whether v1 wants a `network_security_config` dev exception; not + required for the feature.) + +## 5. Conventions (CLAUDE.md) + +- Branch from up-to-date `main`; commit/push as **`wtclaude`** using the token at + `C:\Users\colby\.gitea_token_claude` via `http.extraHeader` (never inline in URL). +- Conventional Commits; end commit messages with the `Co-Authored-By: Claude …` + + `Claude-Session:` trailers. PR template requires ticking the **AI-assisted** box + (tool: Claude Code) and the license box. +- Use `mcp__gitea__*` for PRs/issues. GPL-3.0-or-later. + +## 6. First moves for the fresh session +1. Check whether website#93 / docs#32 have merged (`mcp__gitea__pull_request_read`). +2. In `android-app`: sync `main`, `git checkout -b feature/trusted-devices-mfa`. +3. Explore the app's auth module; implement §3 against the §3.2 contract. +4. Test with the local server + emulator via the §4 cleartext workaround. +5. Open a single `android-app` PR; update `docs/android/PLAN.md` if the app design + deviates from §4.1.1. diff --git a/link/v3.md b/link/v3.md index bd6e92c..e969e84 100644 --- a/link/v3.md +++ b/link/v3.md @@ -115,9 +115,13 @@ CREATE TABLE IF NOT EXISTS shard_feature_visibility ( ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4; ``` -Seeded on boot in `server.js`, one row per feature. **All ten shard features are covered — the four -new ones and the six that already ship — and every default reproduces today's behavior, so the -retrofit is a no-op until an admin changes something.** +**Not** seeded on boot (this changed during implementation): an **absent row means "use the compiled +default"**, so the table starts empty and only ever holds rows an admin has actually touched. The +defaults live in one place — `FEATURES` in `shardVisibility.js` — instead of being duplicated into a +seeder that could drift from it, and a DB blip degrades to those same defaults rather than to +"everything is public". **All ten shard features are covered — the four new ones and the six that +already ship — and every default reproduces today's behavior, so the retrofit is a no-op until an +admin changes something.** | Feature | Default audience | Sensitive fields (default rung) | |---|---|---| @@ -156,10 +160,42 @@ Applied at: 3. **Nav** — `GET /api/v1/public/shard/features` returns only the features the calling viewer can see, so the SPA hides nav entries rather than rendering links that 403. +### 3.6.1 What the first implementation missed (found by the §11 smoke test, fixed) + +Part A shipped enforcement on the SSE path and on `/guilds` + `/governors`, but the **remaining public +REST reads never called into it** — so the same event was projected live and served verbatim from +history. Recorded because each miss is a shape the next phase can repeat: + +- **`/public/shard/feed` returned the stored payload as-is.** `actor.acct` / `actor.webId` were + readable *anonymously* for every logged kind (`player.death`, `mob.killed`, `skill.gain`, + `guild.join`, …) — broader than the §3.1 leak, which was limited to board holders. +- **`/public/shard/idoc` returned `ownerAcct`.** Rule 1 keyed on the exact strings `acct`/`webId`, + but `shapeHouse` flattens the actor into `ownerAcct` / `ownerName` / `ownerSerial`. The lock is now + on the field's **meaning** — a key that is or ends in `acct`/`webId`, case-insensitively — so + flattened spellings are covered and unwritten shapes fail closed. +- **The `houses` field rules were dead config.** Neither `getIdoc` nor `getHouses` projected, so the + panel offered toggles that did nothing. **Every feature's declared fields must name the keys the + read model actually emits**, not just the wire frame's. +- **`/feed` filtered on `PUBLIC_KINDS`**, a module-load constant derived from the compiled defaults, + so live audience changes never reached it. `visibleKinds(level, config)` resolves the readable set + from live config; it deliberately ignores the `stream` flag, which governs SSE fan-out only (market + history stays readable with its firehose off). +- **`shardEvents.db.list` treated an empty `kinds` array as "no filter"** and fell through to an + unfiltered `SELECT`. A fully-gated config would have dumped the whole event log, staff audit + included. An empty allowlist now serves nothing. +- **`projectValue` recursed into every object**, so a `Date` column came back as `{}`. It walks + arrays and plain objects only. The unit tests used JSON fixtures and could not have caught this — + the live read did, which is the argument for §11's smoke test over tests alone. + +**The rule this leaves behind:** *a read path that returns shard data and does not call +`projectFeature` is a bug.* Every new surface in Parts B and C — `/ruleset`, `/points`, `/market`, +`/atlas` — must project, and must gate its kind set on live config rather than on `PUBLIC_KINDS`. + ### 3.7 Admin surface `GET` / `PUT /api/v1/admin/shard/visibility` (admin-only). Validate feature names against the known -set and rungs against the ladder; reject any attempt to set `acct`/`webId` below `admin`. Writes an +set and rungs against the ladder; reject any attempt to set a locked field below `admin` — including +its flattened spellings (`ownerAcct`, `leaderWebId`), see §3.6.1. Writes an `admin.audit`-style row so visibility changes are traceable. New client panel `routes/admin/ShardVisibility.jsx` at `/admin/shard-visibility`, linked from `ShardAdmin.jsx`. diff --git a/website/BACKEND_DESIGN.md b/website/BACKEND_DESIGN.md index c160e74..fb43fb9 100644 --- a/website/BACKEND_DESIGN.md +++ b/website/BACKEND_DESIGN.md @@ -711,6 +711,10 @@ rather than silently ignore: 1. **`acct` and `webId` are admin-only, always.** They are not exposed as configurable fields, and a stored row attempting to loosen them is discarded on read as well as rejected on write. A character name is visible in game; the account behind it and the website user it links to are not. + The lock is on the field's **meaning, not one spelling**: `isLockedField(key)` matches a key that + *is* or *ends in* `acct`/`webId`, case-insensitively, so the flattened forms the read models emit + (`shapeHouse` → `ownerAcct`, `shapeGuild` → `leaderWebId`) are covered too. An exact-key check was + the original implementation and it let `GET /public/shard/idoc` serve `ownerAcct` anonymously. 2. **A kind absent from `KIND_FEATURE` is never broadcast below `admin`.** Fail closed. This is what keeps the kind map a security boundary rather than a convenience filter, and it means a shard that starts emitting an unknown event degrades to staff-only, never to public. @@ -731,14 +735,31 @@ it picks, it fails open on one side.) | Nav | `GET /public/shard/features` returns only what the caller may reach, so the SPA never renders a link that would 403. Presentation only. | Config reads are cached ~5s, so admin changes take effect within seconds **including on already-open -streams**. `PUBLIC_KINDS` still exists and is still exported (`/feed` filtering, `notificationStreams.js`) -but is now **derived** from the kind map rather than hand-maintained, so the two cannot drift. +streams**. `PUBLIC_KINDS` still exists and is still exported (`notificationStreams.js`) but is now +**derived** from the kind map rather than hand-maintained, so the two cannot drift. + +**`PUBLIC_KINDS` is a module-load constant and must not be used to answer "may this caller read this +kind?"** — it is computed from the compiled *defaults*, so it cannot see an admin's changes. Use +`visibleKinds(level, config)`, which resolves against the live config. `/feed` uses it; it originally +used `PUBLIC_KINDS` and consequently kept serving `guild.join` to anonymous callers after an admin had +moved `guilds` to `staff`. `visibleKinds` deliberately ignores the `stream` flag: that governs SSE +fan-out only, so a feature whose live firehose ships off (market) stays readable from stored history. + +**Every read path that returns shard data must call `projectFeature`.** The stored-history endpoints +are not exempt — `/feed` returns the same events the stream does, and returning them unprojected +reopens on the REST side exactly what the stream closes. Relatedly, `shardEvents.db.list` treats an +**empty** `kinds` array as "serve nothing", never "no filter"; the fall-through it used to take would +have turned a fully-gated config into a dump of the entire event log. + +`projectFeature` walks **arrays and plain objects only**. A `Date`, `Buffer` or other class instance +is passed through as a value — rebuilding one key-by-key yields `{}`, which is the difference between +the pure-JSON wire frames and the DB-backed read models whose rows carry real `Date` columns. **Defaults reproduce pre-3.0 behavior exactly**, so installing the framework is a no-op until an admin -changes something — with one deliberate exception, which is the leak it was written to close: -`/public/shard/guilds` and `/public/shard/governors` previously returned the raw stored payload, whose -`leader` / `governor` actors carry `acct` and `webId`. Those fields are now stripped for every caller -below admin. +changes something — with deliberate exceptions, which are the leaks it was written to close. +`/public/shard/guilds`, `/public/shard/governors` and `/public/shard/feed` previously returned the raw +stored payload, whose actors carry `acct` and `webId`; `/public/shard/idoc` returned the flattened +`ownerAcct`. All are now stripped for every caller below admin. --- diff --git a/website/SHARD_VISIBILITY.md b/website/SHARD_VISIBILITY.md index 4f092f8..c8e301f 100644 --- a/website/SHARD_VISIBILITY.md +++ b/website/SHARD_VISIBILITY.md @@ -80,6 +80,11 @@ game to anyone standing next to them; the **account** behind it is not, and neit user it's linked to. Publishing those would disclose something the shard itself doesn't, and would tie a player's in-game identity to their forum identity without their consent. +This rule matches the *meaning* of a field, not one spelling of it. Some responses nest the player +who owns a record (`leader.acct`); others flatten it into the row (`ownerAcct`, `leaderWebId`, +`governorAcct`). Every one of those is locked, and the admin API refuses to configure any of them — +so a new response shape can't quietly reopen the hole by naming the field differently. + **2. Unknown event kinds are never broadcast below admin.** The live stream maps each event kind to a feature. A kind with no mapping — a new event from a shard plugin the site doesn't know yet, say — goes to admins only. It fails closed. This is what keeps the @@ -99,6 +104,15 @@ Three places, one config: - **Navigation** hides links a viewer can't follow, so they don't hit a wall. This is presentation only — the gate is server-side either way. +**Stored history answers the same way the live stream does.** The activity feed reads from the event +log rather than the live stream, but it resolves the *same* question against the *same* config: which +kinds you may read, and which fields survive. So moving a feature up a rung hides it from the history +as well as the stream — there is no back door where yesterday's copy of an event is more revealing +than today's. + +One deliberate asymmetry: turning **live updates** off for a feature stops the push, not the reading. +The marketplace ships this way — its history and its pages are public, only the firehose is off. + Changes take effect within about five seconds, **including on streams that are already open**. You don't need to restart anything. -- 2.49.1 From 4c0ceb1c411b05ac7a9cb3dbf888d117fcf9c08e Mon Sep 17 00:00:00 2001 From: wtclaude Date: Tue, 28 Jul 2026 10:53:01 -0500 Subject: [PATCH 04/19] chore(docs): drop an unrelated working-tree file committed by mistake android/TRUSTED_DEVICES_APP_HANDOFF.md was untracked in the working tree before this branch and was swept in by a `git add -A`. It is not part of this change; untracked here and left on disk. Co-Authored-By: Claude --- android/TRUSTED_DEVICES_APP_HANDOFF.md | 126 ------------------------- 1 file changed, 126 deletions(-) delete mode 100644 android/TRUSTED_DEVICES_APP_HANDOFF.md diff --git a/android/TRUSTED_DEVICES_APP_HANDOFF.md b/android/TRUSTED_DEVICES_APP_HANDOFF.md deleted file mode 100644 index 81ef644..0000000 --- a/android/TRUSTED_DEVICES_APP_HANDOFF.md +++ /dev/null @@ -1,126 +0,0 @@ -# Handoff — Trusted Devices & MFA: remaining work - -> Written 2026-07-22 at the end of the backend/web implementation session, for a -> fresh session to continue. Canonical design: `docs/website/TRUSTED_DEVICES_MFA.md`. -> App-side plan: `docs/android/PLAN.md §4.1.1`. This doc is the "what's left + how". - -## 1. Status at handoff - -**Done, in review, and live-smoke-tested** (real MariaDB + AVD): -- Backend (schema, session service, web + mobile login, self-service + admin - endpoints, invalidation), web client UI, admin front-end UI, OpenAPI spec, - 33 server tests — all in **website PR #93** (branch `feature/trusted-devices-mfa`). -- Docs (BACKEND_DESIGN §3/§4/§6, TRUSTED_DEVICES_MFA.md, PLAN §4.1.1) — **docs PR #32** - (branch `docs/trusted-devices-mfa`). -- Live smoke test passed end-to-end: trusted-device TOTP-skip (web + mobile - `X-Trust-Token`), recovery-code login (single-use), cap 409, admin MFA reset, - audit logging, and a real 2FA login through the Android app on an emulator. - -**Not done — the two remaining items below.** - -## 2. Remaining item A — merge gate (no code) - -- **website#93** and **docs#32** need CI green + review, then merge to `main`. -- CI (`.gitea/workflows/pr-checks.yml`) runs server tests, client build, bot install. -- Nothing to build here; just get them reviewed/merged. The Android work should land - **after** #93 merges so the app builds against the merged contract. - -## 3. Remaining item B — Android app implementation (the real work) - -The backend is fully ready and additive; the existing app is unaffected (verified). -The app just needs to *consume* the new endpoints. Own PR in the **`android-app`** -repo, branch `feature/trusted-devices-mfa`. Package id `com.runicgateway.app`. - -### 3.1 Scope (from PLAN §4.1.1) -1. **Login/TOTP screen:** on the existing `401 { totpRequired }` step, add - - a **"Trust this device"** checkbox, and - - a **"use a recovery code instead"** toggle (send `recoveryCode` instead of `code`). -2. **Trust token storage:** when a login response carries `trustToken`, store it in - **EncryptedSharedPreferences** (same secure store as the bearer tokens — never - plain prefs/logs). On subsequent logins send it as the **`X-Trust-Token`** header - so the server skips the TOTP prompt. -3. **Cap handling:** a login response with `{ trustLimitReached: true, devices }` - means show the device list and prompt the user to revoke one - (`DELETE /auth/me/trusted-devices/:id`) then retry trusting. -4. **Account screens:** - - **Trusted Devices**: list (`GET /auth/me/trusted-devices`), revoke one, untrust - all; "trust this device" (`POST /auth/me/trusted-devices` → returns `trustToken` - for native — store it). - - **Recovery Codes**: show the one-time batch returned by TOTP enable; remaining - count (`GET …/recovery-codes/status`); regenerate (`POST …/recovery-codes/generate`, - password step-up) with a show-once display + copy/share. -5. **Invalidation:** on logout / dead-refresh sign-out / Settings→Server switch, - **clear the stored `trustToken`** alongside the bearer tokens. (A server-side - password change/reset or TOTP disable already revokes it.) -6. **Tests:** JVM `:app:testDebugUnitTest` — DTO decode for the new fields, and - repository logic (trust-token persist/clear, recoveryCode vs code branch). - -### 3.2 Exact API contract the app consumes - -`POST /auth/mobile/login` — body `{ username, password, code?, recoveryCode?, trustDevice?, device_name? }`, optional header `X-Trust-Token: ` -- `200` → `{ accessToken, refreshToken, expiresIn, user:{id,username,role}, trustToken?, trustLimitReached?, devices? }` - - `trustToken` present only when `trustDevice:true` was accepted (store it). - - `trustLimitReached:true` + `devices[]` when at the cap (login still succeeded). -- `401` → `{ totpRequired:true, message }` (missing/invalid 2nd factor) — reveal the - code field (existing behavior); or generic `{ message }` for bad credentials. -- A valid `X-Trust-Token` bound to the user makes a code unnecessary → straight `200`. - -Self-service (Bearer access token): -- `GET /auth/me/trusted-devices` → `[{ id, platform, deviceName, userAgent, createdAt, lastUsedAt, expiresAt }]` -- `POST /auth/me/trusted-devices` body `{ deviceName? }` → `{ trusted:true, trustToken }` (native) | `409 { error:'trusted_device_limit', devices }` -- `DELETE /auth/me/trusted-devices/:id` → `{ revoked:boolean }` -- `DELETE /auth/me/trusted-devices` → `{ revoked:number }` -- `GET /auth/me/account/recovery-codes/status` → `{ remaining:number }` -- `POST /auth/me/account/recovery-codes/generate` body `{ currentPassword? }` → `{ recoveryCodes:[string] }` -- `POST /auth/me/account/totp/enable` body `{ code }` → `{ totp_enabled:true, recoveryCodes:[string] }` (codes shown once) - -The OpenAPI spec (`website/server/swagger/swagger-output.json`, schemas `TrustedDevice`, -`TrustDeviceResult`, `TrustedDeviceLimit`, `RecoveryCodes`) is the source of truth. - -### 3.3 Where it likely goes in the app -The app already handles `totpRequired` (reveals a code field — verified live), so the -auth surface exists. Extend the existing auth Retrofit API + repository + login -ViewModel/screen, add a secure `trustToken` accessor to the encrypted token store, -and add two account screens. Explore `android-app/app/src/main/java/com/runicgateway/app/` -(auth/data/core modules) at the start — don't assume file names. - -## 4. Environment & how-to (verified this session) - -- **DB:** Docker container `uomm-db`, MariaDB on host port **3307**, db `uomysticmoon`, - user `uomm` (password: `docker exec uomm-db printenv MARIADB_PASSWORD`). -- **Run the server:** `cd website/server && node src/server.js` (uses `.env`, already - points at 127.0.0.1:3307). It ensures schema + seeds on boot. For cap testing set - `MAX_TRUSTED_DEVICES=2`; recovery count via `RECOVERY_CODE_COUNT`. - - Pre-existing noise: `uo-link-socket … Unsupported state` decrypt errors are an old - encrypted `uo_link_config` row, unrelated — ignore. -- **Server tests:** `cd website/server && DB_HOST=127.0.0.1 DB_PORT=59999 node --test` - (dead port by design; models stubbed). Client: `cd website/client && npm test`. -- **TOTP codes for manual testing:** from `website/server`, - `node -e "process.stdout.write(require('speakeasy').totp({secret:'',encoding:'base32'}))"`. -- **Android build:** JDK 21; `cd android-app && ./gradlew :app:assembleDebug -Pksp.incremental=false`. - Unit tests: `./gradlew :app:testDebugUnitTest -Pksp.incremental=false`. -- **Emulator:** SDK at `~/AppData/Local/Android/Sdk`; AVDs `Medium_Phone_API_36.1`, - `s22_ultra`. `adb install -r -g app/build/outputs/apk/debug/app-debug.apk`. -- **⚠ Cleartext HTTP gotcha (dev only):** the app blocks plain HTTP to `10.0.2.2` - (`CLEARTEXT communication … not permitted`). Cleartext to `127.0.0.1` **is** allowed, - so for local testing run `adb reverse tcp:3000 tcp:3000` and set the app's server URL - to `http://127.0.0.1:3000` (its default). Production uses HTTPS via the proxy — not a - code issue. (Consider whether v1 wants a `network_security_config` dev exception; not - required for the feature.) - -## 5. Conventions (CLAUDE.md) - -- Branch from up-to-date `main`; commit/push as **`wtclaude`** using the token at - `C:\Users\colby\.gitea_token_claude` via `http.extraHeader` (never inline in URL). -- Conventional Commits; end commit messages with the `Co-Authored-By: Claude …` + - `Claude-Session:` trailers. PR template requires ticking the **AI-assisted** box - (tool: Claude Code) and the license box. -- Use `mcp__gitea__*` for PRs/issues. GPL-3.0-or-later. - -## 6. First moves for the fresh session -1. Check whether website#93 / docs#32 have merged (`mcp__gitea__pull_request_read`). -2. In `android-app`: sync `main`, `git checkout -b feature/trusted-devices-mfa`. -3. Explore the app's auth module; implement §3 against the §3.2 contract. -4. Test with the local server + emulator via the §4 cleartext workaround. -5. Open a single `android-app` PR; update `docs/android/PLAN.md` if the app design - deviates from §4.1.1. -- 2.49.1 From b0a2207c6ad96facd89037655213df7ac3922d9d Mon Sep 17 00:00:00 2001 From: wtclaude Date: Tue, 28 Jul 2026 14:39:53 -0500 Subject: [PATCH 05/19] docs(link): record world.ruleset and mark Protocol 3.0 progress MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Protocol 3.0 order 2 (v3.md §5) is built across all four repos; this is its documentation half, plus the running progress record the plan was missing. v3.md - A progress table at the top and a State column on §9's sequencing table, so "what has landed" is answerable without reading four git logs. Part A (order 1) and world.ruleset (order 2) are marked done; the spawn atlas is next. - §5 gains the implementation notes worth keeping, chiefly: where a system's on/off state is DERIVED rather than configured, read the system's own static instead of inventing a .cfg key (Shadowguard has no Enabled key — it's the TOL expansion gate; Factions is `!ViceVsVirtueSystem.Enabled` by construction in stock ServUO). Also that the plugin CAN be compile-verified despite the "no standalone build" caveat, and how. INTEGRATION.md - The world.ruleset catalog entry and GET /ruleset, with the two things consumers get wrong: caps are in TENTHS (1000 = 100.0), and `connect` exists only if the operator set Bridge.PublicConnectAddress — the shard's real listen address is never published. - §2 now says plainly that v3 has NOT been bumped yet and what that means: sidecars on `edge` report 2 while already carrying some v3 kinds, so do not infer feature availability from the version during this window. PROTOCOL_2.md §10.4 - The deferred "which PvP system does this shard run?" is answered (VvV on, Factions off — and mutually exclusive by construction), and world.systems is marked superseded by world.ruleset, which carries the systems block it asked for. No orphan kind is left behind. BACKEND_DESIGN.md — the shard_ruleset table (why it is stored whole rather than normalized, and why no row means null rather than {}) and the public route. PROJECT_TREE.md is deliberately untouched: sync-project-tree regenerates it on push to main, so it updates itself at the v3 cutover. Co-Authored-By: Claude --- link/INTEGRATION.md | 82 +++++++++++++++++++++++++++++++++++++++ link/PLAN.md | 13 +++++++ link/PROTOCOL_2.md | 17 +++++++- link/v3.md | 61 +++++++++++++++++++++++------ website/BACKEND_DESIGN.md | 20 +++++++++- 5 files changed, 180 insertions(+), 13 deletions(-) diff --git a/link/INTEGRATION.md b/link/INTEGRATION.md index ef9105f..f46e686 100644 --- a/link/INTEGRATION.md +++ b/link/INTEGRATION.md @@ -43,6 +43,20 @@ Pin the version you built against and compare it to the header (or `/health.prot **v2 (Protocol 2.0)** added the account-provisioning surface (§6.x: `POST /accounts/create`, `DELETE /link/{account}`) and the `account.*` events. Outbound event kinds are **additive** — a v1 client that ignores unknown kinds keeps working against the live feed — but the new *endpoints* require a v2 sidecar. If you send `X-UOLink-Version: 1`, calls to the new endpoints are refused with the 409 above. +**v3 (Protocol 3.0) is being built and the version has not been bumped yet.** It is defined as *adds +`world.ruleset`, `points.board`, `vendor.listing` / `vendor.listing.remove`*, and the bump to +`X-UOLink-Version: 3` happens **exactly once**, at the end, when [`v3.md`](v3.md) §4's `edge` → `main` +cutover lands — because a bump is an operator-visible hard break (409 on every protected route, and +the website's WS closes on the `ws.hello` mismatch), so doing it per phase would break the site +repeatedly. + +Until then, sidecars on `edge` still report `2` while already carrying some v3 kinds and endpoints. +That is safe in the direction that matters: event kinds are additive, and a client that ignores +unknown kinds and tolerates a `404` on a not-yet-present endpoint keeps working. What you must **not** +do is infer feature availability from the version number during this window — probe the endpoint, or +treat a missing `world.ruleset` as "this shard hasn't published one". There is deliberately **no +feature-negotiation array**: v3 implies all three kinds. + --- ## 3. Health @@ -315,6 +329,58 @@ The house registry — one row per house, complementing the `house.decay` *trans Render from `GET /houses` (§6) on connect, then keep live with these events. +#### Shard ruleset (Protocol 3.0) + +How the shard is actually configured, published by the shard itself. **Not a sweep** — it changes only +when an operator edits `Config/*.cfg`, so it is emitted once per shard↔sidecar connect (and on +`[bridge reload`), exactly like `server.hello`. + +| kind | fields | notes | +|------|--------|-------| +| `world.ruleset` | `rev`, `shard`, `expansion`, `connect?`, `systems`, `caps`, `housing`, `accounts`, `vetRewards`, `loot`, `vendors`, `champions?`, `treasureMaps`, `vvv?`, `store`, `schedule?` | The whole ruleset, always complete — **never a delta**, so the latest frame replaces the previous one outright. Every block except `shard`/`expansion` is optional and is **omitted when its system is off**, so absence means "not applicable here", not "unknown". | + +`rev` is the shard's FNV-1a of the body: identical `rev` means the ruleset is unchanged and this frame +is just a reconnect re-send, so a consumer can skip the write. It is deliberately **not** +`String.GetHashCode()`, which is seeded per process and would change on every shard restart. + +```json +{"kind":"world.ruleset","rev":"1a2b3c4d","shard":"UOMysticmoon","expansion":"EJ", + "systems":{"cityLoyalty":true,"vvv":true,"factions":false,"siege":false,"chat":true, + "store":true,"dailyRares":true,"honesty":true,"shadowguard":true, + "treasureMaps":true,"vetRewards":true,"testCenter":false}, + "caps":{"skill":1000,"totalSkill":7000,"stat":225,"str":125,"dex":125,"int":125, + "strMax":150,"dexMax":150,"intMax":150}, + "housing":{"accountHouseLimit":1}, + "accounts":{"perIp":3,"charSlots":7,"autoCreate":true}, + "vetRewards":{"enabled":true,"rewardIntervalDays":30}, + "loot":{"feluccaLuckBonus":1000,"feluccaBudgetBonus":100,"feluccaMaxProps":11}, + "vendors":{"restockDelayMinutes":60,"maxSell":500,"economyStockAmount":500}, + "champions":{"powerScrolls":6,"statScrolls":16,"scrollChance":0.1, + "transcendenceChance":50.0,"rankThresholds":[5,10,13]}, + "treasureMaps":{"enabled":true,"lootChance":0.01,"resetDays":30}, + "vvv":{"enabled":true,"startSilver":2000,"enhancedRules":false}, + "store":{"enabled":true,"currencyName":"Sovereigns"}, + "schedule":{"autoSaveEnabled":true,"autoSaveFrequencyMinutes":15,"autoRestartEnabled":false}, + "t":1752489280000} +``` + +**Two things consumers get wrong.** + +1. **`caps.skill` and `caps.totalSkill` are in tenths**, the way ServUO stores them: `1000` is `100.0` + skill and `7000` is `700.0` total. Rendering the raw number is actively misleading. The other caps + (`stat`, `str`, …) are plain integers. +2. **`connect` is present only if the operator set `Bridge.PublicConnectAddress`.** The shard's real + listen address (`Server.cfg`) is never published; nor are `Staff.cfg`, `Email.cfg`, `DataPath.cfg`, + `Bridge.cfg`, `Compiler.cfg`, `Reports.cfg` or `Client.cfg`. The frame is built from an explicit + allowlist in `BridgeRuleset.cs` — `Config.Entries` is never enumerated, because that would sweep in + every key on the server. + +Absent entirely if the shard runs `Bridge.RulesetEnabled=false` or an older plugin. Render from +`GET /ruleset` (§6) on connect, then keep live with this event. + +This **supersedes the `world.systems` frame** sketched in [`PROTOCOL_2.md`](PROTOCOL_2.md) §10.4 and +never implemented; the `systems` block above is what that asked for. + --- ## 5. REST — read queries @@ -658,6 +724,22 @@ GET /houses Every house's latest snapshot — owner→houses map. Served from the sidecar's projection, kept current by the `house.*` stream (§4). Ordered by name. Survives a sidecar restart. +### Shard ruleset (Protocol 3.0) + +``` +GET /ruleset +→ { "ruleset": {"kind":"world.ruleset","rev":"1a2b3c4d","shard":"UOMysticmoon", + "expansion":"EJ","systems":{...},"caps":{...},"accounts":{...}, ... } } +``` + +The shard's published ruleset (§4 for the full frame and its two gotchas). Served from the sidecar's +store, so it **answers while the shard is down** — a rules page that goes blank during a restart is +worse than one that is briefly stale. Keep it current with the `world.ruleset` stream. + +`{"ruleset": null}` means the shard has never published one — an older plugin, or +`Bridge.RulesetEnabled=false`. That is a real answer distinct from a published ruleset, and worth +rendering differently ("not published yet") rather than as an empty ruleset. + --- ## 7. Status codes diff --git a/link/PLAN.md b/link/PLAN.md index fff21e0..cb36025 100644 --- a/link/PLAN.md +++ b/link/PLAN.md @@ -315,6 +315,14 @@ Counts in `hello` are a live snapshot taken on the Core thread, not a cached val 7. **Core edit: `PlayerVendorSale`** (§6). Then the cheat-detection feed. 8. **Cheat signals.** `FastWalk`, `OnPropertyChanged` audit, vendor-sale anomaly detection in the sidecar. +**Beyond 1.0.** Phases above are the 1.0 read/event plane. Protocol 2.0's phasing (provisioning + +world-state boards) is [`PROTOCOL_2.md`](PROTOCOL_2.md) §13; Protocol 3.0's (visibility framework, +shard content and standings) is [`v3.md`](v3.md) §9, which also tracks what has landed. Shipped from +3.0 so far: **Part A** — the visibility framework — and **`world.ruleset`** ([`v3.md`](v3.md) §5), +`BridgeRuleset.cs`, the first bridge stream that is neither an event subscription nor a sweep: it is +emitted once per connect, like `server.hello`, because shard config changes only when an operator +edits a file. + ### Config keys (`Config/Bridge.cfg`) ```ini @@ -328,6 +336,11 @@ EconomySweepSeconds=300 Read in `Configure()` via `Config.Get("Bridge.", default)`. Key scope is the filename: `Bridge.cfg` + `StatSweepSeconds` → `Bridge.StatSweepSeconds`. +The set above is the 1.0 sample, not the current one — every later phase added keys (sweep intervals +for each board, the town-crier/news caps, the admin write plane, account provisioning, and 3.0's +`RulesetEnabled` / `PublicConnectAddress` / `RulesetIncludeSchedule`). **`servuo-plugins/overlay/Config/Bridge.cfg` +is the authoritative, commented list**; `BridgeConfig.cs` holds the defaults. + --- ## 11. Phase 1 acceptance diff --git a/link/PROTOCOL_2.md b/link/PROTOCOL_2.md index bc0dec7..cd9944c 100644 --- a/link/PROTOCOL_2.md +++ b/link/PROTOCOL_2.md @@ -285,6 +285,18 @@ City titles and faction/VvV merchant titles (`CityLoyaltySystem.ApplyCityTitle`, ### 10.4 Factions / Vice vs Virtue +> **Status update (Protocol 3.0, 2026-07-28).** +> +> - **The deferred question is answered.** This shard runs **Vice vs Virtue** (`VvV.cfg Enabled=True`); +> old Factions is off, and in stock ServUO that is not a coincidence — +> `Services/Factions/Core/Faction.cs` sets `Settings.Enabled = !ViceVsVirtueSystem.Enabled`, so the +> two are mutually exclusive by construction. The `vvv.standings` / `vvv.battle` streams below are +> therefore unblocked, but are **not** scoped for 3.0 (see [`v3.md`](v3.md) §2 row 5). +> - **`world.systems` is superseded by `world.ruleset`** ([`v3.md`](v3.md) §5), which shipped in 3.0. +> It was never implemented under this name. `world.ruleset` carries the same +> `systems{cityLoyalty, vvv, factions, …}` sub-object this section asked for, plus the rest of the +> shard's published ruleset, so no orphan kind is left behind. Do not implement `world.systems`. + **Which system is live is a shard decision — verify before building.** Two exist: - **Old Factions** (`Scripts/Services/Factions`): `Faction.Commander` (leader, `Faction.cs:160`), `Faction.Election`, `Faction.Members` (`List`), and faction-controlled **Towns** (`Town.cs` — each town has an owning faction, a sheriff, and finance). Config-gated and, on most modern shards, **off**. @@ -300,7 +312,10 @@ City titles and faction/VvV merchant titles (`CityLoyaltySystem.ApplyCityTitle`, {"kind":"vvv.standings","order":142000,"chaos":138500,"leaderSide":"Order"} ``` -> Start by detecting which system is enabled at boot and streaming only that one; emit a one-time `world.systems` frame (what's on: cityLoyalty, vvv, factions) so the website renders the right panels instead of guessing. +> Start by detecting which system is enabled at boot and streaming only that one. ~~emit a one-time +> `world.systems` frame (what's on: cityLoyalty, vvv, factions) so the website renders the right panels +> instead of guessing.~~ — **superseded: `world.ruleset` already carries that `systems` block** (see the +> status note at the top of this section). ## 11. Further integration points — a menu to pick from diff --git a/link/v3.md b/link/v3.md index e969e84..17e99c8 100644 --- a/link/v3.md +++ b/link/v3.md @@ -1,10 +1,23 @@ # Protocol 3.0 — Shard content, standings & the visibility framework -**Status:** Planned, approved 2026-07-28. Not yet built. All work lands on an `edge` branch in each repo; `edge` → `main` is the v3 cutover. +**Status:** In progress. All work lands on an `edge` branch in each repo; `edge` → `main` is the v3 cutover. **Date:** 2026-07-28 **Codebase:** ServUO 57.4, ``, net48 / x64, Expansion **EJ**. **Companion to** [`PLAN.md`](PLAN.md) (1.0 read/event plane), [`PROTOCOL_2.md`](PROTOCOL_2.md) (2.0 provisioning + world-state streams), [`ADMIN_CONTROLS.md`](ADMIN_CONTROLS.md) (staff write plane), [`INTEGRATION.md`](INTEGRATION.md) (website API). +### Progress + +Each part is marked off here as it lands on `edge`. §9 carries the same state per sequencing row. + +| Order | Part | State | Landed on `edge` | +|---|---|---|---| +| 1 | **A** — visibility framework + actor-leak fix (§3) | ✅ **Done** | website [#109](https://gitea.whitlocktech.com/RunicGateway/website/pulls/109) + [#110](https://gitea.whitlocktech.com/RunicGateway/website/pulls/110), docs [#64](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/64) + [#65](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/65) | +| 2 | **B/1** — `world.ruleset` (§5) | ✅ **Done** | PR-LINKS-B1 | +| 3 | **C** — spawn atlas (§6) | ⬜ **Next** | — | +| 4 | **B/2** — `points.board` (§7) | ⬜ Not started | — | +| 5 | **B/3** — `vendor.listing` (§8) | ⬜ Not started | — | +| 6 | **Cutover** — `PROTOCOL_VERSION` 2→3 (§4) | ⬜ Not started | — | + --- ## 1. Why 3.0 @@ -64,7 +77,12 @@ independently of this work. --- -## 3. Part A — The visibility framework +## 3. Part A — The visibility framework ✅ Done + +*Landed on `edge`: website [#109](https://gitea.whitlocktech.com/RunicGateway/website/pulls/109) (the framework) +and [#110](https://gitea.whitlocktech.com/RunicGateway/website/pulls/110) (the REST-projection gap §3.6.1 +records), docs [#64](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/64) + [#65](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/65). +Smoke-tested across all five rungs per §11.* ### 3.1 The leak this replaces (verified 2026-07-28) @@ -221,7 +239,28 @@ admin-set `uo_link_config.protocol` column — so it happens **exactly once**, a --- -## 5. Part B/1 — `world.ruleset` +## 5. Part B/1 — `world.ruleset` ✅ Done + +*Landed on `edge`. Implementation notes worth keeping:* + +- ***`shadowguard` is derived, not configured.*** `Shadowguard.cfg` carries only `ReadyDuration` and + `RandomizeInstances` — there is no `Enabled` key — so the systems block reports `Core.TOL` + (the expansion gate) instead. Same shape for `factions`: `Factions.cfg` has no `Enabled` either, and + `Services/Factions/Core/Faction.cs` sets `Settings.Enabled = !ViceVsVirtueSystem.Enabled`, so the + frame reads that static rather than inventing a key. **Where a system's on/off state is derived, read + the system's own static; only read `Config.Get` where the .cfg key IS the truth.** +- **`caps.skill` / `caps.totalSkill` are in tenths** (1000 = 100.0), the way ServUO stores them. + Documented in `INTEGRATION.md` and converted in the client, because the raw number is actively + misleading rather than merely unhelpful. +- **`Config.Get` re-parses when the cached type differs.** `InternalGet` caches the parsed value on + the entry and re-parses if `entry.Object is T` fails, so reading `PlayerCaps.SkillCap` as an `int` + where ServUO reads it as a `double` is correct (both parse) — it just re-parses. Harmless, but worth + knowing before assuming a shared cache. +- **The plugin CAN be compile-verified**, contrary to "no standalone build": point Roslyn + (`dotnet sdk/*/Roslyn/bincore/csc.dll`, `/langversion:7.3`, net48 reference assemblies) at the whole + ServUO `Scripts` tree with `overlay/Scripts/Custom/Bridge/*.cs` substituted for the deployed copy, + excluding `Scripts/obj` and `Scripts/bin`. 6,205 files, ~40 s, and it catches every signature error + a boot would. Worth doing before every plugin PR. `PROTOCOL_2.md` §10.4 sketches a `world.systems` capability frame that was never implemented (`grep` returns nothing across all four repos). **`world.ruleset` subsumes it**, carrying a `systems` @@ -534,14 +573,14 @@ inherently up to one full cycle old, and the UI must say so. ## 9. Sequencing -| Order | Part | Repos touched | Wire change | -|---|---|---|---| -| 1 | **A** — visibility framework + actor-leak fix | website, docs | none | -| 2 | **B/1** — `world.ruleset` (§5) | all four | new kind | -| 3 | **C** — spawn atlas (§6) | website, docs | none | -| 4 | **B/2** — `points.board` (§7) | all four | new kind + `char.profile` field | -| 5 | **B/3** — `vendor.listing` (§8) | all four | new kinds | -| 6 | **Cutover** — `PROTOCOL_VERSION` 2→3, `edge` → `main` | all four | the bump | +| Order | Part | Repos touched | Wire change | State | +|---|---|---|---|---| +| 1 | **A** — visibility framework + actor-leak fix | website, docs | none | ✅ Done | +| 2 | **B/1** — `world.ruleset` (§5) | all four | new kind | ✅ Done | +| 3 | **C** — spawn atlas (§6) | website, docs | none | ⬜ **Next** | +| 4 | **B/2** — `points.board` (§7) | all four | new kind + `char.profile` field | ⬜ | +| 5 | **B/3** — `vendor.listing` (§8) | all four | new kinds | ⬜ | +| 6 | **Cutover** — `PROTOCOL_VERSION` 2→3, `edge` → `main` | all four | the bump | ⬜ | --- diff --git a/website/BACKEND_DESIGN.md b/website/BACKEND_DESIGN.md index fb43fb9..7b4c16c 100644 --- a/website/BACKEND_DESIGN.md +++ b/website/BACKEND_DESIGN.md @@ -99,7 +99,7 @@ server/ pages.router.js (2) /public/pages — the draft-preview route precedes /:slug and is deliberately not site-mode gated - shard.router.js (13) /public/shard/* incl. the anonymous + shard.router.js (14) /public/shard/* incl. the anonymous SSE stream; never site-mode gated site.router.js (4) /settings /status /version /contact — the group-root singletons; declares no @@ -358,6 +358,23 @@ analogue to a password — and there is no hash-lookup constraint (verification unused rows and `bcrypt.compare`s each, like password verification). `used_at` is the single-use marker. Cleared wholesale on TOTP disable / password change / password reset. +### shard_ruleset — the shard's published ruleset (Protocol 3.0) + +Singleton row (`id = 1`, CHECK-constrained) holding the latest `world.ruleset` frame: `rev`, +`expansion`, `payload` JSON (the whole frame), `t`, `updated_at`. The shard re-emits the complete +ruleset on every sidecar connect, so this is an **overwrite, not an append** — and the kind is +deliberately **not** in `LOGGED_KINDS`, since logging it would put a duplicate row in `shard_events` +on every reconnect while `server.hello` already marks each of those. + +The frame is stored whole rather than normalized into columns: it is a flat description of server +config that is read as one page, so splitting it up would mean a schema change every time the shard +grows a new block. `rev` (the shard's FNV-1a of the body) and `expansion` are hoisted only because +they are cheap to display — the same payload-plus-hoisted-columns shape `shard_champs` uses. + +**No row means the shard has never published one** (an older plugin, or `Bridge.RulesetEnabled=false`), +served as `null` rather than `{}`: "not published yet" and "published, everything off" are different +answers and the page renders them differently. + ### shard_feature_visibility — per-feature audience config (Protocol 3.0) One row per shard feature: `feature` (PK), `enabled`, `audience` (a rung on the ladder in §6.5), @@ -569,6 +586,7 @@ from the per-route **siteMode** middleware (§5), never from an auth gate. | GET | `/wiki` | list of pages (slug + title) | | GET | `/wiki/:slug` | single page | | POST | `/contact` | (rate-limited) send mail via SMTP; if unconfigured, respond `{fallback:"mailto", email}` | +| GET | `/shard/ruleset` | the shard's own published ruleset (Protocol 3.0 `world.ruleset`): expansion, which optional systems are on, skill/stat caps, account and house limits, champion scroll rules, the save/restart schedule. Served from `shard_ruleset`, so it renders while the shard is down; live via `world.ruleset` on `/shard/stream`. Behind `requireFeature('ruleset')`. **`null`** means the shard has never published one — a real answer, distinct from a published ruleset. `caps.skill` / `caps.totalSkill` are in **tenths** (1000 = 100.0). | | GET | `/shard/features` | the shard features **this caller** may reach plus the audience rung they resolved to (§6.5), so a client hides nav it can't follow. Reports only what the caller can see — the list itself never discloses a gated feature. Consumed by the SPA header and (pending) the Android nav. | Public content GETs pass through the **siteMode** gate (§5). -- 2.49.1 From 09467c67b00261db6d6e454b333eb2bf00aaf206 Mon Sep 17 00:00:00 2001 From: wtclaude Date: Tue, 28 Jul 2026 14:41:54 -0500 Subject: [PATCH 06/19] docs(link): link the world.ruleset PRs from the v3 progress table Co-Authored-By: Claude --- link/v3.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/link/v3.md b/link/v3.md index 17e99c8..6e3b916 100644 --- a/link/v3.md +++ b/link/v3.md @@ -12,7 +12,7 @@ Each part is marked off here as it lands on `edge`. §9 carries the same state p | Order | Part | State | Landed on `edge` | |---|---|---|---| | 1 | **A** — visibility framework + actor-leak fix (§3) | ✅ **Done** | website [#109](https://gitea.whitlocktech.com/RunicGateway/website/pulls/109) + [#110](https://gitea.whitlocktech.com/RunicGateway/website/pulls/110), docs [#64](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/64) + [#65](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/65) | -| 2 | **B/1** — `world.ruleset` (§5) | ✅ **Done** | PR-LINKS-B1 | +| 2 | **B/1** — `world.ruleset` (§5) | ✅ **Done** | servuo-plugins [#3](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/3), link [#17](https://gitea.whitlocktech.com/RunicGateway/link/pulls/17), website [#111](https://gitea.whitlocktech.com/RunicGateway/website/pulls/111), docs [#66](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/66) | | 3 | **C** — spawn atlas (§6) | ⬜ **Next** | — | | 4 | **B/2** — `points.board` (§7) | ⬜ Not started | — | | 5 | **B/3** — `vendor.listing` (§8) | ⬜ Not started | — | @@ -241,7 +241,7 @@ admin-set `uo_link_config.protocol` column — so it happens **exactly once**, a ## 5. Part B/1 — `world.ruleset` ✅ Done -*Landed on `edge`. Implementation notes worth keeping:* +*Landed on `edge`: servuo-plugins [#3](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/3), link [#17](https://gitea.whitlocktech.com/RunicGateway/link/pulls/17), website [#111](https://gitea.whitlocktech.com/RunicGateway/website/pulls/111), docs [#66](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/66). Implementation notes worth keeping:* - ***`shadowguard` is derived, not configured.*** `Shadowguard.cfg` carries only `ReadyDuration` and `RandomizeInstances` — there is no `Enabled` key — so the systems block reports `Core.TOL` -- 2.49.1 From 3fb3f63f255ac77d3f348dc4f24956757264721d Mon Sep 17 00:00:00 2001 From: wtclaude Date: Tue, 28 Jul 2026 16:13:30 -0500 Subject: [PATCH 07/19] docs(website): record the spawn atlas pipeline and what real data changed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Protocol 3.0 order 3 (Part C), docs half of website #112. Part C is website-only — no plugin, no sidecar, no new kinds, no wire change. ## New: website/SPAWN_ATLAS.md The operator-facing reference: the build/import split and why it exists (build needs a ServUO tree, import does not, and the container has the artifact but not the tree), the re-run story, the artifact format, the placement transform, and the three quirks in the source data that are silent when unhandled. Also documents the artwork policy explicitly: **the project ships no creature art and no extraction tooling.** Sprites live in the operator's own client .mul/.uop files and are theirs, not ours to redistribute. `art` is nullable and NULL on every fresh import; an operator who wants art extracts it themselves into the gitignored uploads/atlas/ and maps slugs in a gitignored art map. Text-only is the normal, supported state — not a degraded one. ## New: v3.md §6.1 — what the build against real data changed Six corrections, kept as a diff rather than edited into §6 in place, because each is a trap the next person would otherwise re-enter: 1. **Six facets, not thirteen.** Eodon.xml and the other named-area files carry TerMur/Trammel points; the facet comes from each record's ``. 2. **The XML dependency call resolved: hand-rolled, zero deps.** §6 left fast-xml-parser vs a tokenizer open. 3. **Facet names disagree between sources** — Locations says `Ter Mur`, `` says `TerMur`. Unreconciled the landmark fallback never fires there and every unregioned Ter Mur/Tokuno spawn silently reads "Wilderness". 4. **Spawn type tokens carry XmlSpawner directives** (`Fairy,{RND,4,8}`, `alchemist/z/-50`). Taken literally they invent creatures that do not exist and split real ones in two. 71 of 845 affected; 800 remain after stripping. 5. **The artifact is 1.41 MB, not "well under 1 MB"** — down from 4.40 MB via three encodings. Getting under 1 MB would mean dropping the spawner name. 6. **DELETE, not TRUNCATE** — TRUNCATE is DDL in MariaDB and implicitly commits, which would defeat the all-or-nothing reload the design asked for. §6 also now records that Part C ships as two website PRs: the parsing half is where the correctness risk lives and should not be reviewed inside a 10k-line diff alongside routes and React. ## BACKEND_DESIGN.md The seven atlas tables, the import-owned contract, the four column choices that are traps (`spawn_range`/`grp` reserved words, DELETE vs TRUNCATE, explicit point ids, plain INDEX not FULLTEXT), and the distinction between the configured champion roster and the live champ.update feed. PROJECT_TREE.md is left alone — it is auto-generated by the sync-project-tree workflow. --- - [x] AI-assisted: written with **Claude Code** (Claude Opus 5), reviewed before opening. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01U7CBg11prhLimL9iHSX1bP --- link/v3.md | 60 +++++++++- website/BACKEND_DESIGN.md | 41 +++++++ website/SPAWN_ATLAS.md | 225 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 324 insertions(+), 2 deletions(-) create mode 100644 website/SPAWN_ATLAS.md diff --git a/link/v3.md b/link/v3.md index 6e3b916..79e9939 100644 --- a/link/v3.md +++ b/link/v3.md @@ -13,7 +13,7 @@ Each part is marked off here as it lands on `edge`. §9 carries the same state p |---|---|---|---| | 1 | **A** — visibility framework + actor-leak fix (§3) | ✅ **Done** | website [#109](https://gitea.whitlocktech.com/RunicGateway/website/pulls/109) + [#110](https://gitea.whitlocktech.com/RunicGateway/website/pulls/110), docs [#64](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/64) + [#65](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/65) | | 2 | **B/1** — `world.ruleset` (§5) | ✅ **Done** | servuo-plugins [#3](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/3), link [#17](https://gitea.whitlocktech.com/RunicGateway/link/pulls/17), website [#111](https://gitea.whitlocktech.com/RunicGateway/website/pulls/111), docs [#66](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/66) | -| 3 | **C** — spawn atlas (§6) | ⬜ **Next** | — | +| 3 | **C** — spawn atlas (§6) | 🟡 **Data pipeline done** | website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112) (parsers + CLI + tables); API/client PR next | | 4 | **B/2** — `points.board` (§7) | ⬜ Not started | — | | 5 | **B/3** — `vendor.listing` (§8) | ⬜ Not started | — | | 6 | **Cutover** — `PROTOCOL_VERSION` 2→3 (§4) | ⬜ Not started | — | @@ -329,6 +329,15 @@ frame during verification. **No plugin, no sidecar, no `Bridge.cfg` knob, no new kinds.** Not part of the v3 wire change. +> **Status:** data pipeline landed on `edge` — website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112) +> (parsers, build/import CLI, tables, artifact). API + client pages are the second website PR. +> Part C ships as **two** website PRs, not one: the parsing half is where the correctness risk +> lives, and burying it under routes and React would have meant reviewing it in a 10k-line diff. +> Full operator documentation: [`docs/website/SPAWN_ATLAS.md`](../website/SPAWN_ATLAS.md). +> +> §6.1 below records where the shipped implementation differs from this design. The differences are +> all things the real ServUO data forced, not changes of mind. + **Decision: committed generated artifact + idempotent DB import**, split in two because the build needs the ServUO tree (which the website container does not have) and the import does not. Not runtime import (10.5 MB of XML per boot), not a browser-served blob. @@ -383,6 +392,53 @@ with the tree → commit the regenerated `db/data/spawnAtlas.*.json` → deploy `GET /admin/shard/atlas/status` reports when the DB is behind the artifact. Full detail in `docs/website/SPAWN_ATLAS.md`. +### 6.1 What the build against real data changed + +Six corrections to the design above, from running it against stock ServUO 57.4. Kept as a diff +rather than edited in place, because each one is a trap the next person would otherwise re-enter. + +**1. Six facets, not thirteen.** The design said `spawnAtlas..json ×13`, assuming one facet +per spawn file. There are 13 files but only **6** facets — `Eodon.xml`, `GravewaterLake.xml`, +`TreasuresOfKotl.xml` and the other named-area files carry TerMur/Trammel points. The facet comes +from each record's own ``, never the file name, and the artifact shards 6 ways. + +**2. The XML dependency call: hand-rolled, zero deps.** §6 left `fast-xml-parser` vs a ~120-line +tokenizer open. Resolved as the tokenizer — a deliberate *subset* parser covering only what these +files use. The server keeps zero XML dependencies at any tier. + +**3. Facet names disagree between sources — a silent failure.** `Data/Locations/*.xml` spells them +`Ter Mur` and `Tokuno Islands`; `` and `` say `TerMur` and `Tokuno`. Unreconciled, +the landmark bucket is keyed differently from the points looking it up, so the fallback never fires +and **every unregioned spawn in Ter Mur and Tokuno reads "Wilderness"** — a plausible-looking atlas +that is quietly wrong for two facets. All facet names now pass through `normalizeFacet()`. + +**4. Spawn type tokens carry XmlSpawner directives.** `` types are not always bare class +names: `Fairy,{RND,4,8}`, `alchemist/z/-50`, `Agralem/Name/Agralem`, `greatape,true`. Taken literally +they invent creatures that do not exist *and* split real ones in two, since `Fairy` and +`Fairy,{RND,4,8}` slug apart. 71 of 845 entries were affected; stripping at the first `/` or `,` +leaves **800** real creatures. (The design's "~1,500 creature rows" estimate was high; 800 only +reinforces the plain-`INDEX`-not-`FULLTEXT` call.) + +**5. The artifact is 1.41 MB, not "well under 1 MB".** Dropping the unused `` fields as the +design directed still left 4.40 MB. Three further encodings — `facet` dropped per record, +default-valued fields omitted rather than written as `0`, and `types` as `[name, max]` tuples +(~24,000 entries × 15 bytes of repeated key names) — brought it to 1.41 MB. Getting under 1 MB would +mean dropping the spawner `name`, which is the only human handle on a specific spawner and worth +keeping. `encodePoint()` and `readPoint()` are exact inverses and are round-tripped in tests. + +**6. `DELETE`, not `TRUNCATE`.** The design said "TRUNCATE + batched INSERT in one transaction", +which does not hold: `TRUNCATE` is DDL in MariaDB and implicitly commits, so a mid-import failure +would leave the atlas half-loaded. `DELETE` is transactional, and at ~7k rows the cost is +irrelevant. Point ids are also assigned explicitly rather than by `AUTO_INCREMENT`, because the +join rows need them and `conn.batch()` reports no usable `insertId`. + +**Measured result:** 6,455 points, 800 creatures, 23,927 point/type rows, 387 regions, 558 +landmarks, 25 champion altars. The placement transform resolves **83.2%** of points (3,689 by +region, 1,690 by landmark, 1,086 Wilderness). + +**One thing the design got exactly right:** the point-in-rect transform really is the reason to +build this. "Where does a lizardman spawn?" answers *Shrines, Isamu-Jima, Yew* across three facets. + --- ## 7. Part B/2 — `points.board` @@ -577,7 +633,7 @@ inherently up to one full cycle old, and the UI must say so. |---|---|---|---|---| | 1 | **A** — visibility framework + actor-leak fix | website, docs | none | ✅ Done | | 2 | **B/1** — `world.ruleset` (§5) | all four | new kind | ✅ Done | -| 3 | **C** — spawn atlas (§6) | website, docs | none | ⬜ **Next** | +| 3 | **C** — spawn atlas (§6) | website, docs | none | 🟡 Pipeline done, API/client next | | 4 | **B/2** — `points.board` (§7) | all four | new kind + `char.profile` field | ⬜ | | 5 | **B/3** — `vendor.listing` (§8) | all four | new kinds | ⬜ | | 6 | **Cutover** — `PROTOCOL_VERSION` 2→3, `edge` → `main` | all four | the bump | ⬜ | diff --git a/website/BACKEND_DESIGN.md b/website/BACKEND_DESIGN.md index 7b4c16c..cdffd8c 100644 --- a/website/BACKEND_DESIGN.md +++ b/website/BACKEND_DESIGN.md @@ -388,6 +388,47 @@ feature is ignored (a stale row must not resurrect a removed feature), an invali the default rather than failing open, and a rule touching a locked field (`acct` / `webId`) is discarded. See §6.5. +### shard_spawn_* / shard_regions / shard_landmarks / shard_champion_spawns / shard_atlas_meta — the spawn atlas (Protocol 3.0) + +Static shard **content**, not live shard state. Nothing here comes from the sidecar: the atlas is +built from a ServUO tree by a CLI script, committed as JSON under `server/db/data/`, and loaded with +`npm run atlas:import`. These tables stay populated whether the shard is up or not. Full operator +detail in [`SPAWN_ATLAS.md`](SPAWN_ATLAS.md); the design is `docs/link/v3.md` §6. + +| Table | Key columns | +|---|---| +| `shard_spawn_creatures` | `slug` PK, `name`, `total`, `points`, `facets` JSON, `art` NULL | +| `shard_spawn_points` | `id` PK, `facet`, `name`, `x`, `y`, `width`, `height`, `spawn_range`, `max_count`, `min_delay`, `max_delay`, `tod_start/end/mode`, `region`, `landmark`, `label` | +| `shard_spawn_point_types` | `(point_id, slug)` PK, `max_count` | +| `shard_regions` | `facet`, `name`, `type`, `priority`, `parent`, `rects` JSON | +| `shard_landmarks` | `facet`, `name`, `grp`, `x`, `y`, `z` | +| `shard_champion_spawns` | `slug` PK, `name`, `grp`, `type`, `random_type`, `facet`, `x`, `y`, `z`, `radius`, `label` | +| `shard_atlas_meta` | Singleton (`id = 1`), `payload` JSON, `imported_at` | + +All of them are **import-owned**: `atlas:import` empties and reloads every one inside a single +transaction, so a failed import leaves the previous atlas intact rather than a half-loaded world. +Nothing else writes to them, and nothing holds a foreign key to them — no FKs at all, consistent with +every other `shard_*` table. + +Four column choices worth stating, because each one is a trap: + +- **`spawn_range`, not `range`**, and **`grp`, not `group`** — both are reserved words. +- **`DELETE`, not `TRUNCATE`.** `TRUNCATE` is DDL in MariaDB and implicitly commits, which would + defeat the all-or-nothing reload. At ~7k rows the difference does not matter. +- **Point ids are assigned explicitly**, not left to `AUTO_INCREMENT`: the `shard_spawn_point_types` + rows need to know them, and `conn.batch()` reports no usable `insertId` for a multi-row insert. +- **Plain `INDEX` on `name`, deliberately not `FULLTEXT`.** ~800 creature rows makes a `LIKE` scan + free, and FULLTEXT's minimum token length would break searches for names like "orc". + +`shard_champion_spawns` is the *configured* altar roster ("there is an Unholy Terror altar in +Deceit"). The live `champ.update` feed in `shard_champs` is the separate answer to "it is on level 3 +right now". Both exist; they are not the same data. + +**`shard_spawn_creatures.art` is always NULL on a fresh import.** The project ships no creature +artwork: sprites live in the operator's own client `.mul`/`.uop` files and are theirs, not ours to +redistribute. An operator supplies art via a gitignored map plus images under the (already +gitignored) `server/uploads/atlas/`. Text-only is the normal, supported state. + --- ## 4. API contract diff --git a/website/SPAWN_ATLAS.md b/website/SPAWN_ATLAS.md new file mode 100644 index 0000000..57e735e --- /dev/null +++ b/website/SPAWN_ATLAS.md @@ -0,0 +1,225 @@ +# Spawn atlas + +**Status:** Data pipeline landed on `edge` (website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112)); API and client pages follow in a second PR. +**Design:** [`docs/link/v3.md` §6](../link/v3.md) — Protocol 3.0 Part C. + +The spawn atlas is a browsable catalogue of what the shard *contains*: which +creatures spawn, where, how many, and which champion altars are configured. It +answers "where do I find a lizardman?" with **"Shrines, Yew, Isamu-Jima"** rather +than with a list of raw coordinates. + +## What it is not + +The atlas is **static shard content, not live shard state.** + +- It does **not** come from the sidecar. Nothing here touches the bridge, and + there is no event kind, no wire change and no `PROTOCOL_VERSION` bump for it. + Part C is website-only. +- It stays fully populated while the shard is down. +- Its champion table (`shard_champion_spawns`) is the *configured roster* — + "there is an Unholy Terror altar in Deceit". The live `champ.update` feed in + `shard_champs` is the separate, sidecar-fed answer to "it is on level 3 right + now". Both exist; do not conflate them. + +Routes live at `/api/v1/public/atlas`, deliberately **not** under `/shard`, +because `/shard/*` means sidecar-dependent. + +## The two commands + +Building needs a ServUO tree. Importing does not. That split is the whole design: +the website container ships the committed artifact but has no ServUO tree, so it +can import but never build. + +```bash +# On a machine that has the ServUO tree (writes server/db/data/spawnAtlas.*.json) +cd website/server +npm run atlas:build -- --servuo /path/to/ServUO + +# Anywhere, including the deployed container +npm run atlas:import +``` + +`atlas:build` flags: + +| Flag | Default | Meaning | +|---|---|---| +| `--servuo` | *(required)* | ServUO server root — the directory holding `Spawns/`, `Data/`, `Config/` | +| `--out` | `server/db/data` | Where to write the artifact | +| `--landmark-radius` | `200` | Max tile distance for the landmark fallback | + +`atlas:import` takes `--dir` (default `server/db/data`). + +## Operator re-run story + +Spawns changed → rebuild → commit → deploy → import. + +1. `npm run atlas:build -- --servuo ` on a machine with the tree. +2. Commit the regenerated `server/db/data/spawnAtlas.*.json`. +3. Deploy. +4. `npm run atlas:import`, or `POST /api/v1/admin/shard/atlas/import`. + +`shard_atlas_meta` stores a sha256 per source file, so +`GET /api/v1/admin/shard/atlas/status` reports when the database is behind the +committed artifact. **Build stays CLI-only** — there is no admin button that +reads a ServUO tree. + +## Sources + +| File | Count (stock ServUO 57.4) | Used for | +|---|---|---| +| `Spawns/*.xml` | 13 files, ~10.5 MB | Every spawner: location, size, delays, time-of-day, creature types | +| `Data/Regions.xml` | 129 KB, nested | Named regions and their rectangles | +| `Data/Locations/*.xml` | 6 files | Landmarks (dungeon levels, town markers) | +| `Config/ChampionSpawns.xml` | 4.8 KB | Configured champion altars | + +**There are 13 spawn files but only 6 facets.** `Eodon.xml`, +`GravewaterLake.xml`, `TreasuresOfKotl.xml` and the other named-area files hold +TerMur/Trammel points. The facet always comes from each record's own ``, +never from the file name, and the artifact shards by facet — Felucca, Trammel, +Ilshenar, Malas, Tokuno, TerMur. + +## How a coordinate becomes a place name + +This is the transform the atlas exists for, in `resolveRegion()`: + +1. The highest-`priority` named region whose rectangle contains the point. Ties + break toward the **smallest** rect, so a specific room wins over the + dungeon-wide rect enclosing it. +2. Otherwise the nearest landmark within `--landmark-radius` tiles, labelled by + its **group** ("Covetous"), not its individual marker ("Level 1"). +3. Otherwise `"Wilderness"`. + +The radius cap in step 2 is what keeps step 3 reachable. Without it the nearest +landmark is always *some* landmark however far away, and open countryside gets +labelled with a dungeon on the far side of the map. + +Against stock ServUO this resolves **83.2%** of points (5,369 of 6,455): 3,689 by +region, 1,690 by landmark, 1,086 Wilderness. + +## Three quirks in the source data + +Each of these is silent if unhandled — the atlas still builds, it is just wrong. + +**Facet names disagree between sources.** `Data/Locations/*.xml` spells them +`Ter Mur` and `Tokuno Islands`, while `` and `` say `TerMur` and +`Tokuno`. Unreconciled, the landmark bucket for those two facets is keyed +differently from the points looking it up, so the fallback never fires and every +unregioned spawn in Ter Mur and Tokuno reads "Wilderness". All facet names are +canonicalised through `normalizeFacet()`; unknown facets pass through unchanged +so a custom shard facet still gets an atlas. + +**Spawn type tokens carry XmlSpawner directives.** The `` type is not +always a bare class name: + +``` +Fairy,{RND,4,8} alchemist/z/-50 Agralem/Name/Agralem +GargishRouser,1 greatape,true GargishRefugee/hue/34532 +``` + +Taken literally these invent creatures that do not exist *and* split real ones in +two, because `Fairy` and `Fairy,{RND,4,8}` slug apart into separate entries. 71 of +845 were affected. Everything from the first `/` or `,` is stripped, leaving 800 +real creatures. + +**Case is inconsistent across files.** The same creature is `Lizardman` in one +file and `lizardman` in another. Slugging collapses them correctly, but the +display name is chosen deterministically — most common spelling wins, ties break +to the more capitalised form, then alphabetically — because otherwise it would +depend on file read order and produce a spurious artifact diff on every unrelated +rebuild. + +## Artifact format + +`server/db/data/`, all committed: + +| File | Contents | +|---|---| +| `spawnAtlas.meta.json` | Indented. Build time, counts, sha256 per source file | +| `spawnAtlas.index.json` | Compact. Facets, creatures, regions, landmarks, champions | +| `spawnAtlas..json` × 6 | Compact. That facet's spawn points | + +**1.41 MB total**, down from 4.40 MB. The design budgeted "well under 1 MB", +which turned out optimistic for 6,455 points; three encodings closed most of the +gap: + +- `facet` is dropped per record — the shard file names it once at the top. +- Fields at their default are omitted rather than written as `0`. Most spawners + are a single point with no time-of-day gating, so `width`, `height`, `range` + and the three `tod*` fields are zero on the large majority of records. +- `types` are `[name, max]` tuples. There are ~24,000 type entries and + `{"type":"Orc","max":1}` spends 15 bytes apiece restating two key names that + never vary. + +`label` is not stored at all — it is exactly `region || landmark || "Wilderness"` +and the importer recomputes it. + +`buildSpawnAtlas.encodePoint()` and `importSpawnAtlas.readPoint()` are exact +inverses, round-tripped in `test/spawnAtlas.build.test.js`. **Change one, change +both.** The artifact never reaches the browser; the browser sees only paginated +API responses. + +## Artwork — operator-supplied, never shipped + +**This project ships no creature art and no extraction tooling, and never will.** +UO sprites live in the operator's own client `.mul`/`.uop` files. They are the +operator's, not ours to redistribute. + +The atlas is fully functional as text. `shard_spawn_creatures.art` is nullable +and is NULL on every fresh import; pages render without images, which is the +normal and supported state, not a degraded one. + +An operator who wants art: + +1. Extracts it from **their own** client files (UOFiddler, ClassicUO tooling, or + any art extractor). +2. Drops the images under `server/uploads/atlas/`. +3. Copies `server/db/data/spawnAtlas.art.example.json` to `spawnAtlas.art.json` + and maps creature slugs to file names. +4. Re-runs `npm run atlas:import`. + +Both `spawnAtlas.art.json` and `server/uploads/` are gitignored, so neither the +map nor the images can be committed by accident. + +## Tables + +All six are **import-owned**: `atlas:import` empties and reloads them in one +transaction. Nothing else writes to them and nothing holds a foreign key to them +— no FKs at all, consistent with every other `shard_*` table. Full column +listings in [`BACKEND_DESIGN.md`](BACKEND_DESIGN.md). + +| Table | Rows (stock) | Notes | +|---|---|---| +| `shard_spawn_creatures` | 800 | `slug` PK; `total` = sum of each type's own max; nullable `art` | +| `shard_spawn_points` | 6,455 | `spawn_range`, since `range` is reserved in MariaDB | +| `shard_spawn_point_types` | 23,927 | The many-to-many; one spawner commonly carries six types | +| `shard_regions` | 387 | Flattened out of the nesting; `rects` JSON | +| `shard_landmarks` | 558 | `grp`, since `group` is reserved in SQL | +| `shard_champion_spawns` | 25 | Configured altars, not the live feed | +| `shard_atlas_meta` | 1 | Singleton (`id = 1`); source hashes for the drift check | + +`shard_spawn_creatures.name` carries a plain `INDEX`, deliberately **not +`FULLTEXT`**: ~800 rows makes a `LIKE` scan free, and FULLTEXT's minimum token +length would break searches for names like "orc". + +The importer uses `DELETE`, not `TRUNCATE` — `TRUNCATE` is DDL in MariaDB and +would implicitly commit, defeating the all-or-nothing reload. Point ids are +assigned explicitly rather than left to `AUTO_INCREMENT`, because the join rows +need to know them and `conn.batch()` reports no usable `insertId`. + +## Parsing notes + +`server/src/utils/spawnAtlasParse.js` is **pure and fs-free**, so CI covers it +with no ServUO tree. It adds **zero dependencies**. + +- `Regions.xml`, `Locations/*.xml` and `ChampionSpawns.xml` genuinely nest, and + get a small hand-rolled **subset** tokenizer — elements, attributes, + self-closing tags, comments, the XML declaration, CDATA, and the five + predefined entities plus numeric refs. It is not a general-purpose XML parser + and must not be reused as one. +- The ~10.5 MB of `Spawns/*.xml` never touches that tokenizer. Those records are + flat, so they get a streaming regex sweep instead; a DOM would allocate a node + per element across ~40 fields on every record to keep 14 of them. **Do not put + the Points files through a DOM parser.** +- `` is `Type:MX=n:SB=…` segments joined by `:OBJ=`. Split on `:OBJ=` + *first* — a naive `split(':')` shreds it. A single Trammel point carries six + types. -- 2.49.1 From ff1c2064a51ab234b2a9906cc90053204f174520 Mon Sep 17 00:00:00 2001 From: wtclaude Date: Tue, 28 Jul 2026 16:44:39 -0500 Subject: [PATCH 08/19] docs(website): the atlas reads the shard's tree on every boot, not a snapshot MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Follows the redesign in website #112. Two decisions from the original §6 were rejected in review and replaced; the docs now describe what was actually built. **The committed artifact is gone.** A shard's maps change over its life, so a snapshot in the repo silently drifts from the world players actually see. The ServUO tree is the single source of truth and the atlas is re-derived on every server boot, hash-gated so an unchanged tree costs one read pass and no write. **Nothing may name a facet.** The first implementation carried a lookup table of the six stock UO facets. A shard may add facets, replace them outright, or rename them when its maps are updated, and a built-in list mishandles all three silently. Reconciliation is now by matching against the facet set discovered from the shard's own data, with an unmatched name keeping its own rather than being forced into a wrong bucket. ## Changes - **`website/SPAWN_ATLAS.md`** rewritten: the two ideas that shape the design, how to configure the tree path, the boot flow as a decision tree, the approve/reject flow, and the code layout. The artwork policy is unchanged and still explicit — no art ever ships, operators extract their own from their own client files. - **`link/v3.md` §6.1 (new)** records the two rejected decisions plus the two boot-path contracts. The old "what real data changed" list becomes §6.2. §6's now-superseded passages — the artifact bullet, the payload budget, the operator re-run story — are marked rather than deleted, so the reasoning stays legible. - **`website/BACKEND_DESIGN.md`** documents `shard_atlas_pending` and the two contracts that make it safe: a facet removal is staged for a human, and the boot refresh can never block startup. The two contracts are the part worth reviewing. Losing a facet is indistinguishable at boot from a half-copied or mid-update tree, so it is staged rather than applied; and no failure mode of the atlas — missing path, unreadable mount, malformed file, database error — is allowed to stop the site coming up. --- - [x] AI-assisted: written with **Claude Code** (Claude Opus 5), reviewed before opening. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01U7CBg11prhLimL9iHSX1bP --- link/v3.md | 82 +++++++++----- website/BACKEND_DESIGN.md | 32 +++++- website/SPAWN_ATLAS.md | 226 ++++++++++++++++++++++---------------- 3 files changed, 214 insertions(+), 126 deletions(-) diff --git a/link/v3.md b/link/v3.md index 79e9939..f62c426 100644 --- a/link/v3.md +++ b/link/v3.md @@ -335,22 +335,25 @@ frame during verification. > lives, and burying it under routes and React would have meant reviewing it in a 10k-line diff. > Full operator documentation: [`docs/website/SPAWN_ATLAS.md`](../website/SPAWN_ATLAS.md). > -> §6.1 below records where the shipped implementation differs from this design. The differences are -> all things the real ServUO data forced, not changes of mind. +> **§6 below is the original design and is partly superseded.** §6.1 records two decisions that were +> rejected in review and replaced (the committed artifact, and the fixed facet list); §6.2 records +> the corrections the real ServUO data forced. Read both before trusting §6. -**Decision: committed generated artifact + idempotent DB import**, split in two because the build -needs the ServUO tree (which the website container does not have) and the import does not. Not -runtime import (10.5 MB of XML per boot), not a browser-served blob. +**Decision (revised at implementation time): the shard's ServUO tree is the single source of truth, +re-derived on every server boot.** The original plan here was a committed generated artifact plus an +idempotent import. That was rejected in review for two reasons, recorded in §6.1: a snapshot in the +repo goes stale as a shard's maps change, and the design leaned on a fixed facet list that no shard +is obliged to keep. Still not a browser-served blob; still parsed server-side only. New in `website/server/`: - `src/utils/spawnAtlasParse.js` — **pure functions, no fs**, so they are unit-testable in CI without a ServUO tree: `parseObjects2()`, `parsePoints()`, `parseRegions()`, `parseLocations()`, `resolveRegion()`. -- `scripts/buildSpawnAtlas.js` (`--servuo --out db/data/`) and `scripts/importSpawnAtlas.js` - (TRUNCATE + batched INSERT in one transaction); `package.json` scripts `atlas:build`, `atlas:import`. -- `db/data/spawnAtlas..json` ×13 + `spawnAtlas.index.json` (creatures, champions, regions, - landmarks, meta with per-source-file hashes). +- ~~`scripts/buildSpawnAtlas.js` and a committed `db/data/spawnAtlas.*.json` artifact~~ — dropped, + see §6.1 R1. Replaced by `src/utils/spawnAtlasSource.js` (the only thing that reads a ServUO tree, + shared by the boot path and the CLI) and a `scripts/importSpawnAtlas.js` that is a thin CLI over + the model. `package.json` gains `atlas:import` only. - `src/model/shardAtlas/{shardAtlas.db.js,shardAtlas.model.js}` following the `shardState` split. - `src/router/v1/public/atlas.{router,controller}.js`; `test/spawnAtlas.parse.test.js`. @@ -381,18 +384,48 @@ stays CLI-only.** Client: `routes/public/Atlas.jsx` (`/site/atlas`) and `AtlasCreature.jsx` (`/site/atlas/:slug`). -**Payload risk** — a monolithic artifact would be 2–3 MB of committed JSON. Shard per facet and drop -every `` field the site cannot use (`UniqueId`, all trigger/refractory/proximity/sequential -fields, sound ids), keeping Name/Map/X/Y/W/H/Range/MaxCount/MinDelay/MaxDelay/TOD*/types — well under -1 MB. The artifact never reaches the browser; the browser sees only paginated API responses. +**Payload risk** — *superseded by §6.1 R1; nothing is committed.* The field selection it describes +still applies at parse time: every `` field the site cannot use (`UniqueId`, all +trigger/refractory/proximity/sequential fields, sound ids) is dropped, keeping +Name/Map/X/Y/W/H/Range/MaxCount/MinDelay/MaxDelay/TOD*/types. Parsed data never reaches the browser; +the browser sees only paginated API responses. -**Operator re-run story** — spawns changed → `npm run atlas:build -- --servuo ` on a machine -with the tree → commit the regenerated `db/data/spawnAtlas.*.json` → deploy → `npm run atlas:import` -(or `POST /admin/shard/atlas/import`). `shard_atlas_meta.source` holds per-file hashes, so -`GET /admin/shard/atlas/status` reports when the DB is behind the artifact. Full detail in -`docs/website/SPAWN_ATLAS.md`. +**Operator re-run story** — *revised by §6.1 R1.* Spawns changed → restart, or +`npm run atlas:import` / `POST /admin/shard/atlas/import` to apply without one. `shard_atlas_meta` +holds a sha256 per source file, so the server can tell on boot whether anything changed, and +`GET /admin/shard/atlas/status` reports drift. If the change would remove a facet it is staged for +approval rather than applied (§6.1 R3). Full detail in `docs/website/SPAWN_ATLAS.md`. -### 6.1 What the build against real data changed +### 6.1 What implementation changed + +Two design decisions in §6 were rejected in review and replaced; the rest are corrections the real +ServUO data forced. Kept as a diff rather than edited in place, because each is a trap the next +person would otherwise re-enter. + +**R1. The committed artifact is gone — the tree is re-parsed on every boot.** §6 proposed building a +generated artifact, committing it, and importing it. Two problems. A shard's maps change over its +life, so a snapshot in the repo silently drifts from the world players actually see; and the build/ +import split existed only to work around the website container not having a tree, which is a +deployment question (mount it) rather than a reason to freeze data. The server now hashes the source +files on boot and re-derives the atlas when they differ. `scripts/buildSpawnAtlas.js`, the 1.41 MB +artifact, and the whole encode/decode seam it needed are deleted. + +**R2. Nothing may name a facet.** The first implementation carried a lookup table of the six stock +UO facets to reconcile the spelling drift between sources. A shard may add facets, replace them +outright, or rename them when its maps are updated, and a built-in list mishandles all three +silently. Reconciliation is now by *matching* against the facet set discovered from the shard's own +spawn and region data — exact key, then prefix in either direction — with an unmatched name keeping +its own rather than being forced into a wrong bucket. + +**R3. Two contracts on the boot path.** It never blocks startup: no path, an unreadable mount, a +malformed file or a database error is caught and logged, and the site comes up serving whatever +atlas it had. And a refresh that would REMOVE a facet is never applied automatically — facet loss +is indistinguishable at boot from a half-copied or mid-update tree, so it is staged in +`shard_atlas_pending` for an admin to approve or reject. Only the decision is stored (source hashes ++ the facet diff, a few KB); approving re-parses, so what lands matches the tree at approval time. +A rejection is remembered against those hashes so it does not re-prompt every restart. + +### 6.2 What the build against real data changed Six corrections to the design above, from running it against stock ServUO 57.4. Kept as a diff rather than edited in place, because each one is a trap the next person would otherwise re-enter. @@ -419,12 +452,11 @@ they invent creatures that do not exist *and* split real ones in two, since `Fai leaves **800** real creatures. (The design's "~1,500 creature rows" estimate was high; 800 only reinforces the plain-`INDEX`-not-`FULLTEXT` call.) -**5. The artifact is 1.41 MB, not "well under 1 MB".** Dropping the unused `` fields as the -design directed still left 4.40 MB. Three further encodings — `facet` dropped per record, -default-valued fields omitted rather than written as `0`, and `types` as `[name, max]` tuples -(~24,000 entries × 15 bytes of repeated key names) — brought it to 1.41 MB. Getting under 1 MB would -mean dropping the spawner `name`, which is the only human handle on a specific spawner and worth -keeping. `encodePoint()` and `readPoint()` are exact inverses and are round-tripped in tests. +**5. The artifact would have been 1.41 MB, not "well under 1 MB" — and is now moot.** Dropping the +unused `` fields as the design directed still left 4.40 MB; three further encodings brought +it to 1.41 MB, and getting under 1 MB would have meant dropping the spawner `name`. The size budget +in §6 was simply optimistic for 6,455 points. Superseded by §6.1 R1: there is no artifact, so there +is no payload to budget and no encode/decode seam to keep in sync. **6. `DELETE`, not `TRUNCATE`.** The design said "TRUNCATE + batched INSERT in one transaction", which does not hold: `TRUNCATE` is DDL in MariaDB and implicitly commits, so a mid-import failure diff --git a/website/BACKEND_DESIGN.md b/website/BACKEND_DESIGN.md index cdffd8c..cb7743e 100644 --- a/website/BACKEND_DESIGN.md +++ b/website/BACKEND_DESIGN.md @@ -391,9 +391,15 @@ discarded. See §6.5. ### shard_spawn_* / shard_regions / shard_landmarks / shard_champion_spawns / shard_atlas_meta — the spawn atlas (Protocol 3.0) Static shard **content**, not live shard state. Nothing here comes from the sidecar: the atlas is -built from a ServUO tree by a CLI script, committed as JSON under `server/db/data/`, and loaded with -`npm run atlas:import`. These tables stay populated whether the shard is up or not. Full operator -detail in [`SPAWN_ATLAS.md`](SPAWN_ATLAS.md); the design is `docs/link/v3.md` §6. +derived from the shard's own ServUO tree, re-read on **every server boot** and hash-gated so an +unchanged tree costs one read pass and no write. Nothing is precomputed and committed — a shard's +maps change over its life, and a snapshot in the repo would silently drift from the world players +actually see. These tables stay populated whether the shard is up or not. Full operator detail in +[`SPAWN_ATLAS.md`](SPAWN_ATLAS.md); the design is `docs/link/v3.md` §6. + +**No facet name appears anywhere in the code.** A shard may add facets, replace them, or rename them +when its maps are updated; the facet set is discovered from the tree, and the loose spellings in +`Data/Locations` are matched against it rather than looked up in a table. | Table | Key columns | |---|---| @@ -403,13 +409,27 @@ detail in [`SPAWN_ATLAS.md`](SPAWN_ATLAS.md); the design is `docs/link/v3.md` § | `shard_regions` | `facet`, `name`, `type`, `priority`, `parent`, `rects` JSON | | `shard_landmarks` | `facet`, `name`, `grp`, `x`, `y`, `z` | | `shard_champion_spawns` | `slug` PK, `name`, `grp`, `type`, `random_type`, `facet`, `x`, `y`, `z`, `radius`, `label` | -| `shard_atlas_meta` | Singleton (`id = 1`), `payload` JSON, `imported_at` | +| `shard_atlas_meta` | Singleton (`id = 1`), `payload` JSON (counts + a sha256 per source file), `imported_at` | +| `shard_atlas_pending` | Singleton (`id = 1`), `status` (`pending`/`rejected`), `payload` JSON, `detected_at` | -All of them are **import-owned**: `atlas:import` empties and reloads every one inside a single -transaction, so a failed import leaves the previous atlas intact rather than a half-loaded world. +The first seven are **import-owned**: a refresh empties and reloads every one inside a single +transaction, so a failed reload leaves the previous atlas intact rather than a half-loaded world. Nothing else writes to them, and nothing holds a foreign key to them — no FKs at all, consistent with every other `shard_*` table. +**`shard_atlas_pending` is the security-relevant one.** A refresh that would REMOVE a facet is never +applied automatically: facet loss is indistinguishable at boot from a half-copied or mid-update tree, +so it is staged here for an admin to approve or reject, and **startup is never blocked by it**. Only +the decision is stored — source hashes plus the facet diff, a few KB — and approving re-parses the +tree, so a multi-megabyte blob never lands in the database and what gets applied matches the tree at +approval time. A rejection is remembered against those exact hashes so a declined refresh does not +re-prompt on every restart. Everything else (new facets, renamed regions, changed spawns) applies +immediately, since none of it can destroy data an operator would miss. + +The boot refresh is **best-effort by contract**: no configured path, an unreadable mount, a malformed +file or a database error is caught and logged, and the site comes up serving whatever atlas it had. +The tree path comes from the `spawn_atlas_servuo_path` setting, falling back to `SERVUO_PATH`. + Four column choices worth stating, because each one is a trap: - **`spawn_range`, not `range`**, and **`grp`, not `group`** — both are reserved words. diff --git a/website/SPAWN_ATLAS.md b/website/SPAWN_ATLAS.md index 57e735e..2e9591b 100644 --- a/website/SPAWN_ATLAS.md +++ b/website/SPAWN_ATLAS.md @@ -8,6 +8,19 @@ creatures spawn, where, how many, and which champion altars are configured. It answers "where do I find a lizardman?" with **"Shrines, Yew, Isamu-Jima"** rather than with a list of raw coordinates. +## Two things that shape the whole design + +**The shard's ServUO tree is the single source of truth.** Nothing is +precomputed and committed to the repository. A shard's maps change over its +lifetime — facets get added, replaced, or renamed — and a snapshot in the repo +would silently drift from the world players actually see. The atlas is therefore +re-derived from the tree **on every server boot**. + +**Facets are not a fixed list.** Nothing in the codebase names Felucca, Trammel, +or any other stock facet. The facet set is whatever the shard's own files +declare, discovered at parse time. A shard running entirely custom maps gets +exactly the same treatment as a stock one, with no code change. + ## What it is not The atlas is **static shard content, not live shard state.** @@ -24,44 +37,83 @@ The atlas is **static shard content, not live shard state.** Routes live at `/api/v1/public/atlas`, deliberately **not** under `/shard`, because `/shard/*` means sidecar-dependent. -## The two commands +## Configuring the tree -Building needs a ServUO tree. Importing does not. That split is the whole design: -the website container ships the committed artifact but has no ServUO tree, so it -can import but never build. +The website needs to be able to *read* the ServUO tree — same host, a bind mount, +or a shared volume. Two ways to point at it, the setting winning over the +environment: -```bash -# On a machine that has the ServUO tree (writes server/db/data/spawnAtlas.*.json) -cd website/server -npm run atlas:build -- --servuo /path/to/ServUO +| Source | Notes | +|---|---| +| `spawn_atlas_servuo_path` setting | Admin-editable; changes take effect on the next refresh without a redeploy | +| `SERVUO_PATH` env var | The deploy-time default, since the path usually describes a mount the deployment sets up | -# Anywhere, including the deployed container -npm run atlas:import +With neither set the atlas is simply skipped — the site runs normally without +one. + +## The boot path + +On every start the server hashes the source files and compares them against what +is loaded. Unchanged (the normal case on a restart) costs one read pass, ~120 ms, +and no database write. A real change costs a ~400 ms parse and a reload. + +Two contracts govern it: + +**1. It never blocks startup.** No configured path, an unreadable mount, a +malformed file, a database error — every one is caught and logged, and the site +comes up serving whatever atlas it already had. + +**2. A facet disappearing is never applied automatically.** Losing a facet looks +exactly like a half-copied or mid-update tree, and boot cannot tell that apart +from a real map change. That refresh is *staged* for a human instead. Everything +else — new facets, renamed regions, changed spawns — applies immediately, since +none of it can destroy something an operator would miss. + +``` +boot + └─ path configured? no ──▶ skip + └─ tree readable? no ──▶ warn, carry on + └─ hashes changed? no ──▶ done (nothing parsed) + └─ parse + └─ a facet would be removed? + no ──▶ import + yes ──▶ stage for admin review; atlas unchanged ``` -`atlas:build` flags: +### Approving or rejecting a staged refresh -| Flag | Default | Meaning | -|---|---|---| -| `--servuo` | *(required)* | ServUO server root — the directory holding `Spawns/`, `Data/`, `Config/` | -| `--out` | `server/db/data` | Where to write the artifact | -| `--landmark-radius` | `200` | Max tile distance for the landmark fallback | +Only the *decision* is stored, never the parsed world — a few KB of source hashes +plus the facet diff. Approving **re-parses** the tree, so what lands matches the +tree at approval time rather than at boot, and a multi-megabyte blob never sits +in the database. -`atlas:import` takes `--dir` (default `server/db/data`). +A rejection is remembered against those exact source hashes, so a declined +refresh does not re-prompt on every restart. Change the tree and the hashes +differ, which asks again. -## Operator re-run story +From the admin panel (second PR), or from the CLI: -Spawns changed → rebuild → commit → deploy → import. +```bash +cd website/server +npm run atlas:import -- --status # what is loaded, and what is pending +npm run atlas:import -- --approve # apply the staged refresh +npm run atlas:import -- --reject # keep the current atlas, dismiss it +``` -1. `npm run atlas:build -- --servuo ` on a machine with the tree. -2. Commit the regenerated `server/db/data/spawnAtlas.*.json`. -3. Deploy. -4. `npm run atlas:import`, or `POST /api/v1/admin/shard/atlas/import`. +## The CLI -`shard_atlas_meta` stores a sha256 per source file, so -`GET /api/v1/admin/shard/atlas/status` reports when the database is behind the -committed artifact. **Build stays CLI-only** — there is no admin button that -reads a ServUO tree. +The server refreshes itself on boot, so this is for applying a map change +*without* a restart, and for the approve/reject flow above. + +```bash +npm run atlas:import # import if the tree differs +npm run atlas:import -- --servuo # override the path for this run +npm run atlas:import -- --force # reimport even if unchanged +``` + +`--servuo` is a per-run override and deliberately does **not** persist — changing +where the atlas permanently reads from is an admin action, not a side effect of a +one-off import. ## Sources @@ -72,11 +124,10 @@ reads a ServUO tree. | `Data/Locations/*.xml` | 6 files | Landmarks (dungeon levels, town markers) | | `Config/ChampionSpawns.xml` | 4.8 KB | Configured champion altars | -**There are 13 spawn files but only 6 facets.** `Eodon.xml`, +**A stock tree has 13 spawn files but only 6 facets.** `Eodon.xml`, `GravewaterLake.xml`, `TreasuresOfKotl.xml` and the other named-area files hold TerMur/Trammel points. The facet always comes from each record's own ``, -never from the file name, and the artifact shards by facet — Felucca, Trammel, -Ilshenar, Malas, Tokuno, TerMur. +never from the file name. ## How a coordinate becomes a place name @@ -85,16 +136,17 @@ This is the transform the atlas exists for, in `resolveRegion()`: 1. The highest-`priority` named region whose rectangle contains the point. Ties break toward the **smallest** rect, so a specific room wins over the dungeon-wide rect enclosing it. -2. Otherwise the nearest landmark within `--landmark-radius` tiles, labelled by - its **group** ("Covetous"), not its individual marker ("Level 1"). +2. Otherwise the nearest landmark within the landmark radius (200 tiles by + default), labelled by its **group** ("Covetous"), not its individual marker + ("Level 1"). 3. Otherwise `"Wilderness"`. The radius cap in step 2 is what keeps step 3 reachable. Without it the nearest landmark is always *some* landmark however far away, and open countryside gets labelled with a dungeon on the far side of the map. -Against stock ServUO this resolves **83.2%** of points (5,369 of 6,455): 3,689 by -region, 1,690 by landmark, 1,086 Wilderness. +Against stock ServUO this resolves **83.2%** of points (5,369 of 6,455): 3,681 by +region, 1,688 by landmark, 1,086 Wilderness. ## Three quirks in the source data @@ -102,11 +154,17 @@ Each of these is silent if unhandled — the atlas still builds, it is just wron **Facet names disagree between sources.** `Data/Locations/*.xml` spells them `Ter Mur` and `Tokuno Islands`, while `` and `` say `TerMur` and -`Tokuno`. Unreconciled, the landmark bucket for those two facets is keyed -differently from the points looking it up, so the fallback never fires and every -unregioned spawn in Ter Mur and Tokuno reads "Wilderness". All facet names are -canonicalised through `normalizeFacet()`; unknown facets pass through unchanged -so a custom shard facet still gets an atlas. +`Tokuno`. Unreconciled, the landmark bucket is keyed differently from the points +looking it up, so the fallback never fires and every unregioned spawn on those +facets reads "Wilderness". + +This is reconciled **by matching, not by a lookup table** — there is no list of +facet names anywhere. `facetKey()` collapses spelling differences (lowercase, +alphanumerics only), and `resolveFacetName()` matches a loose spelling against +the canonical set discovered from the shard's own spawn and region data, by exact +key then by prefix in either direction. A name matching nothing keeps its own +name: forcing a wrong match would file a real custom facet's landmarks under the +wrong facet, which is worse than leaving it alone. **Spawn type tokens carry XmlSpawner directives.** The `` type is not always a bare class name: @@ -125,38 +183,35 @@ real creatures. file and `lizardman` in another. Slugging collapses them correctly, but the display name is chosen deterministically — most common spelling wins, ties break to the more capitalised form, then alphabetically — because otherwise it would -depend on file read order and produce a spurious artifact diff on every unrelated -rebuild. +depend on file read order and change on an unrelated restart. -## Artifact format +## Tables -`server/db/data/`, all committed: +All are **import-owned**: a refresh empties and reloads them in one transaction, +so a failed reload leaves the previous atlas intact rather than a half-loaded +world. Nothing else writes to them and nothing holds a foreign key to them — no +FKs at all, consistent with every other `shard_*` table. Full column listings in +[`BACKEND_DESIGN.md`](BACKEND_DESIGN.md). -| File | Contents | -|---|---| -| `spawnAtlas.meta.json` | Indented. Build time, counts, sha256 per source file | -| `spawnAtlas.index.json` | Compact. Facets, creatures, regions, landmarks, champions | -| `spawnAtlas..json` × 6 | Compact. That facet's spawn points | +| Table | Rows (stock) | Notes | +|---|---|---| +| `shard_spawn_creatures` | 800 | `slug` PK; `total` = sum of each type's own max; nullable `art` | +| `shard_spawn_points` | 6,455 | `spawn_range`, since `range` is reserved in MariaDB | +| `shard_spawn_point_types` | 23,927 | The many-to-many; one spawner commonly carries six types | +| `shard_regions` | 387 | Flattened out of the nesting; `rects` JSON | +| `shard_landmarks` | 558 | `grp`, since `group` is reserved in SQL | +| `shard_champion_spawns` | 25 | Configured altars, not the live feed | +| `shard_atlas_meta` | 1 | Singleton; source hashes, for the change check | +| `shard_atlas_pending` | 0–1 | Singleton; a staged refresh awaiting admin review | -**1.41 MB total**, down from 4.40 MB. The design budgeted "well under 1 MB", -which turned out optimistic for 6,455 points; three encodings closed most of the -gap: +`shard_spawn_creatures.name` carries a plain `INDEX`, deliberately **not +`FULLTEXT`**: ~800 rows makes a `LIKE` scan free, and FULLTEXT's minimum token +length would break searches for names like "orc". -- `facet` is dropped per record — the shard file names it once at the top. -- Fields at their default are omitted rather than written as `0`. Most spawners - are a single point with no time-of-day gating, so `width`, `height`, `range` - and the three `tod*` fields are zero on the large majority of records. -- `types` are `[name, max]` tuples. There are ~24,000 type entries and - `{"type":"Orc","max":1}` spends 15 bytes apiece restating two key names that - never vary. - -`label` is not stored at all — it is exactly `region || landmark || "Wilderness"` -and the importer recomputes it. - -`buildSpawnAtlas.encodePoint()` and `importSpawnAtlas.readPoint()` are exact -inverses, round-tripped in `test/spawnAtlas.build.test.js`. **Change one, change -both.** The artifact never reaches the browser; the browser sees only paginated -API responses. +The reload uses `DELETE`, not `TRUNCATE` — `TRUNCATE` is DDL in MariaDB and would +implicitly commit, defeating the all-or-nothing guarantee. Point ids are assigned +explicitly rather than left to `AUTO_INCREMENT`, because the join rows need them +and `conn.batch()` reports no usable `insertId`. ## Artwork — operator-supplied, never shipped @@ -175,41 +230,22 @@ An operator who wants art: 2. Drops the images under `server/uploads/atlas/`. 3. Copies `server/db/data/spawnAtlas.art.example.json` to `spawnAtlas.art.json` and maps creature slugs to file names. -4. Re-runs `npm run atlas:import`. +4. Restarts, or runs `npm run atlas:import -- --force`. Both `spawnAtlas.art.json` and `server/uploads/` are gitignored, so neither the map nor the images can be committed by accident. -## Tables +## Code layout -All six are **import-owned**: `atlas:import` empties and reloads them in one -transaction. Nothing else writes to them and nothing holds a foreign key to them -— no FKs at all, consistent with every other `shard_*` table. Full column -listings in [`BACKEND_DESIGN.md`](BACKEND_DESIGN.md). +| File | Role | +|---|---| +| `src/utils/spawnAtlasParse.js` | **Pure and fs-free** parsers, so CI covers them with no ServUO tree. Zero dependencies. | +| `src/utils/spawnAtlasSource.js` | The only thing that reads a ServUO tree; shared by the boot path and the CLI | +| `src/model/shardAtlas/shardAtlas.db.js` | The one-transaction replace | +| `src/model/shardAtlas/shardAtlas.model.js` | The refresh decision, staging, approve/reject | +| `scripts/importSpawnAtlas.js` | Thin CLI over the model | -| Table | Rows (stock) | Notes | -|---|---|---| -| `shard_spawn_creatures` | 800 | `slug` PK; `total` = sum of each type's own max; nullable `art` | -| `shard_spawn_points` | 6,455 | `spawn_range`, since `range` is reserved in MariaDB | -| `shard_spawn_point_types` | 23,927 | The many-to-many; one spawner commonly carries six types | -| `shard_regions` | 387 | Flattened out of the nesting; `rects` JSON | -| `shard_landmarks` | 558 | `grp`, since `group` is reserved in SQL | -| `shard_champion_spawns` | 25 | Configured altars, not the live feed | -| `shard_atlas_meta` | 1 | Singleton (`id = 1`); source hashes for the drift check | - -`shard_spawn_creatures.name` carries a plain `INDEX`, deliberately **not -`FULLTEXT`**: ~800 rows makes a `LIKE` scan free, and FULLTEXT's minimum token -length would break searches for names like "orc". - -The importer uses `DELETE`, not `TRUNCATE` — `TRUNCATE` is DDL in MariaDB and -would implicitly commit, defeating the all-or-nothing reload. Point ids are -assigned explicitly rather than left to `AUTO_INCREMENT`, because the join rows -need to know them and `conn.batch()` reports no usable `insertId`. - -## Parsing notes - -`server/src/utils/spawnAtlasParse.js` is **pure and fs-free**, so CI covers it -with no ServUO tree. It adds **zero dependencies**. +Parsing notes: - `Regions.xml`, `Locations/*.xml` and `ChampionSpawns.xml` genuinely nest, and get a small hand-rolled **subset** tokenizer — elements, attributes, -- 2.49.1 From be7e1a69ce7ef23b1208789a4087fbfa78e2562b Mon Sep 17 00:00:00 2001 From: wtclaude Date: Tue, 28 Jul 2026 19:51:50 -0500 Subject: [PATCH 09/19] docs(website): the spawn atlas API, the admin panel, and the delay-unit trap MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Docs half of website #113 (Protocol 3.0 Part C, second website PR). Carries the atlas rewrite that missed #67: that PR merged before the "derive from the tree on every boot" commit was pushed, so `edge` currently describes the build/import-artifact design that was rejected in review, not what shipped in website #112. It lands here. New in SPAWN_ATLAS.md: the six public routes and five admin ones, and three behaviours that read as bugs unless they are written down — an unreadable tree answers 200 with status "unavailable" rather than 500 (refresh reports outcomes so boot is never blocked by a bad tree, and the contract is preserved at the API), setting the ServUO path deliberately does not import, and `points` is a count while `spawners` is the list. Also the delay-unit trap: XmlSpawner stores MinDelay/MaxDelay in minutes OR seconds per record, decided by that record's own DelayInSec flag, so a `5` is five minutes on one spawner and five seconds on the next. Both are plausible respawn times, which is what makes it silent. 170 of 6,455 stock spawners are second-flagged. And PARSER_VERSION, which exists because hashing the tree alone would strand an install whose maps never change on whatever an older parser derived. BACKEND_DESIGN.md gains the routes, the router-map entry, and the parser-version rule. v3.md marks Part C done and records in 6.3 what the API half found. api-route-inventory.json refreshed from the live manifest — it had drifted to 200 routes before this PR (real count was 204) and is now 215. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01U7CBg11prhLimL9iHSX1bP --- link/v3.md | 42 ++++++++++++-- website/BACKEND_DESIGN.md | 23 +++++++- website/SPAWN_ATLAS.md | 97 +++++++++++++++++++++++++++++++- website/api-route-inventory.json | 60 ++++++++++++++++++++ 4 files changed, 214 insertions(+), 8 deletions(-) diff --git a/link/v3.md b/link/v3.md index f62c426..d23b6a7 100644 --- a/link/v3.md +++ b/link/v3.md @@ -13,7 +13,7 @@ Each part is marked off here as it lands on `edge`. §9 carries the same state p |---|---|---|---| | 1 | **A** — visibility framework + actor-leak fix (§3) | ✅ **Done** | website [#109](https://gitea.whitlocktech.com/RunicGateway/website/pulls/109) + [#110](https://gitea.whitlocktech.com/RunicGateway/website/pulls/110), docs [#64](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/64) + [#65](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/65) | | 2 | **B/1** — `world.ruleset` (§5) | ✅ **Done** | servuo-plugins [#3](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/3), link [#17](https://gitea.whitlocktech.com/RunicGateway/link/pulls/17), website [#111](https://gitea.whitlocktech.com/RunicGateway/website/pulls/111), docs [#66](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/66) | -| 3 | **C** — spawn atlas (§6) | 🟡 **Data pipeline done** | website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112) (parsers + CLI + tables); API/client PR next | +| 3 | **C** — spawn atlas (§6) | ✅ **Done** | website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112) (parsers + CLI + tables) + [#113](https://gitea.whitlocktech.com/RunicGateway/website/pulls/113) (API + pages + admin panel), docs [#67](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/67) + [#68](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/68) | | 4 | **B/2** — `points.board` (§7) | ⬜ Not started | — | | 5 | **B/3** — `vendor.listing` (§8) | ⬜ Not started | — | | 6 | **Cutover** — `PROTOCOL_VERSION` 2→3 (§4) | ⬜ Not started | — | @@ -329,8 +329,10 @@ frame during verification. **No plugin, no sidecar, no `Bridge.cfg` knob, no new kinds.** Not part of the v3 wire change. -> **Status:** data pipeline landed on `edge` — website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112) -> (parsers, build/import CLI, tables, artifact). API + client pages are the second website PR. +> **Status:** complete on `edge` — website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112) +> (parsers, import CLI, tables) and [#113](https://gitea.whitlocktech.com/RunicGateway/website/pulls/113) +> (the six public routes, the five admin ones, `/site/atlas` + `/site/atlas/:slug`, and the +> Admin → Spawn Atlas panel). > Part C ships as **two** website PRs, not one: the parsing half is where the correctness risk > lives, and burying it under routes and React would have meant reviewing it in a 10k-line diff. > Full operator documentation: [`docs/website/SPAWN_ATLAS.md`](../website/SPAWN_ATLAS.md). @@ -471,6 +473,38 @@ region, 1,690 by landmark, 1,086 Wilderness). **One thing the design got exactly right:** the point-in-rect transform really is the reason to build this. "Where does a lizardman spawn?" answers *Shrines, Isamu-Jima, Yew* across three facets. +### 6.3 What the API/client half added + +The second website PR built the six public routes, the five admin ones, `/site/atlas` + +`/site/atlas/:slug`, and the Admin → Spawn Atlas panel. Three things it changed or established: + +**1. Respawn delays were being read in the wrong unit — sometimes.** XmlSpawner writes +`MinDelay`/`MaxDelay` in minutes and switches to seconds only when a delay does not divide into +whole minutes, flagging that per record with `DelayInSec` +(`XmlSpawner2.cs:7462-7480`, read back at `:6345-6358`). So a `5` means five *minutes* on one +spawner and five *seconds* on the next, both plausible, and the pipeline stored the raw number. +170 of 6,455 stock spawners are second-flagged — few enough to look like noise on a page and be +believed. The parser now normalises to **seconds**, and the API and UI carry seconds throughout. +*This is the class of bug §6.2 is a list of: the atlas still builds, it is just quietly wrong.* + +**2. The hash gate needed a parser version, and this generalises.** Fixing the parse exposed that +"has the tree changed?" is the wrong question on its own — an install whose maps never change would +have kept serving the old readings forever, because the only thing compared was the tree. +`spawnAtlasSource.PARSER_VERSION` is stored in `shard_atlas_meta` beside the source hashes, and a +mismatch counts as drift. Any future parse correction lands on the next boot without an operator +having to know it happened. **Bump it whenever the parser derives different data from identical +files.** + +**3. `points` is a count; `spawners` is the list.** The first cut of the detail route spread the +creature row and then set `points` to the array of spawn points — the same key meaning a number on +the search route and an array on the detail route. Renamed before it shipped, and worth recording +because the two names are one letter apart in meaning and it reads as correct. + +**On projection.** The `atlas` feature declares no sensitive fields, so `projectFeature` is a no-op +on every one of these routes today. Every handler calls it anyway, per §3.6.1's rule — the point of +the rule is that the *first* field that needs gating is covered by construction rather than by a +retrofit nobody remembers to do. + --- ## 7. Part B/2 — `points.board` @@ -665,7 +699,7 @@ inherently up to one full cycle old, and the UI must say so. |---|---|---|---|---| | 1 | **A** — visibility framework + actor-leak fix | website, docs | none | ✅ Done | | 2 | **B/1** — `world.ruleset` (§5) | all four | new kind | ✅ Done | -| 3 | **C** — spawn atlas (§6) | website, docs | none | 🟡 Pipeline done, API/client next | +| 3 | **C** — spawn atlas (§6) | website, docs | none | ✅ Done | | 4 | **B/2** — `points.board` (§7) | all four | new kind + `char.profile` field | ⬜ | | 5 | **B/3** — `vendor.listing` (§8) | all four | new kinds | ⬜ | | 6 | **Cutover** — `PROTOCOL_VERSION` 2→3, `edge` → `main` | all four | the bump | ⬜ | diff --git a/website/BACKEND_DESIGN.md b/website/BACKEND_DESIGN.md index cb7743e..fd26513 100644 --- a/website/BACKEND_DESIGN.md +++ b/website/BACKEND_DESIGN.md @@ -101,6 +101,10 @@ server/ deliberately not site-mode gated shard.router.js (14) /public/shard/* incl. the anonymous SSE stream; never site-mode gated + atlas.router.js (6) /public/atlas/* — the spawn atlas. + NOT under /shard: nothing here + touches the sidecar, and unlike + /shard it IS site-mode gated site.router.js (4) /settings /status /version /contact — the group-root singletons; declares no router-level middleware @@ -409,7 +413,7 @@ when its maps are updated; the facet set is discovered from the tree, and the lo | `shard_regions` | `facet`, `name`, `type`, `priority`, `parent`, `rects` JSON | | `shard_landmarks` | `facet`, `name`, `grp`, `x`, `y`, `z` | | `shard_champion_spawns` | `slug` PK, `name`, `grp`, `type`, `random_type`, `facet`, `x`, `y`, `z`, `radius`, `label` | -| `shard_atlas_meta` | Singleton (`id = 1`), `payload` JSON (counts + a sha256 per source file), `imported_at` | +| `shard_atlas_meta` | Singleton (`id = 1`), `payload` JSON (counts, a sha256 per source file, `parserVersion`), `imported_at` | | `shard_atlas_pending` | Singleton (`id = 1`), `status` (`pending`/`rejected`), `payload` JSON, `detected_at` | The first seven are **import-owned**: a refresh empties and reloads every one inside a single @@ -430,6 +434,11 @@ The boot refresh is **best-effort by contract**: no configured path, an unreadab file or a database error is caught and logged, and the site comes up serving whatever atlas it had. The tree path comes from the `spawn_atlas_servuo_path` setting, falling back to `SERVUO_PATH`. +**A refresh re-derives when the tree changed OR the parser did.** `spawnAtlasSource.PARSER_VERSION` +is stored in `shard_atlas_meta` beside the source hashes and bumped whenever the parser produces +different data from identical files. Hashing the tree alone would strand an install whose maps never +change on whatever an older build derived — a corrected parse would ship and never reach the data. + Four column choices worth stating, because each one is a trap: - **`spawn_range`, not `range`**, and **`grp`, not `group`** — both are reserved words. @@ -463,7 +472,7 @@ are authoritative, and they answer different questions: | Artifact | Source of truth for | Generated by | |---|---|---| -| `server/routes.manifest.json` — mirrored as [api-route-inventory.json](./api-route-inventory.json) | **What URLs exist.** 200 public routes + 2 on the internal listener, sorted, method + path only. | `npm run routes:manifest`, by walking the live Express stack | +| `server/routes.manifest.json` — mirrored as [api-route-inventory.json](./api-route-inventory.json) | **What URLs exist.** 215 public routes + 2 on the internal listener, sorted, method + path only. | `npm run routes:manifest`, by walking the live Express stack | | `server/swagger/swagger-output.json` — served at `/api/docs` | **What each route means.** Parameters, bodies, response codes, security. | `npm run swagger`, from `#swagger.*` annotations | The split is deliberate: Swagger is annotation-derived, so an unannotated route is invisible in it and @@ -649,6 +658,12 @@ from the per-route **siteMode** middleware (§5), never from an auth gate. | POST | `/contact` | (rate-limited) send mail via SMTP; if unconfigured, respond `{fallback:"mailto", email}` | | GET | `/shard/ruleset` | the shard's own published ruleset (Protocol 3.0 `world.ruleset`): expansion, which optional systems are on, skill/stat caps, account and house limits, champion scroll rules, the save/restart schedule. Served from `shard_ruleset`, so it renders while the shard is down; live via `world.ruleset` on `/shard/stream`. Behind `requireFeature('ruleset')`. **`null`** means the shard has never published one — a real answer, distinct from a published ruleset. `caps.skill` / `caps.totalSkill` are in **tenths** (1000 = 100.0). | | GET | `/shard/features` | the shard features **this caller** may reach plus the audience rung they resolved to (§6.5), so a client hides nav it can't follow. Reports only what the caller can see — the list itself never discloses a gated feature. Consumed by the SPA header and (pending) the Android nav. | +| GET | `/atlas/creatures?q=&facet=&limit=&offset=` | the bestiary, most numerous first, with an unpaginated `total`. Static content parsed from the shard's ServUO tree — **not** sidecar-backed, which is why the atlas sits outside `/shard`, and unlike `/shard/*` it **is** site-mode gated. Behind `requireFeature('atlas')`. `?facet=` is matched exactly and never validated against a list (no facet name exists in the code); the filter is an `EXISTS` over the points rather than a JSON path or `JSON_SEARCH` built from caller input, whose `%`/`_` wildcards would make `?facet=%` match everything. | +| GET | `/atlas/creatures/:slug` | one creature: `places` (the point-in-rect aggregate — "lizardman → Shrines, Isamu-Jima, Yew"), `spawners` (the bounded raw list, with `spawnersTruncated`), `alsoHere`. **`points` is a COUNT and `spawners` is the LIST** — named apart so one key never means a number on one route and an array on another. `minDelay`/`maxDelay` are in **seconds**, normalised at parse time from the source's per-record minutes-or-seconds. 404 = no such creature in this atlas. | +| GET | `/atlas/regions?facet=&q=` | named regions and the rectangles that placed each spawner | +| GET | `/atlas/landmarks?facet=&q=` | points of interest, labelled by `group` ("Covetous", not "Level 1") | +| GET | `/atlas/champions?facet=` | the **configured** altar roster. Not `/shard/champs`, which is the live board. | +| GET | `/atlas/meta` | facets, counts and when the atlas was parsed. Game-world facts only — the ServUO path, source hashes and any pending refresh are operator detail and live on the admin route. | Public content GETs pass through the **siteMode** gate (§5). @@ -692,6 +707,10 @@ file a route sits in — that is the property the route manifest freezes. | GET | `/users/:id/trusted-devices` | list a user's active trusted devices (never tokens) | | DELETE | `/users/:id/trusted-devices` · `…/:deviceId` | revoke all / one of a user's trusted devices (logs `admin.trusted_device.revoke[_all]`) | | POST | `/users/:id/mfa/reset` | recover a locked-out user: disable TOTP + revoke all trusted devices + clear recovery codes (logs `admin.user.totp.reset`) | +| GET | `/shard/atlas` | spawn-atlas status (`adminOnly`): the ServUO path, whether the tree is readable, whether it has drifted from what is loaded, counts, facets, and any refresh staged for review. The public `/atlas/meta` reports the game world only; the filesystem detail is here. | +| POST | `/shard/atlas/import` | re-import without restarting; `{force}` ignores the hash gate. **An unreadable tree answers 200 with `status:"unavailable"`, not 500** — `refresh()` reports outcomes rather than throwing (the boot path must never be blocked by a bad tree) and that contract is preserved at the API. | +| POST | `/shard/atlas/approve` · `/shard/atlas/reject` | answer a refresh staged because it would REMOVE a facet. Approving **re-parses** the tree, so what lands matches it at approval time; rejecting is remembered against those source hashes so it does not re-prompt every restart. 404 when nothing is staged. | +| PUT | `/shard/atlas/path` | point the atlas at a different tree (persisted as `spawn_atlas_servuo_path`, which wins over `SERVUO_PATH`). Blank clears it. Deliberately **does not import** — moving the mount and reloading the world are separate decisions — and returns fresh status so the panel can offer the import next. | Every admin write logs to `activity_log`. diff --git a/website/SPAWN_ATLAS.md b/website/SPAWN_ATLAS.md index 2e9591b..f74d251 100644 --- a/website/SPAWN_ATLAS.md +++ b/website/SPAWN_ATLAS.md @@ -1,6 +1,6 @@ # Spawn atlas -**Status:** Data pipeline landed on `edge` (website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112)); API and client pages follow in a second PR. +**Status:** Complete on `edge` — data pipeline in website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112), API + pages in website [#113](https://gitea.whitlocktech.com/RunicGateway/website/pulls/113). **Design:** [`docs/link/v3.md` §6](../link/v3.md) — Protocol 3.0 Part C. The spawn atlas is a browsable catalogue of what the shard *contains*: which @@ -91,7 +91,7 @@ A rejection is remembered against those exact source hashes, so a declined refresh does not re-prompt on every restart. Change the tree and the hashes differ, which asks again. -From the admin panel (second PR), or from the CLI: +From **Admin → Spawn Atlas**, or from the CLI: ```bash cd website/server @@ -259,3 +259,96 @@ Parsing notes: - `` is `Type:MX=n:SB=…` segments joined by `:OBJ=`. Split on `:OBJ=` *first* — a naive `split(':')` shreds it. A single Trammel point carries six types. +- **Respawn delays are stored in two different units, per record.** XmlSpawner + writes `MinDelay`/`MaxDelay` in minutes, and switches to seconds only when a + spawner's delay does not divide into whole minutes — flagging that with + `DelayInSec` on the same record. A `5` therefore means five *minutes* on one + spawner and five *seconds* on the next, and both are plausible respawn times, + so a reader assuming either unit is silently wrong about the other. Stock + ServUO 57.4 has ~170 second-flagged spawners out of 6,455. The parser + normalises everything to **seconds**; the API and UI carry seconds throughout. + +### The parser version + +`spawnAtlasSource.js` exports `PARSER_VERSION`, stored in `shard_atlas_meta` +alongside the source hashes and bumped whenever the parser derives **different +data from identical files** — a fixed misreading, a new field, a changed unit. + +A refresh re-derives when the tree changed **or** the parser did. Hashing the +tree alone would be a trap: an install whose maps never change would keep serving +whatever an older build derived, indefinitely, and a deploy that corrects the +parse would never reach the data. A version mismatch counts as drift, so the +correction lands on the next boot without an operator having to know it happened. + +## The API + +Everything is served from MariaDB. Nothing on this path touches the sidecar, so +the pages stay complete while the shard is down — which is why the routes sit at +`/api/v1/public/atlas` and **not** under `/public/shard`, where a prefix means +"sidecar-dependent". Unlike `/shard/*`, they *are* `siteMode`-gated, like +`/posts` and `/wiki`: a bestiary is site content and follows site content's rules. + +Every route carries `requireFeature('atlas')` — **404** when an admin has +disabled the feature (its pages must not reveal that it exists) and **403** when +the caller sits below its configured audience. The default is `anonymous`, so the +gates are inert until an admin changes something. Responses are field-projected +like every other shard read; `atlas` declares no sensitive fields today, and the +projection call is there so the first one that does is covered by construction +rather than by a retrofit ([`v3.md` §3.6.1](../link/v3.md)). + +| Route | Answers | +|---|---| +| `GET /atlas/creatures?q=&facet=&limit=&offset=` | The bestiary, most numerous first, paginated with an unpaginated `total` | +| `GET /atlas/creatures/:slug?facet=&points=` | One creature: `places`, `spawners`, `alsoHere` | +| `GET /atlas/regions?facet=&q=` | Named regions and their rectangles | +| `GET /atlas/landmarks?facet=&q=` | Points of interest, labelled by `group` | +| `GET /atlas/champions?facet=` | The configured altar roster | +| `GET /atlas/meta` | Facets, counts and when the atlas was parsed | + +Two shapes worth knowing: + +- **`places` is the aggregate the atlas exists for.** "Lizardman → Shrines, + Isamu-Jima, Yew", grouped in SQL rather than by summing 6,455 point rows in + Node. `spawners` is the raw list underneath it, bounded, with + `spawnersTruncated` saying when it was cut. +- **`points` is a COUNT, `spawners` is the LIST.** The two are named apart + deliberately: the same key meaning a number on the search route and an array on + the detail route is the kind of thing a client only discovers in production. + +`GET /atlas/meta` reports the **game world only**. The ServUO path, the per-file +hashes and any pending refresh describe the operator's filesystem, and live on +the admin route instead. + +A facet is never validated against a list — nothing in the codebase names one. +`?facet=` is length-bounded and matched exactly, so an unknown name returns an +empty result rather than an error. The filter is an `EXISTS` over the points and +deliberately not a JSON path or `JSON_SEARCH` built from caller input: that +function treats `%` and `_` as wildcards, which would make `?facet=%` match +everything. + +## The admin panel + +**Admin → Spawn Atlas** (`/admin/shard-atlas`, admin-only — it reads a path on +the server's filesystem and replaces every atlas table, which is closer to a +deploy action than to moderation). + +| Route | Does | +|---|---| +| `GET /admin/shard/atlas` | Status: path, readable, drift, counts, facets, pending | +| `POST /admin/shard/atlas/import` | Import now; `{ force: true }` ignores the hash gate | +| `POST /admin/shard/atlas/approve` | Apply a staged refresh, facet loss and all | +| `POST /admin/shard/atlas/reject` | Keep the current atlas; remember the decision | +| `PUT /admin/shard/atlas/path` | Point the atlas at a different tree | + +Three behaviours that are deliberate: + +- **An unreadable tree is a 200, not a 500.** `refresh()` reports outcomes rather + than throwing, because the boot path must never be stopped by a bad tree, and + that contract is preserved at the API. The panel says *"The tree could not be + read: …"*; a 500 would say only that something broke. +- **Setting the path does not import.** Moving the mount and reloading the world + are separate decisions, and an operator fixing a typo should not have a + multi-thousand-row replace happen under them. The response carries fresh status + so the panel can offer the import as the next step. +- **Every action is written to the admin activity log** (`shard.atlas.import` / + `.approve` / `.reject` / `.path`). diff --git a/website/api-route-inventory.json b/website/api-route-inventory.json index 63a0207..f42fcb4 100644 --- a/website/api-route-inventory.json +++ b/website/api-route-inventory.json @@ -257,6 +257,26 @@ "method": "GET", "path": "/api/v1/admin/shard/accounts" }, + { + "method": "GET", + "path": "/api/v1/admin/shard/atlas" + }, + { + "method": "POST", + "path": "/api/v1/admin/shard/atlas/approve" + }, + { + "method": "POST", + "path": "/api/v1/admin/shard/atlas/import" + }, + { + "method": "PUT", + "path": "/api/v1/admin/shard/atlas/path" + }, + { + "method": "POST", + "path": "/api/v1/admin/shard/atlas/reject" + }, { "method": "GET", "path": "/api/v1/admin/shard/audit" @@ -313,6 +333,14 @@ "method": "GET", "path": "/api/v1/admin/shard/vendors/:account" }, + { + "method": "GET", + "path": "/api/v1/admin/shard/visibility" + }, + { + "method": "PUT", + "path": "/api/v1/admin/shard/visibility" + }, { "method": "PUT", "path": "/api/v1/admin/site-mode" @@ -705,6 +733,30 @@ "method": "GET", "path": "/api/v1/player/shard/vendors/:account" }, + { + "method": "GET", + "path": "/api/v1/public/atlas/champions" + }, + { + "method": "GET", + "path": "/api/v1/public/atlas/creatures" + }, + { + "method": "GET", + "path": "/api/v1/public/atlas/creatures/:slug" + }, + { + "method": "GET", + "path": "/api/v1/public/atlas/landmarks" + }, + { + "method": "GET", + "path": "/api/v1/public/atlas/meta" + }, + { + "method": "GET", + "path": "/api/v1/public/atlas/regions" + }, { "method": "POST", "path": "/api/v1/public/contact" @@ -737,6 +789,10 @@ "method": "GET", "path": "/api/v1/public/shard/economy" }, + { + "method": "GET", + "path": "/api/v1/public/shard/features" + }, { "method": "GET", "path": "/api/v1/public/shard/feed" @@ -769,6 +825,10 @@ "method": "GET", "path": "/api/v1/public/shard/presence" }, + { + "method": "GET", + "path": "/api/v1/public/shard/ruleset" + }, { "method": "GET", "path": "/api/v1/public/shard/status" -- 2.49.1 From 8e857a9c8d896c90d886964dce7459c4b6aa718e Mon Sep 17 00:00:00 2001 From: wtclaude Date: Tue, 28 Jul 2026 21:05:04 -0500 Subject: [PATCH 10/19] docs(link): points.board, the leaderboards API, and what a real shard changed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Protocol 3.0 §7 lands across servuo-plugins, link and website; this is the matching documentation. INTEGRATION.md - points.board in the event catalog: one frame per system, never a delta, no points.remove (the shard's system set is fixed at startup). Four gotchas called out, all of them things a consumer gets wrong by default: 1. maxPoints: 0 means UNCAPPED, not "zero allowed" — and on a real shard it is the COMMON case, so anything rendering points/maxPoints must special-case it or divide by zero on the happy path. 2. nameString is usually null, with a cliloc in nameNumber — so humanising the system key is the primary display path, not a fallback. 3. players counts players actually holding points, not table size: ten of the ~25 systems keep a zero-point row per character ever created, so the raw count would report the shard's whole census. 4. Entries carry serial + name only, never acct/webId. - The char.profile `points` block, and why `rank` is absent by default. - GET /points and /points/:system, including why 404 (never published) and 200-with-empty-top (published, nobody scored) are different answers. v3.md - B/2 marked done in both the progress table and §9. - NEW §7.5, "what the run against a real shard changed" — the same record §6.1 and §6.2 keep. Four corrections the plan could not have anticipated from reading PointsSystem.cs, the sharpest being that (long)double.MaxValue is an unchecked conversion yielding long.MinValue, which published "maxPoints": -9223372036854775808 on the first live sweep. Also records that GetEntry/GetPoints mutate the world on AutoAdd systems and so cannot be used in a read model, and the one deliberate deviation from §7.4: the visibility field rule must key on the wire's `name`, not the descriptive `characterName`, or it is silently inert. BACKEND_DESIGN.md — shard_points_boards (including why the top-N list stays in the payload and why listing orders by COALESCE(name, system)), plus the two new public routes. PLAN.md — 3.0 phasing brought current: the spawn atlas and points.board added to what has shipped, and the Points* keys noted in the config-key paragraph. PROJECT_TREE.md files are deliberately untouched — they are CI-generated and say so. Co-Authored-By: Claude --- link/INTEGRATION.md | 89 +++++++++++++++++++++++++++++++++++++++ link/PLAN.md | 13 ++++-- link/v3.md | 48 +++++++++++++++++++-- website/BACKEND_DESIGN.md | 30 +++++++++++++ 4 files changed, 173 insertions(+), 7 deletions(-) diff --git a/link/INTEGRATION.md b/link/INTEGRATION.md index f46e686..697235f 100644 --- a/link/INTEGRATION.md +++ b/link/INTEGRATION.md @@ -381,6 +381,74 @@ Absent entirely if the shard runs `Bridge.RulesetEnabled=false` or an older plug This **supersedes the `world.systems` frame** sketched in [`PROTOCOL_2.md`](PROTOCOL_2.md) §10.4 and never implemented; the `systems` block above is what that asked for. +#### Points / loyalty leaderboards (Protocol 3.0) + +ServUO carries ~25 separate point currencies — Queen's Loyalty, Void Pool, Casino, Clean Up Britannia, +the nine city loyalties, Blackthorn, the Doom / Khaldun / Kotl treasure systems — every one a standing +players accumulate over months, and none of them visible outside an in-game gump before 3.0. + +A diff sweep (default 300 s), **one frame per system** rather than one large frame for all of them, +matching `champ.update` / `guild.update`. A system is emitted only when its top N or its participant +count actually changes. + +| kind | fields | notes | +|------|--------|-------| +| `points.board` | `system`, `nameString`, `nameNumber`, `maxPoints`, `showOnGump`, `players`, `top[]` | One system's complete board — **never a delta**. The latest frame for a `system` replaces the previous one outright. `top[]` entries are `{rank, serial, name, points}`. | + +`system` is the shard's own `PointsType` enum name (`QueensLoyalty`, `CleanUpBritannia`, …) and is the +board's stable key. There is deliberately **no `points.remove`**: the set of systems is fixed at startup +by `PointsSystem.Configure`, so a system cannot disappear at runtime — the same argument `city.update` +makes for cities. + +```json +{"kind":"points.board","system":"QueensLoyalty", + "nameString":"Queen's Loyalty","nameNumber":1114938, + "maxPoints":15000,"showOnGump":true,"players":842, + "top":[{"rank":1,"serial":"0x1A2B","name":"Darrow","points":29500}, + {"rank":2,"serial":"0x1A2C","name":"Mireille","points":21000}], + "t":1752489280000} +``` + +**Four things consumers get wrong.** + +1. **`maxPoints` of `0` means UNCAPPED, not "zero points allowed".** ServUO's idiom for an uncapped + system is `double.MaxValue` (`DespiseCrystals`, `ShameCrystals` and `VoidPool` all use it), which + the plugin normalises to `0` rather than emitting a nonsense integer. On a real shard **most + systems are uncapped**, so a UI that renders `points / maxPoints` must special-case this or it will + divide by zero on the common path. +2. **`nameString` is usually `null`.** The shard's `Name` is a `TextDefinition`, which may carry a + literal *or* a cliloc id, and in practice most systems use the cliloc — so `nameNumber` is set and + `nameString` is `null`. Resolve clilocs consumer-side; failing that, humanising the `system` key + ("CleanUpBritannia" → "Clean Up Britannia") reads better than showing a bare number. This is the + same contract `titles.reward` already documents. +3. **`players` counts players who actually hold points**, not the size of the system's table. Ten of + the ~25 systems have `AutoAdd = true` and therefore keep a zero-point row for every character that + has ever logged in, so the raw table size would report the shard's entire character census as that + system's participants. +4. **Entries carry `serial` and `name` only — never `acct` or `webId`.** A board is the widest-audience + surface the bridge has, so the account name of every ranked player deliberately does not cross the + wire; resolve serial → site user from your own link mirror if you need it. + +Absent entirely if the shard runs `Bridge.PointsLeaderboardEnabled=false` or an older plugin. Render +from `GET /points` (§6) on connect, then keep live with this event. + +##### `char.profile` gains a `points` block + +Read-model enrichment on the existing kind — there is **no** request kind for one character's points, +the same precedent `titles` set in [`PROTOCOL_2.md`](PROTOCOL_2.md) §10.3: + +```json +"points":[{"system":"QueensLoyalty","nameString":"Queen's Loyalty","nameNumber":1114938, + "points":29500,"maxPoints":15000}] +``` + +Systems where the character has no entry, or an entry at zero, are **omitted** — otherwise every sheet +would carry ~25 zeroes. `maxPoints` follows the same `0 == uncapped` rule as the board. + +`rank` is **absent by default** and appears only when the shard runs `Bridge.PointsProfileRank=true`: +a points lookup stops at the character's own row, but a rank must count every row that beats them, in +every system, on every profile build. Derive rank from `points.board` instead for anyone in the top N. + --- ## 5. REST — read queries @@ -740,6 +808,27 @@ worse than one that is briefly stale. Keep it current with the `world.ruleset` s `Bridge.RulesetEnabled=false`. That is a real answer distinct from a published ruleset, and worth rendering differently ("not published yet") rather than as an empty ruleset. +### Points / loyalty leaderboards (Protocol 3.0) + +``` +GET /points +→ { "boards": [ {"kind":"points.board","system":"QueensLoyalty","nameString":"Queen's Loyalty", + "nameNumber":1114938,"maxPoints":15000,"showOnGump":true,"players":842, + "top":[{"rank":1,"serial":"0x1A2B","name":"Darrow","points":29500}, ...],"t":...}, ... ] } + +GET /points/{system} # e.g. /points/QueensLoyalty +→ {"kind":"points.board","system":"QueensLoyalty", ... } +``` + +Every system's latest board, or one by its `PointsType` name (§4 for the frame and its four gotchas). +Served from the sidecar's projection, kept current by the `points.board` stream, ordered by display +name. Survives a sidecar restart — which matters more here than for live state, since these are +standings built over months and blanking them during a restart reads as data loss. + +`GET /points/{system}` returns **404** for a system the shard has never published (an unknown name, or +one excluded by `Bridge.PointsSystems`). That is distinct from a published board nobody has scored in +yet, which is **200** with an empty `top[]` — and the two are worth rendering differently. + --- ## 7. Status codes diff --git a/link/PLAN.md b/link/PLAN.md index cb36025..709fead 100644 --- a/link/PLAN.md +++ b/link/PLAN.md @@ -318,10 +318,14 @@ Counts in `hello` are a live snapshot taken on the Core thread, not a cached val **Beyond 1.0.** Phases above are the 1.0 read/event plane. Protocol 2.0's phasing (provisioning + world-state boards) is [`PROTOCOL_2.md`](PROTOCOL_2.md) §13; Protocol 3.0's (visibility framework, shard content and standings) is [`v3.md`](v3.md) §9, which also tracks what has landed. Shipped from -3.0 so far: **Part A** — the visibility framework — and **`world.ruleset`** ([`v3.md`](v3.md) §5), -`BridgeRuleset.cs`, the first bridge stream that is neither an event subscription nor a sweep: it is +3.0 so far: **Part A** — the visibility framework — **`world.ruleset`** ([`v3.md`](v3.md) §5), +`BridgeRuleset.cs`, the first bridge stream that is neither an event subscription nor a sweep (it is emitted once per connect, like `server.hello`, because shard config changes only when an operator -edits a file. +edits a file) — the **spawn atlas** ([`v3.md`](v3.md) §6), which is website-only and touches no wire +at all — and **`points.board`** ([`v3.md`](v3.md) §7), `BridgePoints.cs`, the loyalty/points +leaderboards. `BridgePoints` is the widest read the bridge performs: ten of ServUO's ~25 point systems +keep a row for every character ever created, so it selects the top N in a single bounded pass rather +than sorting, and runs on a deliberately slow 300 s interval. ### Config keys (`Config/Bridge.cfg`) @@ -338,7 +342,8 @@ Read in `Configure()` via `Config.Get("Bridge.", default)`. Key scope is The set above is the 1.0 sample, not the current one — every later phase added keys (sweep intervals for each board, the town-crier/news caps, the admin write plane, account provisioning, and 3.0's -`RulesetEnabled` / `PublicConnectAddress` / `RulesetIncludeSchedule`). **`servuo-plugins/overlay/Config/Bridge.cfg` +`RulesetEnabled` / `PublicConnectAddress` / `RulesetIncludeSchedule`, and the `Points*` block). +**`servuo-plugins/overlay/Config/Bridge.cfg` is the authoritative, commented list**; `BridgeConfig.cs` holds the defaults. --- diff --git a/link/v3.md b/link/v3.md index d23b6a7..5b6862d 100644 --- a/link/v3.md +++ b/link/v3.md @@ -14,7 +14,7 @@ Each part is marked off here as it lands on `edge`. §9 carries the same state p | 1 | **A** — visibility framework + actor-leak fix (§3) | ✅ **Done** | website [#109](https://gitea.whitlocktech.com/RunicGateway/website/pulls/109) + [#110](https://gitea.whitlocktech.com/RunicGateway/website/pulls/110), docs [#64](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/64) + [#65](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/65) | | 2 | **B/1** — `world.ruleset` (§5) | ✅ **Done** | servuo-plugins [#3](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/3), link [#17](https://gitea.whitlocktech.com/RunicGateway/link/pulls/17), website [#111](https://gitea.whitlocktech.com/RunicGateway/website/pulls/111), docs [#66](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/66) | | 3 | **C** — spawn atlas (§6) | ✅ **Done** | website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112) (parsers + CLI + tables) + [#113](https://gitea.whitlocktech.com/RunicGateway/website/pulls/113) (API + pages + admin panel), docs [#67](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/67) + [#68](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/68) | -| 4 | **B/2** — `points.board` (§7) | ⬜ Not started | — | +| 4 | **B/2** — `points.board` (§7) | ✅ **Done** | servuo-plugins [#4](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/4), link [#18](https://gitea.whitlocktech.com/RunicGateway/link/pulls/18), website [#114](https://gitea.whitlocktech.com/RunicGateway/website/pulls/114), docs [#69](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/69) | | 5 | **B/3** — `vendor.listing` (§8) | ⬜ Not started | — | | 6 | **Cutover** — `PROTOCOL_VERSION` 2→3 (§4) | ⬜ Not started | — | @@ -507,7 +507,12 @@ retrofit nobody remembers to do. --- -## 7. Part B/2 — `points.board` +## 7. Part B/2 — `points.board` ✅ Done + +*Landed on `edge`: servuo-plugins [#4](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/4), +link [#18](https://gitea.whitlocktech.com/RunicGateway/link/pulls/18), website [#114](https://gitea.whitlocktech.com/RunicGateway/website/pulls/114), +docs [#69](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/69). Verified against the real ServUO tree +per §11 — see §7.5 for what that run changed.* Two deliverables: a diff sweep for the boards, and a `points` block folded into `char.profile` — the `PROTOCOL_2.md` §10.3 `titles` precedent (read-model enrichment, no new request kind). @@ -590,6 +595,43 @@ Client — NEW `routes/public/Leaderboards.jsx` at `/site/leaderboards`; a "Loya added to `components/CharacterSheet.jsx`, one edit serving both `PlayerCharacter.jsx` and `AdminCharacter.jsx`. +### 7.5 What the run against a real shard changed + +The plan above was written from reading `PointsSystem.cs`. Booting the actual shard (ServUO 57.4, a +43,011-mobile world) and letting one sweep run corrected four things — all of them invisible to a +fake-shard test, because a fake shard emits whatever the spec says it should. + +1. **`maxPoints` overflowed to `long.MinValue`.** `MaxPoints` is a `double`, and ServUO's idiom for an + uncapped system is `double.MaxValue` — which `DespiseCrystals`, `ShameCrystals` and `VoidPool` all + use. `(long)double.MaxValue` in C# is an **unchecked** conversion: it does not throw, it yields + `long.MinValue`, and the first real sweep published + `"maxPoints": -9223372036854775808` for three of the five live boards. Fixed with `Cap()` / + `Score()` converters that normalise anything unrepresentable to `0`, which is now the wire's + documented **"uncapped"** value. Worth stating plainly because it inverts the obvious reading: + **on a real shard, `maxPoints: 0` is the common case, not an edge case**, so any UI dividing by it + must special-case it. +2. **`nameString` is usually `null`.** Most systems define their `Name` as a cliloc rather than a + literal: four of the five boards on the live shard came back `nameString: null` with only + `nameNumber` set. The humanise-the-`system`-key fallback is therefore the *primary* display path, + not a defensive nicety, and both the leaderboards page and the character sheet lead with it. +3. **`GetEntry`/`GetPoints` cannot be used in the read model.** `GetEntry(from, create: false)` still + calls `AddEntry` when the system has `AutoAdd` (`PointsSystem.cs:207`) — it **mutates the world**. + Ten of the ~25 systems have `AutoAdd = true`, so a profile built with the obvious accessor would + have appended up to ten rows to the points save file every time anyone viewed a character sheet. + `BridgeProfile.WritePoints` hand-rolls a read-only scan instead, and says so loudly. +4. **`players` had to be redefined.** §7.2 called for "the entry count", but those same ten `AutoAdd` + systems hold a zero-point row per character ever created — so the raw count reports the shard's + whole census as one system's participants. It is now the number of players actually holding points, + which is both the honest number and a strictly better diff signal (it moves when someone scores, + not when someone logs in for the first time). + +One deviation from the plan as written, for the same class of reason: §7.4 named the per-field +visibility rule `characterName`, but `projectValue` matches on the **literal JSON key**, and the wire +key is `name`. A rule under the descriptive name would have been silently inert — an admin tightening +character names would have got no enforcement and no error, exactly the failure §3.6.1 records for the +flattened `ownerAcct`. `FEATURES.leaderboards.fields` therefore keys on `name`, with a test that fails +if it is renamed back. + --- ## 8. Part B/3 — `vendor.listing` @@ -700,7 +742,7 @@ inherently up to one full cycle old, and the UI must say so. | 1 | **A** — visibility framework + actor-leak fix | website, docs | none | ✅ Done | | 2 | **B/1** — `world.ruleset` (§5) | all four | new kind | ✅ Done | | 3 | **C** — spawn atlas (§6) | website, docs | none | ✅ Done | -| 4 | **B/2** — `points.board` (§7) | all four | new kind + `char.profile` field | ⬜ | +| 4 | **B/2** — `points.board` (§7) | all four | new kind + `char.profile` field | ✅ Done | | 5 | **B/3** — `vendor.listing` (§8) | all four | new kinds | ⬜ | | 6 | **Cutover** — `PROTOCOL_VERSION` 2→3, `edge` → `main` | all four | the bump | ⬜ | diff --git a/website/BACKEND_DESIGN.md b/website/BACKEND_DESIGN.md index fd26513..4304050 100644 --- a/website/BACKEND_DESIGN.md +++ b/website/BACKEND_DESIGN.md @@ -379,6 +379,34 @@ they are cheap to display — the same payload-plus-hoisted-columns shape `shard served as `null` rather than `{}`: "not published yet" and "published, everything off" are different answers and the page renders them differently. +### shard_points_boards — points / loyalty leaderboards (Protocol 3.0) + +One row per point system, keyed by the shard's own `PointsType` name (`QueensLoyalty`, +`CleanUpBritannia`, …). The shard carries ~25 of these, each a standing players build over months. +Columns: `system` (PK), `name`, `name_cliloc`, `max_points`, `players`, `show_on_gump`, `payload` JSON +(the whole `points.board` frame), `t`, `updated_at`. + +**The top-N list stays inside `payload`** rather than being normalized into a `shard_points_entries` +table. It is a fixed-size list (10 by default) that is only ever read whole — exactly like +`shard_governors.candidates` — so normalizing buys nothing until something needs a per-character +reverse lookup, and a character's own standings already ride inside `char.profile` instead. + +Board state, not events: `points.board` is **not** in `LOGGED_KINDS`, for the same reason +`guild.update` isn't. The shard emits a frame every time anyone's score moves a top ten, so logging +would grow `shard_events` without bound for something whose only interesting value is its latest +version. There is also **no delete path** — the shard's set of systems is fixed at startup, so there is +no `points.remove` to mirror. + +Two values carry non-obvious meanings, both set by the plugin and both documented in +[`link/INTEGRATION.md`](../link/INTEGRATION.md) §4: + +- **`max_points = 0` means uncapped**, and on a real shard that is the *common* case (ServUO's + uncapped idiom is `double.MaxValue`, which the plugin normalises to 0). Anything rendering + `points / max_points` must special-case it. +- **`name` is usually NULL**, with `name_cliloc` set instead — most systems name themselves with a + cliloc rather than a literal. Listing therefore orders by `COALESCE(name, system)`, so boards + awaiting cliloc resolution sort by their own key rather than clumping together under NULL. + ### shard_feature_visibility — per-feature audience config (Protocol 3.0) One row per shard feature: `feature` (PK), `enabled`, `audience` (a rung on the ladder in §6.5), @@ -657,6 +685,8 @@ from the per-route **siteMode** middleware (§5), never from an auth gate. | GET | `/wiki/:slug` | single page | | POST | `/contact` | (rate-limited) send mail via SMTP; if unconfigured, respond `{fallback:"mailto", email}` | | GET | `/shard/ruleset` | the shard's own published ruleset (Protocol 3.0 `world.ruleset`): expansion, which optional systems are on, skill/stat caps, account and house limits, champion scroll rules, the save/restart schedule. Served from `shard_ruleset`, so it renders while the shard is down; live via `world.ruleset` on `/shard/stream`. Behind `requireFeature('ruleset')`. **`null`** means the shard has never published one — a real answer, distinct from a published ruleset. `caps.skill` / `caps.totalSkill` are in **tenths** (1000 = 100.0). | +| GET | `/shard/points` | every points/loyalty leaderboard the shard publishes (Protocol 3.0 `points.board`) — Queen's Loyalty, Void Pool, the nine city loyalties, Clean Up Britannia, … Served from `shard_points_boards`, so it renders while the shard is down; live via `points.board` on `/shard/stream`. Behind `requireFeature('leaderboards')`, ordered by display name. **`maxPoints: 0` means uncapped** (the common case), and `nameString` is usually `null` with `nameNumber` holding a cliloc — resolve client-side or humanise the `system` key. | +| GET | `/shard/points/:system` | one board by the shard's `PointsType` name (e.g. `QueensLoyalty`); `:system` must match `/^[A-Za-z][A-Za-z0-9_]{0,47}$/` or **400** before any query runs. **404** = the shard has never published that system, which is distinct from a published board nobody has scored in yet (**200** with an empty `top`). | | GET | `/shard/features` | the shard features **this caller** may reach plus the audience rung they resolved to (§6.5), so a client hides nav it can't follow. Reports only what the caller can see — the list itself never discloses a gated feature. Consumed by the SPA header and (pending) the Android nav. | | GET | `/atlas/creatures?q=&facet=&limit=&offset=` | the bestiary, most numerous first, with an unpaginated `total`. Static content parsed from the shard's ServUO tree — **not** sidecar-backed, which is why the atlas sits outside `/shard`, and unlike `/shard/*` it **is** site-mode gated. Behind `requireFeature('atlas')`. `?facet=` is matched exactly and never validated against a list (no facet name exists in the code); the filter is an `EXISTS` over the points rather than a JSON path or `JSON_SEARCH` built from caller input, whose `%`/`_` wildcards would make `?facet=%` match everything. | | GET | `/atlas/creatures/:slug` | one creature: `places` (the point-in-rect aggregate — "lizardman → Shrines, Isamu-Jima, Yew"), `spawners` (the bounded raw list, with `spawnersTruncated`), `alsoHere`. **`points` is a COUNT and `spawners` is the LIST** — named apart so one key never means a number on one route and an array on another. `minDelay`/`maxDelay` are in **seconds**, normalised at parse time from the source's per-record minutes-or-seconds. 404 = no such creature in this atlas. | -- 2.49.1 From be9f5019fa0985edd448ed341dc1effa36753994 Mon Sep 17 00:00:00 2001 From: wtclaude Date: Wed, 29 Jul 2026 04:22:05 -0500 Subject: [PATCH 11/19] =?UTF-8?q?docs(link):=20the=20cliloc=20table,=20and?= =?UTF-8?q?=20why=20=C2=A78.6's=20recommendation=20was=20not=20implementab?= =?UTF-8?q?le?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Protocol 3.0 §8.6 resolved as its own website-only change, landing ahead of §8 so the marketplace ships with real item names. Matching documentation for website #TBD. NEW website/CLILOCS.md — operator-facing: why the conversion step exists, how to convert, how to configure the path, the refresh contract, what gets stored and how names are applied. link/v3.md §8.6 rewritten. Two things in the original recommendation turned out to be wrong, and both are recorded because the reasoning generalises: 1. The committed db/data/clilocs.json artifact predates the Part C corrections (§6.1) and violates both — no committed snapshot of derived content, and nothing EA-derived ever shipped. UO's strings are EA's, exactly as the creature sprites are. 2. "scripts/buildClilocs.js reads the UO client's Cliloc.enu" is not possible. EVERY current client ships its cliloc files compressed (first DWORD's high byte 0x8E, the Mythic container); the plain layout is what those files looked like before that change, and parsing one as the other does not fail cleanly — it yields ~19k records with negative ids, 1,722 distinct keys out of 19,508, one 62 KB "string", and a truncation somewhere in the middle. ServUO's own Ultima.StringList cannot read it either, so VendorSearch.GetItemName is already inert on such a shard and the work could not be pushed to the plugin. That second point also retires an open question in §8.2: the warning never to call GetItemName in the market sweep costs us nothing we could otherwise have had, because the in-game Vendor Search gump has the same gap. Three traps found by building it are recorded: StringList.SaveStringList RE-COMPRESSES on save (its output is byte-identical to its compressed input, because its purpose is round-tripping a file back into the client); trimming a text line before splitting silently drops the ~half of a table that is empty strings; and Number('') is 0, not NaN. Also updated: - Progress and §9 sequencing tables: order 5 split into 5a (this, website only) and 5b (the four-repo wire change). - website/BACKEND_DESIGN.md — shard_clilocs / shard_cliloc_meta, the three admin routes, and why there is no staged-approval flow and no public route. - link/INTEGRATION.md — the char.profile field note now says explicitly not to expect the shard to resolve clilocs, and points at CLILOCS.md. - §10 documentation obligations list CLILOCS.md. Documentation only. Every claim was written after the corresponding behaviour was observed running: the compressed-format finding and the parse failures come from the real client files on this machine, and the counts (123,490 parsed → 67,496 stored) and timings from importing them into the live MariaDB. PROJECT_TREE.md files are deliberately untouched — they are CI-generated by the sync-project-tree workflow and say so in their header. Co-Authored-By: Claude --- link/INTEGRATION.md | 2 +- link/v3.md | 73 +++++++++++-- website/BACKEND_DESIGN.md | 48 +++++++++ website/CLILOCS.md | 216 ++++++++++++++++++++++++++++++++++++++ 4 files changed, 327 insertions(+), 12 deletions(-) create mode 100644 website/CLILOCS.md diff --git a/link/INTEGRATION.md b/link/INTEGRATION.md index 697235f..1b5ba20 100644 --- a/link/INTEGRATION.md +++ b/link/INTEGRATION.md @@ -488,7 +488,7 @@ Full character sheet: stats, all trained skills, worn equipment with flattened i Field notes: - `skills[].base` is trained value, `value` includes item/temp bonuses, `cap` is the cap. **Do not assume `base <= cap`** — GM characters can exceed it. - `equipment[].mods` is a flattened map of every non-zero AOS attribute on the item (weapon or armor). Empty `{}` for plain items. -- Item names are usually **clilocs**, not strings: use `name` when present, otherwise resolve `cliloc` against a UO cliloc table on the site. +- Item names are usually **clilocs**, not strings: use `name` when present, otherwise resolve `cliloc` against a UO cliloc table on the site. **Do not expect the shard to resolve them for you** — on any modern client ServUO's own `Ultima.StringList` cannot read the client's compressed cliloc files, so `VendorSearch.GetItemName` returns `item.Name` and the in-game Vendor Search gump has the same gap. Building that table is a consumer-side job; the website's is described in [`website/CLILOCS.md`](../website/CLILOCS.md). - `titles` (Protocol 2.0): `selected` is the index into `reward` currently displayed (`-1` if none). `fameKarma`/`skill` are computed display titles, omitted when the character has none. `reward` entries may be a **cliloc number as a string** or a literal string — resolve numeric ones against your cliloc table, same as item names. - Errors: unknown account → **404** `{"kind":"bridge.error","reason":"unknown account"}`; bad slot → **404**/**400** similarly. diff --git a/link/v3.md b/link/v3.md index 5b6862d..3ef0cc5 100644 --- a/link/v3.md +++ b/link/v3.md @@ -15,9 +15,14 @@ Each part is marked off here as it lands on `edge`. §9 carries the same state p | 2 | **B/1** — `world.ruleset` (§5) | ✅ **Done** | servuo-plugins [#3](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/3), link [#17](https://gitea.whitlocktech.com/RunicGateway/link/pulls/17), website [#111](https://gitea.whitlocktech.com/RunicGateway/website/pulls/111), docs [#66](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/66) | | 3 | **C** — spawn atlas (§6) | ✅ **Done** | website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112) (parsers + CLI + tables) + [#113](https://gitea.whitlocktech.com/RunicGateway/website/pulls/113) (API + pages + admin panel), docs [#67](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/67) + [#68](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/68) | | 4 | **B/2** — `points.board` (§7) | ✅ **Done** | servuo-plugins [#4](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/4), link [#18](https://gitea.whitlocktech.com/RunicGateway/link/pulls/18), website [#114](https://gitea.whitlocktech.com/RunicGateway/website/pulls/114), docs [#69](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/69) | -| 5 | **B/3** — `vendor.listing` (§8) | ⬜ Not started | — | +| 5a | **B/3 dependency** — cliloc table (§8.6) | 🟨 In review | website [#TBD](https://gitea.whitlocktech.com/RunicGateway/website/pulls), docs [#TBD](https://gitea.whitlocktech.com/RunicGateway/docs/pulls) | +| 5b | **B/3** — `vendor.listing` (§8) | ⬜ Not started | — | | 6 | **Cutover** — `PROTOCOL_VERSION` 2→3 (§4) | ⬜ Not started | — | +Order 5 split in two once §8.6's cliloc dependency turned out to be a client-format problem rather +than a parser (see §8.6). 5a is website-only and lands first so the marketplace ships with real item +names; 5b is the four-repo wire change. + --- ## 1. Why 3.0 @@ -715,17 +720,62 @@ admin can turn the stream on. `uoLinkSocket` paginates `/market` on reconnect, b `/market/vendors/:serial`, behind `requireFeature('market')`. **Rate-limit it** — this is the first genuinely expensive public endpoint; `express-rate-limit` is already a dependency. -### 8.6 The open dependency — cliloc names +### 8.6 The open dependency — cliloc names ✅ Resolved (shipped ahead of §8) -`CharacterSheet.jsx:14-15` already documents the gap ("without a cliloc table on the site we can only -show literals") and renders equipment as `id {itemId}`. Search-by-name needs that table. +`CharacterSheet.jsx:14-15` documented the gap ("without a cliloc table on the site we can only +show literals") and rendered equipment as `id {itemId}`. Search-by-name needs that table. -- **Recommended:** `scripts/buildClilocs.js` reads the UO client's `Cliloc.enu` → committed - `db/data/clilocs.json`; ingest denormalizes into `shard_vendor_items.display_name`. Same - build-artifact pattern as §6, and it **also fixes the character sheet**. -- **Fallback:** ship with item-art + price + region filters, and name search only over renamed items. +**Resolved as its own website-only change, landed BEFORE the market so `/site/market` ships with real +item names.** Full design and operator guide: [`docs/website/CLILOCS.md`](../website/CLILOCS.md). +Ingest denormalizes the resolved name into `shard_vendor_items.display_name` as planned. -This decision is the reason §8 is sequenced last. +Two things in the original recommendation above turned out to be wrong, and both are worth recording +because the reasoning generalises. + +**1. The committed `db/data/clilocs.json` artifact was dropped.** It predates the two Part C +corrections (§6.1) and violates both: no committed snapshot of derived content, and nothing +EA-derived ever shipped. UO's strings are EA's, exactly as the creature sprites are. Replaced with +the §6 pattern instead — parse on every boot from an operator-configured path, hash-gated, output +gitignored, `PARSER_VERSION` counted as drift. + +**2. `scripts/buildClilocs.js reads the UO client's Cliloc.enu` is not possible, and the reason +matters.** **Every current client ships its cliloc files COMPRESSED** — all four `Cliloc.*` files +open with a DWORD whose high byte is `0x8E`, the "Mythic" container. The plain layout (`02 00 00 00 +01 00`, then `{int32 number, byte flag, uint16 length, UTF-8}`) is what those files looked like +*before* that change. Parsing a compressed file as plain does not fail cleanly: it yields ~19k +"records" with negative ids, 1,722 distinct keys out of 19,508, one 62 KB "string", and a truncation +somewhere in the middle. + +Decompressing means porting an inverse-BWT coder with a 1 KB frequency header — a few hundred lines +whose failure mode is plausible-looking garbage rather than an error. Two facts closed off the +alternatives: + +- **ServUO cannot read it either.** Its bundled `Ultima.StringList` implements only the plain layout, + so on a modern client `VendorSearch.StringList` is null and `VendorSearch.GetItemName` returns + `item.Name`. **The in-game Vendor Search gump has the same gap** — which also means §8.2's warning + never to call `GetItemName` in the sweep costs us nothing we could otherwise have had. +- The shard therefore cannot supply names on our behalf, so this could not be pushed to the plugin. + +⇒ **the operator converts once, from their own client, and the site reads the result.** Accepted +shapes are the plain binary layout and a `numbertext` export; the site sniffs which. +`server/tools/cliloc-export/` drives UOFiddler's `Ultima.dll` (the decompressor that already exists) +and writes the plain form. A shard that never converts is fully supported — names render as ids, +exactly as before. + +Three traps found by building it, all recorded in `CLILOCS.md`: + +- **`StringList.SaveStringList` RE-COMPRESSES on save.** It looks exactly like the export path and is + not; its output is byte-identical to its compressed input, because its purpose is round-tripping a + file back into the client. +- **Trimming a text line before splitting silently drops half the table.** Roughly half of a real + cliloc table is empty strings (ids the client reserves), exported as `1005008`. Trimming eats + the trailing separator, leaving a bare number that then looks like a header row — 55,994 of 123,490 + entries vanished, and the import still looked successful. +- **`Number('')` is `0`, not `NaN`.** A line starting with a separator imports as a bogus cliloc 0 + unless the empty field is rejected explicitly. + +Blank entries are dropped at import (123,490 parsed → **67,496** stored), which also makes the binary +and text paths converge on identical content. ### 8.7 Client @@ -743,7 +793,8 @@ inherently up to one full cycle old, and the UI must say so. | 2 | **B/1** — `world.ruleset` (§5) | all four | new kind | ✅ Done | | 3 | **C** — spawn atlas (§6) | website, docs | none | ✅ Done | | 4 | **B/2** — `points.board` (§7) | all four | new kind + `char.profile` field | ✅ Done | -| 5 | **B/3** — `vendor.listing` (§8) | all four | new kinds | ⬜ | +| 5a | **B/3 dependency** — cliloc table (§8.6) | website, docs | none | 🟨 In review | +| 5b | **B/3** — `vendor.listing` (§8) | all four | new kinds | ⬜ | | 6 | **Cutover** — `PROTOCOL_VERSION` 2→3, `edge` → `main` | all four | the bump | ⬜ | --- @@ -761,7 +812,7 @@ inherently up to one full cycle old, and the UI must say so. in the security section. - NEW `website/SHARD_VISIBILITY.md` — admin-facing: what each feature exposes, what each rung means, what cannot be loosened. -- NEW `website/SPAWN_ATLAS.md`, NEW `website/MARKETPLACE.md`. +- NEW `website/SPAWN_ATLAS.md`, NEW `website/CLILOCS.md`, NEW `website/MARKETPLACE.md`. - `PROJECT_TREE.md` in each touched repo. - `npm run swagger` **and** `npm run routes:manifest` on every route-touching PR — both are committed artifacts, and `test/routeManifest.test.js` fails on drift. diff --git a/website/BACKEND_DESIGN.md b/website/BACKEND_DESIGN.md index 4304050..ba13de7 100644 --- a/website/BACKEND_DESIGN.md +++ b/website/BACKEND_DESIGN.md @@ -486,6 +486,51 @@ artwork: sprites live in the operator's own client `.mul`/`.uop` files and are t redistribute. An operator supplies art via a gitignored map plus images under the (already gitignored) `server/uploads/atlas/`. Text-only is the normal, supported state. +### shard_clilocs / shard_cliloc_meta — UO's localization table (Protocol 3.0) + +Items on the wire carry a `LabelNumber`, not a name. The bridge has always sent it — +`char.profile.equipment.cliloc`, reward titles as a cliloc number in string form, and one per +marketplace listing — but with no table to resolve it against, the character sheet could only render +`id 1023721` where the game renders "quarter staff". + +| Table | Shape | +|---|---| +| `shard_clilocs` | `number` INT PK, `flag`, `text` TEXT | +| `shard_cliloc_meta` | Singleton (`id = 1`), `payload` JSON (source file, sha256, count, `parserVersion`), `imported_at` | + +Import-owned and all-or-nothing in one transaction, same contract as the atlas — including **`DELETE`, +not `TRUNCATE`**, for the same reason. + +**Sourced from a file the operator converts once from their own UO client**, at a path from the +`cliloc_client_path` setting falling back to `UO_CLIENT_PATH`. Nothing client-derived is committed: +UO's strings are EA's, exactly as the creature sprites are. A shard with nothing configured is fully +supported — names render as ids. Full design and operator guide: [`CLILOCS.md`](CLILOCS.md). + +The conversion step is not avoidable: **every current client ships its cliloc files compressed** +(first DWORD's high byte `0x8E`), and ServUO's own bundled `Ultima.StringList` cannot read that +either — so the shard cannot supply names on our behalf. The plain layout and a delimited text export +are both accepted, sniffed by header rather than extension. + +Three decisions worth stating: + +- **`text` is TEXT, not VARCHAR.** Long property descriptions reach 12 KB. The index that matters for + marketplace search is the denormalized `shard_vendor_items.display_name`, not this table. +- **Blank entries are dropped at import** — 123,490 parsed → **67,496** stored. Roughly half a cliloc + table is empty strings for ids the client reserves and never uses; a row that resolves to no name is + indistinguishable from no row at all, and dropping them makes the binary and text imports converge + on identical content. +- **No staged-approval flow, unlike the atlas.** The atlas escalates facet loss because a half-copied + tree and a real map change are indistinguishable from inside the process. A cliloc file is one file + with one hash, and a partial copy makes the parser fail on a truncated record — the ambiguity the + atlas must escalate is one this parser simply detects, so it refuses the import and leaves the + previous table serving. + +**Resolution is server-side and there is no public route.** The table is never served *as* a table: +67k rows would dwarf any page using them, and the Android client consumes the same already-resolved +JSON. `resolveMany()` returns only ids that resolved to something displayable — placeholders like +`~1_val~` are stripped, since the bridge sends the id and never the property packet that carries the +arguments — and it never throws, because a cliloc lookup is decoration on a character sheet. + --- ## 4. API contract @@ -741,6 +786,9 @@ file a route sits in — that is the property the route manifest freezes. | POST | `/shard/atlas/import` | re-import without restarting; `{force}` ignores the hash gate. **An unreadable tree answers 200 with `status:"unavailable"`, not 500** — `refresh()` reports outcomes rather than throwing (the boot path must never be blocked by a bad tree) and that contract is preserved at the API. | | POST | `/shard/atlas/approve` · `/shard/atlas/reject` | answer a refresh staged because it would REMOVE a facet. Approving **re-parses** the tree, so what lands matches it at approval time; rejecting is remembered against those source hashes so it does not re-prompt every restart. 404 when nothing is staged. | | PUT | `/shard/atlas/path` | point the atlas at a different tree (persisted as `spawn_atlas_servuo_path`, which wins over `SERVUO_PATH`). Blank clears it. Deliberately **does not import** — moving the mount and reloading the world are separate decisions — and returns fresh status so the panel can offer the import next. | +| GET | `/shard/clilocs` | cliloc-table status (`adminOnly`): the configured path, the file actually resolved (the path may be a directory), readability, drift against what is loaded, and the entry count. `configured:false` is a supported state — item names then render as ids. No public counterpart: the table is never served *as* a table. | +| POST | `/shard/clilocs/import` | reload after a client patch; `{force}` ignores the hash gate. **A missing file — or the likely mistake of pointing at the client's own COMPRESSED `Cliloc.enu` — answers 200 with `status:"unavailable"` and a `code`, not 500.** `COMPRESSED` is called out by name: a 500 would say only "something broke", and the operator needs to be told which file to convert. | +| PUT | `/shard/clilocs/path` | point the site at a different cliloc file or directory (persisted as `cliloc_client_path`, which wins over `UO_CLIENT_PATH`). Blank clears it. Deliberately **does not import**, same reasoning as the atlas path. | Every admin write logs to `activity_log`. diff --git a/website/CLILOCS.md b/website/CLILOCS.md new file mode 100644 index 0000000..a7b5b49 --- /dev/null +++ b/website/CLILOCS.md @@ -0,0 +1,216 @@ +# Cliloc table (item and title names) + +**Status:** Complete on `edge` — website [#TBD](https://gitea.whitlocktech.com/RunicGateway/website/pulls). +**Design:** [`docs/link/v3.md` §8.6](../link/v3.md) — Protocol 3.0, the dependency Part B/3 was sequenced behind. + +A "cliloc" is UO's localization table: an integer id mapped to a display string. +**Items on the wire carry a `LabelNumber`, not a name.** The bridge has always +sent that number — `char.profile.equipment` has a `cliloc` field, reward titles +arrive as a cliloc number in string form, and every marketplace listing carries +one — but the site had no table to look it up in, so a character sheet could only +render `id 1023721` where the game renders **"quarter staff"**. + +The number was never the missing piece. The table was. + +## Why the operator has to convert the file + +This is the awkward part, and it is not avoidable: + +**Every current UO client ships its cliloc files compressed.** The four +`Cliloc.*` files in a modern client all begin with a DWORD whose high byte is +`0x8E` — the "Mythic" compressed container. The plain layout this site parses is +what those files looked like *before* that change. + +Decompressing it means an inverse-BWT coder with a frequency header — a few +hundred lines of bit-level work whose failure mode is plausible-looking garbage +rather than an error. The site has no business carrying that at runtime. + +Two facts make the alternatives worse, not better: + +- **ServUO cannot read it either.** Its bundled `Ultima.StringList` implements + only the plain layout, so on a modern client `VendorSearch.StringList` is null + and `VendorSearch.GetItemName` returns `item.Name` — usually nothing. The + shard cannot supply names on our behalf; the in-game Vendor Search gump has the + same gap. +- **Nothing client-derived may be committed.** UO's strings are EA's. The repo + ships no string table for the same reason it ships no artwork and no map + snapshot — see [`SPAWN_ATLAS.md`](SPAWN_ATLAS.md). + +So the conversion happens **once, on the operator's machine, against their own +client**, and the site reads the result from a path it is given. A shard that +never does this is in a fully supported state: names render as ids, exactly as +they did before the table existed. + +## Converting + +Either format below is accepted; the site sniffs which one it was handed. + +| Format | Fidelity | Notes | +|---|---|---| +| **Plain binary** (recommended) | Exact | 6-byte header, then `{int32 number, byte flag, uint16 length, UTF-8}` records | +| Delimited text | Loses leading/trailing whitespace | `numbertext` per line; a header row, blank lines and `#` comments are ignored | + +The whitespace caveat is real but cosmetic: ~1,300 of the 123,490 entries in a +stock `Cliloc.enu` are label prefixes like `"max = "` whose trailing space is +meaningful when the client concatenates a value onto them. Nothing on this site +concatenates, and every consumer passes through `displayText()`, which trims. + +### Using the bundled tool + +`server/tools/cliloc-export/` is a small .NET console app that drives +[UOFiddler](https://github.com/polserver/UOFiddler)'s `Ultima.dll` — the +decompressor that already exists and is already maintained — and writes the plain +format. It loads that DLL **reflectively** so it compiles against any SDK, and it +writes the records by hand because UOFiddler's own `SaveStringList` *re-compresses* +on save (its purpose is round-tripping a file back into the client, so its output +is byte-identical to its input — a trap worth knowing about). + +```bash +cd website/server/tools/cliloc-export +dotnet build -c Release + +# binary (recommended) +dotnet run -- "/Ultima.dll" "/Cliloc.enu" /srv/uo-data/clilocs.plain + +# or tab-delimited +dotnet run -- "/Ultima.dll" "/Cliloc.enu" /srv/uo-data/clilocs.tsv --tsv +``` + +A UOFiddler GUI export works equally well — anything producing one of the two +shapes above is fine. + +## Configuring the path + +Two ways to point at the converted file, the setting winning over the +environment: + +| Source | Notes | +|---|---| +| `cliloc_client_path` setting | Admin-editable (Admin → Shard); takes effect on the next refresh without a redeploy | +| `UO_CLIENT_PATH` env var | The deploy-time default, since the path usually describes a mount the deployment sets up | + +The value may be **the file itself or a directory to search**, because both are +natural answers to "where is it". A directory is searched case-insensitively (the +client writes `Cliloc.enu` on Windows; the site usually runs on Linux) for, in +order: `clilocs.tsv`, `clilocs.csv`, `cliloc.plain.enu`, `cliloc.enu.plain`, +`clilocs.txt`, `cliloc.enu`. + +That ordering puts explicitly-converted names first on purpose. Pointing the +setting straight at an unconverted client directory finds `cliloc.enu`, which is +compressed — and the site says so by name rather than failing obscurely: + +``` +status: unavailable +code: COMPRESSED +reason: This is a compressed (Mythic-format) cliloc file, which the site cannot + read. Convert it to the plain format first — see docs/website/CLILOCS.md. +``` + +## Refresh contract + +Identical in shape to the spawn atlas, and for the same reasons: + +- **It never blocks startup.** No path, an unreadable file, a wrong-format file, + a database error — all caught and logged. The site comes up either way. +- **Hash-gated.** The boot path hashes the file and skips the parse entirely when + it matches what is loaded, which is every restart that did not follow a client + patch. Measured on a stock table: **14 ms** for the no-op, **663 ms** for a full + parse and replace. +- **A `PARSER_VERSION` bump also counts as drift**, so a corrected parse reaches + an install whose client never patches. + +### Why there is no staged-approval flow + +The atlas stages a refresh that would *remove a facet*, because a half-copied +tree and a real map change are indistinguishable from inside the process. A +cliloc file is one file with one hash, and its realistic corruption — a partial +copy — makes the parser fail on a truncated record instead of yielding a +plausible-but-short table. **The ambiguity the atlas has to escalate to a human is +one this parser can simply detect**, so it refuses the import and leaves the +previous table serving. Verified: a file truncated to half its length reports + +``` +code: TRUNCATED +reason: Truncated record header at byte 2486759 (74909 entries read) +``` + +and the 67,496 rows already loaded are untouched. + +## What gets stored + +| | | +|---|---| +| Parsed from a stock `Cliloc.enu` | **123,490** entries | +| Of those, empty strings | **55,994** (ids the client reserves and never uses) | +| Stored in `shard_clilocs` | **67,496** | + +Blank entries are dropped at import. A row resolving to no name is +indistinguishable from no row at all to every caller, and dropping them makes the +binary and text imports converge on **identical** content — the binary format +carries the blanks explicitly and a text export may or may not, depending on the +tool. Verified: both formats import to the same 67,496 rows with the same keys. + +`text` is `TEXT`, not `VARCHAR`: the long property descriptions reach 12 KB, and +silently truncating them would be worse than storing them. The index that matters +for marketplace search is on the denormalized `shard_vendor_items.display_name`, +not here. + +## How names are applied + +**Resolution happens server-side.** The table is never served *as* a table and +there is no public route for it. Two reasons: 67k rows would dwarf any page that +used them, and the Android client consumes the same JSON and would otherwise need +its own copy. + +`resolveMany()` takes a batch of ids and returns a `Map` holding only those that +resolved to something displayable, so "no such id" and "id with no usable name" +collapse into one branch at the call site. It never throws — a cliloc lookup is +decoration on someone's character sheet, and a database blip must not fail the +sheet. A capped in-process cache fronts it; measured cold **4.2 ms**, warm +**0.015 ms**. + +### `displayText()` + +Cliloc strings interpolate arguments the client pulls from an item's property +list — `~1_val~`, `~2_NAME~`. **We never have those**: the bridge sends the id, +not the packet. So a name carrying them is reduced to what is actually knowable. + +| Raw | Displayed | +|---|---| +| `quarter staff` | `quarter staff` | +| `cold damage ~1_val~%` | `cold damage` | +| `[~1_stuff~]` | *(nothing — the whole string was the argument)* | +| `50%` | `50%` | + +The trailing `%` in row two is dropped **because a placeholder was removed** — it +is the unit belonging to the number we never had. Row four shows why that +condition matters: stripping `%` unconditionally would corrupt a string that +legitimately ends in one. + +### Consumers + +- **Character sheet equipment.** `enrichCharProfile` attaches `clilocName` to each + item. A player-given `name` always wins — "Bob's lucky axe" must not be + relabelled "hatchet" — and the client re-states that precedence. +- **Reward titles.** `titles.rewardResolved` is a parallel array with the numeric + entries turned into words (`null` where nothing resolved). The sheet used to + *skip* numeric reward titles entirely, having no way to render them. +- **Marketplace listings** (Protocol 3.0 §8) denormalize the resolved name into + `shard_vendor_items.display_name` so search can index it. + +## Admin surface + +All admin-only, alongside the atlas under Admin → Shard: + +| Route | Purpose | +|---|---| +| `GET /api/v1/admin/shard/clilocs` | Path, resolved file, readability, drift, entry count | +| `POST /api/v1/admin/shard/clilocs/import` | Reload after a client patch; `{ "force": true }` reimports an unchanged file | +| `PUT /api/v1/admin/shard/clilocs/path` | Set the path; blank disables resolution | + +A refresh **result is not an exception**: a missing file, or the likely mistake of +pointing at the client's own compressed `Cliloc.enu`, answers `200` with +`status: "unavailable"` and a reason. A `500` would say only "something broke"; +the operator needs to be told which file to convert. Setting the path +deliberately does **not** import as a side effect — the response carries the +refreshed status so the panel can offer that as the next step. -- 2.49.1 From e3aabf9e3e85bae733af8b752447764318f33f81 Mon Sep 17 00:00:00 2001 From: wtclaude Date: Wed, 29 Jul 2026 04:25:06 -0500 Subject: [PATCH 12/19] docs(link): fill in the cliloc PR numbers, correct the cliloc file count MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The progress table and CLILOCS.md carried #TBD placeholders until the PRs existed; they now point at website #115 and docs #70. Also corrects "all four Cliloc.* files" to eight (chs, cht, deu, enu, esp, fra, jpn, kor) in both v3.md §8.6 and CLILOCS.md — the compression marker was checked against every one of them, and understating the sample weakens the claim it supports. Co-Authored-By: Claude --- link/v3.md | 4 ++-- website/CLILOCS.md | 11 ++++++----- 2 files changed, 8 insertions(+), 7 deletions(-) diff --git a/link/v3.md b/link/v3.md index 3ef0cc5..7f108e6 100644 --- a/link/v3.md +++ b/link/v3.md @@ -15,7 +15,7 @@ Each part is marked off here as it lands on `edge`. §9 carries the same state p | 2 | **B/1** — `world.ruleset` (§5) | ✅ **Done** | servuo-plugins [#3](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/3), link [#17](https://gitea.whitlocktech.com/RunicGateway/link/pulls/17), website [#111](https://gitea.whitlocktech.com/RunicGateway/website/pulls/111), docs [#66](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/66) | | 3 | **C** — spawn atlas (§6) | ✅ **Done** | website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112) (parsers + CLI + tables) + [#113](https://gitea.whitlocktech.com/RunicGateway/website/pulls/113) (API + pages + admin panel), docs [#67](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/67) + [#68](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/68) | | 4 | **B/2** — `points.board` (§7) | ✅ **Done** | servuo-plugins [#4](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/4), link [#18](https://gitea.whitlocktech.com/RunicGateway/link/pulls/18), website [#114](https://gitea.whitlocktech.com/RunicGateway/website/pulls/114), docs [#69](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/69) | -| 5a | **B/3 dependency** — cliloc table (§8.6) | 🟨 In review | website [#TBD](https://gitea.whitlocktech.com/RunicGateway/website/pulls), docs [#TBD](https://gitea.whitlocktech.com/RunicGateway/docs/pulls) | +| 5a | **B/3 dependency** — cliloc table (§8.6) | 🟨 In review | website [#115](https://gitea.whitlocktech.com/RunicGateway/website/pulls/115), docs [#70](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/70) | | 5b | **B/3** — `vendor.listing` (§8) | ⬜ Not started | — | | 6 | **Cutover** — `PROTOCOL_VERSION` 2→3 (§4) | ⬜ Not started | — | @@ -739,7 +739,7 @@ the §6 pattern instead — parse on every boot from an operator-configured path gitignored, `PARSER_VERSION` counted as drift. **2. `scripts/buildClilocs.js reads the UO client's Cliloc.enu` is not possible, and the reason -matters.** **Every current client ships its cliloc files COMPRESSED** — all four `Cliloc.*` files +matters.** **Every current client ships its cliloc files COMPRESSED** — all eight `Cliloc.*` files open with a DWORD whose high byte is `0x8E`, the "Mythic" container. The plain layout (`02 00 00 00 01 00`, then `{int32 number, byte flag, uint16 length, UTF-8}`) is what those files looked like *before* that change. Parsing a compressed file as plain does not fail cleanly: it yields ~19k diff --git a/website/CLILOCS.md b/website/CLILOCS.md index a7b5b49..3e97f43 100644 --- a/website/CLILOCS.md +++ b/website/CLILOCS.md @@ -1,6 +1,6 @@ # Cliloc table (item and title names) -**Status:** Complete on `edge` — website [#TBD](https://gitea.whitlocktech.com/RunicGateway/website/pulls). +**Status:** Complete on `edge` — website [#115](https://gitea.whitlocktech.com/RunicGateway/website/pulls/115), docs [#70](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/70). **Design:** [`docs/link/v3.md` §8.6](../link/v3.md) — Protocol 3.0, the dependency Part B/3 was sequenced behind. A "cliloc" is UO's localization table: an integer id mapped to a display string. @@ -16,10 +16,11 @@ The number was never the missing piece. The table was. This is the awkward part, and it is not avoidable: -**Every current UO client ships its cliloc files compressed.** The four -`Cliloc.*` files in a modern client all begin with a DWORD whose high byte is -`0x8E` — the "Mythic" compressed container. The plain layout this site parses is -what those files looked like *before* that change. +**Every current UO client ships its cliloc files compressed.** All eight +`Cliloc.*` files in a modern client (`chs`, `cht`, `deu`, `enu`, `esp`, `fra`, +`jpn`, `kor`) begin with a DWORD whose high byte is `0x8E` — the "Mythic" +compressed container. The plain layout this site parses is what those files +looked like *before* that change. Decompressing it means an inverse-BWT coder with a frequency header — a few hundred lines of bit-level work whose failure mode is plausible-looking garbage -- 2.49.1 From ee0c146d7aa68017d0c43d6a6c4863962180e4f8 Mon Sep 17 00:00:00 2001 From: wtclaude Date: Wed, 29 Jul 2026 06:46:43 -0500 Subject: [PATCH 13/19] docs(link): cliloc overlays for shard-added and shard-edited items MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Shards edit items and add new ones, carrying cliloc ids no stock client table has. The cliloc table now reads a SET of sources rather than one file — a base plus every operator-maintained overlay under `custom/`, hash-gated together and re-read on every boot, exactly as §6 reads the ServUO tree. Matching docs for website #115. website/CLILOCS.md gains a "Shard-added and shard-edited items" section: the directory layout, merge precedence, the per-source breakdown an operator uses to confirm an overlay took effect, and why `custom/` is a convention we chose rather than one discovered from the shard — ServUO has no server-side notion of a custom cliloc, they live in the patched client a shard distributes and nothing in the tree declares them. Scale recorded: the live shard's script tree references 16,434 cliloc ids and only 37 are absent from stock, which is why this is an overlay and not a second table. "Why there is no staged-approval flow" is replaced by "Two ways a refresh is refused", because the set brings back the hazard a single file did not have. A corrupt source fails the parse loudly; a source that has VANISHED parses perfectly and imports a table quietly missing everything it contributed. That is the same ambiguity §6 stages a facet removal for, so it is staged here too (`needsReview`, `{approve:true}` to accept) — as a flag rather than §6's approve/reject pair, because the atlas stores a pending decision SO THAT approving re-parses, and here nothing is stored. Two more traps recorded in §8.6 (now five), both found by running a shard-style overlay rather than another stock-table fixture: - Tidying punctuation unconditionally corrupts real names — a custom "Runic Gateway Sigil (v2)" rendered as "(v2". Stripping leftover brackets is right after a placeholder is removed and wrong otherwise, the same condition the `%` rule already had. - Source labels must be forward-slashed and root-relative or the same directory fingerprints differently on Windows and Linux and every boot looks like a change — the identical bug §6 records. BACKEND_DESIGN.md: the source set, the two refusals, and the updated admin route notes (`approve`, `missingSources`, overlays found beside a file path). Documentation only. Every claim was written after the behaviour was observed: the counts come from the live shard's script tree, and the refusal/approve flow and per-source breakdown are copied from real runs in website #115. PROJECT_TREE.md files are deliberately untouched — CI-generated by the sync-project-tree workflow. Co-Authored-By: Claude --- link/v3.md | 26 +++++++- website/BACKEND_DESIGN.md | 37 ++++++++---- website/CLILOCS.md | 123 +++++++++++++++++++++++++++++++------- 3 files changed, 151 insertions(+), 35 deletions(-) diff --git a/link/v3.md b/link/v3.md index 7f108e6..362718d 100644 --- a/link/v3.md +++ b/link/v3.md @@ -762,7 +762,25 @@ shapes are the plain binary layout and a `numbertext` export; the site and writes the plain form. A shard that never converts is fully supported — names render as ids, exactly as before. -Three traps found by building it, all recorded in `CLILOCS.md`: +**Shards edit items and add new ones**, and those carry ids no stock client table has — so this reads +a **set** of sources, not one file, hash-gated together and re-read on every boot exactly as §6 reads +the ServUO tree: a base (the converted client table) plus every overlay under `custom/`, later +winning. Adding one custom item therefore never means re-exporting a 5 MB client file. Measured on +the live shard for scale: its script tree references **16,434** cliloc ids and only **37** are absent +from stock — tens against a 67k base, which is why an overlay and not a second table. `custom/` is the +one convention here that is ours rather than the shard's, because **ServUO has no server-side notion +of a custom cliloc**: they live in the patched client a shard distributes, and nothing in the tree +declares them. + +That set also brings back a hazard a single file did not have, and §8.6 answers it the way §6 does. A +corrupt source fails the parse loudly, but a source that has **vanished** parses perfectly and imports +a table quietly missing everything it contributed — an unmounted volume is indistinguishable from a +deliberate deletion. So it is **staged, not applied** (`status: 'needsReview'`), reported by both the +import and `status()`, and accepted with `{approve:true}`. It is a flag rather than §6's +approve/reject pair because the atlas stores a pending decision *so that approving re-parses*; here +nothing is stored, so re-reading at approval time is automatic. + +Five traps found by building it, all recorded in `CLILOCS.md`: - **`StringList.SaveStringList` RE-COMPRESSES on save.** It looks exactly like the export path and is not; its output is byte-identical to its compressed input, because its purpose is round-tripping a @@ -773,6 +791,12 @@ Three traps found by building it, all recorded in `CLILOCS.md`: entries vanished, and the import still looked successful. - **`Number('')` is `0`, not `NaN`.** A line starting with a separator imports as a bogus cliloc 0 unless the empty field is rejected explicitly. +- **Tidying punctuation unconditionally corrupts real names.** Stripping leftover brackets is right + after a placeholder is removed (`[~1_stuff~]` → nothing) and wrong otherwise: a shard's custom + `"Runic Gateway Sigil (v2)"` rendered as `"(v2"`. Same shape as the `%` rule. **Found only by + running a shard-style overlay through it** — every stock-table fixture passed. +- **Source labels must be forward-slashed and root-relative**, or the same directory fingerprints + differently on Windows and Linux and every boot looks like a change. The identical bug §6 records. Blank entries are dropped at import (123,490 parsed → **67,496** stored), which also makes the binary and text paths converge on identical content. diff --git a/website/BACKEND_DESIGN.md b/website/BACKEND_DESIGN.md index ba13de7..3172b92 100644 --- a/website/BACKEND_DESIGN.md +++ b/website/BACKEND_DESIGN.md @@ -501,10 +501,19 @@ marketplace listing — but with no table to resolve it against, the character s Import-owned and all-or-nothing in one transaction, same contract as the atlas — including **`DELETE`, not `TRUNCATE`**, for the same reason. -**Sourced from a file the operator converts once from their own UO client**, at a path from the -`cliloc_client_path` setting falling back to `UO_CLIENT_PATH`. Nothing client-derived is committed: -UO's strings are EA's, exactly as the creature sprites are. A shard with nothing configured is fully -supported — names render as ids. Full design and operator guide: [`CLILOCS.md`](CLILOCS.md). +**Sourced from files the operator supplies**, at a path from the `cliloc_client_path` setting falling +back to `UO_CLIENT_PATH`. Nothing client-derived is committed: UO's strings are EA's, exactly as the +creature sprites are. A shard with nothing configured is fully supported — names render as ids. Full +design and operator guide: [`CLILOCS.md`](CLILOCS.md). + +**It reads a SET of sources, not one file**, because shards edit items and add new ones and those +carry cliloc ids no stock client table has. A base (the converted client table) plus every overlay +under `custom/` are re-read on every boot and hash-gated **together**, exactly as the atlas re-reads +`Regions.xml` + `Locations/*.xml` + `Spawns/*.xml` + `ChampionSpawns.xml`. Later sources win, so an +overlay both adds ids and overrides stock ones, and adding one custom item never means re-exporting a +5 MB client file. Scale, measured on the live shard: its script tree references 16,434 cliloc ids and +only 37 are absent from stock — tens of entries against a 67k base, which is why this is an overlay +and not a second table. The conversion step is not avoidable: **every current client ships its cliloc files compressed** (first DWORD's high byte `0x8E`), and ServUO's own bundled `Ultima.StringList` cannot read that @@ -519,11 +528,15 @@ Three decisions worth stating: table is empty strings for ids the client reserves and never uses; a row that resolves to no name is indistinguishable from no row at all, and dropping them makes the binary and text imports converge on identical content. -- **No staged-approval flow, unlike the atlas.** The atlas escalates facet loss because a half-copied - tree and a real map change are indistinguishable from inside the process. A cliloc file is one file - with one hash, and a partial copy makes the parser fail on a truncated record — the ambiguity the - atlas must escalate is one this parser simply detects, so it refuses the import and leaves the - previous table serving. +- **Two refusals, one of them the atlas's.** A corrupt source fails the parse on a truncated record, + so it is caught outright and leaves the previous table serving. But a source that has **vanished** + parses perfectly and imports a table quietly missing everything it contributed — an unmounted volume + and a deliberate deletion are indistinguishable from here, which is precisely the ambiguity the + atlas stages a facet removal for. So it is escalated: `status: 'needsReview'`, nothing applied, + `missingSources` reported by both the import and `status()`, and an admin accepts it with + `{approve:true}`. That is a flag rather than the atlas's approve/reject pair because the atlas + stores a pending decision so that approving **re-parses** the tree; here nothing is stored, so + re-reading at approval time is automatic. **Resolution is server-side and there is no public route.** The table is never served *as* a table: 67k rows would dwarf any page using them, and the Android client consumes the same already-resolved @@ -786,9 +799,9 @@ file a route sits in — that is the property the route manifest freezes. | POST | `/shard/atlas/import` | re-import without restarting; `{force}` ignores the hash gate. **An unreadable tree answers 200 with `status:"unavailable"`, not 500** — `refresh()` reports outcomes rather than throwing (the boot path must never be blocked by a bad tree) and that contract is preserved at the API. | | POST | `/shard/atlas/approve` · `/shard/atlas/reject` | answer a refresh staged because it would REMOVE a facet. Approving **re-parses** the tree, so what lands matches it at approval time; rejecting is remembered against those source hashes so it does not re-prompt every restart. 404 when nothing is staged. | | PUT | `/shard/atlas/path` | point the atlas at a different tree (persisted as `spawn_atlas_servuo_path`, which wins over `SERVUO_PATH`). Blank clears it. Deliberately **does not import** — moving the mount and reloading the world are separate decisions — and returns fresh status so the panel can offer the import next. | -| GET | `/shard/clilocs` | cliloc-table status (`adminOnly`): the configured path, the file actually resolved (the path may be a directory), readability, drift against what is loaded, and the entry count. `configured:false` is a supported state — item names then render as ids. No public counterpart: the table is never served *as* a table. | -| POST | `/shard/clilocs/import` | reload after a client patch; `{force}` ignores the hash gate. **A missing file — or the likely mistake of pointing at the client's own COMPRESSED `Cliloc.enu` — answers 200 with `status:"unavailable"` and a `code`, not 500.** `COMPRESSED` is called out by name: a 500 would say only "something broke", and the operator needs to be told which file to convert. | -| PUT | `/shard/clilocs/path` | point the site at a different cliloc file or directory (persisted as `cliloc_client_path`, which wins over `UO_CLIENT_PATH`). Blank clears it. Deliberately **does not import**, same reasoning as the atlas path. | +| GET | `/shard/clilocs` | cliloc-table status (`adminOnly`): every source found now (base first, then `custom/` overlays in merge order), what each contributed at the last import, readability, drift across the set, the entry count, and `missingSources`. `configured:false` is a supported state — item names then render as ids. No public counterpart: the table is never served *as* a table. | +| POST | `/shard/clilocs/import` | reload after a client patch or an overlay edit; `{force}` ignores the hash gate, `{approve}` accepts a **vanished** source (refused by default — see the table notes above). **A missing path — or the likely mistake of pointing at the client's own COMPRESSED `Cliloc.enu` — answers 200 with `status:"unavailable"` and a `code`, not 500.** `COMPRESSED` is called out by name: a 500 would say only "something broke", and the operator needs to be told which file to convert. | +| PUT | `/shard/clilocs/path` | point the site at a different cliloc base file or directory (persisted as `cliloc_client_path`, which wins over `UO_CLIENT_PATH`). Overlays are read from `custom/` beside it either way. Blank clears it. Deliberately **does not import**, same reasoning as the atlas path. | Every admin write logs to `activity_log`. diff --git a/website/CLILOCS.md b/website/CLILOCS.md index 3e97f43..e0747f5 100644 --- a/website/CLILOCS.md +++ b/website/CLILOCS.md @@ -80,21 +80,74 @@ dotnet run -- "/Ultima.dll" "/Cliloc.enu" /srv/uo-data/cli A UOFiddler GUI export works equally well — anything producing one of the two shapes above is fine. +## Shard-added and shard-edited items + +**Shards edit items and add new ones**, and those carry cliloc ids no stock +client table has. The table is therefore built from a **set** of sources, all +re-read on every boot and hash-gated together — the same shape as the spawn +atlas, which reads `Regions.xml` + `Locations/*.xml` + `Spawns/*.xml` + +`ChampionSpawns.xml` and merges them: + +``` +/ + clilocs.plain ← base: the converted client table + custom/ + 01-uomysticmoon.tsv ← overlays: shard additions and overrides + 02-events.tsv +``` + +Overlays use the same delimited-text format, are read in **sorted order**, and +**later sources win** — so an overlay both *adds* ids the client never had and +*overrides* stock ones the shard has re-purposed. Any `.tsv`, `.csv`, `.txt`, +`.enu` or `.plain` file in `custom/` is picked up; anything else (a `README.md`, +say) is ignored. + +Adding, editing or removing any overlay counts as drift, so a new custom item +needs only a file edit and a restart — or the admin panel's Import button. +**Adding one item never means re-exporting a 5 MB client file.** + +The import result reports what each source contributed, which is how you confirm +an overlay took effect — `overrode: 0` on a file meant to re-label stock items +says it did not: + +```json +"sources": [ + { "label": "clilocs.plain", "kind": "base", "entries": 123490, "added": 123490, "overrode": 0 }, + { "label": "custom/uomysticmoon.tsv", "kind": "custom", "entries": 2, "added": 1, "overrode": 1 } +] +``` + +**Why a convention rather than discovery.** Everywhere else this pipeline follows +the shard's own files, but **ServUO has no server-side notion of a custom +cliloc** — they live in the patched client a shard distributes to its players, +and nothing in the tree declares them. There is nothing to discover, so `custom/` +is the one thing here that is our convention rather than the shard's. (An +operator who *does* patch their client cliloc needs no overlay at all: convert +the patched file and their edits are simply in the base.) + +Measured on the live shard for scale: its script tree references **16,434** cliloc +ids and only **37** are absent from the stock client table — tens of entries +against a 67k base, which is what makes an overlay the right shape rather than a +second full table. + ## Configuring the path -Two ways to point at the converted file, the setting winning over the -environment: +Two ways to point at the sources, the setting winning over the environment: | Source | Notes | |---|---| | `cliloc_client_path` setting | Admin-editable (Admin → Shard); takes effect on the next refresh without a redeploy | | `UO_CLIENT_PATH` env var | The deploy-time default, since the path usually describes a mount the deployment sets up | -The value may be **the file itself or a directory to search**, because both are -natural answers to "where is it". A directory is searched case-insensitively (the -client writes `Cliloc.enu` on Windows; the site usually runs on Linux) for, in -order: `clilocs.tsv`, `clilocs.csv`, `cliloc.plain.enu`, `cliloc.enu.plain`, -`clilocs.txt`, `cliloc.enu`. +The value may be **the base file itself or a directory to search**, because both +are natural answers to "where is it". Overlays are read from a `custom/` +directory beside the base **either way** — pointing at a file does not forfeit +them. + +A directory is searched case-insensitively (the client writes `Cliloc.enu` on +Windows; the site usually runs on Linux) for, in order: `clilocs.tsv`, +`clilocs.csv`, `clilocs.plain`, `cliloc.plain`, `cliloc.plain.enu`, +`cliloc.enu.plain`, `clilocs.txt`, `cliloc.enu`. That ordering puts explicitly-converted names first on purpose. Pointing the setting straight at an unconverted client directory finds `cliloc.enu`, which is @@ -120,22 +173,44 @@ Identical in shape to the spawn atlas, and for the same reasons: - **A `PARSER_VERSION` bump also counts as drift**, so a corrected parse reaches an install whose client never patches. -### Why there is no staged-approval flow +### Two ways a refresh is refused -The atlas stages a refresh that would *remove a facet*, because a half-copied -tree and a real map change are indistinguishable from inside the process. A -cliloc file is one file with one hash, and its realistic corruption — a partial -copy — makes the parser fail on a truncated record instead of yielding a -plausible-but-short table. **The ambiguity the atlas has to escalate to a human is -one this parser can simply detect**, so it refuses the import and leaves the -previous table serving. Verified: a file truncated to half its length reports +**A corrupt file** — the realistic failure for any single source — makes the +parser fail on a truncated record rather than yield a plausible-but-short table, +so it is caught outright. Verified: a file truncated to half its length reports ``` code: TRUNCATED reason: Truncated record header at byte 2486759 (74909 entries read) ``` -and the 67,496 rows already loaded are untouched. +and the rows already loaded are untouched. A malformed overlay names the file it +came from (`custom/broken.tsv: No cliloc entries found…`), because "which of my +six overlay files is broken" is otherwise a guessing game. + +**A source that has VANISHED** is the hazard a single file did not have. It +parses perfectly and imports a table quietly missing everything that file +contributed — and an unmounted volume looks exactly like a deliberate deletion +from here. This is the same ambiguity the atlas stages a facet removal for, so it +is escalated rather than applied: + +``` +status: needsReview +reason: 1 previously-loaded cliloc source(s) are missing; + the existing table is unchanged +missingSources: ["custom/uomysticmoon.tsv"] +``` + +`status()` reports `missingSources` too, so the panel can show it before anyone +clicks Import. An admin accepts it by re-running the import with +`{ "approve": true }`. + +**Why that is a flag and not the atlas's approve/reject pair.** The atlas stores +a pending decision in its own table so that approving *re-parses the tree*, which +is what keeps a multi-megabyte blob out of the database and makes the applied +result match the tree at approval time. Here nothing is stored, so re-reading at +approval time is automatic — the decision is a single boolean on the import an +admin was already going to run. ## What gets stored @@ -182,11 +257,15 @@ not the packet. So a name carrying them is reduced to what is actually knowable. | `cold damage ~1_val~%` | `cold damage` | | `[~1_stuff~]` | *(nothing — the whole string was the argument)* | | `50%` | `50%` | +| `Runic Gateway Sigil (v2)` | `Runic Gateway Sigil (v2)` | -The trailing `%` in row two is dropped **because a placeholder was removed** — it -is the unit belonging to the number we never had. Row four shows why that -condition matters: stripping `%` unconditionally would corrupt a string that -legitimately ends in one. +**Punctuation is only tidied when a placeholder was actually removed.** The +trailing `%` in row two is the unit belonging to the number we never had, and the +brackets in row three only ever wrapped the argument — but a string with no +placeholder has no such debris, and trimming it anyway corrupts real names. Rows +four and five are the ones that caught it: a shard's custom +`"Runic Gateway Sigil (v2)"` rendered as `"(v2"` while the bracket trim was +unconditional. ### Consumers @@ -205,8 +284,8 @@ All admin-only, alongside the atlas under Admin → Shard: | Route | Purpose | |---|---| -| `GET /api/v1/admin/shard/clilocs` | Path, resolved file, readability, drift, entry count | -| `POST /api/v1/admin/shard/clilocs/import` | Reload after a client patch; `{ "force": true }` reimports an unchanged file | +| `GET /api/v1/admin/shard/clilocs` | Sources found, what each contributed at the last import, readability, drift, entry count, `missingSources` | +| `POST /api/v1/admin/shard/clilocs/import` | Reload after a client patch or an overlay edit; `{ "force": true }` reimports an unchanged set, `{ "approve": true }` accepts a vanished source | | `PUT /api/v1/admin/shard/clilocs/path` | Set the path; blank disables resolution | A refresh **result is not an exception**: a missing file, or the likely mistake of -- 2.49.1 From 6ce60a82c336db52f90a4ec9dc39b3d89343e31d Mon Sep 17 00:00:00 2001 From: wtclaude Date: Wed, 29 Jul 2026 09:52:06 -0500 Subject: [PATCH 14/19] =?UTF-8?q?docs(link):=20the=20player-vendor=20marke?= =?UTF-8?q?tplace=20(Protocol=203.0=20=C2=A78)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Documents order 5b across the four repos, and records what building it changed about §8 as designed. - NEW website/MARKETPLACE.md — the operator guide: what the pages must say out loud and why, the privacy contract (the player's in-game Vendor Search toggle wins, and no admin setting overrides it), the Bridge.cfg knobs and how they trade against each other, and the measured sweep costs. - INTEGRATION.md — catalog entry for vendor.listing / vendor.listing.remove with its six consumer gotchas, and the GET /market REST section (the sidecar's only paged read, and why it orders by serial rather than shop name). - BACKEND_DESIGN.md — shard_vendors / shard_vendor_items, the routes, and the marketplace search as the only rate-limited public read. - SHARD_VISIBILITY.md — why the market's fields default to Everyone (the in-game gump already shows exactly that set), why location is one setting covering four things, and why hiding the owner name without the owner id achieves nothing. - PLAN.md — the amortized round-robin as the one sweep pattern the bridge did not previously have, and an update to §7's cliloc note: pushing name resolution to the plugin was never an option, because ServUO cannot read a modern client's compressed cliloc files either. - v3.md §8.8 — the four things the build settled differently, chief among them that §8.1's FLAT location payload would have made Part A's pre-wired market.location rule inert, exactly like the characterName miss one part earlier. Co-Authored-By: Claude --- README.md | 3 + link/INTEGRATION.md | 101 ++++++++++++++++++++++ link/PLAN.md | 23 ++++- link/v3.md | 79 +++++++++++++++-- website/BACKEND_DESIGN.md | 53 +++++++++++- website/MARKETPLACE.md | 167 ++++++++++++++++++++++++++++++++++++ website/SHARD_VISIBILITY.md | 22 ++++- 7 files changed, 440 insertions(+), 8 deletions(-) create mode 100644 website/MARKETPLACE.md diff --git a/README.md b/README.md index 8dab127..fe4d540 100644 --- a/README.md +++ b/README.md @@ -20,6 +20,9 @@ ci/ cross-cutting CI/quality notes | [HERO_EDITOR.md](website/HERO_EDITOR.md) | Hero canvas editor feature spec | | [WIKI_UPGRADE.md](website/WIKI_UPGRADE.md) | Wiki subsystem upgrade notes | | [SHARD_VISIBILITY.md](website/SHARD_VISIBILITY.md) | Who sees which shard data — the admin-configurable audience framework | +| [SPAWN_ATLAS.md](website/SPAWN_ATLAS.md) | The bestiary / spawn atlas: what the shard contains, parsed from its own ServUO tree | +| [CLILOCS.md](website/CLILOCS.md) | UO's id → name table: converting one from your client so items have names | +| [MARKETPLACE.md](website/MARKETPLACE.md) | The player-vendor index: how it is gathered, what it costs, how to tune it | | [website-README.md](website/website-README.md) | Snapshot of the website repo's README (setup/run reference) | | [PROJECT_TREE.md](website/PROJECT_TREE.md) | Auto-generated snapshot of the repo's tracked file layout | diff --git a/link/INTEGRATION.md b/link/INTEGRATION.md index 1b5ba20..fe3b9de 100644 --- a/link/INTEGRATION.md +++ b/link/INTEGRATION.md @@ -449,6 +449,78 @@ would carry ~25 zeroes. `maxPoints` follows the same `0 == uncapped` rule as the a points lookup stops at the character's own row, but a rank must count every row that beats them, in every system, on every profile build. Derive rank from `points.board` instead for anyone in the top N. +#### Player-vendor marketplace (Protocol 3.0) + +The shard-wide shop index: every player vendor's shop name, owner, location and priced inventory — +the same set the in-game **Vendor Search** gump reads, published so a site can offer the same search +from outside the game. + +An **amortized round-robin diff sweep**, not a snapshot RPC, and the distinction is load-bearing: +`rpc.rs::try_route` correlates a reply on the FIRST frame carrying a matching `reqId`, so a chunked +reply sharing one `reqId` would deliver chunk 1 to the HTTP caller and leak chunks 2..N onto the +broadcast feed. A whole-world snapshot could not fit in one frame inside the 10 s reply timeout +either. The per-account `vendor.snapshot` RPC (§5) is unaffected and still serves the player portal. + +Each tick inventories at most `Bridge.MarketSweepBatch` vendors (default 25) starting from a +persistent cursor, so **per-tick cost is bounded independently of world size**; full coverage takes +`ceil(vendors / batch) × MarketSweepSeconds`. A vendor is emitted only when its contents, prices, +shop name or location actually change. + +| kind | fields | notes | +|------|--------|-------| +| `vendor.listing` | `serial`, `shopName`, `ownerSerial`, `ownerName`, `location{}`, `count`, `total`, `truncated`, `items[]` | One vendor's complete shop — **never a delta**. The latest frame for a `serial` replaces the previous one outright. | +| `vendor.listing.remove` | `serial` | The shop is gone from the index: dismissed, expired, or its owner switched off the in-game Vendor Search flag. | + +```json +{"kind":"vendor.listing","serial":"0x40001234", + "shopName":"Darrow's Bargains","ownerSerial":"0x1A2B","ownerName":"Darrow", + "location":{"map":"Trammel","x":1421,"y":1699,"z":0, + "region":"Britain","house":"Darrow's Villa"}, + "count":2,"total":2,"truncated":false, + "items":[{"serial":"0x40012ABC","itemId":3922,"hue":0,"amount":1, + "price":25000,"name":null,"cliloc":1023721}, + {"serial":"0x40012ABD","itemId":7026,"hue":1157,"amount":3, + "price":500,"name":"a shard sigil","cliloc":1041243}], + "t":1752489280000} +``` + +**Six things consumers get wrong.** + +1. **`name` is `null` for nearly every item; `cliloc` is the real label.** Items carry a + `LabelNumber`, not a name. The plugin deliberately never calls `VendorSearch.GetItemName`, which + builds an `ObjectPropertyList`, serialises it and byte-parses the packet **per item** — a + multi-hundred-millisecond stall across a full pass. (It would not work anyway: every current + client ships its cliloc files compressed and ServUO's bundled `Ultima.StringList` cannot read + them, so the in-game gump has the same gap.) Resolve clilocs consumer-side; a non-null `name` is a + player-set literal and is strictly more specific, so **prefer it over the cliloc**. +2. **`location` is one nested object, and it may be absent entirely.** It is nested so that a + consumer gating vendor whereabouts gates one field rather than five that can drift apart — the + website's `market.location` rule removes the whole object. Treat a missing `location` as "not + published", not as an error. +3. **`truncated` means the shop holds more than the frame carries.** `count` is what was published, + `total` is what the shop actually holds, capped by `Bridge.MarketMaxListings` (default 250). A + commodity reseller with thousands of stacked resources is real and an uncapped frame for one is + measured in megabytes. Say "showing 250 of 3,104" rather than presenting a partial shop as + complete. +4. **`child: true` means the price buys the ENCLOSING CONTAINER.** ServUO prices a container as a + unit and everything inside inherits that price with no `VendorItem` of its own; `DoSearch` + surfaces the same flag. A UI that prints the container's price against each item inside it is + lying about the shard. +5. **Opted-out vendors are absent, and that is a privacy control.** `pv.VendorSearch` is the player's + own in-game toggle and the sweep honours it — hide your vendor in game and it is hidden here too. + The same goes for `Map.Internal` and a null backpack, matching `DoSearch`. Process + `vendor.listing.remove` promptly: it is how a player *revoking* that consent reaches you. +6. **Prices are inherently stale, by design.** The round-robin sweep means a shop can be a full cycle + behind. Any UI over this must say how old the data may be — the website derives it from the oldest + vendor row. + +Entries carry `ownerSerial`/`ownerName` and **never `acct` or `webId`**, the same rule `points.board` +follows. Absent entirely if the shard runs `Bridge.MarketEnabled=false` or an older plugin. Render +from `GET /market` (§6) on connect, then keep live with these events — though note that a live +firehose of whole vendor inventories is the largest stream the bridge produces, and a consumer that +only needs a browsable index (as the website does) is better served by the REST read plus the +periodic re-sweep. + --- ## 5. REST — read queries @@ -829,6 +901,35 @@ standings built over months and blanking them during a restart reads as data los one excluded by `Bridge.PointsSystems`). That is distinct from a published board nobody has scored in yet, which is **200** with an empty `top[]` — and the two are worth rendering differently. +### Player-vendor marketplace (Protocol 3.0) + +``` +GET /market?limit=200&offset=0 +→ { "vendors": [ {"kind":"vendor.listing","serial":"0x40001234", + "shopName":"Darrow's Bargains","ownerSerial":"0x1A2B","ownerName":"Darrow", + "location":{"map":"Trammel","x":1421,"y":1699,"z":0, + "region":"Britain","house":"Darrow's Villa"}, + "count":2,"total":2,"truncated":false,"items":[ ... ],"t":...}, ... ], + "total": 137, "limit": 200, "offset": 0 } +``` + +Every vendor's latest shop, exactly as `vendor.listing` published it (§4 for the frame and its six +gotchas). Served from the sidecar's projection, so it answers while the shard is down. + +**This is the only PAGED read the sidecar serves**, because it is the only board that can be a whole +world's inventory. `limit` is clamped to 1..1000 (default 200); `total` is returned so a caller knows +when to stop rather than paging until it sees a short page, which would race a concurrent sweep. +Ordering is by **serial**, not by shop name — a serial is stable while a shop name is renameable, so +a rename mid-walk cannot make a vendor skip or repeat a page. + +The route is `/market` and deliberately **not** `/vendors`: `/vendors/{account}` next door is the +per-account RPC (§5), and two routes a prefix apart meaning "this player's shops" and "every shop on +the shard" is a trap nobody wins. + +Frames are served **verbatim**, owner names and coordinates included. That is not an oversight: the +sidecar defines no audiences. Deciding who may see what is the consuming site's job — see +[`v3.md`](v3.md) §3 for how the website does it. + --- ## 7. Status codes diff --git a/link/PLAN.md b/link/PLAN.md index 709fead..b054421 100644 --- a/link/PLAN.md +++ b/link/PLAN.md @@ -284,6 +284,15 @@ Counts in `hello` are a live snapshot taken on the Core thread, not a cached val `Item.Name` is frequently `null`; the display name is `LabelNumber`, a cliloc id. **There is no `Data/Cliloc.enu` in this repo** — `BRIDGE_FINDINGS.md` §IV.4 is wrong about this. Cliloc data lives in the client install, which `DataPath` resolves to `D:\Games\Electronic Arts\Ultima Online Classic\`. Ship **both** `name` (when non-null) and `cliloc`, and resolve the number **on the website** against a cliloc map. That avoids a server-side dependency on the client directory. +**Update (3.0).** That recommendation held, and the reason it had to hold turned out to be stronger +than "avoids a dependency": **ServUO cannot resolve clilocs either.** Every current client ships its +`Cliloc.*` files compressed, and the bundled `Ultima.StringList` reads only the older plain layout — +so `VendorSearch.StringList` is null and `VendorSearch.GetItemName` returns `item.Name` on any modern +shard. The in-game Vendor Search gump has the same gap, which is why `vendor.listing` never calls it. +Pushing name resolution to the plugin was never an option. See [`v3.md`](v3.md) §8.6 and +`docs/website/CLILOCS.md` for how the site gets a table instead (the operator converts one from their +own client, once). + --- ## 8. Corrections to `BRIDGE_FINDINGS.md` @@ -327,6 +336,17 @@ leaderboards. `BridgePoints` is the widest read the bridge performs: ten of Serv keep a row for every character ever created, so it selects the top N in a single bounded pass rather than sorting, and runs on a deliberately slow 300 s interval. +Also shipped: **`vendor.listing`** ([`v3.md`](v3.md) §8), `BridgeMarket.cs`, the shard-wide +player-vendor index. It introduces the one sweep pattern the bridge did not previously have — an +**amortized round-robin**. Every other sweep walks its whole collection per tick, which is fine for +tens of houses or a fixed set of point systems and is not fine for a world of shops whose inventories +recurse into containers. `BridgeMarket` inventories at most `MarketSweepBatch` vendors per tick from +a persistent cursor, so the per-tick cost is bounded by the batch rather than by world size, and full +coverage takes `ceil(vendors / batch) x MarketSweepSeconds`. Measured at **15.4 ms** for a cold tick +of 25 vendors x 40 listings and **0.3 ms** in steady state (the per-vendor diff), on a shard of 209k +items / 43k mobiles. It is also the first stream to honour a per-player privacy toggle: ServUO's own +`PlayerVendor.VendorSearch` flag, so a shop hidden in game is hidden on the site. + ### Config keys (`Config/Bridge.cfg`) ```ini @@ -342,7 +362,8 @@ Read in `Configure()` via `Config.Get("Bridge.", default)`. Key scope is The set above is the 1.0 sample, not the current one — every later phase added keys (sweep intervals for each board, the town-crier/news caps, the admin write plane, account provisioning, and 3.0's -`RulesetEnabled` / `PublicConnectAddress` / `RulesetIncludeSchedule`, and the `Points*` block). +`RulesetEnabled` / `PublicConnectAddress` / `RulesetIncludeSchedule`, and the `Points*` and `Market*` +blocks). **`servuo-plugins/overlay/Config/Bridge.cfg` is the authoritative, commented list**; `BridgeConfig.cs` holds the defaults. diff --git a/link/v3.md b/link/v3.md index 362718d..b9d4abd 100644 --- a/link/v3.md +++ b/link/v3.md @@ -15,8 +15,8 @@ Each part is marked off here as it lands on `edge`. §9 carries the same state p | 2 | **B/1** — `world.ruleset` (§5) | ✅ **Done** | servuo-plugins [#3](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/3), link [#17](https://gitea.whitlocktech.com/RunicGateway/link/pulls/17), website [#111](https://gitea.whitlocktech.com/RunicGateway/website/pulls/111), docs [#66](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/66) | | 3 | **C** — spawn atlas (§6) | ✅ **Done** | website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112) (parsers + CLI + tables) + [#113](https://gitea.whitlocktech.com/RunicGateway/website/pulls/113) (API + pages + admin panel), docs [#67](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/67) + [#68](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/68) | | 4 | **B/2** — `points.board` (§7) | ✅ **Done** | servuo-plugins [#4](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/4), link [#18](https://gitea.whitlocktech.com/RunicGateway/link/pulls/18), website [#114](https://gitea.whitlocktech.com/RunicGateway/website/pulls/114), docs [#69](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/69) | -| 5a | **B/3 dependency** — cliloc table (§8.6) | 🟨 In review | website [#115](https://gitea.whitlocktech.com/RunicGateway/website/pulls/115), docs [#70](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/70) | -| 5b | **B/3** — `vendor.listing` (§8) | ⬜ Not started | — | +| 5a | **B/3 dependency** — cliloc table (§8.6) | ✅ **Done** | website [#115](https://gitea.whitlocktech.com/RunicGateway/website/pulls/115), docs [#70](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/70) | +| 5b | **B/3** — `vendor.listing` (§8) | 🟨 In review | servuo-plugins [#5](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/5), link [#19](https://gitea.whitlocktech.com/RunicGateway/link/pulls/19), website [#116](https://gitea.whitlocktech.com/RunicGateway/website/pulls/116), docs [#71](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/71) | | 6 | **Cutover** — `PROTOCOL_VERSION` 2→3 (§4) | ⬜ Not started | — | Order 5 split in two once §8.6's cliloc dependency turned out to be a client-format problem rather @@ -639,7 +639,7 @@ if it is renamed back. --- -## 8. Part B/3 — `vendor.listing` +## 8. Part B/3 — `vendor.listing` 🟨 In review ### 8.1 It cannot be an RPC, and this is load-bearing @@ -807,6 +807,75 @@ and text paths converge on identical content. driven by `staleAt` (the oldest `shard_vendors.updated_at`). The round-robin sweep means data is inherently up to one full cycle old, and the UI must say so. +Shipped with a second page, `routes/public/MarketVendor.jsx` at `/site/market/vendors/:serial` — +where a search result points. It is the only surface that can render the two states the result list +cannot: a `truncated` shop (*"showing 250 of 3,104 — this shop holds more than the shard +publishes"*) and a `location` an admin has gated away, which is a real answer rather than an empty +coordinate. + +### 8.8 What the build changed + +Four things the implementation settled differently from §8 as written, all of them found by building +against the live shard. + +**1. `location` is a nested object, not flat `map`/`x`/`y`/`region`.** §8.1's payload sketch had them +flat, and it would have made `market.location` — a rule Part A pre-wired — **inert**, exactly like +the `characterName` miss §7.5 records: `projectValue` matches literal JSON keys, so there is no +`location` key for the rule to match. Flat keys would have needed five rules that could drift apart. +Nesting makes one rule hide the facet, the coordinates, the region and the house together, on the +live frame and the stored read model alike, because both now spell it the same way. + +The other pre-wired rule, `market.ownerName`, checked out — it is a real key on the frame. Owner is +written as flat `ownerSerial`/`ownerName` rather than through `BridgeJson.Actor`, which would add +`acct` and `webId`; same argument `points.board` makes. `ownerSerial` was **added** to the +configurable fields alongside `ownerName`, because an admin who hides the owner's name and leaves a +serial every other board resolves back to that name has not hidden anything. + +**2. The per-vendor diff signature is the full listing set, not §8.3's `count | Σ(serial ^ price)`.** +That hash collides on the single most common change a shop makes: two items swapping prices, which +is what re-pricing looks like. The signature is built over the same buffer the frame is written +from, in the same order, so a match really does mean an identical frame. + +**3. There is no `payload` column on `shard_vendors`.** §8.5 implied the board pattern (whole frame +in JSON, columns hoisted for display). It does not apply here: the items ARE the searchable rows, so +they are normalized into `shard_vendor_items` and there is nothing left worth duplicating. The +sidecar keeps the whole blob, because outage resilience is its job and search is not. + +**4. Sweep cost is reported, and a slow tick warns.** The batch cap is a *claim* about per-tick cost, +and an operator tuning `MarketSweepBatch` was otherwise tuning blind. `[bridge status` now carries +`lastMs`/`maxMs`, and a tick over 50 ms prints a rate-limited warning naming the knob. + +Measured on the live shard (27 vendors × 40 listings, 209k items / 43k mobiles): + +| | | +|---|---| +| First tick — 25 vendors emitted cold | **15.4 ms** | +| Second tick — the remaining 2 | **3.4 ms** | +| Steady state — nothing changed | **0.3 ms** | +| Website `/market` search over 1,040 listings | 1,040 total, names resolved | +| Cliloc re-resolution pass over 1,040 rows | **50 ms** | + +The diff is what makes the steady state ~free; the batch cap is what bounds the cold case. Note the +arithmetic the warning exists for: at the default cap of 250 listings, a batch of 25 **full** shops +is 6,250 items ≈ 95 ms — over budget. Real shops hold tens, which is why 25 is the default, but a +shard of commodity resellers should lower the batch, and now it will be told to. + +Two smaller things worth not rediscovering: + +- **`BridgeJson.Escape` takes a NON-NULL string** — it dereferences `value.Length` immediately — and + `BridgeJson.Str` writes its own `,"key":` prefix, so neither serves a value inside a hand-built + object. Nearly everything this frame writes is legitimately null (an item's plain `Name` is null + for almost every item; a vendor in the street has no house), so that is the common path, not an + edge case. `BridgeMarket.Text()` is the two-line writer that was missing. +- **The ServUO console writes in the OS code page**, so an em dash in a `Console.WriteLine` renders + as `???` in the log an operator would paste into an issue. Bridge console output is ASCII. + +Search-side, one thing the site had to fix rather than inherit: `%` and `_` in a user's query are +**LIKE** metacharacters, not SQL ones, so parameterization does not neutralize them — a search for +`%` would otherwise match every listing on the shard. `shardMarket.db.js` escapes them. (The atlas's +`LIKE` searches predate this and have the same shape over a much smaller table; worth a follow-up, +not a blocker here.) + --- ## 9. Sequencing @@ -817,8 +886,8 @@ inherently up to one full cycle old, and the UI must say so. | 2 | **B/1** — `world.ruleset` (§5) | all four | new kind | ✅ Done | | 3 | **C** — spawn atlas (§6) | website, docs | none | ✅ Done | | 4 | **B/2** — `points.board` (§7) | all four | new kind + `char.profile` field | ✅ Done | -| 5a | **B/3 dependency** — cliloc table (§8.6) | website, docs | none | 🟨 In review | -| 5b | **B/3** — `vendor.listing` (§8) | all four | new kinds | ⬜ | +| 5a | **B/3 dependency** — cliloc table (§8.6) | website, docs | none | ✅ Done | +| 5b | **B/3** — `vendor.listing` (§8) | all four | new kinds | 🟨 In review | | 6 | **Cutover** — `PROTOCOL_VERSION` 2→3, `edge` → `main` | all four | the bump | ⬜ | --- diff --git a/website/BACKEND_DESIGN.md b/website/BACKEND_DESIGN.md index 3172b92..409569c 100644 --- a/website/BACKEND_DESIGN.md +++ b/website/BACKEND_DESIGN.md @@ -407,6 +407,50 @@ Two values carry non-obvious meanings, both set by the plugin and both documente cliloc rather than a literal. Listing therefore orders by `COALESCE(name, system)`, so boards awaiting cliloc resolution sort by their own key rather than clumping together under NULL. +### shard_vendors / shard_vendor_items — the player-vendor marketplace (Protocol 3.0) + +The shard-wide shop index, fed by `vendor.listing` / `vendor.listing.remove`. One row per player +vendor and one per priced listing. Full operator detail in [`MARKETPLACE.md`](MARKETPLACE.md); the +design is `docs/link/v3.md` §8. + +| Table | Shape | +|---|---| +| `shard_vendors` | `serial` (PK), `shop_name`, `owner_serial`, `owner_name`, `map`/`x`/`y`/`z`, `region`, `house`, `item_count`, `item_total`, `truncated`, `t`, `updated_at`. Indexes on owner, map, region and `updated_at`. | +| `shard_vendor_items` | `id` (PK), `vendor_serial`, `serial`, `item_id`, `hue`, `amount`, `price`, `name`, `cliloc`, `display_name`, `child`. Indexes on `vendor_serial`, `price`, `item_id`, `display_name`, and `(display_name, price)`. | + +**Ingest is per-vendor and authoritative**: the frame is the whole shop, so ingest is +delete-then-insert of that vendor's listings inside one transaction. All-or-nothing matters +specifically because the two writes are "the shop" and "what is in it" — a failure between them +leaves a shop advertising an inventory it no longer has, which is visibly wrong and indistinguishable +from a genuinely empty shop. No foreign keys, consistent with every other `shard_*` table. + +**There is deliberately no `payload` column**, unlike `shard_points_boards` directly above. The +board's top-N is a fixed-size list read whole, so it lives in JSON; here the items *are* the +searchable rows, so they are normalized and nothing is left worth duplicating. The sidecar keeps the +whole blob — outage resilience is its job, search is ours. + +Market state, not events: neither kind is in `LOGGED_KINDS`, and this is the strongest case of the +three v3 kinds. One frame carries up to 250 listings and the sweep re-emits a shop on any price +change, so logging would turn `shard_events` into a price history nobody reads. + +Two columns carry non-obvious meanings: + +- **`item_count` vs `item_total`.** `item_count` is what the frame published; `item_total` is what + the shop actually holds. They differ when `truncated` — the shard caps listings per frame + (`Bridge.MarketMaxListings`, 250 by default), and a commodity reseller with thousands of stacks + genuinely exceeds it. Any UI must show both or it presents a partial shop as complete. +- **`display_name` is denormalized at ingest**, resolved from the item's literal `name` (preferred — + a player set it, so it is more specific) else its `cliloc` against `shard_clilocs`. Resolving at + query time would put the cliloc table on the hot path and make search-by-name impossible. Because + the shard's diff sweep will not re-send an unchanged shop just because the site learned what its + items are called, **a cliloc import triggers a bulk re-resolution** of this column (after a boot + import and after an admin import; ~50 ms per thousand rows, never throws). + +`updated_at` is written explicitly on every upsert rather than left to `ON UPDATE CURRENT_TIMESTAMP`, +which MariaDB does not fire when every column is written back unchanged. A shop re-published +identically is still *freshly confirmed*, and without this the staleness banner would age a perfectly +current shop forever. + ### shard_feature_visibility — per-feature audience config (Protocol 3.0) One row per shard feature: `feature` (PK), `enabled`, `audience` (a rung on the ladder in §6.5), @@ -745,6 +789,9 @@ from the per-route **siteMode** middleware (§5), never from an auth gate. | GET | `/shard/ruleset` | the shard's own published ruleset (Protocol 3.0 `world.ruleset`): expansion, which optional systems are on, skill/stat caps, account and house limits, champion scroll rules, the save/restart schedule. Served from `shard_ruleset`, so it renders while the shard is down; live via `world.ruleset` on `/shard/stream`. Behind `requireFeature('ruleset')`. **`null`** means the shard has never published one — a real answer, distinct from a published ruleset. `caps.skill` / `caps.totalSkill` are in **tenths** (1000 = 100.0). | | GET | `/shard/points` | every points/loyalty leaderboard the shard publishes (Protocol 3.0 `points.board`) — Queen's Loyalty, Void Pool, the nine city loyalties, Clean Up Britannia, … Served from `shard_points_boards`, so it renders while the shard is down; live via `points.board` on `/shard/stream`. Behind `requireFeature('leaderboards')`, ordered by display name. **`maxPoints: 0` means uncapped** (the common case), and `nameString` is usually `null` with `nameNumber` holding a cliloc — resolve client-side or humanise the `system` key. | | GET | `/shard/points/:system` | one board by the shard's `PointsType` name (e.g. `QueensLoyalty`); `:system` must match `/^[A-Za-z][A-Za-z0-9_]{0,47}$/` or **400** before any query runs. **404** = the shard has never published that system, which is distinct from a published board nobody has scored in yet (**200** with an empty `top`). | +| GET | `/shard/market?q=&minPrice=&maxPrice=&itemId=&map=®ion=&sort=&limit=&offset=` | search the player-vendor marketplace (Protocol 3.0 `vendor.listing`). Returns **listings**, not vendors — "who sells X and for how much" is the question, and a vendor-shaped result would make every caller flatten the shops back out. Served from `shard_vendors` + `shard_vendor_items`, so it renders while the shard is down. Behind `requireFeature('market')` **and rate-limited** — the first genuinely expensive public read on the site (a `LIKE` scan plus a `COUNT` over what is typically the largest `shard_*` table, reachable with no session). `sort ∈ {price_asc, price_desc, recent}`. `q` matches the resolved display name **or** the item's literal name, with `%`/`_` escaped: they are `LIKE` metacharacters, not SQL ones, so parameterization alone would let `?q=%` match every listing on the shard. Every response repeats `staleAt` (the oldest vendor row) because the shard sweeps round-robin — a banner that ages with the results it labels, not one fetched once. | +| GET | `/shard/market/meta` | index size, staleness (`staleAt`/`freshAt`) and which facets and regions actually hold vendors, so a client builds its filters without running a search it will discard. | +| GET | `/shard/market/vendors/:serial` | one shop and its listings; `:serial` must match `/^0x[0-9A-Fa-f]{1,16}$/` or **400** before any query runs. **404** = a serial the index has never seen, which also covers a vendor since dismissed or hidden — to an anonymous caller those are the same answer, and distinguishing them would leak that a hidden vendor exists. `truncated` (with `total` exceeding `count`) means the shop holds more than the shard publishes per frame. | | GET | `/shard/features` | the shard features **this caller** may reach plus the audience rung they resolved to (§6.5), so a client hides nav it can't follow. Reports only what the caller can see — the list itself never discloses a gated feature. Consumed by the SPA header and (pending) the Android nav. | | GET | `/atlas/creatures?q=&facet=&limit=&offset=` | the bestiary, most numerous first, with an unpaginated `total`. Static content parsed from the shard's ServUO tree — **not** sidecar-backed, which is why the atlas sits outside `/shard`, and unlike `/shard/*` it **is** site-mode gated. Behind `requireFeature('atlas')`. `?facet=` is matched exactly and never validated against a list (no facet name exists in the code); the filter is an `EXISTS` over the points rather than a JSON path or `JSON_SEARCH` built from caller input, whose `%`/`_` wildcards would make `?facet=%` match everything. | | GET | `/atlas/creatures/:slug` | one creature: `places` (the point-in-rect aggregate — "lizardman → Shrines, Isamu-Jima, Yew"), `spawners` (the bounded raw list, with `spawnersTruncated`), `alsoHere`. **`points` is a COUNT and `spawners` is the LIST** — named apart so one key never means a number on one route and an array on another. `minDelay`/`maxDelay` are in **seconds**, normalised at parse time from the source's per-record minutes-or-seconds. 404 = no such creature in this atlas. | @@ -835,7 +882,11 @@ who"; `activity_log` provides the history feed. - **Cookie**: `httpOnly`, `sameSite=Lax`, `path=/`, and **`secure` decided per-request** (`COOKIE_SECURE=auto` → `secure: req.secure`). - **Trusted-device MFA.** A second, separate httpOnly cookie (`rg_trust`, default 30d) — opaque, sha256-hashed server-side in `trusted_devices` — lets a browser/app **skip the TOTP step** (never the password) on future logins. It is a server-side, per-row-revocable record (never a JWT claim), so the stateless session JWT is unchanged and trust stays revocable. It only ever gates the **second factor**; it deliberately outlives logout, and is cleared on untrust / password change / password reset / TOTP disable. **Recovery codes** (bcrypt, single-use) are the 2FA-lockout fallback. All admin trusted-device/MFA actions and the self actions (`auth.login.trusted_device`, `account.trusted_device.*`, `account.recovery_code*`, `admin.trusted_device.*`, `admin.user.totp.reset`) are audit-logged. See `docs/website/TRUSTED_DEVICES_MFA.md`. This is the key to dual access: the cookie is `Secure` when reached through Pangolin (HTTPS, `X-Forwarded-Proto: https`) but **not** `Secure` when reached directly over the LAN IP on plain HTTP — so login works in both. `COOKIE_SECURE=true|false` can force it. Requires `trust proxy` (below). `localhost:5173` (Vite) and `localhost:3000` are same-site, so the cookie flows in dev too. - **bcrypt** hashing (cost 10+); plaintext passwords never stored, logged, or returned. -- **Rate limiting** (`express-rate-limit`) on `/auth/login` and `/public/contact`. +- **Rate limiting** (`express-rate-limit`) on `/auth/login`, `/public/contact`, and — the only limited + *read* — `/public/shard/market` and `/public/shard/market/vendors/:serial` (60/min/IP). Every other + public read is an indexed lookup of bounded size; the marketplace search is a `LIKE` scan plus a + `COUNT` over the largest `shard_*` table, anonymous by default, so it is the one public GET that is + worth money to serve. - **Validation** (`express-validator`) on all writes; centralized error handler. - **helmet** with a Content-Security-Policy tuned for the built React SPA. The policies now live in **`server/src/config/csp.js`** (`app.js` only wires them up): diff --git a/website/MARKETPLACE.md b/website/MARKETPLACE.md new file mode 100644 index 0000000..5212b9f --- /dev/null +++ b/website/MARKETPLACE.md @@ -0,0 +1,167 @@ +# Marketplace — the player-vendor index + +**Status:** On `edge` — servuo-plugins [#5](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/5), link [#19](https://gitea.whitlocktech.com/RunicGateway/link/pulls/19), website [#116](https://gitea.whitlocktech.com/RunicGateway/website/pulls/116). +**Design:** [`docs/link/v3.md` §8](../link/v3.md) — Protocol 3.0 Part B/3. +**Depends on:** [`CLILOCS.md`](CLILOCS.md) — without a cliloc table, listings render as item ids. + +The marketplace is a searchable index of every player vendor on the shard: what +each shop is selling, for how much, and where it is standing. It is the same set +the in-game **Vendor Search** gump reads, offered from outside the game — so a +player can find the vanquishing kryss they want before logging in, and someone +who does not play at all can see that the economy exists. + +Page: `/site/market`, plus `/site/market/vendors/:serial` for one shop. + +## Three things the pages must say out loud + +Everything below follows from how the data is gathered, and each has a visible +consequence the UI is required to surface. + +**1. The prices are not live.** The shard sweeps vendors **round-robin** — at +most `Bridge.MarketSweepBatch` shops per tick — so a given shop can be a full +cycle behind. The page carries a *"prices last refreshed N minutes ago"* banner +driven by the **oldest** vendor row, not the newest: the one stale shop is the +one that wastes somebody's trip. + +**2. A shop can be truncated.** `Bridge.MarketMaxListings` (250 by default) caps +how many listings one frame carries. A commodity reseller with thousands of +stacked resources is a real thing, and an uncapped frame for one is measured in +megabytes. Over the cap the shop reports `truncated`, and the vendor page says +*"showing 250 of 3,104 — this shop holds more than the shard publishes"* rather +than presenting a partial shop as complete. + +**3. An item may have no name.** Items on the wire carry a cliloc id, not a name. +On a shard whose operator has not converted a cliloc table +([`CLILOCS.md`](CLILOCS.md)) the honest render is the item id — never an invented +label, which would be indistinguishable from a real one. + +## Privacy: the player's own toggle wins + +Only vendors whose owner left the in-game **Vendor Search** flag ON are ever sent +to the site. A player who hides their shop in game is hidden here too, and no +admin setting overrides that. When they hide one that was already indexed, the +shard emits `vendor.listing.remove` and the row is deleted — so revoking consent +takes effect, it does not merely stop refreshing. + +Shop name, owner character name and location default to **Everyone**, because the +stock Vendor Search gump already shows exactly that set to any player in game. +They remain admin-configurable; see [`SHARD_VISIBILITY.md`](SHARD_VISIBILITY.md). +Account names and website user ids never cross the wire at all. + +## How it is put together + +``` +ServUO uo-link sidecar website +────── ─────────────── ─────── +BridgeMarket.cs vendors table shard_vendors + round-robin sweep ──────► (whole frame blob) ──────► shard_vendor_items + per-vendor diff GET /market (paged) + display_name + vendor.listing resolved at ingest + vendor.listing.remove +``` + +**The shard side** walks at most `MarketSweepBatch` vendors per tick from a +persistent cursor, diffs each against what it last published, and emits a whole +frame for any shop that moved. Per-tick cost is therefore bounded by the batch, +not by how many vendors the world holds — full coverage takes +`ceil(vendors / batch) × MarketSweepSeconds`. + +**The sidecar** stores each frame whole and serves `GET /market`, its only paged +read. It normalizes nothing and defines no audiences: it is a dumb forwarder, and +search is the website's job. + +**The website** splits each frame into a vendor row and its listings, replacing +that vendor's whole listing set inside one transaction (the frame is +authoritative for that vendor, never a delta). Item names are resolved against +the cliloc table **on the way in** and stored denormalized, which is what makes +search-by-name possible and keeps the cliloc table off the hot path. + +## Operating it + +Everything is in `Config/Bridge.cfg` on the shard. There is nothing to configure +on the website. + +| Setting | Default | What it does | +|---|---|---| +| `MarketEnabled` | `true` | Master switch. Off publishes nothing; the page shows an empty index. | +| `MarketSweepSeconds` | `60` | Tick interval. | +| `MarketSweepBatch` | `25` | Vendors inventoried per tick. Clamped 1..500. | +| `MarketMaxListings` | `250` | Per-shop listing cap, after which `truncated`. Clamped 1..5000. | + +**Faster coverage vs. per-tick cost.** Lowering `MarketSweepSeconds` or raising +`MarketSweepBatch` both refresh the index sooner and both cost more per tick. +The expensive part is the item walk, which recurses into every container a vendor +is selling — so a shard of big shops should raise the interval rather than the +batch. + +`[bridge status` reports the sweep, including `lastMs` and `maxMs`: + +``` +market(enabled=True sweeps=42 scanned=108 emitted=27 removed=0 skipped=0 + truncated=0 tracked=27 vendors=27 cursor=2 batch=25 lastMs=0.31 maxMs=15.40) +``` + +A tick over **50 ms** prints a rate-limited warning naming the knob: + +``` +[Bridge] market sweep took 82.4 ms (budget 50 ms) - lower Bridge.MarketSweepBatch (now 25) if this persists +``` + +Measured on a shard with 27 vendors × 40 listings (209k items, 43k mobiles): +**15.4 ms** for the first cold tick of 25 vendors, **0.3 ms** in steady state — +the diff is what makes an unchanged world nearly free. Note the arithmetic: 25 +*full* shops at the 250-listing cap is 6,250 items ≈ 95 ms, over budget. Real +shops hold tens, which is why 25 is the default and why the warning exists. + +`[bridge sweepnow` runs one tick immediately; `[bridge reload` re-reads the +settings above without a restart. + +## Names arriving late + +Item names come from the cliloc table, and the market sweep will **not** re-send +an unchanged shop just because the site learned what its items are called. So a +cliloc import triggers a bulk re-resolution of every stored listing — otherwise +an operator who configures clilocs after the first sweep would see item ids until +every shop happened to change on its own. It runs after a boot import and after +an admin import, takes ~50 ms per thousand listings, and never throws: a failure +leaves names exactly as they were. + +## API + +All under `/api/v1/public/shard`, gated by the `market` feature and +**rate-limited** — these are the first genuinely expensive public reads on the +site (a `LIKE` scan plus a `COUNT` over what is typically the largest `shard_*` +table, reachable with no session). + +| Route | What | +|---|---| +| `GET /market` | Search. Returns **listings**, not vendors — "who sells X and for how much" is the question. `?q=&minPrice=&maxPrice=&itemId=&map=®ion=&sort=&limit=&offset=`, `sort ∈ {price_asc, price_desc, recent}`. | +| `GET /market/meta` | Index size, staleness, and which facets and regions actually hold vendors — so a client builds its filters without running a search it will discard. | +| `GET /market/vendors/:serial` | One shop and its listings. **404** for a serial the index has never seen, which also covers a vendor since dismissed or hidden — to an anonymous caller those are the same answer. | + +`q` matches the resolved display name **or** the item's own literal name, because +an item with a player-set name (most of what is worth searching for on a +player-run shard) may carry a generic cliloc. `%` and `_` in a query are escaped: +they are `LIKE` metacharacters, not SQL ones, so parameterization alone would let +a search for `%` match every listing on the shard. + +Full schemas are in the OpenAPI spec (`ShardMarketPage`, `ShardMarketVendor`, +`ShardMarketMeta`, `ShardMarketListing`, `ShardMarketLocation`). + +## Tables + +`shard_vendors` (one row per shop) and `shard_vendor_items` (one row per priced +listing). Both are ingest-owned; nothing else writes to them. No foreign keys, +in keeping with every other `shard_*` table — the ingest transaction is what +keeps them consistent, and an FK would turn a malformed frame into a failed write +rather than a dropped row. + +There is deliberately **no `payload` column** on `shard_vendors`, unlike the +points board next door. The board's top-N is a fixed-size list read whole, so it +lives in JSON; here the items *are* the searchable rows, so they are normalized +and there is nothing left worth duplicating. The sidecar keeps the whole blob, +because outage resilience is its job. + +`shard_vendor_items.display_name` is denormalized and indexed (alone, and +composite with `price` for "cheapest matching X"). See "Names arriving late" +above for how it is kept current. diff --git a/website/SHARD_VISIBILITY.md b/website/SHARD_VISIBILITY.md index c8e301f..31254f9 100644 --- a/website/SHARD_VISIBILITY.md +++ b/website/SHARD_VISIBILITY.md @@ -60,12 +60,32 @@ board while holding back one column. See the table in §3. | **Shard rules** | Skill/stat caps, house limits, vet rewards, the ruleset | Everyone | Connect address → Everyone | | **Spawn atlas** | Bestiary and spawn locations (static content) | Everyone | — | | **Leaderboards** | Point and loyalty standings | Everyone | Character names → Everyone | -| **Marketplace** | The shard-wide player-vendor index | Everyone, **live updates off** | Vendor owner name → Everyone · Location → Everyone | +| **Marketplace** | The shard-wide player-vendor index | Everyone, **live updates off** | Vendor owner name → Everyone · Vendor owner character id → Everyone · In-game location → Everyone | **Why the marketplace ships with live updates off.** A live feed of every vendor's full inventory would be the single largest thing the site sends. No page needs it — the marketplace is a search over stored data with a “prices last refreshed N minutes ago” stamp. Turn it on only if you want it. +**Why the marketplace's fields default to Everyone.** A vendor's shop name, its owner's character +name and where it is standing are *already* visible to every player in game: the stock Vendor Search +gump surfaces exactly that set to anyone who opens it. Publishing them on the site is not a new +disclosure. They stay configurable because a shard may still prefer to keep its economy behind a +login — and because "already public in game" is a judgement about your shard, not ours. + +**Location is one setting covering four things.** Hiding it removes the facet, the coordinates, the +region *and* the house name together. That is deliberate: those are four ways of saying the same +thing, and a setting that hid the coordinates while publishing the house name would not have hidden +anything. + +**Hiding the owner name also hides the owner character id.** They are separate settings so you can +be explicit, but leaving the id published while hiding the name achieves nothing — the leaderboards +and guild boards resolve that same id back to a character name. Set both. + +**What hiding a vendor cannot do.** Only vendors whose owner left the in-game *Vendor Search* flag ON +are ever sent to the site, so a player who hides their shop in game is hidden here too — and no +setting on this page can override that. It works the other way as well: these settings control who +sees the index, not whether players can find each other's shops in game. + **Why house owner/price default to Staff.** The public Houses page has always been a "where are the falling houses" board — location only. Owner and price are the staff view. That split is preserved. -- 2.49.1 From 71207cef1646e9eeab3441089b61c4a2c4996459 Mon Sep 17 00:00:00 2001 From: wtclaude Date: Wed, 29 Jul 2026 18:04:00 -0500 Subject: [PATCH 15/19] docs(link): the Protocol 3.0 cutover (v3.md order 6) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit INTEGRATION.md was written for the window that just closed -- it told integrators the version had NOT been bumped yet and that a sidecar on `edge` reports 2 while already carrying v3 kinds. That guidance is now wrong in the direction that matters, so the version section states 3 (header, /health, ws.hello, the 409 example and the §8 worked example) and replaces the "until then" paragraph with what a v2 integration actually has to do to upgrade: change the constant it sends, and nothing else, because nothing that existed in v2 changed shape. v3.md gains §4.1 for what the bump touches and, more importantly, WHY the website's boot migration is gated on a marker row: schema.sql is re-run on every boot and uo_link_config.protocol is admin-editable, so an ungated UPDATE would silently un-pin an operator running an older sidecar. That is the one piece of the cutover a reader could not infer from the code being one constant. Progress tables: 5b done, 6 in review. Co-Authored-By: Claude --- link/INTEGRATION.md | 38 +++++++++++++++++++------------------- link/PLAN.md | 6 ++++++ link/v3.md | 43 ++++++++++++++++++++++++++++++++++++++----- 3 files changed, 63 insertions(+), 24 deletions(-) diff --git a/link/INTEGRATION.md b/link/INTEGRATION.md index fe3b9de..a9fa80d 100644 --- a/link/INTEGRATION.md +++ b/link/INTEGRATION.md @@ -31,31 +31,31 @@ Missing or wrong token → **401** `{"error":"missing or invalid auth token"}`. The wire protocol is versioned so a mismatch is caught immediately instead of failing weirdly. -- Every response carries an **`X-UOLink-Version: 2`** header. -- `GET /health` and the WebSocket `ws.hello` frame include `"protocol": 2`. -- **Optionally**, send `X-UOLink-Version: 2` on your requests. If it disagrees with the sidecar, the request is rejected **409 Conflict**: +- Every response carries an **`X-UOLink-Version: 3`** header. +- `GET /health` and the WebSocket `ws.hello` frame include `"protocol": 3`. +- **Optionally**, send `X-UOLink-Version: 3` on your requests. If it disagrees with the sidecar, the request is rejected **409 Conflict**: ```json - { "error": "protocol version mismatch", "sidecar_protocol": 2, "client_protocol": "1" } + { "error": "protocol version mismatch", "sidecar_protocol": 3, "client_protocol": "2" } ``` Pin the version you built against and compare it to the header (or `/health.protocol`) at startup. **v2 (Protocol 2.0)** added the account-provisioning surface (§6.x: `POST /accounts/create`, `DELETE /link/{account}`) and the `account.*` events. Outbound event kinds are **additive** — a v1 client that ignores unknown kinds keeps working against the live feed — but the new *endpoints* require a v2 sidecar. If you send `X-UOLink-Version: 1`, calls to the new endpoints are refused with the 409 above. -**v3 (Protocol 3.0) is being built and the version has not been bumped yet.** It is defined as *adds -`world.ruleset`, `points.board`, `vendor.listing` / `vendor.listing.remove`*, and the bump to -`X-UOLink-Version: 3` happens **exactly once**, at the end, when [`v3.md`](v3.md) §4's `edge` → `main` -cutover lands — because a bump is an operator-visible hard break (409 on every protected route, and -the website's WS closes on the `ws.hello` mismatch), so doing it per phase would break the site -repeatedly. +**v3 (Protocol 3.0)** adds `world.ruleset`, `points.board` and `vendor.listing` / +`vendor.listing.remove`, with the `GET /ruleset`, `/points` and `/market` reads that serve them from +the sidecar's store. Same shape as the v2 bump: the event kinds are additive, so a v2 client that +ignores unknown kinds keeps working against the live feed, but the three new endpoints require a v3 +sidecar. There is deliberately **no feature-negotiation array** — v3 implies all three kinds, so the +version number alone tells you what is available. -Until then, sidecars on `edge` still report `2` while already carrying some v3 kinds and endpoints. -That is safe in the direction that matters: event kinds are additive, and a client that ignores -unknown kinds and tolerates a `404` on a not-yet-present endpoint keeps working. What you must **not** -do is infer feature availability from the version number during this window — probe the endpoint, or -treat a missing `world.ruleset` as "this shard hasn't published one". There is deliberately **no -feature-negotiation array**: v3 implies all three kinds. +**Upgrading a v2 integration.** The bump is an operator-visible hard break in one direction only: a +client still declaring `2` gets a 409 on every protected route and, on the WebSocket, a closed +connection on the `ws.hello` mismatch. So update the pinned version at the same time you deploy the +v3 sidecar. Nothing that existed in v2 changed shape, so that is the whole migration — the website +does it with a one-shot boot migration of its `uo_link_config.protocol` row ([`v3.md`](v3.md) §4.1); +a third-party client changes the constant it sends. --- @@ -68,7 +68,7 @@ GET /health (no auth) ```json { "status": "ok", // "ok" when plugin connected AND db reachable, else "degraded" - "protocol": 1, + "protocol": 3, "plugin_connected": true, // is the shard link up right now? "database": "ok", // "ok" | "error" "uptime": "3d 12h", @@ -91,7 +91,7 @@ A push-only stream of game events as they happen. You do **not** send commands o **On connect**, the first frame is: ```json -{ "kind": "ws.hello", "protocol": 1 } +{ "kind": "ws.hello", "protocol": 3 } ``` **Then** a continuous stream of event frames, each with at least `t` (epoch ms) and `kind`. Route on `kind`. @@ -955,7 +955,7 @@ sidecar defines no audiences. Deciding who may see what is the consuming site's A typical character page: ```js -const H = { "Authorization": `Bearer ${TOKEN}`, "X-UOLink-Version": "2" }; +const H = { "Authorization": `Bearer ${TOKEN}`, "X-UOLink-Version": "3" }; // 1. render the roster const roster = await fetch(`${BASE}/roster/${account}`, { headers: H }).then(r => r.json()); diff --git a/link/PLAN.md b/link/PLAN.md index b054421..5bf9b88 100644 --- a/link/PLAN.md +++ b/link/PLAN.md @@ -347,6 +347,12 @@ of 25 vendors x 40 listings and **0.3 ms** in steady state (the per-vendor diff) items / 43k mobiles. It is also the first stream to honour a per-player privacy toggle: ServUO's own `PlayerVendor.VendorSearch` flag, so a shop hidden in game is hidden on the site. +That completes 3.0's feature work, so the last step is the version itself: `PROTOCOL_VERSION` **2 → +3** and the coordinated `edge` → `main` merge across all four repos ([`v3.md`](v3.md) §4 and §4.1). +The bump is deliberately the *only* thing that happens at that moment — v3 adds kinds and endpoints +but changes nothing that already existed in v2 — so the operator-visible break is limited to +re-pinning the version, which the website does for itself in a one-shot boot migration. + ### Config keys (`Config/Bridge.cfg`) ```ini diff --git a/link/v3.md b/link/v3.md index b9d4abd..aa8b3c1 100644 --- a/link/v3.md +++ b/link/v3.md @@ -1,6 +1,6 @@ # Protocol 3.0 — Shard content, standings & the visibility framework -**Status:** In progress. All work lands on an `edge` branch in each repo; `edge` → `main` is the v3 cutover. +**Status:** Feature-complete on `edge`; the cutover (order 6) is in review. All work lands on an `edge` branch in each repo; `edge` → `main` is the v3 cutover. **Date:** 2026-07-28 **Codebase:** ServUO 57.4, ``, net48 / x64, Expansion **EJ**. **Companion to** [`PLAN.md`](PLAN.md) (1.0 read/event plane), [`PROTOCOL_2.md`](PROTOCOL_2.md) (2.0 provisioning + world-state streams), [`ADMIN_CONTROLS.md`](ADMIN_CONTROLS.md) (staff write plane), [`INTEGRATION.md`](INTEGRATION.md) (website API). @@ -16,8 +16,8 @@ Each part is marked off here as it lands on `edge`. §9 carries the same state p | 3 | **C** — spawn atlas (§6) | ✅ **Done** | website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112) (parsers + CLI + tables) + [#113](https://gitea.whitlocktech.com/RunicGateway/website/pulls/113) (API + pages + admin panel), docs [#67](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/67) + [#68](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/68) | | 4 | **B/2** — `points.board` (§7) | ✅ **Done** | servuo-plugins [#4](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/4), link [#18](https://gitea.whitlocktech.com/RunicGateway/link/pulls/18), website [#114](https://gitea.whitlocktech.com/RunicGateway/website/pulls/114), docs [#69](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/69) | | 5a | **B/3 dependency** — cliloc table (§8.6) | ✅ **Done** | website [#115](https://gitea.whitlocktech.com/RunicGateway/website/pulls/115), docs [#70](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/70) | -| 5b | **B/3** — `vendor.listing` (§8) | 🟨 In review | servuo-plugins [#5](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/5), link [#19](https://gitea.whitlocktech.com/RunicGateway/link/pulls/19), website [#116](https://gitea.whitlocktech.com/RunicGateway/website/pulls/116), docs [#71](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/71) | -| 6 | **Cutover** — `PROTOCOL_VERSION` 2→3 (§4) | ⬜ Not started | — | +| 5b | **B/3** — `vendor.listing` (§8) | ✅ **Done** | servuo-plugins [#5](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/5), link [#19](https://gitea.whitlocktech.com/RunicGateway/link/pulls/19), website [#116](https://gitea.whitlocktech.com/RunicGateway/website/pulls/116), docs [#71](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/71) | +| 6 | **Cutover** — `PROTOCOL_VERSION` 2→3 (§4) | 🟨 In review | the bump: link [#PR_LINK_BUMP](https://gitea.whitlocktech.com/RunicGateway/link/pulls/PR_LINK_BUMP), website [#PR_SITE_BUMP](https://gitea.whitlocktech.com/RunicGateway/website/pulls/PR_SITE_BUMP), docs [#PR_DOCS_BUMP](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/PR_DOCS_BUMP) — then `edge` → `main`: servuo-plugins [#PR_PLUG_CUT](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/PR_PLUG_CUT), link [#PR_LINK_CUT](https://gitea.whitlocktech.com/RunicGateway/link/pulls/PR_LINK_CUT), website [#PR_SITE_CUT](https://gitea.whitlocktech.com/RunicGateway/website/pulls/PR_SITE_CUT), docs [#PR_DOCS_CUT](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/PR_DOCS_CUT) | Order 5 split in two once §8.6's cliloc dependency turned out to be a client-format problem rather than a parser (see §8.6). 5a is website-only and lands first so the marketplace ships with real item @@ -242,6 +242,39 @@ admin-set `uo_link_config.protocol` column — so it happens **exactly once**, a from 2 to 3, so the cutover doesn't require a manual admin edit. `UOLINK_PROTOCOL` still overrides. - No feature-negotiation array anywhere — v3 implies all three kinds. +### 4.1 What the bump actually touches + +The version lives in five places, and all five move together: + +| Where | Change | +|---|---| +| `link/sidecar/src/main.rs` | `PROTOCOL_VERSION` 2 → 3 (with the v3 note beside the v2 one), plus the sidecar README's worked example | +| `website/server/db/schema.sql` | `uo_link_config.protocol` column default 1 → 3, plus the boot migration below | +| `website/server/src/model/uoLinkConfig/uoLinkConfig.model.js` | `DEFAULT_PROTOCOL` — what a site with nothing saved yet declares | +| `website/server/src/utils/uoLinkClient.js` + `uoLinkSocket.js` | the `config.protocol || …` fallbacks, so an unset value can never quietly send `1` and 409 with a confusing message | +| `website/client/.../ShardAdmin.jsx`, `website/.env.example` | the admin form's initial value and the documented env default | + +**The migration has to be one-shot, and that is the only subtle part.** `schema.sql` is re-run on +*every* boot (`utils/db.js::ensureSchema`), and every other statement in its migration block is an +idempotent `ADD COLUMN IF NOT EXISTS` / `MODIFY`. A bare `UPDATE uo_link_config SET protocol = 3` +would not be idempotent in the sense that matters: `protocol` is **admin-editable**, so an operator +who deliberately pins an older sidecar in Admin → Shard would silently be un-pinned on the next +restart. It is therefore gated on a marker row in `settings`: + +```sql +ALTER TABLE uo_link_config MODIFY COLUMN protocol INT NOT NULL DEFAULT 3; +UPDATE uo_link_config SET protocol = 3 + WHERE id = 1 AND protocol < 3 + AND NOT EXISTS (SELECT 1 FROM settings WHERE `key` = 'uo_link_protocol_3_migrated'); +INSERT IGNORE INTO settings (`key`, value) VALUES ('uo_link_protocol_3_migrated', '1'); +``` + +The marker is written *after* the `UPDATE`, so the first boot on the new build migrates and every +later boot is a no-op. A fresh install has no `uo_link_config` row to update and simply gets the +marker plus the new column default. `protocol < 3` rather than `= 2` so an install that never left +the old default of `1` is carried across too — it could not have been talking to a v2 sidecar +anyway. + --- ## 5. Part B/1 — `world.ruleset` ✅ Done @@ -887,8 +920,8 @@ not a blocker here.) | 3 | **C** — spawn atlas (§6) | website, docs | none | ✅ Done | | 4 | **B/2** — `points.board` (§7) | all four | new kind + `char.profile` field | ✅ Done | | 5a | **B/3 dependency** — cliloc table (§8.6) | website, docs | none | ✅ Done | -| 5b | **B/3** — `vendor.listing` (§8) | all four | new kinds | 🟨 In review | -| 6 | **Cutover** — `PROTOCOL_VERSION` 2→3, `edge` → `main` | all four | the bump | ⬜ | +| 5b | **B/3** — `vendor.listing` (§8) | all four | new kinds | ✅ Done | +| 6 | **Cutover** — `PROTOCOL_VERSION` 2→3, `edge` → `main` | all four | the bump | 🟨 In review | --- -- 2.49.1 From 32def88c4e33185826778f483fa1b45bf06ab48c Mon Sep 17 00:00:00 2001 From: wtclaude Date: Wed, 29 Jul 2026 18:09:50 -0500 Subject: [PATCH 16/19] docs(link): fill in the cutover PR numbers The order-6 row was written before the seven PRs existed. Same follow-up as the cliloc row got. Co-Authored-By: Claude --- link/v3.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/link/v3.md b/link/v3.md index aa8b3c1..cb393e8 100644 --- a/link/v3.md +++ b/link/v3.md @@ -17,7 +17,7 @@ Each part is marked off here as it lands on `edge`. §9 carries the same state p | 4 | **B/2** — `points.board` (§7) | ✅ **Done** | servuo-plugins [#4](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/4), link [#18](https://gitea.whitlocktech.com/RunicGateway/link/pulls/18), website [#114](https://gitea.whitlocktech.com/RunicGateway/website/pulls/114), docs [#69](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/69) | | 5a | **B/3 dependency** — cliloc table (§8.6) | ✅ **Done** | website [#115](https://gitea.whitlocktech.com/RunicGateway/website/pulls/115), docs [#70](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/70) | | 5b | **B/3** — `vendor.listing` (§8) | ✅ **Done** | servuo-plugins [#5](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/5), link [#19](https://gitea.whitlocktech.com/RunicGateway/link/pulls/19), website [#116](https://gitea.whitlocktech.com/RunicGateway/website/pulls/116), docs [#71](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/71) | -| 6 | **Cutover** — `PROTOCOL_VERSION` 2→3 (§4) | 🟨 In review | the bump: link [#PR_LINK_BUMP](https://gitea.whitlocktech.com/RunicGateway/link/pulls/PR_LINK_BUMP), website [#PR_SITE_BUMP](https://gitea.whitlocktech.com/RunicGateway/website/pulls/PR_SITE_BUMP), docs [#PR_DOCS_BUMP](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/PR_DOCS_BUMP) — then `edge` → `main`: servuo-plugins [#PR_PLUG_CUT](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/PR_PLUG_CUT), link [#PR_LINK_CUT](https://gitea.whitlocktech.com/RunicGateway/link/pulls/PR_LINK_CUT), website [#PR_SITE_CUT](https://gitea.whitlocktech.com/RunicGateway/website/pulls/PR_SITE_CUT), docs [#PR_DOCS_CUT](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/PR_DOCS_CUT) | +| 6 | **Cutover** — `PROTOCOL_VERSION` 2→3 (§4) | 🟨 In review | the bump: link [#20](https://gitea.whitlocktech.com/RunicGateway/link/pulls/20), website [#117](https://gitea.whitlocktech.com/RunicGateway/website/pulls/117), docs [#72](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/72) — then `edge` → `main`: servuo-plugins [#6](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/6), link [#21](https://gitea.whitlocktech.com/RunicGateway/link/pulls/21), website [#118](https://gitea.whitlocktech.com/RunicGateway/website/pulls/118), docs [#73](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/73) | Order 5 split in two once §8.6's cliloc dependency turned out to be a client-format problem rather than a parser (see §8.6). 5a is website-only and lands first so the marketplace ships with real item -- 2.49.1 From 5a091157d67aa147526b98feeeff67a9f197da6a Mon Sep 17 00:00:00 2001 From: wtclaude Date: Thu, 30 Jul 2026 00:46:16 -0500 Subject: [PATCH 17/19] =?UTF-8?q?docs(android):=20scope=20M11=20=E2=80=94?= =?UTF-8?q?=20Protocol=203.0=20shard=20parity=20for=20the=20app?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The v3 work added four shard features and an admin-configurable visibility framework the Android client knows nothing about. v3.md §10 deferred the app side as a follow-up; re-examining it before the cutover found the gap is wider than nav hiding: - no consumer for any of ruleset / leaderboards / market / atlas, - no `points` block on the character sheet (§7.3), - no cliloc-resolved item names (§8.6), and - shard nav gated on session role alone, so an admin who disables a feature or raises its audience leaves the app rendering entries that 404/403 into a generic error where the web client hides them. Scoped as PLAN.md §9 M11 in two PRs (the visibility rules + read-model adds, then the four screens), with the traps a real shard exposes recorded inline: uncapped `maxPoints: 0`, cliloc-named boards with a null `nameString`, skill caps in tenths, the required market staleness banner, the market stream being off by default, atlas delays in seconds, and `points`-count vs `spawners`-list. edge → main is held until both land so web and app surface the same shard on the same day. Neither PR is coupled to the merge order — on a pre-v3 website every new route and /public/shard/features 404s and the app falls back to today's behavior — so holding the cutover is a schedule decision, not a technical dependency. Also records two things verified as already correct, so they are not re-derived: the app's SSE request rides the authenticated client (same audience rung as the same account on web), and every shard DTO is nullable-with-defaults (field projection cannot cause a decode failure). Co-Authored-By: Claude --- android/PLAN.md | 111 +++++++++++++++++++++++++++++++++++++++++++++--- link/v3.md | 27 ++++++++++-- 2 files changed, 129 insertions(+), 9 deletions(-) diff --git a/android/PLAN.md b/android/PLAN.md index 8891a0d..47782a1 100644 --- a/android/PLAN.md +++ b/android/PLAN.md @@ -1,6 +1,6 @@ # Android App — Plan -Status: **M0–M7 landed; M7 (push notifications) both parts done — Part 1 backend (website#78) and Part 2 app (Android-app#15) plus a small `push.ntfyUrl` settings addition (website#79). Remaining: set the shard's `NTFY_*` deploy config so push lights up, and cut the v1 tag. M9 (native SSO login) is now underway backend-first — the Mobile SSO Authorization Bridge is being built in `website/` + `docs/` ahead of the app-side client (§4.2, §9 M9); custom-scheme callback only for now, App Links deferred (see [`APP_LINKS.md`](./APP_LINKS.md)).** This document is the +Status: **M0–M7 landed; M7 (push notifications) both parts done — Part 1 backend (website#78) and Part 2 app (Android-app#15) plus a small `push.ntfyUrl` settings addition (website#79). Remaining: set the shard's `NTFY_*` deploy config so push lights up, and cut the v1 tag. M9 (native SSO login) is now underway backend-first — the Mobile SSO Authorization Bridge is being built in `website/` + `docs/` ahead of the app-side client (§4.2, §9 M9); custom-scheme callback only for now, App Links deferred (see [`APP_LINKS.md`](./APP_LINKS.md)). **M11 (Protocol 3.0 shard parity)** is scoped and next: the app sees none of the four shard features v3 added (`ruleset`, `leaderboards`, `market`, `atlas`) and does not consult `GET /public/shard/features`, so it gates shard nav on session role alone while an admin can switch any of those surfaces off or raise its audience — the v3 `edge` → `main` cutover is held until it lands (§9 M11).** This document is the design contract for the `RunicGateway/Android-app` repo. It was written before implementation so the API changes it depends on could be landed in `website/` and `docs/` first. The authoritative API reference is the committed OpenAPI spec at `website/server/swagger/swagger-output.json` (regenerated @@ -631,6 +631,8 @@ not rank). | News & content | everyone | `/public/posts/:category`, `/public/pages/:slug` | | Wiki | everyone | `/public/wiki`, `/public/wiki/categories`, `/public/wiki/tags`, `/public/wiki/:slug` | | Shard (live) | everyone | `/public/shard/*` + `/public/shard/stream` (SSE) | +| **Rules / Leaderboards / Market** | everyone, *if the shard publishes them* | `/public/shard/{ruleset,points,market}` (M11) | +| **Atlas** (bestiary) | everyone, *if the shard publishes it* | `/public/atlas/*` (M11) | | Contact | everyone | `/public/contact` | | **My Account** | signed-in | `/player/account/*` (or `/admin/account/*` for staff — see §6.4) | | **My Characters / Vendors / Houses** | `player` (linked) | `/player/shard/*` | @@ -639,6 +641,12 @@ not rank). Guidelines: - The menu is **declarative + data-driven**, not a pile of `if role ==` checks — one list of entries with a `minAccess`/`requiredCapability` field, filtered by the session. +- **Session role is not the only gate on shard surfaces (M11).** Every shard-derived feature is + *admin-configurable* — it can be switched off or raised to a higher audience rung — so a shard entry + is filtered by the session role **and** by `GET /public/shard/features`, which reports the features + the caller may actually reach. While that answer is unknown (in flight, or the lookup failed) the app + shows everything: the server gates regardless, and a nav that flickers in on every load is worse than + a link that briefly `403`s. - Never hide the fact that more exists behind auth in a way that misleads; anonymous users see public groups and a "Sign in" affordance. - The server is the source of truth: a hidden/greyed item is a UX convenience; every gated call still @@ -663,9 +671,15 @@ Guidelines: ### 6.2 Public shard (live) - Status/online/feed/economy/champs/guilds/governors(+history)/presence/houses/idoc — the `/public/shard/*` GETs. -- **Live updates** — subscribe to `GET /public/shard/stream` (SSE, safe kinds only) and patch the - in-memory boards in place (champ/guild/city/house/presence update+remove frames). Reconnect with - backoff; fall back to poll if SSE drops. +- **Live updates** — subscribe to `GET /public/shard/stream` (SSE) and patch the in-memory boards in + place (champ/guild/city/house/presence update+remove frames). Reconnect with backoff; fall back to + poll if SSE drops. What arrives on the stream is **resolved from the caller's audience rung at + subscribe time**, not from a fixed allowlist (Protocol 3.0 §3.6) — the stream request carries the + bearer like every other call, so a signed-in app session sees exactly what the same account sees on + the web. +- **Visibility + the Protocol 3.0 surfaces (M11)** — `GET /public/shard/features` drives which of these + the menu offers; `GET /public/shard/{ruleset,points,points/:system,market,market/meta,market/vendors/:serial}` + and `GET /public/atlas/*` are the new reads. Full contract and traps in §9 M11. ### 6.3 Player self-service & game data (bearer) - **Account** — `GET /player/account`; `PATCH /player/account/username`; @@ -675,6 +689,11 @@ Guidelines: - **My game data** — `GET /player/shard/roster/:account`, `/char/:serial`, `/vendors/:account`, `/sales`, `/houses`. All ownership-checked server-side; a `503` means shard/sidecar down → show an "offline, retry" state (see §7). +- **The character sheet carries two things the app does not yet read (M11):** the `points` block + (per-character loyalty/points standings, Protocol 3.0 §7.3) and the server-resolved cliloc names on + `equipment[].clilocName` / `titles.rewardResolved` (§8.6). Both are served **ungated** on this route — + a character's own standings are self-service data and do not depend on the public `leaderboards` + feature being visible, which is the behavior the app must mirror rather than re-gate. - **Presentation is text-only for v1.** Character sheets and vendor listings render as data/text — no item icons or paperdoll art. A richer "pretty paperdoll" view is a **future** enhancement (pending the art/asset work on the platform side) and is explicitly out of the first release. @@ -883,10 +902,90 @@ push, and Play (M6–M8) follow the designed app. shard-write actions degrade gracefully when the sidecar is offline. Excluded: hero/CMS block editor, Discord-bot config, uo-link config, OAuth-provider setup. +12. **M11 — Protocol 3.0 shard parity** (post-v1; scoped 2026-07-30). The website's Protocol 3.0 work + added four shard features and, with them, an **admin-configurable visibility framework** the app + knows nothing about. `link/v3.md` §10 deferred the app side as a follow-up; it is now scoped + deliberately, and **the v3 `edge` → `main` cutover is held until both parts land** so web and app + surface the same shard on the same day (decided 2026-07-30). + + Neither part is coupled to the cutover *merge order*, which is what makes holding it a schedule + decision rather than a technical one: against a pre-v3 website every new route and + `/public/shard/features` simply `404`s, and each consumer below falls back to exactly today's + behavior. The app declares no protocol version and never talks to the sidecar. + + - **Part 1 — the visibility rules + the read-model adds.** The security-shaped half, reviewed on + its own: + - `GET /public/shard/features` → `{ level, features[] }`: the features **this caller** may reach. + A new singleton cache mirrors the web client's (`lib/useShardFeatures.js`): per-viewer but + stable for a session, invalidated on sign-in/out and on a server switch. + - `MenuEntry` gains `feature: String?` beside its existing `access`, so the one declarative menu + (§5) filters on the session role **and** the shard's live feature config. While the lookup is + in flight or has failed, **show everything** — the same deliberate fail-open the web client + takes, because the server gates regardless and a nav that flickers in on every load is worse + than a link that briefly `403`s. The gate is server-side; hiding is presentation. + - **`404` and `403` mean different things here** and neither is a generic error: + `requireFeature` `404`s a *disabled* feature (deliberately not disclosing that it exists) and + `403`s a viewer *below its audience*. Both render "not available on this shard", alongside the + existing `503` = shard offline (§7). + - **The `level` from `/features` is authoritative — do not re-derive the rung from the role.** + The server's ladder is `anonymous → logged_in → player → staff → admin`, where `player` means + *a linked game account* and staff always satisfy `player` (the same superset rule `Menu.kt` + already encodes as `isPlayer || isStaff`). + - **`char.profile.points`** → the "Loyalty & Points" block the web character sheet gained: + `CharProfileDto.points[{system, nameString, points, maxPoints, rank?}]`. Three traps, all of + them things a real shard does and a fake one does not (`v3.md` §7.5): `maxPoints == 0` means + **uncapped** and is the *common* case, so nothing may divide by it; `nameString` is usually + `null` because most systems name themselves with a cliloc, making the humanise-the-`system`-key + path the **primary** one rather than a fallback; and `rank` is absent unless the shard runs + `PointsProfileRank=true` — absent and "unranked" are different answers. + - **Cliloc-resolved names** (`v3.md` §8.6, already live on the website): `EquipmentDto` gains + `name` + `clilocName` and `TitlesDto` gains `rewardResolved`, so equipment stops rendering as a + layer or a bare id. Precedence is `name → clilocName → layer`: a player-given name outranks the + resolved type name, and the server applies the same order. A shard with no cliloc table + configured sends neither field and the sheet renders exactly as it does today. + - `ActorDto` keeps its `acct` / `webId` fields (nullable, so nothing breaks) but its KDoc stops + describing them as available: they are **locked to the admin rung**, always, and stripped from + every response below it. + - **Part 2 — the four new screens**, each hidden by its feature name in the menu: + - **Rules** — `GET /public/shard/ruleset` (`ruleset`). A `null` body means "the shard has not + published its ruleset yet", which is a different state from the feature being disabled. Every + block is optional and omitted when its system is off. **`caps.skill` / `caps.totalSkill` are in + tenths** (1000 = 100.0) and must be converted — the raw number is actively misleading, not + merely unhelpful. Live via the `world.ruleset` frame, which is on the public stream by default. + - **Leaderboards** — `GET /public/shard/points`, `/points/:system` (`leaderboards`). The same + `maxPoints`/`nameString` traps as the profile block. Live via `points.board`. + - **Market** — `GET /public/shard/market` (`q`, `minPrice`, `maxPrice`, `itemId`, `map`, `region`, + `sort`, `limit`, `offset`), `/market/meta` for the filter options + staleness, and + `/market/vendors/:serial` (`market`). Four things this screen must get right: it is the site's + first **rate-limited** public endpoint, so handle `429` the way the contact form does; the + *"prices last refreshed N minutes ago"* banner is **required, not decoration** — the shard + sweeps vendors round-robin, so a listing can legitimately be a full cycle stale and a page + implying live prices sends people to an item that sold twenty minutes ago; a `truncated` shop + must say so; and `location` is a **nested object** that an admin may gate away entirely, which + the vendor screen renders as "hidden by the shard" (a real answer) rather than as blank + coordinates — same for `ownerName` / `ownerSerial`. **The `market` SSE fan-out is off by + default** (a live firehose of vendor inventories would be the site's biggest bandwidth + consumer), so the screen is a plain paginated read and must never depend on live frames. + - **Atlas** — `GET /public/atlas/{creatures,creatures/:slug,regions,landmarks,champions,meta}` + (`atlas`). Note the path: `/public/atlas`, **not** `/public/shard` — the atlas is static shard + *content*, not live shard *state*, and unlike `/shard/*` it **is** `siteMode`-gated like + `/posts` and `/wiki`, so a site in maintenance mode withholds it independently of the sidecar. + Two units/naming traps from `v3.md` §6.3: respawn delays are **seconds** throughout, and + `points` is a *count* on the search route while `spawners` is the *list* on the detail route. + - **Verification** — the five-rung walk (`anonymous`, `logged_in`, `player`, `staff`, `admin`) + against a local website on the cutover branch, per + [`../link/v3.md`](../link/v3.md) §11 and the shard-visibility smoke harness; plus one pass with + **every feature disabled** in Admin → Shard Visibility, confirming the app *hides* each surface + instead of erroring on it. Unit tests cover the menu filter (role × feature set), the + `404`/`403`/`503` mapping, and DTO decode for each new shape. + - **Excluded**, in the same class as M10's exclusions: the admin *configuration* panels — Shard + Visibility, Spawn Atlas and Cliloc import — alongside the hero/CMS block editor, Discord-bot + config, uo-link config and OAuth-provider setup. + ### Deferred (not a milestone) -- **`/api/mobile` facade migration + app-version floor** — briefly planned as M11 (2026-07-22), now - **deferred with no app work scheduled**. The website's router refactor is being done in place with +- **`/api/mobile` facade migration + app-version floor** — briefly planned as its own milestone + (2026-07-22), now **deferred with no app work scheduled**. The website's router refactor is being done in place with every URL byte-identical and `/api/v1` is not being retired, so the app's ~70 hardcoded `api/v1/…` endpoints, its SSE path, and its SSO URLs keep working untouched. If the mobile contract ever needs to diverge from web, the migration comes back — starting from a one-line alias mount on the server, diff --git a/link/v3.md b/link/v3.md index cb393e8..284f886 100644 --- a/link/v3.md +++ b/link/v3.md @@ -23,6 +23,11 @@ Order 5 split in two once §8.6's cliloc dependency turned out to be a client-fo than a parser (see §8.6). 5a is website-only and lands first so the marketplace ships with real item names; 5b is the four-repo wire change. +**The `edge` → `main` half of order 6 is held for Android parity** (decided 2026-07-30, see §10): the +app sees none of the four new features and gates shard nav on session role alone, so merging the +cutover first would ship a shard whose app client silently disagrees with the web client about what is +public. The **bump** PRs into `edge` are unaffected and merge normally. + --- ## 1. Why 3.0 @@ -921,7 +926,7 @@ not a blocker here.) | 4 | **B/2** — `points.board` (§7) | all four | new kind + `char.profile` field | ✅ Done | | 5a | **B/3 dependency** — cliloc table (§8.6) | website, docs | none | ✅ Done | | 5b | **B/3** — `vendor.listing` (§8) | all four | new kinds | ✅ Done | -| 6 | **Cutover** — `PROTOCOL_VERSION` 2→3, `edge` → `main` | all four | the bump | 🟨 In review | +| 6 | **Cutover** — `PROTOCOL_VERSION` 2→3, `edge` → `main` | all four | the bump | 🟨 In review — `edge` → `main` held for Android parity (§10) | --- @@ -943,8 +948,24 @@ not a blocker here.) - `npm run swagger` **and** `npm run routes:manifest` on every route-touching PR — both are committed artifacts, and `test/routeManifest.test.js` fails on drift. -**Follow-up, not scoped for 3.0:** the Android app consumes the same public/player shard API and will -need `/public/shard/features` to hide its own nav. Track separately against `android-app/`. +**Android parity — now scoped, and it gates the cutover (decided 2026-07-30).** This was written as a +"track separately" follow-up. It was re-examined before the cutover and the gap is wider than nav +hiding: the app consumes the same public/player shard API but has **no consumer for any of the four new +features** (`ruleset`, `leaderboards`, `market`, `atlas`), no `points` block on its character sheet, no +cliloc-resolved item names (§8.6), and — the part that matters for §3 — **it gates shard navigation on +session role alone**, so an admin who disables a feature or raises its audience leaves the app +rendering entries that `404`/`403` into a generic error where the web client hides them. + +Two things were verified as already correct and are recorded so they are not re-derived: the app's SSE +request rides the same authenticated OkHttp client as every other call, so an app session resolves to +the same audience rung as the same account on the web; and every shard DTO in the app is +nullable-with-defaults, so field projection strips fields without a deserialization failure. + +Scoped as **M11 in [`../android/PLAN.md`](../android/PLAN.md) §9**, two PRs (the visibility rules + +read-model adds, then the four screens). `edge` → `main` is held until both land, so web and app +surface the same shard on the same day. Neither PR is coupled to the merge order — on a pre-v3 website +every new route and `/public/shard/features` `404`s and the app falls back to today's behavior — so +holding the cutover is a schedule decision, not a technical dependency. --- -- 2.49.1 From afcdb373eca209df307289e6a4fae1f2113665f5 Mon Sep 17 00:00:00 2001 From: wtclaude Date: Thu, 30 Jul 2026 03:34:29 -0500 Subject: [PATCH 18/19] docs(website): an operator runbook for extracting from your own UO client CLILOCS.md and SPAWN_ATLAS.md each explain WHY the operator has to supply something out of their own client, but neither says how. UOFIDDLER.md is the missing procedure: where to get UOFiddler, which two files in the zip matter, which runtime it needs, where Cliloc.enu actually lives, the conversion, how to point the site at the result, and how to confirm it took. Verified end to end on a stock Windows box: UOFiddler 4.22.2 (Ultima.dll is net10.0), .NET SDK 9.0.312 building the net8.0 converter, RollForward carrying it onto runtime 10.0.8, and the site's own parser reading the output back. Corrects one claim while doing it. CLILOCS.md said a UOFiddler GUI export "works equally well"; it does not. Its Cliloc tab writes `Number;Text;Flag` -- three columns, flag LAST -- and parseClilocText splits on the first separator only, so the flag is absorbed into the name and every item renders as `quarter staff;0`. The parser already handles `number,flag,text` with the flag in the middle, but a trailing `;0` is indistinguishable from a name that genuinely ends that way, so this stays a documented `sed` on the operator's side rather than a heuristic that would corrupt real names. Co-Authored-By: Claude --- README.md | 1 + website/CLILOCS.md | 17 ++- website/SPAWN_ATLAS.md | 3 +- website/UOFIDDLER.md | 282 +++++++++++++++++++++++++++++++++++++++++ 4 files changed, 300 insertions(+), 3 deletions(-) create mode 100644 website/UOFIDDLER.md diff --git a/README.md b/README.md index fe4d540..679624b 100644 --- a/README.md +++ b/README.md @@ -22,6 +22,7 @@ ci/ cross-cutting CI/quality notes | [SHARD_VISIBILITY.md](website/SHARD_VISIBILITY.md) | Who sees which shard data — the admin-configurable audience framework | | [SPAWN_ATLAS.md](website/SPAWN_ATLAS.md) | The bestiary / spawn atlas: what the shard contains, parsed from its own ServUO tree | | [CLILOCS.md](website/CLILOCS.md) | UO's id → name table: converting one from your client so items have names | +| [UOFIDDLER.md](website/UOFIDDLER.md) | **Operator runbook** — step-by-step extraction from your own UO client (cliloc table, creature art) | | [MARKETPLACE.md](website/MARKETPLACE.md) | The player-vendor index: how it is gathered, what it costs, how to tune it | | [website-README.md](website/website-README.md) | Snapshot of the website repo's README (setup/run reference) | | [PROJECT_TREE.md](website/PROJECT_TREE.md) | Auto-generated snapshot of the repo's tracked file layout | diff --git a/website/CLILOCS.md b/website/CLILOCS.md index e0747f5..f2e1413 100644 --- a/website/CLILOCS.md +++ b/website/CLILOCS.md @@ -44,6 +44,11 @@ they did before the table existed. ## Converting +> **Step-by-step operator instructions — where to get UOFiddler, where your +> client files are, and how to verify the import — are in +> [`UOFIDDLER.md`](UOFIDDLER.md).** This section covers the formats and the +> reasoning behind them. + Either format below is accepted; the site sniffs which one it was handed. | Format | Fidelity | Notes | @@ -77,8 +82,16 @@ dotnet run -- "/Ultima.dll" "/Cliloc.enu" /srv/uo-data/cli dotnet run -- "/Ultima.dll" "/Cliloc.enu" /srv/uo-data/clilocs.tsv --tsv ``` -A UOFiddler GUI export works equally well — anything producing one of the two -shapes above is fine. +A UOFiddler GUI export works too, but **not unmodified**: its Cliloc tab writes +`Number;Text;Flag` — three columns, the flag *last* — and the parser reads +`numbertext`, so the trailing field is absorbed into the name and +every item renders as `quarter staff;0`. Stripping it is one `sed`, given in +[`UOFIDDLER.md`](UOFIDDLER.md) §Route B. + +The parser already tolerates `number,flag,text`, with the flag in the *middle*. +It is not extended to cover the trailing form because a final `;0` is +indistinguishable from a name that genuinely ends that way — a heuristic there +would corrupt real names to save the operator one command. ## Shard-added and shard-edited items diff --git a/website/SPAWN_ATLAS.md b/website/SPAWN_ATLAS.md index f74d251..343ad62 100644 --- a/website/SPAWN_ATLAS.md +++ b/website/SPAWN_ATLAS.md @@ -223,7 +223,8 @@ The atlas is fully functional as text. `shard_spawn_creatures.art` is nullable and is NULL on every fresh import; pages render without images, which is the normal and supported state, not a degraded one. -An operator who wants art: +An operator who wants art — step-by-step, with the UOFiddler side spelled out, in +[`UOFIDDLER.md`](UOFIDDLER.md) §Part 2: 1. Extracts it from **their own** client files (UOFiddler, ClassicUO tooling, or any art extractor). diff --git a/website/UOFIDDLER.md b/website/UOFIDDLER.md new file mode 100644 index 0000000..96d8e69 --- /dev/null +++ b/website/UOFIDDLER.md @@ -0,0 +1,282 @@ +# Extracting from your own UO client (UOFiddler) + +**Audience:** the shard operator, once, at setup time. +**Related:** [`CLILOCS.md`](CLILOCS.md) (why the cliloc conversion is unavoidable), +[`SPAWN_ATLAS.md`](SPAWN_ATLAS.md) (where creature art fits). + +Two features read data that **only exists inside a UO client**, and a UO client's +files are EA's, not ours to redistribute. So neither this repo nor any image we +publish can ship them — the operator extracts from **their own** client, once, +and points the site at the result. + +| Feature | What it needs | Required? | Without it | +|---|---|---|---| +| **Item / title names** ([`CLILOCS.md`](CLILOCS.md)) | `Cliloc.enu`, converted | No | Names render as raw ids — `id 1023721` instead of *quarter staff* | +| **Creature art** ([`SPAWN_ATLAS.md`](SPAWN_ATLAS.md)) | Sprites from `.mul`/`.uop` | No | Atlas pages render as text, which is the normal state | + +**Both are optional and neither is load-bearing.** A shard that never does any of +this is fully supported. Do part one and skip part two if art is not worth your +time — they share only the tool. + +Everything you extract stays **outside the repository**: the converted cliloc +file lives at a path you choose, and `spawnAtlas.art.json` plus `server/uploads/` +are gitignored, so none of it can be committed by accident. + +--- + +## Part 0 — Get UOFiddler + +[UOFiddler](https://github.com/polserver/UOFiddler) is the community client-file +editor. We use it because its `Ultima.dll` already contains the cliloc +decompressor, maintained by people who do this for a living. + +1. Download the latest release zip from + — one asset, named + `UOFiddler-.zip` (4.22.2 is ~2 MB). +2. Extract it. The zip contains a single top-level folder, and the two files that + matter are at **its root**: + + ``` + UOFiddler-4.22.2/ + Ultima.dll ← the decompressor (Part 1 needs this path) + UoFiddler.exe ← the GUI (Part 2 needs this) + plugins/ + … + ``` + +3. **Runtime:** UOFiddler 4.22.2 is built for **.NET 10**. Running `UoFiddler.exe` + needs the .NET 10 **Desktop** Runtime (Windows only); loading `Ultima.dll` from + the converter in Part 1 needs the .NET 10 runtime. Install from + . + +### Finding your client files + +The cliloc file is in your **UO client installation directory**, not in your +ServUO tree — the shard server has no copy of it. Look for `Cliloc.enu` (English; +the other seven are `chs`, `cht`, `deu`, `esp`, `fra`, `jpn`, `kor`) beside +`art.mul` / `artLegacyMUL.uop`. The EA Classic Client's default location is: + +``` +C:\Program Files (x86)\Electronic Arts\Ultima Online Classic\ +``` + +**If your shard distributes its own patched client to players, use that copy.** +Any cliloc edits you shipped to players are then already in the base table and +you need no overlay for them (see [`CLILOCS.md`](CLILOCS.md) §Shard-added and +shard-edited items). + +--- + +## Part 1 — Convert the cliloc table + +**Goal:** turn the client's compressed `Cliloc.enu` into a file the site can +read, and point the site at it. + +The site cannot read `Cliloc.enu` directly. Every modern client compresses it +(the "Mythic" container), and so does ServUO's own bundled `Ultima.StringList` — +which is why the shard cannot supply names on our behalf either. The full +reasoning is in [`CLILOCS.md`](CLILOCS.md) §Why the operator has to convert the +file; this section is just the procedure. + +Two routes. **The bundled tool is the recommended one** — the GUI export needs a +fixup step, described below. + +### Route A — the bundled converter (recommended) + +Needs a .NET SDK (any version 8 or newer — the project targets `net8.0` and rolls +forward, so whatever you have works) **plus** the .NET 10 runtime from Part 0, +which is what actually loads `Ultima.dll`. + +```bash +cd website/server/tools/cliloc-export +dotnet build -c Release + +# plain binary — recommended, exact +dotnet run -c Release -- \ + "/path/to/UOFiddler-4.22.2/Ultima.dll" \ + "/path/to/UO client/Cliloc.enu" \ + /srv/uo-data/clilocs.plain + +# or tab-delimited text, if you want to eyeball or hand-edit it +dotnet run -c Release -- \ + "/path/to/UOFiddler-4.22.2/Ultima.dll" \ + "/path/to/UO client/Cliloc.enu" \ + /srv/uo-data/clilocs.tsv --tsv +``` + +Expected output for a stock English client: + +``` +wrote 123490 entries to /srv/uo-data/clilocs.plain (maxTextBytes=12150, skippedOversize=0) +``` + +**Sanity-check that number.** A stock `Cliloc.enu` is ~123,000 entries. A few +hundred means it read something else and you should not ship the result. The +tool exits non-zero and says `no entries were written — is that a cliloc file?` +when it gets nothing at all. + +The conversion runs on whatever machine has the client (usually Windows), and the +site reads the output wherever it runs — so **copy the output file to the server** +if those are different machines. It is a single self-contained file (~5 MB); the +`--tsv` form is larger but diff-able. + +
+Errors you may hit + +| Message | Cause | +|---|---| +| `Ultima.StringList not found — is that really UOFiddler's Ultima.dll?` | First argument points at some other `Ultima.dll` (ServUO ships one too — it is **not** the same assembly and cannot do this) | +| `You must install .NET to run this application` | Missing the .NET 10 runtime from Part 0 step 3 | +| `Unexpected Ultima.StringList API` | UOFiddler older than 4.21 | +| `usage: clilocexport …` | Fewer than three arguments | + +
+ +### Route B — the UOFiddler GUI + +Use this if you would rather not install a .NET SDK. **It needs one extra step**, +so do not skip the fixup. + +1. Launch `UoFiddler.exe` and point it at your client directory when it asks + (or **Options → Path Settings**). +2. Open the **Cliloc** tab and use its **export to CSV** action. +3. It writes `CliLoc.csv` to UOFiddler's configured output path, in **three** + columns with a header row: + + ``` + Number;Text;Flag + 1023721;quarter staff;0 + ``` + +4. **Strip the trailing flag column.** The site's text parser reads + `numbertext`, so that third field is otherwise absorbed into the name + and every item on the site renders as `quarter staff;0`. + + ```bash + sed -E 's/;[0-9]+$//' CliLoc.csv > clilocs.csv + ``` + + ```powershell + Get-Content CliLoc.csv | + ForEach-Object { $_ -replace ';\d+$','' } | + Set-Content -Encoding utf8 clilocs.csv + ``` + + The header row needs no removal — a line whose first field is not an integer + is skipped. Blank entries (`1005008;`) survive the fixup correctly and are + dropped at import, as intended. + +5. Copy `clilocs.csv` to the server. + +**Why the fixup is not just done for us:** the parser already handles +`number,flag,text` — the flag in the *middle*, which is what several exports +emit. UOFiddler puts it at the *end*, where it is indistinguishable from a name +that genuinely ends in `;0`. One `sed` on the operator's side beats a parser +heuristic that would corrupt real names. + +### Point the site at it + +Two ways, the setting winning over the environment: + +| Where | How | +|---|---| +| **Admin → Shard → cliloc path** | Takes effect on the next refresh, no redeploy | +| `UO_CLIENT_PATH` env var | The deploy-time default | + +The value may be **the file itself or a directory to search** — both are natural +answers to "where is it", and overlays are picked up either way. + +Setting the path deliberately does **not** import as a side effect. Click +**Import** (or `POST /api/v1/admin/shard/clilocs/import`) to load it. + +### Verify + +`GET /api/v1/admin/shard/clilocs`, or the Admin → Shard panel, reports what each +source contributed: + +```json +"sources": [ + { "label": "clilocs.plain", "kind": "base", "entries": 123490, "added": 123490, "overrode": 0 } +] +``` + +Roughly **67,500 rows stored** from a stock table is correct — about half a +cliloc table is empty strings for ids the client reserves and never uses. + +Then load any character sheet with equipment: items should show names rather than +`id 1023721`. + +
+What a refusal means + +A bad file answers `200` with a `status` and a named reason, not a `500` — you +need to be told *which file* to fix. + +| `code` | Meaning | +|---|---| +| `COMPRESSED` | You pointed at the raw client `Cliloc.enu`. Convert it — this whole page. | +| `TRUNCATED` | Half-copied file. Re-copy; the loaded table is untouched. | +| `EMPTY` | A text source with no parseable rows — the file is named in the reason. | +| `status: needsReview` + `missingSources` | A previously-loaded source has vanished (unmounted volume? deliberate deletion?). Nothing changes until you re-import with `{ "approve": true }`. | + +
+ +### Custom items — do *not* re-export for these + +Shard-added items carry ids no client table has. Drop a small delimited file in a +`custom/` directory beside the base file and re-import: + +``` +/srv/uo-data/ + clilocs.plain ← base, from this guide + custom/ + 01-uomysticmoon.tsv ← your additions and overrides +``` + +Files are read in sorted order and **later sources win**, so an overlay both adds +new ids and overrides stock ones you have re-purposed. **Adding one item never +means re-exporting a 5 MB client file.** Details in [`CLILOCS.md`](CLILOCS.md). + +--- + +## Part 2 — Creature art for the spawn atlas (optional) + +**Goal:** put sprites on atlas pages. Purely cosmetic — the atlas is fully +functional as text, and `art` is NULL on every fresh import. + +**This project ships no art and no art-extraction tooling, and never will.** + +1. In `UoFiddler.exe` (paths configured as in Route B step 1), open the + **Animations** tab for creature sprites — or **Items** for object art — find + the creature, and export as PNG. Right-click an entry for its export options, + or use the tab's *Export All* action for a batch. (4.22.2 added an export + option to the Animation tab's thumbnail list, which is the convenient one + here.) +2. Put the images under `server/uploads/atlas/`. +3. Copy `server/db/data/spawnAtlas.art.example.json` to `spawnAtlas.art.json` in + the same directory and map creature slugs to file names: + + ```json + { + "lizardman": "lizardman.png", + "orc": "orc.png" + } + ``` + + **Keys are the slugs the atlas API reports**, derived from the type names in + your own shard's `Spawns/*.xml` — read them off the atlas rather than guessing. + A creature with no entry renders without art, which is the default. + +4. Restart, or `npm run atlas:import -- --force`. + +The art map is re-read on every atlas refresh, so adding one image is an edit plus +a refresh. Both `spawnAtlas.art.json` and `server/uploads/` are gitignored. + +--- + +## Licensing, briefly + +UO's strings and sprites are EA's. Extracting from **your own** client for +**your own** shard is the arrangement here; redistributing the extracted files is +not something this project does or can advise on. That is the whole reason this +page exists instead of a download link. -- 2.49.1 From 1444c7741396abdc37f06aa43e3be9ed27001b69 Mon Sep 17 00:00:00 2001 From: wtclaude Date: Sat, 1 Aug 2026 00:58:53 -0500 Subject: [PATCH 19/19] docs(link): the shard-name fallback, the atlas places shape, and two traps MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Records what the live Protocol 3.0 smoke test (ServUO + sidecar + website + AVD) turned up, so none of it has to be rediscovered. v3.md §5.3 — the ruleset `shard` field now falls back to the instance's own name when the shard publishes ServUO's stock "My Shard", why that is done at ingest rather than on read (the frame is also broadcast live), and why the backfill snapshot must go through the dispatcher instead of writing state directly: a direct call made it a second writer that skipped the normalization. v3.md §7.4 — an unscored board renders a placeholder row rather than a blank card, and why it is deliberately not shaped like a real entry. PLAN.md §9 M11 — `places` is a list of {facet,label,spawners,maxAlive} OBJECTS, not of place-name strings, and typing it `List` makes the whole detail route fail to decode while the request itself returns 200. Adds the rule that came out of it: decode tests must feed real captured JSON, because the fakes build DTOs in Kotlin and can never catch a wire mismatch. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01U7CBg11prhLimL9iHSX1bP --- android/PLAN.md | 14 +++++++++++--- link/v3.md | 27 +++++++++++++++++++++++++-- 2 files changed, 36 insertions(+), 5 deletions(-) diff --git a/android/PLAN.md b/android/PLAN.md index 47782a1..af75c06 100644 --- a/android/PLAN.md +++ b/android/PLAN.md @@ -970,14 +970,22 @@ push, and Play (M6–M8) follow the designed app. (`atlas`). Note the path: `/public/atlas`, **not** `/public/shard` — the atlas is static shard *content*, not live shard *state*, and unlike `/shard/*` it **is** `siteMode`-gated like `/posts` and `/wiki`, so a site in maintenance mode withholds it independently of the sidecar. - Two units/naming traps from `v3.md` §6.3: respawn delays are **seconds** throughout, and - `points` is a *count* on the search route while `spawners` is the *list* on the detail route. + Three traps. Two are units/naming, from `v3.md` §6.3: respawn delays are **seconds** + throughout, and `points` is a *count* on the search route while `spawners` is the *list* on + the detail route. The third is a **shape**: `places` is a list of + `{facet, label, spawners, maxAlive}` **objects**, not of place-name strings — it is the + aggregate the screen exists to show ("Shrines, Isamu-Jima, Yew"), it arrives only on the + detail route, and typing it `List` makes that whole route fail to decode while the + request itself returns `200`. - **Verification** — the five-rung walk (`anonymous`, `logged_in`, `player`, `staff`, `admin`) against a local website on the cutover branch, per [`../link/v3.md`](../link/v3.md) §11 and the shard-visibility smoke harness; plus one pass with **every feature disabled** in Admin → Shard Visibility, confirming the app *hides* each surface instead of erroring on it. Unit tests cover the menu filter (role × feature set), the - `404`/`403`/`503` mapping, and DTO decode for each new shape. + `404`/`403`/`503` mapping, and DTO decode for each new shape. **Decode tests must feed real + captured JSON**, not DTOs built in Kotlin: the fakes under `data/api/fake/` construct objects + directly, so they can never catch a wire/type mismatch — which is how the `places` shape above + shipped past a green suite. - **Excluded**, in the same class as M10's exclusions: the admin *configuration* panels — Shard Visibility, Spawn Atlas and Cliloc import — alongside the hero/CMS block editor, Discord-bot config, uo-link config and OAuth-provider setup. diff --git a/link/v3.md b/link/v3.md index 284f886..cd79d2a 100644 --- a/link/v3.md +++ b/link/v3.md @@ -350,8 +350,11 @@ Sidecar — `store.rs`: singleton `ruleset(id CHECK(id=1), rev, json, updated_t) `main.rs`: new arm in the board-projection match; `web.rs`: `GET /ruleset` served from the store, so it answers during a shard outage (`PROTOCOL_2.md` §12.2). -Website — `uoLinkClient.getRuleset()`; `uoLinkSocket.backfill()` (object-shaped, so follow the -`getPresence()` block's explicit form, not the array-only `snapshot()` helper); `shardIngest.js` → +Website — `uoLinkClient.getRuleset()`; `uoLinkSocket.backfill()` (object-shaped, so it cannot use the +array-only `snapshot()` helper — but it **must still go through `shardIngest.ingest()`**, as +`ingestEach` does, rather than calling `shardState.setRuleset` directly: the two arrival orders have +to produce the same stored frame, and a direct call quietly made backfill a second writer that +skipped the normalization below); `shardIngest.js` → `shardState.setRuleset`, **not** in `LOGGED_KINDS` (it re-arrives every reconnect and `server.hello` already marks those); `KIND_FEATURE['world.ruleset'] = 'ruleset'`; `shard_ruleset` singleton table (`rev`, `expansion`, `payload JSON`, `t`); `GET /public/shard/ruleset` behind @@ -360,6 +363,17 @@ already marks those); `KIND_FEATURE['world.ruleset'] = 'ruleset'`; `shard_rulese Client — NEW `routes/public/Rules.jsx` at `/site/rules`, alongside `/site/champs|guilds|governors|houses`; live via `useShardFeed({ filter: new Set(['world.ruleset']) })`. +**The `shard` field falls back to the instance's own name.** ServUO ships `Server.cfg` with +`Name=My Shard`, so an operator who never edited it publishes that verbatim — which is the shard +saying *unnamed*, not naming anything, and the rules page then reads "My Shard" under a header +carrying the real one. `shardIngest` substitutes `settings.getInstanceName()` (the admin-editable +site title, else `BRAND_NAME` — the same resolution `getPublic().brand.name` uses, so one install +never shows two names) when `shard` is absent, blank, or exactly the stock default, matched +case-insensitively and trim-tolerantly but only as a **whole** value: a shard genuinely called +*"My Shard Reborn"* has named itself and keeps it. Applied at **ingest**, not on read, because the +ruleset is also broadcast live — the same object goes to the SSE fan-out, so a read-time +substitution would be undone by the next reconnect's frame. + ### 5.4 Risk Perf is nil (~3 KB per connect). The only real risk is publishing a secret, mitigated by the explicit @@ -638,6 +652,15 @@ Client — NEW `routes/public/Leaderboards.jsx` at `/site/leaderboards`; a "Loya added to `components/CharacterSheet.jsx`, one edit serving both `PlayerCharacter.jsx` and `AdminCharacter.jsx`. +**An unscored board still renders a row.** Most systems on a young shard have `top: []`, and a page +of blank cards reads as broken rather than as new — so a board with no entries shows a single +placeholder bearing the **instance's own name** with an em dash where a score goes, above the +existing "nobody has earned points here yet" line. It is deliberately **not** shaped like an entry — +no rank, no medal, no bar, muted — because a placeholder that looked like a real standing would be a +fabricated one; the first real entry replaces it outright. Purely presentational: the API keeps +sending an empty `top`, so no consumer ever receives an invented row. Web and app render it the same +way (`Leaderboards.jsx`, `LeaderboardsScreen.kt`). + ### 7.5 What the run against a real shard changed The plan above was written from reading `PointsSystem.cs`. Booting the actual shard (ServUO 57.4, a -- 2.49.1