10 Commits

Author SHA1 Message Date
1b7da860b5 Merge remote-tracking branch 'origin/edge' into docs/spawn-atlas 2026-07-28 16:47:54 -05:00
ff1c2064a5 docs(website): the atlas reads the shard's tree on every boot, not a snapshot
Follows the redesign in website #112. Two decisions from the original §6 were
rejected in review and replaced; the docs now describe what was actually built.

**The committed artifact is gone.** A shard's maps change over its life, so a
snapshot in the repo silently drifts from the world players actually see. The
ServUO tree is the single source of truth and the atlas is re-derived on every
server boot, hash-gated so an unchanged tree costs one read pass and no write.

**Nothing may name a facet.** The first implementation carried a lookup table of
the six stock UO facets. A shard may add facets, replace them outright, or rename
them when its maps are updated, and a built-in list mishandles all three
silently. Reconciliation is now by matching against the facet set discovered from
the shard's own data, with an unmatched name keeping its own rather than being
forced into a wrong bucket.

## Changes

- **`website/SPAWN_ATLAS.md`** rewritten: the two ideas that shape the design,
  how to configure the tree path, the boot flow as a decision tree, the
  approve/reject flow, and the code layout. The artwork policy is unchanged and
  still explicit — no art ever ships, operators extract their own from their own
  client files.
- **`link/v3.md` §6.1 (new)** records the two rejected decisions plus the two
  boot-path contracts. The old "what real data changed" list becomes §6.2. §6's
  now-superseded passages — the artifact bullet, the payload budget, the operator
  re-run story — are marked rather than deleted, so the reasoning stays legible.
- **`website/BACKEND_DESIGN.md`** documents `shard_atlas_pending` and the two
  contracts that make it safe: a facet removal is staged for a human, and the
  boot refresh can never block startup.

The two contracts are the part worth reviewing. Losing a facet is
indistinguishable at boot from a half-copied or mid-update tree, so it is staged
rather than applied; and no failure mode of the atlas — missing path, unreadable
mount, malformed file, database error — is allowed to stop the site coming up.

---

- [x] AI-assisted: written with **Claude Code** (Claude Opus 5), reviewed before opening.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U7CBg11prhLimL9iHSX1bP
2026-07-28 16:44:39 -05:00
10ae129b94 Merge pull request 'docs(website): record the spawn atlas pipeline and what real data changed' (#67) from docs/spawn-atlas into edge
Reviewed-on: #67
Reviewed-by: Colby Whitlock <whitlocktech@gmail.com>
2026-07-28 21:16:06 +00:00
3fb3f63f25 docs(website): record the spawn atlas pipeline and what real data changed
Protocol 3.0 order 3 (Part C), docs half of website #112. Part C is website-only
— no plugin, no sidecar, no new kinds, no wire change.

## New: website/SPAWN_ATLAS.md

The operator-facing reference: the build/import split and why it exists (build
needs a ServUO tree, import does not, and the container has the artifact but not
the tree), the re-run story, the artifact format, the placement transform, and
the three quirks in the source data that are silent when unhandled.

Also documents the artwork policy explicitly: **the project ships no creature
art and no extraction tooling.** Sprites live in the operator's own client
.mul/.uop files and are theirs, not ours to redistribute. `art` is nullable and
NULL on every fresh import; an operator who wants art extracts it themselves into
the gitignored uploads/atlas/ and maps slugs in a gitignored art map. Text-only
is the normal, supported state — not a degraded one.

## New: v3.md §6.1 — what the build against real data changed

Six corrections, kept as a diff rather than edited into §6 in place, because
each is a trap the next person would otherwise re-enter:

1. **Six facets, not thirteen.** Eodon.xml and the other named-area files carry
   TerMur/Trammel points; the facet comes from each record's `<Map>`.
2. **The XML dependency call resolved: hand-rolled, zero deps.** §6 left
   fast-xml-parser vs a tokenizer open.
3. **Facet names disagree between sources** — Locations says `Ter Mur`, `<Map>`
   says `TerMur`. Unreconciled the landmark fallback never fires there and every
   unregioned Ter Mur/Tokuno spawn silently reads "Wilderness".
4. **Spawn type tokens carry XmlSpawner directives** (`Fairy,{RND,4,8}`,
   `alchemist/z/-50`). Taken literally they invent creatures that do not exist
   and split real ones in two. 71 of 845 affected; 800 remain after stripping.
5. **The artifact is 1.41 MB, not "well under 1 MB"** — down from 4.40 MB via
   three encodings. Getting under 1 MB would mean dropping the spawner name.
6. **DELETE, not TRUNCATE** — TRUNCATE is DDL in MariaDB and implicitly commits,
   which would defeat the all-or-nothing reload the design asked for.

§6 also now records that Part C ships as two website PRs: the parsing half is
where the correctness risk lives and should not be reviewed inside a 10k-line
diff alongside routes and React.

## BACKEND_DESIGN.md

The seven atlas tables, the import-owned contract, the four column choices that
are traps (`spawn_range`/`grp` reserved words, DELETE vs TRUNCATE, explicit point
ids, plain INDEX not FULLTEXT), and the distinction between the configured
champion roster and the live champ.update feed.

PROJECT_TREE.md is left alone — it is auto-generated by the sync-project-tree
workflow.

---

- [x] AI-assisted: written with **Claude Code** (Claude Opus 5), reviewed before opening.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U7CBg11prhLimL9iHSX1bP
2026-07-28 16:13:30 -05:00
e9ecdc0ecb Merge pull request 'docs(link): record world.ruleset and mark Protocol 3.0 progress' (#66) from docs/world-ruleset into edge
Reviewed-on: #66
Reviewed-by: Colby Whitlock <whitlocktech@gmail.com>
2026-07-28 20:34:38 +00:00
09467c67b0 docs(link): link the world.ruleset PRs from the v3 progress table
Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-28 14:41:54 -05:00
b0a2207c6a docs(link): record world.ruleset and mark Protocol 3.0 progress
Protocol 3.0 order 2 (v3.md §5) is built across all four repos; this is its
documentation half, plus the running progress record the plan was missing.

v3.md
  - A progress table at the top and a State column on §9's sequencing table, so
    "what has landed" is answerable without reading four git logs. Part A (order
    1) and world.ruleset (order 2) are marked done; the spawn atlas is next.
  - §5 gains the implementation notes worth keeping, chiefly: where a system's
    on/off state is DERIVED rather than configured, read the system's own static
    instead of inventing a .cfg key (Shadowguard has no Enabled key — it's the
    TOL expansion gate; Factions is `!ViceVsVirtueSystem.Enabled` by
    construction in stock ServUO). Also that the plugin CAN be compile-verified
    despite the "no standalone build" caveat, and how.

INTEGRATION.md
  - The world.ruleset catalog entry and GET /ruleset, with the two things
    consumers get wrong: caps are in TENTHS (1000 = 100.0), and `connect` exists
    only if the operator set Bridge.PublicConnectAddress — the shard's real
    listen address is never published.
  - §2 now says plainly that v3 has NOT been bumped yet and what that means:
    sidecars on `edge` report 2 while already carrying some v3 kinds, so do not
    infer feature availability from the version during this window.

PROTOCOL_2.md §10.4
  - The deferred "which PvP system does this shard run?" is answered (VvV on,
    Factions off — and mutually exclusive by construction), and world.systems is
    marked superseded by world.ruleset, which carries the systems block it asked
    for. No orphan kind is left behind.

BACKEND_DESIGN.md — the shard_ruleset table (why it is stored whole rather than
normalized, and why no row means null rather than {}) and the public route.

PROJECT_TREE.md is deliberately untouched: sync-project-tree regenerates it on
push to main, so it updates itself at the v3 cutover.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-28 14:39:53 -05:00
bd9718a859 Merge pull request 'docs(shard): record the REST projection gap the Part A smoke test found' (#65) from docs/shard-visibility-rest-projection into edge
Reviewed-on: #65
Reviewed-by: Colby Whitlock <whitlocktech@gmail.com>
2026-07-28 15:56:20 +00:00
4c0ceb1c41 chore(docs): drop an unrelated working-tree file committed by mistake
android/TRUSTED_DEVICES_APP_HANDOFF.md was untracked in the working tree
before this branch and was swept in by a `git add -A`. It is not part of
this change; untracked here and left on disk.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-28 10:53:01 -05:00
35ad440bad docs(shard): record the REST projection gap the Part A smoke test found
The live five-rung smoke test of the visibility framework found that Part
A enforced it on the SSE path and on /guilds + /governors, but not on the
remaining public REST reads - so one event was projected live and served
verbatim from stored history.

link/v3.md gains 3.6.1 with the full list (the anonymous acct/webId leak
on /feed, the flattened ownerAcct on /idoc, the dead `houses` field
rules, /feed ignoring live config, the empty-allowlist fall-through, and
the Date-to-{} projection bug), plus the rule it leaves behind: a read
path that returns shard data and does not project is a bug, and every new
Part B/C surface must gate its kind set on live config rather than on
PUBLIC_KINDS.

3.5 also corrected: the table is NOT seeded on boot. An absent row means
"use the compiled default", which keeps the defaults in one place instead
of duplicating them into a seeder that could drift.

BACKEND_DESIGN.md 6.5 records the same as a security contract: rule 1
locks a field by meaning rather than spelling; PUBLIC_KINDS is a
module-load constant and must not answer per-caller questions;
projectFeature walks arrays and plain objects only.

SHARD_VISIBILITY.md gets the admin-facing version - that stored history
answers the same way the live stream does, and that turning live updates
off stops the push, not the reading.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-28 10:52:13 -05:00
7 changed files with 687 additions and 39 deletions

View File

@@ -43,6 +43,20 @@ Pin the version you built against and compare it to the header (or `/health.prot
**v2 (Protocol 2.0)** added the account-provisioning surface (§6.x: `POST /accounts/create`, `DELETE /link/{account}`) and the `account.*` events. Outbound event kinds are **additive** — a v1 client that ignores unknown kinds keeps working against the live feed — but the new *endpoints* require a v2 sidecar. If you send `X-UOLink-Version: 1`, calls to the new endpoints are refused with the 409 above.
**v3 (Protocol 3.0) is being built and the version has not been bumped yet.** It is defined as *adds
`world.ruleset`, `points.board`, `vendor.listing` / `vendor.listing.remove`*, and the bump to
`X-UOLink-Version: 3` happens **exactly once**, at the end, when [`v3.md`](v3.md) §4's `edge` → `main`
cutover lands — because a bump is an operator-visible hard break (409 on every protected route, and
the website's WS closes on the `ws.hello` mismatch), so doing it per phase would break the site
repeatedly.
Until then, sidecars on `edge` still report `2` while already carrying some v3 kinds and endpoints.
That is safe in the direction that matters: event kinds are additive, and a client that ignores
unknown kinds and tolerates a `404` on a not-yet-present endpoint keeps working. What you must **not**
do is infer feature availability from the version number during this window — probe the endpoint, or
treat a missing `world.ruleset` as "this shard hasn't published one". There is deliberately **no
feature-negotiation array**: v3 implies all three kinds.
---
## 3. Health
@@ -315,6 +329,58 @@ The house registry — one row per house, complementing the `house.decay` *trans
Render from `GET /houses` (§6) on connect, then keep live with these events.
#### Shard ruleset (Protocol 3.0)
How the shard is actually configured, published by the shard itself. **Not a sweep** — it changes only
when an operator edits `Config/*.cfg`, so it is emitted once per shard↔sidecar connect (and on
`[bridge reload`), exactly like `server.hello`.
| kind | fields | notes |
|------|--------|-------|
| `world.ruleset` | `rev`, `shard`, `expansion`, `connect?`, `systems`, `caps`, `housing`, `accounts`, `vetRewards`, `loot`, `vendors`, `champions?`, `treasureMaps`, `vvv?`, `store`, `schedule?` | The whole ruleset, always complete — **never a delta**, so the latest frame replaces the previous one outright. Every block except `shard`/`expansion` is optional and is **omitted when its system is off**, so absence means "not applicable here", not "unknown". |
`rev` is the shard's FNV-1a of the body: identical `rev` means the ruleset is unchanged and this frame
is just a reconnect re-send, so a consumer can skip the write. It is deliberately **not**
`String.GetHashCode()`, which is seeded per process and would change on every shard restart.
```json
{"kind":"world.ruleset","rev":"1a2b3c4d","shard":"UOMysticmoon","expansion":"EJ",
"systems":{"cityLoyalty":true,"vvv":true,"factions":false,"siege":false,"chat":true,
"store":true,"dailyRares":true,"honesty":true,"shadowguard":true,
"treasureMaps":true,"vetRewards":true,"testCenter":false},
"caps":{"skill":1000,"totalSkill":7000,"stat":225,"str":125,"dex":125,"int":125,
"strMax":150,"dexMax":150,"intMax":150},
"housing":{"accountHouseLimit":1},
"accounts":{"perIp":3,"charSlots":7,"autoCreate":true},
"vetRewards":{"enabled":true,"rewardIntervalDays":30},
"loot":{"feluccaLuckBonus":1000,"feluccaBudgetBonus":100,"feluccaMaxProps":11},
"vendors":{"restockDelayMinutes":60,"maxSell":500,"economyStockAmount":500},
"champions":{"powerScrolls":6,"statScrolls":16,"scrollChance":0.1,
"transcendenceChance":50.0,"rankThresholds":[5,10,13]},
"treasureMaps":{"enabled":true,"lootChance":0.01,"resetDays":30},
"vvv":{"enabled":true,"startSilver":2000,"enhancedRules":false},
"store":{"enabled":true,"currencyName":"Sovereigns"},
"schedule":{"autoSaveEnabled":true,"autoSaveFrequencyMinutes":15,"autoRestartEnabled":false},
"t":1752489280000}
```
**Two things consumers get wrong.**
1. **`caps.skill` and `caps.totalSkill` are in tenths**, the way ServUO stores them: `1000` is `100.0`
skill and `7000` is `700.0` total. Rendering the raw number is actively misleading. The other caps
(`stat`, `str`, …) are plain integers.
2. **`connect` is present only if the operator set `Bridge.PublicConnectAddress`.** The shard's real
listen address (`Server.cfg`) is never published; nor are `Staff.cfg`, `Email.cfg`, `DataPath.cfg`,
`Bridge.cfg`, `Compiler.cfg`, `Reports.cfg` or `Client.cfg`. The frame is built from an explicit
allowlist in `BridgeRuleset.cs` — `Config.Entries` is never enumerated, because that would sweep in
every key on the server.
Absent entirely if the shard runs `Bridge.RulesetEnabled=false` or an older plugin. Render from
`GET /ruleset` (§6) on connect, then keep live with this event.
This **supersedes the `world.systems` frame** sketched in [`PROTOCOL_2.md`](PROTOCOL_2.md) §10.4 and
never implemented; the `systems` block above is what that asked for.
---
## 5. REST — read queries
@@ -658,6 +724,22 @@ GET /houses
Every house's latest snapshot — owner→houses map. Served from the sidecar's projection, kept current by the `house.*` stream (§4). Ordered by name. Survives a sidecar restart.
### Shard ruleset (Protocol 3.0)
```
GET /ruleset
→ { "ruleset": {"kind":"world.ruleset","rev":"1a2b3c4d","shard":"UOMysticmoon",
"expansion":"EJ","systems":{...},"caps":{...},"accounts":{...}, ... } }
```
The shard's published ruleset (§4 for the full frame and its two gotchas). Served from the sidecar's
store, so it **answers while the shard is down** — a rules page that goes blank during a restart is
worse than one that is briefly stale. Keep it current with the `world.ruleset` stream.
`{"ruleset": null}` means the shard has never published one — an older plugin, or
`Bridge.RulesetEnabled=false`. That is a real answer distinct from a published ruleset, and worth
rendering differently ("not published yet") rather than as an empty ruleset.
---
## 7. Status codes

View File

@@ -315,6 +315,14 @@ Counts in `hello` are a live snapshot taken on the Core thread, not a cached val
7. **Core edit: `PlayerVendorSale`** (§6). Then the cheat-detection feed.
8. **Cheat signals.** `FastWalk`, `OnPropertyChanged` audit, vendor-sale anomaly detection in the sidecar.
**Beyond 1.0.** Phases above are the 1.0 read/event plane. Protocol 2.0's phasing (provisioning +
world-state boards) is [`PROTOCOL_2.md`](PROTOCOL_2.md) §13; Protocol 3.0's (visibility framework,
shard content and standings) is [`v3.md`](v3.md) §9, which also tracks what has landed. Shipped from
3.0 so far: **Part A** — the visibility framework — and **`world.ruleset`** ([`v3.md`](v3.md) §5),
`BridgeRuleset.cs`, the first bridge stream that is neither an event subscription nor a sweep: it is
emitted once per connect, like `server.hello`, because shard config changes only when an operator
edits a file.
### Config keys (`Config/Bridge.cfg`)
```ini
@@ -328,6 +336,11 @@ EconomySweepSeconds=300
Read in `Configure()` via `Config.Get<T>("Bridge.<Key>", default)`. Key scope is the filename: `Bridge.cfg` + `StatSweepSeconds``Bridge.StatSweepSeconds`.
The set above is the 1.0 sample, not the current one — every later phase added keys (sweep intervals
for each board, the town-crier/news caps, the admin write plane, account provisioning, and 3.0's
`RulesetEnabled` / `PublicConnectAddress` / `RulesetIncludeSchedule`). **`servuo-plugins/overlay/Config/Bridge.cfg`
is the authoritative, commented list**; `BridgeConfig.cs` holds the defaults.
---
## 11. Phase 1 acceptance

View File

@@ -285,6 +285,18 @@ City titles and faction/VvV merchant titles (`CityLoyaltySystem.ApplyCityTitle`,
### 10.4 Factions / Vice vs Virtue
> **Status update (Protocol 3.0, 2026-07-28).**
>
> - **The deferred question is answered.** This shard runs **Vice vs Virtue** (`VvV.cfg Enabled=True`);
> old Factions is off, and in stock ServUO that is not a coincidence —
> `Services/Factions/Core/Faction.cs` sets `Settings.Enabled = !ViceVsVirtueSystem.Enabled`, so the
> two are mutually exclusive by construction. The `vvv.standings` / `vvv.battle` streams below are
> therefore unblocked, but are **not** scoped for 3.0 (see [`v3.md`](v3.md) §2 row 5).
> - **`world.systems` is superseded by `world.ruleset`** ([`v3.md`](v3.md) §5), which shipped in 3.0.
> It was never implemented under this name. `world.ruleset` carries the same
> `systems{cityLoyalty, vvv, factions, …}` sub-object this section asked for, plus the rest of the
> shard's published ruleset, so no orphan kind is left behind. Do not implement `world.systems`.
**Which system is live is a shard decision — verify before building.** Two exist:
- **Old Factions** (`Scripts/Services/Factions`): `Faction.Commander` (leader, `Faction.cs:160`), `Faction.Election`, `Faction.Members` (`List<PlayerState>`), and faction-controlled **Towns** (`Town.cs` — each town has an owning faction, a sheriff, and finance). Config-gated and, on most modern shards, **off**.
@@ -300,7 +312,10 @@ City titles and faction/VvV merchant titles (`CityLoyaltySystem.ApplyCityTitle`,
{"kind":"vvv.standings","order":142000,"chaos":138500,"leaderSide":"Order"}
```
> Start by detecting which system is enabled at boot and streaming only that one; emit a one-time `world.systems` frame (what's on: cityLoyalty, vvv, factions) so the website renders the right panels instead of guessing.
> Start by detecting which system is enabled at boot and streaming only that one. ~~emit a one-time
> `world.systems` frame (what's on: cityLoyalty, vvv, factions) so the website renders the right panels
> instead of guessing.~~ — **superseded: `world.ruleset` already carries that `systems` block** (see the
> status note at the top of this section).
## 11. Further integration points — a menu to pick from

View File

@@ -1,10 +1,23 @@
# Protocol 3.0 — Shard content, standings & the visibility framework
**Status:** Planned, approved 2026-07-28. Not yet built. All work lands on an `edge` branch in each repo; `edge``main` is the v3 cutover.
**Status:** In progress. All work lands on an `edge` branch in each repo; `edge``main` is the v3 cutover.
**Date:** 2026-07-28
**Codebase:** ServUO 57.4, `<servuo>`, net48 / x64, Expansion **EJ**.
**Companion to** [`PLAN.md`](PLAN.md) (1.0 read/event plane), [`PROTOCOL_2.md`](PROTOCOL_2.md) (2.0 provisioning + world-state streams), [`ADMIN_CONTROLS.md`](ADMIN_CONTROLS.md) (staff write plane), [`INTEGRATION.md`](INTEGRATION.md) (website API).
### Progress
Each part is marked off here as it lands on `edge`. §9 carries the same state per sequencing row.
| Order | Part | State | Landed on `edge` |
|---|---|---|---|
| 1 | **A** — visibility framework + actor-leak fix (§3) | ✅ **Done** | website [#109](https://gitea.whitlocktech.com/RunicGateway/website/pulls/109) + [#110](https://gitea.whitlocktech.com/RunicGateway/website/pulls/110), docs [#64](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/64) + [#65](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/65) |
| 2 | **B/1**`world.ruleset` (§5) | ✅ **Done** | servuo-plugins [#3](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/3), link [#17](https://gitea.whitlocktech.com/RunicGateway/link/pulls/17), website [#111](https://gitea.whitlocktech.com/RunicGateway/website/pulls/111), docs [#66](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/66) |
| 3 | **C** — spawn atlas (§6) | 🟡 **Data pipeline done** | website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112) (parsers + CLI + tables); API/client PR next |
| 4 | **B/2**`points.board` (§7) | ⬜ Not started | — |
| 5 | **B/3**`vendor.listing` (§8) | ⬜ Not started | — |
| 6 | **Cutover**`PROTOCOL_VERSION` 2→3 (§4) | ⬜ Not started | — |
---
## 1. Why 3.0
@@ -64,7 +77,12 @@ independently of this work.
---
## 3. Part A — The visibility framework
## 3. Part A — The visibility framework ✅ Done
*Landed on `edge`: website [#109](https://gitea.whitlocktech.com/RunicGateway/website/pulls/109) (the framework)
and [#110](https://gitea.whitlocktech.com/RunicGateway/website/pulls/110) (the REST-projection gap §3.6.1
records), docs [#64](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/64) + [#65](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/65).
Smoke-tested across all five rungs per §11.*
### 3.1 The leak this replaces (verified 2026-07-28)
@@ -115,9 +133,13 @@ CREATE TABLE IF NOT EXISTS shard_feature_visibility (
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
```
Seeded on boot in `server.js`, one row per feature. **All ten shard features are covered — the four
new ones and the six that already ship — and every default reproduces today's behavior, so the
retrofit is a no-op until an admin changes something.**
**Not** seeded on boot (this changed during implementation): an **absent row means "use the compiled
default"**, so the table starts empty and only ever holds rows an admin has actually touched. The
defaults live in one place — `FEATURES` in `shardVisibility.js` — instead of being duplicated into a
seeder that could drift from it, and a DB blip degrades to those same defaults rather than to
"everything is public". **All ten shard features are covered — the four new ones and the six that
already ship — and every default reproduces today's behavior, so the retrofit is a no-op until an
admin changes something.**
| Feature | Default audience | Sensitive fields (default rung) |
|---|---|---|
@@ -156,10 +178,42 @@ Applied at:
3. **Nav**`GET /api/v1/public/shard/features` returns only the features the calling viewer can
see, so the SPA hides nav entries rather than rendering links that 403.
### 3.6.1 What the first implementation missed (found by the §11 smoke test, fixed)
Part A shipped enforcement on the SSE path and on `/guilds` + `/governors`, but the **remaining public
REST reads never called into it** — so the same event was projected live and served verbatim from
history. Recorded because each miss is a shape the next phase can repeat:
- **`/public/shard/feed` returned the stored payload as-is.** `actor.acct` / `actor.webId` were
readable *anonymously* for every logged kind (`player.death`, `mob.killed`, `skill.gain`,
`guild.join`, …) — broader than the §3.1 leak, which was limited to board holders.
- **`/public/shard/idoc` returned `ownerAcct`.** Rule 1 keyed on the exact strings `acct`/`webId`,
but `shapeHouse` flattens the actor into `ownerAcct` / `ownerName` / `ownerSerial`. The lock is now
on the field's **meaning** — a key that is or ends in `acct`/`webId`, case-insensitively — so
flattened spellings are covered and unwritten shapes fail closed.
- **The `houses` field rules were dead config.** Neither `getIdoc` nor `getHouses` projected, so the
panel offered toggles that did nothing. **Every feature's declared fields must name the keys the
read model actually emits**, not just the wire frame's.
- **`/feed` filtered on `PUBLIC_KINDS`**, a module-load constant derived from the compiled defaults,
so live audience changes never reached it. `visibleKinds(level, config)` resolves the readable set
from live config; it deliberately ignores the `stream` flag, which governs SSE fan-out only (market
history stays readable with its firehose off).
- **`shardEvents.db.list` treated an empty `kinds` array as "no filter"** and fell through to an
unfiltered `SELECT`. A fully-gated config would have dumped the whole event log, staff audit
included. An empty allowlist now serves nothing.
- **`projectValue` recursed into every object**, so a `Date` column came back as `{}`. It walks
arrays and plain objects only. The unit tests used JSON fixtures and could not have caught this —
the live read did, which is the argument for §11's smoke test over tests alone.
**The rule this leaves behind:** *a read path that returns shard data and does not call
`projectFeature` is a bug.* Every new surface in Parts B and C — `/ruleset`, `/points`, `/market`,
`/atlas` — must project, and must gate its kind set on live config rather than on `PUBLIC_KINDS`.
### 3.7 Admin surface
`GET` / `PUT /api/v1/admin/shard/visibility` (admin-only). Validate feature names against the known
set and rungs against the ladder; reject any attempt to set `acct`/`webId` below `admin`. Writes an
set and rungs against the ladder; reject any attempt to set a locked field below `admin` — including
its flattened spellings (`ownerAcct`, `leaderWebId`), see §3.6.1. Writes an
`admin.audit`-style row so visibility changes are traceable. New client panel
`routes/admin/ShardVisibility.jsx` at `/admin/shard-visibility`, linked from `ShardAdmin.jsx`.
@@ -185,7 +239,28 @@ admin-set `uo_link_config.protocol` column — so it happens **exactly once**, a
---
## 5. Part B/1 — `world.ruleset`
## 5. Part B/1 — `world.ruleset` ✅ Done
*Landed on `edge`: servuo-plugins [#3](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/3), link [#17](https://gitea.whitlocktech.com/RunicGateway/link/pulls/17), website [#111](https://gitea.whitlocktech.com/RunicGateway/website/pulls/111), docs [#66](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/66). Implementation notes worth keeping:*
- ***`shadowguard` is derived, not configured.*** `Shadowguard.cfg` carries only `ReadyDuration` and
`RandomizeInstances` — there is no `Enabled` key — so the systems block reports `Core.TOL`
(the expansion gate) instead. Same shape for `factions`: `Factions.cfg` has no `Enabled` either, and
`Services/Factions/Core/Faction.cs` sets `Settings.Enabled = !ViceVsVirtueSystem.Enabled`, so the
frame reads that static rather than inventing a key. **Where a system's on/off state is derived, read
the system's own static; only read `Config.Get` where the .cfg key IS the truth.**
- **`caps.skill` / `caps.totalSkill` are in tenths** (1000 = 100.0), the way ServUO stores them.
Documented in `INTEGRATION.md` and converted in the client, because the raw number is actively
misleading rather than merely unhelpful.
- **`Config.Get` re-parses when the cached type differs.** `InternalGet<T>` caches the parsed value on
the entry and re-parses if `entry.Object is T` fails, so reading `PlayerCaps.SkillCap` as an `int`
where ServUO reads it as a `double` is correct (both parse) — it just re-parses. Harmless, but worth
knowing before assuming a shared cache.
- **The plugin CAN be compile-verified**, contrary to "no standalone build": point Roslyn
(`dotnet sdk/*/Roslyn/bincore/csc.dll`, `/langversion:7.3`, net48 reference assemblies) at the whole
ServUO `Scripts` tree with `overlay/Scripts/Custom/Bridge/*.cs` substituted for the deployed copy,
excluding `Scripts/obj` and `Scripts/bin`. 6,205 files, ~40 s, and it catches every signature error
a boot would. Worth doing before every plugin PR.
`PROTOCOL_2.md` §10.4 sketches a `world.systems` capability frame that was never implemented
(`grep` returns nothing across all four repos). **`world.ruleset` subsumes it**, carrying a `systems`
@@ -254,19 +329,31 @@ frame during verification.
**No plugin, no sidecar, no `Bridge.cfg` knob, no new kinds.** Not part of the v3 wire change.
**Decision: committed generated artifact + idempotent DB import**, split in two because the build
needs the ServUO tree (which the website container does not have) and the import does not. Not
runtime import (10.5 MB of XML per boot), not a browser-served blob.
> **Status:** data pipeline landed on `edge` — website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112)
> (parsers, build/import CLI, tables, artifact). API + client pages are the second website PR.
> Part C ships as **two** website PRs, not one: the parsing half is where the correctness risk
> lives, and burying it under routes and React would have meant reviewing it in a 10k-line diff.
> Full operator documentation: [`docs/website/SPAWN_ATLAS.md`](../website/SPAWN_ATLAS.md).
>
> **§6 below is the original design and is partly superseded.** §6.1 records two decisions that were
> rejected in review and replaced (the committed artifact, and the fixed facet list); §6.2 records
> the corrections the real ServUO data forced. Read both before trusting §6.
**Decision (revised at implementation time): the shard's ServUO tree is the single source of truth,
re-derived on every server boot.** The original plan here was a committed generated artifact plus an
idempotent import. That was rejected in review for two reasons, recorded in §6.1: a snapshot in the
repo goes stale as a shard's maps change, and the design leaned on a fixed facet list that no shard
is obliged to keep. Still not a browser-served blob; still parsed server-side only.
New in `website/server/`:
- `src/utils/spawnAtlasParse.js`**pure functions, no fs**, so they are unit-testable in CI without
a ServUO tree: `parseObjects2()`, `parsePoints()`, `parseRegions()`, `parseLocations()`,
`resolveRegion()`.
- `scripts/buildSpawnAtlas.js` (`--servuo <path> --out db/data/`) and `scripts/importSpawnAtlas.js`
(TRUNCATE + batched INSERT in one transaction); `package.json` scripts `atlas:build`, `atlas:import`.
- `db/data/spawnAtlas.<facet>.json` ×13 + `spawnAtlas.index.json` (creatures, champions, regions,
landmarks, meta with per-source-file hashes).
- ~~`scripts/buildSpawnAtlas.js` and a committed `db/data/spawnAtlas.*.json` artifact~~ — dropped,
see §6.1 R1. Replaced by `src/utils/spawnAtlasSource.js` (the only thing that reads a ServUO tree,
shared by the boot path and the CLI) and a `scripts/importSpawnAtlas.js` that is a thin CLI over
the model. `package.json` gains `atlas:import` only.
- `src/model/shardAtlas/{shardAtlas.db.js,shardAtlas.model.js}` following the `shardState` split.
- `src/router/v1/public/atlas.{router,controller}.js`; `test/spawnAtlas.parse.test.js`.
@@ -297,16 +384,92 @@ stays CLI-only.**
Client: `routes/public/Atlas.jsx` (`/site/atlas`) and `AtlasCreature.jsx` (`/site/atlas/:slug`).
**Payload risk**a monolithic artifact would be 23 MB of committed JSON. Shard per facet and drop
every `<Points>` field the site cannot use (`UniqueId`, all trigger/refractory/proximity/sequential
fields, sound ids), keeping Name/Map/X/Y/W/H/Range/MaxCount/MinDelay/MaxDelay/TOD*/types — well under
1 MB. The artifact never reaches the browser; the browser sees only paginated API responses.
**Payload risk***superseded by §6.1 R1; nothing is committed.* The field selection it describes
still applies at parse time: every `<Points>` field the site cannot use (`UniqueId`, all
trigger/refractory/proximity/sequential fields, sound ids) is dropped, keeping
Name/Map/X/Y/W/H/Range/MaxCount/MinDelay/MaxDelay/TOD*/types. Parsed data never reaches the browser;
the browser sees only paginated API responses.
**Operator re-run story**spawns changed → `npm run atlas:build -- --servuo <path>` on a machine
with the tree → commit the regenerated `db/data/spawnAtlas.*.json` → deploy `npm run atlas:import`
(or `POST /admin/shard/atlas/import`). `shard_atlas_meta.source` holds per-file hashes, so
`GET /admin/shard/atlas/status` reports when the DB is behind the artifact. Full detail in
`docs/website/SPAWN_ATLAS.md`.
**Operator re-run story***revised by §6.1 R1.* Spawns changed → restart, or
`npm run atlas:import` / `POST /admin/shard/atlas/import` to apply without one. `shard_atlas_meta`
holds a sha256 per source file, so the server can tell on boot whether anything changed, and
`GET /admin/shard/atlas/status` reports drift. If the change would remove a facet it is staged for
approval rather than applied (§6.1 R3). Full detail in `docs/website/SPAWN_ATLAS.md`.
### 6.1 What implementation changed
Two design decisions in §6 were rejected in review and replaced; the rest are corrections the real
ServUO data forced. Kept as a diff rather than edited in place, because each is a trap the next
person would otherwise re-enter.
**R1. The committed artifact is gone — the tree is re-parsed on every boot.** §6 proposed building a
generated artifact, committing it, and importing it. Two problems. A shard's maps change over its
life, so a snapshot in the repo silently drifts from the world players actually see; and the build/
import split existed only to work around the website container not having a tree, which is a
deployment question (mount it) rather than a reason to freeze data. The server now hashes the source
files on boot and re-derives the atlas when they differ. `scripts/buildSpawnAtlas.js`, the 1.41 MB
artifact, and the whole encode/decode seam it needed are deleted.
**R2. Nothing may name a facet.** The first implementation carried a lookup table of the six stock
UO facets to reconcile the spelling drift between sources. A shard may add facets, replace them
outright, or rename them when its maps are updated, and a built-in list mishandles all three
silently. Reconciliation is now by *matching* against the facet set discovered from the shard's own
spawn and region data — exact key, then prefix in either direction — with an unmatched name keeping
its own rather than being forced into a wrong bucket.
**R3. Two contracts on the boot path.** It never blocks startup: no path, an unreadable mount, a
malformed file or a database error is caught and logged, and the site comes up serving whatever
atlas it had. And a refresh that would REMOVE a facet is never applied automatically — facet loss
is indistinguishable at boot from a half-copied or mid-update tree, so it is staged in
`shard_atlas_pending` for an admin to approve or reject. Only the decision is stored (source hashes
+ the facet diff, a few KB); approving re-parses, so what lands matches the tree at approval time.
A rejection is remembered against those hashes so it does not re-prompt every restart.
### 6.2 What the build against real data changed
Six corrections to the design above, from running it against stock ServUO 57.4. Kept as a diff
rather than edited in place, because each one is a trap the next person would otherwise re-enter.
**1. Six facets, not thirteen.** The design said `spawnAtlas.<facet>.json ×13`, assuming one facet
per spawn file. There are 13 files but only **6** facets — `Eodon.xml`, `GravewaterLake.xml`,
`TreasuresOfKotl.xml` and the other named-area files carry TerMur/Trammel points. The facet comes
from each record's own `<Map>`, never the file name, and the artifact shards 6 ways.
**2. The XML dependency call: hand-rolled, zero deps.** §6 left `fast-xml-parser` vs a ~120-line
tokenizer open. Resolved as the tokenizer — a deliberate *subset* parser covering only what these
files use. The server keeps zero XML dependencies at any tier.
**3. Facet names disagree between sources — a silent failure.** `Data/Locations/*.xml` spells them
`Ter Mur` and `Tokuno Islands`; `<Map>` and `<Facet name>` say `TerMur` and `Tokuno`. Unreconciled,
the landmark bucket is keyed differently from the points looking it up, so the fallback never fires
and **every unregioned spawn in Ter Mur and Tokuno reads "Wilderness"** — a plausible-looking atlas
that is quietly wrong for two facets. All facet names now pass through `normalizeFacet()`.
**4. Spawn type tokens carry XmlSpawner directives.** `<Objects2>` types are not always bare class
names: `Fairy,{RND,4,8}`, `alchemist/z/-50`, `Agralem/Name/Agralem`, `greatape,true`. Taken literally
they invent creatures that do not exist *and* split real ones in two, since `Fairy` and
`Fairy,{RND,4,8}` slug apart. 71 of 845 entries were affected; stripping at the first `/` or `,`
leaves **800** real creatures. (The design's "~1,500 creature rows" estimate was high; 800 only
reinforces the plain-`INDEX`-not-`FULLTEXT` call.)
**5. The artifact would have been 1.41 MB, not "well under 1 MB" — and is now moot.** Dropping the
unused `<Points>` fields as the design directed still left 4.40 MB; three further encodings brought
it to 1.41 MB, and getting under 1 MB would have meant dropping the spawner `name`. The size budget
in §6 was simply optimistic for 6,455 points. Superseded by §6.1 R1: there is no artifact, so there
is no payload to budget and no encode/decode seam to keep in sync.
**6. `DELETE`, not `TRUNCATE`.** The design said "TRUNCATE + batched INSERT in one transaction",
which does not hold: `TRUNCATE` is DDL in MariaDB and implicitly commits, so a mid-import failure
would leave the atlas half-loaded. `DELETE` is transactional, and at ~7k rows the cost is
irrelevant. Point ids are also assigned explicitly rather than by `AUTO_INCREMENT`, because the
join rows need them and `conn.batch()` reports no usable `insertId`.
**Measured result:** 6,455 points, 800 creatures, 23,927 point/type rows, 387 regions, 558
landmarks, 25 champion altars. The placement transform resolves **83.2%** of points (3,689 by
region, 1,690 by landmark, 1,086 Wilderness).
**One thing the design got exactly right:** the point-in-rect transform really is the reason to
build this. "Where does a lizardman spawn?" answers *Shrines, Isamu-Jima, Yew* across three facets.
---
@@ -498,14 +661,14 @@ inherently up to one full cycle old, and the UI must say so.
## 9. Sequencing
| Order | Part | Repos touched | Wire change |
|---|---|---|---|
| 1 | **A** — visibility framework + actor-leak fix | website, docs | none |
| 2 | **B/1**`world.ruleset` (§5) | all four | new kind |
| 3 | **C** — spawn atlas (§6) | website, docs | none |
| 4 | **B/2**`points.board` (§7) | all four | new kind + `char.profile` field |
| 5 | **B/3**`vendor.listing` (§8) | all four | new kinds |
| 6 | **Cutover**`PROTOCOL_VERSION` 2→3, `edge``main` | all four | the bump |
| Order | Part | Repos touched | Wire change | State |
|---|---|---|---|---|
| 1 | **A** — visibility framework + actor-leak fix | website, docs | none | ✅ Done |
| 2 | **B/1**`world.ruleset` (§5) | all four | new kind | ✅ Done |
| 3 | **C** — spawn atlas (§6) | website, docs | none | 🟡 Pipeline done, API/client next |
| 4 | **B/2**`points.board` (§7) | all four | new kind + `char.profile` field | ⬜ |
| 5 | **B/3**`vendor.listing` (§8) | all four | new kinds | ⬜ |
| 6 | **Cutover**`PROTOCOL_VERSION` 2→3, `edge``main` | all four | the bump | ⬜ |
---

View File

@@ -99,7 +99,7 @@ server/
pages.router.js (2) /public/pages — the draft-preview
route precedes /:slug and is
deliberately not site-mode gated
shard.router.js (13) /public/shard/* incl. the anonymous
shard.router.js (14) /public/shard/* incl. the anonymous
SSE stream; never site-mode gated
site.router.js (4) /settings /status /version /contact —
the group-root singletons; declares no
@@ -358,6 +358,23 @@ analogue to a password — and there is no hash-lookup constraint (verification
unused rows and `bcrypt.compare`s each, like password verification). `used_at` is the single-use
marker. Cleared wholesale on TOTP disable / password change / password reset.
### shard_ruleset — the shard's published ruleset (Protocol 3.0)
Singleton row (`id = 1`, CHECK-constrained) holding the latest `world.ruleset` frame: `rev`,
`expansion`, `payload` JSON (the whole frame), `t`, `updated_at`. The shard re-emits the complete
ruleset on every sidecar connect, so this is an **overwrite, not an append** — and the kind is
deliberately **not** in `LOGGED_KINDS`, since logging it would put a duplicate row in `shard_events`
on every reconnect while `server.hello` already marks each of those.
The frame is stored whole rather than normalized into columns: it is a flat description of server
config that is read as one page, so splitting it up would mean a schema change every time the shard
grows a new block. `rev` (the shard's FNV-1a of the body) and `expansion` are hoisted only because
they are cheap to display — the same payload-plus-hoisted-columns shape `shard_champs` uses.
**No row means the shard has never published one** (an older plugin, or `Bridge.RulesetEnabled=false`),
served as `null` rather than `{}`: "not published yet" and "published, everything off" are different
answers and the page renders them differently.
### shard_feature_visibility — per-feature audience config (Protocol 3.0)
One row per shard feature: `feature` (PK), `enabled`, `audience` (a rung on the ladder in §6.5),
@@ -371,6 +388,67 @@ feature is ignored (a stale row must not resurrect a removed feature), an invali
the default rather than failing open, and a rule touching a locked field (`acct` / `webId`) is
discarded. See §6.5.
### shard_spawn_* / shard_regions / shard_landmarks / shard_champion_spawns / shard_atlas_meta — the spawn atlas (Protocol 3.0)
Static shard **content**, not live shard state. Nothing here comes from the sidecar: the atlas is
derived from the shard's own ServUO tree, re-read on **every server boot** and hash-gated so an
unchanged tree costs one read pass and no write. Nothing is precomputed and committed — a shard's
maps change over its life, and a snapshot in the repo would silently drift from the world players
actually see. These tables stay populated whether the shard is up or not. Full operator detail in
[`SPAWN_ATLAS.md`](SPAWN_ATLAS.md); the design is `docs/link/v3.md` §6.
**No facet name appears anywhere in the code.** A shard may add facets, replace them, or rename them
when its maps are updated; the facet set is discovered from the tree, and the loose spellings in
`Data/Locations` are matched against it rather than looked up in a table.
| Table | Key columns |
|---|---|
| `shard_spawn_creatures` | `slug` PK, `name`, `total`, `points`, `facets` JSON, `art` NULL |
| `shard_spawn_points` | `id` PK, `facet`, `name`, `x`, `y`, `width`, `height`, `spawn_range`, `max_count`, `min_delay`, `max_delay`, `tod_start/end/mode`, `region`, `landmark`, `label` |
| `shard_spawn_point_types` | `(point_id, slug)` PK, `max_count` |
| `shard_regions` | `facet`, `name`, `type`, `priority`, `parent`, `rects` JSON |
| `shard_landmarks` | `facet`, `name`, `grp`, `x`, `y`, `z` |
| `shard_champion_spawns` | `slug` PK, `name`, `grp`, `type`, `random_type`, `facet`, `x`, `y`, `z`, `radius`, `label` |
| `shard_atlas_meta` | Singleton (`id = 1`), `payload` JSON (counts + a sha256 per source file), `imported_at` |
| `shard_atlas_pending` | Singleton (`id = 1`), `status` (`pending`/`rejected`), `payload` JSON, `detected_at` |
The first seven are **import-owned**: a refresh empties and reloads every one inside a single
transaction, so a failed reload leaves the previous atlas intact rather than a half-loaded world.
Nothing else writes to them, and nothing holds a foreign key to them — no FKs at all, consistent with
every other `shard_*` table.
**`shard_atlas_pending` is the security-relevant one.** A refresh that would REMOVE a facet is never
applied automatically: facet loss is indistinguishable at boot from a half-copied or mid-update tree,
so it is staged here for an admin to approve or reject, and **startup is never blocked by it**. Only
the decision is stored — source hashes plus the facet diff, a few KB — and approving re-parses the
tree, so a multi-megabyte blob never lands in the database and what gets applied matches the tree at
approval time. A rejection is remembered against those exact hashes so a declined refresh does not
re-prompt on every restart. Everything else (new facets, renamed regions, changed spawns) applies
immediately, since none of it can destroy data an operator would miss.
The boot refresh is **best-effort by contract**: no configured path, an unreadable mount, a malformed
file or a database error is caught and logged, and the site comes up serving whatever atlas it had.
The tree path comes from the `spawn_atlas_servuo_path` setting, falling back to `SERVUO_PATH`.
Four column choices worth stating, because each one is a trap:
- **`spawn_range`, not `range`**, and **`grp`, not `group`** — both are reserved words.
- **`DELETE`, not `TRUNCATE`.** `TRUNCATE` is DDL in MariaDB and implicitly commits, which would
defeat the all-or-nothing reload. At ~7k rows the difference does not matter.
- **Point ids are assigned explicitly**, not left to `AUTO_INCREMENT`: the `shard_spawn_point_types`
rows need to know them, and `conn.batch()` reports no usable `insertId` for a multi-row insert.
- **Plain `INDEX` on `name`, deliberately not `FULLTEXT`.** ~800 creature rows makes a `LIKE` scan
free, and FULLTEXT's minimum token length would break searches for names like "orc".
`shard_champion_spawns` is the *configured* altar roster ("there is an Unholy Terror altar in
Deceit"). The live `champ.update` feed in `shard_champs` is the separate answer to "it is on level 3
right now". Both exist; they are not the same data.
**`shard_spawn_creatures.art` is always NULL on a fresh import.** The project ships no creature
artwork: sprites live in the operator's own client `.mul`/`.uop` files and are theirs, not ours to
redistribute. An operator supplies art via a gitignored map plus images under the (already
gitignored) `server/uploads/atlas/`. Text-only is the normal, supported state.
---
## 4. API contract
@@ -569,6 +647,7 @@ from the per-route **siteMode** middleware (§5), never from an auth gate.
| GET | `/wiki` | list of pages (slug + title) |
| GET | `/wiki/:slug` | single page |
| POST | `/contact` | (rate-limited) send mail via SMTP; if unconfigured, respond `{fallback:"mailto", email}` |
| GET | `/shard/ruleset` | the shard's own published ruleset (Protocol 3.0 `world.ruleset`): expansion, which optional systems are on, skill/stat caps, account and house limits, champion scroll rules, the save/restart schedule. Served from `shard_ruleset`, so it renders while the shard is down; live via `world.ruleset` on `/shard/stream`. Behind `requireFeature('ruleset')`. **`null`** means the shard has never published one — a real answer, distinct from a published ruleset. `caps.skill` / `caps.totalSkill` are in **tenths** (1000 = 100.0). |
| GET | `/shard/features` | the shard features **this caller** may reach plus the audience rung they resolved to (§6.5), so a client hides nav it can't follow. Reports only what the caller can see — the list itself never discloses a gated feature. Consumed by the SPA header and (pending) the Android nav. |
Public content GETs pass through the **siteMode** gate (§5).
@@ -711,6 +790,10 @@ rather than silently ignore:
1. **`acct` and `webId` are admin-only, always.** They are not exposed as configurable fields, and a
stored row attempting to loosen them is discarded on read as well as rejected on write. A character
name is visible in game; the account behind it and the website user it links to are not.
The lock is on the field's **meaning, not one spelling**: `isLockedField(key)` matches a key that
*is* or *ends in* `acct`/`webId`, case-insensitively, so the flattened forms the read models emit
(`shapeHouse``ownerAcct`, `shapeGuild``leaderWebId`) are covered too. An exact-key check was
the original implementation and it let `GET /public/shard/idoc` serve `ownerAcct` anonymously.
2. **A kind absent from `KIND_FEATURE` is never broadcast below `admin`.** Fail closed. This is what
keeps the kind map a security boundary rather than a convenience filter, and it means a shard that
starts emitting an unknown event degrades to staff-only, never to public.
@@ -731,14 +814,31 @@ it picks, it fails open on one side.)
| Nav | `GET /public/shard/features` returns only what the caller may reach, so the SPA never renders a link that would 403. Presentation only. |
Config reads are cached ~5s, so admin changes take effect within seconds **including on already-open
streams**. `PUBLIC_KINDS` still exists and is still exported (`/feed` filtering, `notificationStreams.js`)
but is now **derived** from the kind map rather than hand-maintained, so the two cannot drift.
streams**. `PUBLIC_KINDS` still exists and is still exported (`notificationStreams.js`) but is now
**derived** from the kind map rather than hand-maintained, so the two cannot drift.
**`PUBLIC_KINDS` is a module-load constant and must not be used to answer "may this caller read this
kind?"** — it is computed from the compiled *defaults*, so it cannot see an admin's changes. Use
`visibleKinds(level, config)`, which resolves against the live config. `/feed` uses it; it originally
used `PUBLIC_KINDS` and consequently kept serving `guild.join` to anonymous callers after an admin had
moved `guilds` to `staff`. `visibleKinds` deliberately ignores the `stream` flag: that governs SSE
fan-out only, so a feature whose live firehose ships off (market) stays readable from stored history.
**Every read path that returns shard data must call `projectFeature`.** The stored-history endpoints
are not exempt — `/feed` returns the same events the stream does, and returning them unprojected
reopens on the REST side exactly what the stream closes. Relatedly, `shardEvents.db.list` treats an
**empty** `kinds` array as "serve nothing", never "no filter"; the fall-through it used to take would
have turned a fully-gated config into a dump of the entire event log.
`projectFeature` walks **arrays and plain objects only**. A `Date`, `Buffer` or other class instance
is passed through as a value — rebuilding one key-by-key yields `{}`, which is the difference between
the pure-JSON wire frames and the DB-backed read models whose rows carry real `Date` columns.
**Defaults reproduce pre-3.0 behavior exactly**, so installing the framework is a no-op until an admin
changes something — with one deliberate exception, which is the leak it was written to close:
`/public/shard/guilds` and `/public/shard/governors` previously returned the raw stored payload, whose
`leader` / `governor` actors carry `acct` and `webId`. Those fields are now stripped for every caller
below admin.
changes something — with deliberate exceptions, which are the leaks it was written to close.
`/public/shard/guilds`, `/public/shard/governors` and `/public/shard/feed` previously returned the raw
stored payload, whose actors carry `acct` and `webId`; `/public/shard/idoc` returned the flattened
`ownerAcct`. All are now stripped for every caller below admin.
---

View File

@@ -80,6 +80,11 @@ game to anyone standing next to them; the **account** behind it is not, and neit
user it's linked to. Publishing those would disclose something the shard itself doesn't, and would
tie a player's in-game identity to their forum identity without their consent.
This rule matches the *meaning* of a field, not one spelling of it. Some responses nest the player
who owns a record (`leader.acct`); others flatten it into the row (`ownerAcct`, `leaderWebId`,
`governorAcct`). Every one of those is locked, and the admin API refuses to configure any of them —
so a new response shape can't quietly reopen the hole by naming the field differently.
**2. Unknown event kinds are never broadcast below admin.**
The live stream maps each event kind to a feature. A kind with no mapping — a new event from a shard
plugin the site doesn't know yet, say — goes to admins only. It fails closed. This is what keeps the
@@ -99,6 +104,15 @@ Three places, one config:
- **Navigation** hides links a viewer can't follow, so they don't hit a wall. This is presentation
only — the gate is server-side either way.
**Stored history answers the same way the live stream does.** The activity feed reads from the event
log rather than the live stream, but it resolves the *same* question against the *same* config: which
kinds you may read, and which fields survive. So moving a feature up a rung hides it from the history
as well as the stream — there is no back door where yesterday's copy of an event is more revealing
than today's.
One deliberate asymmetry: turning **live updates** off for a feature stops the push, not the reading.
The marketplace ships this way — its history and its pages are public, only the firehose is off.
Changes take effect within about five seconds, **including on streams that are already open**. You
don't need to restart anything.

261
website/SPAWN_ATLAS.md Normal file
View File

@@ -0,0 +1,261 @@
# Spawn atlas
**Status:** Data pipeline landed on `edge` (website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112)); API and client pages follow in a second PR.
**Design:** [`docs/link/v3.md` §6](../link/v3.md) — Protocol 3.0 Part C.
The spawn atlas is a browsable catalogue of what the shard *contains*: which
creatures spawn, where, how many, and which champion altars are configured. It
answers "where do I find a lizardman?" with **"Shrines, Yew, Isamu-Jima"** rather
than with a list of raw coordinates.
## Two things that shape the whole design
**The shard's ServUO tree is the single source of truth.** Nothing is
precomputed and committed to the repository. A shard's maps change over its
lifetime — facets get added, replaced, or renamed — and a snapshot in the repo
would silently drift from the world players actually see. The atlas is therefore
re-derived from the tree **on every server boot**.
**Facets are not a fixed list.** Nothing in the codebase names Felucca, Trammel,
or any other stock facet. The facet set is whatever the shard's own files
declare, discovered at parse time. A shard running entirely custom maps gets
exactly the same treatment as a stock one, with no code change.
## What it is not
The atlas is **static shard content, not live shard state.**
- It does **not** come from the sidecar. Nothing here touches the bridge, and
there is no event kind, no wire change and no `PROTOCOL_VERSION` bump for it.
Part C is website-only.
- It stays fully populated while the shard is down.
- Its champion table (`shard_champion_spawns`) is the *configured roster*
"there is an Unholy Terror altar in Deceit". The live `champ.update` feed in
`shard_champs` is the separate, sidecar-fed answer to "it is on level 3 right
now". Both exist; do not conflate them.
Routes live at `/api/v1/public/atlas`, deliberately **not** under `/shard`,
because `/shard/*` means sidecar-dependent.
## Configuring the tree
The website needs to be able to *read* the ServUO tree — same host, a bind mount,
or a shared volume. Two ways to point at it, the setting winning over the
environment:
| Source | Notes |
|---|---|
| `spawn_atlas_servuo_path` setting | Admin-editable; changes take effect on the next refresh without a redeploy |
| `SERVUO_PATH` env var | The deploy-time default, since the path usually describes a mount the deployment sets up |
With neither set the atlas is simply skipped — the site runs normally without
one.
## The boot path
On every start the server hashes the source files and compares them against what
is loaded. Unchanged (the normal case on a restart) costs one read pass, ~120 ms,
and no database write. A real change costs a ~400 ms parse and a reload.
Two contracts govern it:
**1. It never blocks startup.** No configured path, an unreadable mount, a
malformed file, a database error — every one is caught and logged, and the site
comes up serving whatever atlas it already had.
**2. A facet disappearing is never applied automatically.** Losing a facet looks
exactly like a half-copied or mid-update tree, and boot cannot tell that apart
from a real map change. That refresh is *staged* for a human instead. Everything
else — new facets, renamed regions, changed spawns — applies immediately, since
none of it can destroy something an operator would miss.
```
boot
└─ path configured? no ──▶ skip
└─ tree readable? no ──▶ warn, carry on
└─ hashes changed? no ──▶ done (nothing parsed)
└─ parse
└─ a facet would be removed?
no ──▶ import
yes ──▶ stage for admin review; atlas unchanged
```
### Approving or rejecting a staged refresh
Only the *decision* is stored, never the parsed world — a few KB of source hashes
plus the facet diff. Approving **re-parses** the tree, so what lands matches the
tree at approval time rather than at boot, and a multi-megabyte blob never sits
in the database.
A rejection is remembered against those exact source hashes, so a declined
refresh does not re-prompt on every restart. Change the tree and the hashes
differ, which asks again.
From the admin panel (second PR), or from the CLI:
```bash
cd website/server
npm run atlas:import -- --status # what is loaded, and what is pending
npm run atlas:import -- --approve # apply the staged refresh
npm run atlas:import -- --reject # keep the current atlas, dismiss it
```
## The CLI
The server refreshes itself on boot, so this is for applying a map change
*without* a restart, and for the approve/reject flow above.
```bash
npm run atlas:import # import if the tree differs
npm run atlas:import -- --servuo <path> # override the path for this run
npm run atlas:import -- --force # reimport even if unchanged
```
`--servuo` is a per-run override and deliberately does **not** persist — changing
where the atlas permanently reads from is an admin action, not a side effect of a
one-off import.
## Sources
| File | Count (stock ServUO 57.4) | Used for |
|---|---|---|
| `Spawns/*.xml` | 13 files, ~10.5 MB | Every spawner: location, size, delays, time-of-day, creature types |
| `Data/Regions.xml` | 129 KB, nested | Named regions and their rectangles |
| `Data/Locations/*.xml` | 6 files | Landmarks (dungeon levels, town markers) |
| `Config/ChampionSpawns.xml` | 4.8 KB | Configured champion altars |
**A stock tree has 13 spawn files but only 6 facets.** `Eodon.xml`,
`GravewaterLake.xml`, `TreasuresOfKotl.xml` and the other named-area files hold
TerMur/Trammel points. The facet always comes from each record's own `<Map>`,
never from the file name.
## How a coordinate becomes a place name
This is the transform the atlas exists for, in `resolveRegion()`:
1. The highest-`priority` named region whose rectangle contains the point. Ties
break toward the **smallest** rect, so a specific room wins over the
dungeon-wide rect enclosing it.
2. Otherwise the nearest landmark within the landmark radius (200 tiles by
default), labelled by its **group** ("Covetous"), not its individual marker
("Level 1").
3. Otherwise `"Wilderness"`.
The radius cap in step 2 is what keeps step 3 reachable. Without it the nearest
landmark is always *some* landmark however far away, and open countryside gets
labelled with a dungeon on the far side of the map.
Against stock ServUO this resolves **83.2%** of points (5,369 of 6,455): 3,681 by
region, 1,688 by landmark, 1,086 Wilderness.
## Three quirks in the source data
Each of these is silent if unhandled — the atlas still builds, it is just wrong.
**Facet names disagree between sources.** `Data/Locations/*.xml` spells them
`Ter Mur` and `Tokuno Islands`, while `<Map>` and `<Facet name>` say `TerMur` and
`Tokuno`. Unreconciled, the landmark bucket is keyed differently from the points
looking it up, so the fallback never fires and every unregioned spawn on those
facets reads "Wilderness".
This is reconciled **by matching, not by a lookup table** — there is no list of
facet names anywhere. `facetKey()` collapses spelling differences (lowercase,
alphanumerics only), and `resolveFacetName()` matches a loose spelling against
the canonical set discovered from the shard's own spawn and region data, by exact
key then by prefix in either direction. A name matching nothing keeps its own
name: forcing a wrong match would file a real custom facet's landmarks under the
wrong facet, which is worse than leaving it alone.
**Spawn type tokens carry XmlSpawner directives.** The `<Objects2>` type is not
always a bare class name:
```
Fairy,{RND,4,8} alchemist/z/-50 Agralem/Name/Agralem
GargishRouser,1 greatape,true GargishRefugee/hue/34532
```
Taken literally these invent creatures that do not exist *and* split real ones in
two, because `Fairy` and `Fairy,{RND,4,8}` slug apart into separate entries. 71 of
845 were affected. Everything from the first `/` or `,` is stripped, leaving 800
real creatures.
**Case is inconsistent across files.** The same creature is `Lizardman` in one
file and `lizardman` in another. Slugging collapses them correctly, but the
display name is chosen deterministically — most common spelling wins, ties break
to the more capitalised form, then alphabetically — because otherwise it would
depend on file read order and change on an unrelated restart.
## Tables
All are **import-owned**: a refresh empties and reloads them in one transaction,
so a failed reload leaves the previous atlas intact rather than a half-loaded
world. Nothing else writes to them and nothing holds a foreign key to them — no
FKs at all, consistent with every other `shard_*` table. Full column listings in
[`BACKEND_DESIGN.md`](BACKEND_DESIGN.md).
| Table | Rows (stock) | Notes |
|---|---|---|
| `shard_spawn_creatures` | 800 | `slug` PK; `total` = sum of each type's own max; nullable `art` |
| `shard_spawn_points` | 6,455 | `spawn_range`, since `range` is reserved in MariaDB |
| `shard_spawn_point_types` | 23,927 | The many-to-many; one spawner commonly carries six types |
| `shard_regions` | 387 | Flattened out of the nesting; `rects` JSON |
| `shard_landmarks` | 558 | `grp`, since `group` is reserved in SQL |
| `shard_champion_spawns` | 25 | Configured altars, not the live feed |
| `shard_atlas_meta` | 1 | Singleton; source hashes, for the change check |
| `shard_atlas_pending` | 01 | Singleton; a staged refresh awaiting admin review |
`shard_spawn_creatures.name` carries a plain `INDEX`, deliberately **not
`FULLTEXT`**: ~800 rows makes a `LIKE` scan free, and FULLTEXT's minimum token
length would break searches for names like "orc".
The reload uses `DELETE`, not `TRUNCATE``TRUNCATE` is DDL in MariaDB and would
implicitly commit, defeating the all-or-nothing guarantee. Point ids are assigned
explicitly rather than left to `AUTO_INCREMENT`, because the join rows need them
and `conn.batch()` reports no usable `insertId`.
## Artwork — operator-supplied, never shipped
**This project ships no creature art and no extraction tooling, and never will.**
UO sprites live in the operator's own client `.mul`/`.uop` files. They are the
operator's, not ours to redistribute.
The atlas is fully functional as text. `shard_spawn_creatures.art` is nullable
and is NULL on every fresh import; pages render without images, which is the
normal and supported state, not a degraded one.
An operator who wants art:
1. Extracts it from **their own** client files (UOFiddler, ClassicUO tooling, or
any art extractor).
2. Drops the images under `server/uploads/atlas/`.
3. Copies `server/db/data/spawnAtlas.art.example.json` to `spawnAtlas.art.json`
and maps creature slugs to file names.
4. Restarts, or runs `npm run atlas:import -- --force`.
Both `spawnAtlas.art.json` and `server/uploads/` are gitignored, so neither the
map nor the images can be committed by accident.
## Code layout
| File | Role |
|---|---|
| `src/utils/spawnAtlasParse.js` | **Pure and fs-free** parsers, so CI covers them with no ServUO tree. Zero dependencies. |
| `src/utils/spawnAtlasSource.js` | The only thing that reads a ServUO tree; shared by the boot path and the CLI |
| `src/model/shardAtlas/shardAtlas.db.js` | The one-transaction replace |
| `src/model/shardAtlas/shardAtlas.model.js` | The refresh decision, staging, approve/reject |
| `scripts/importSpawnAtlas.js` | Thin CLI over the model |
Parsing notes:
- `Regions.xml`, `Locations/*.xml` and `ChampionSpawns.xml` genuinely nest, and
get a small hand-rolled **subset** tokenizer — elements, attributes,
self-closing tags, comments, the XML declaration, CDATA, and the five
predefined entities plus numeric refs. It is not a general-purpose XML parser
and must not be reused as one.
- The ~10.5 MB of `Spawns/*.xml` never touches that tokenizer. Those records are
flat, so they get a streaming regex sweep instead; a DOM would allocate a node
per element across ~40 fields on every record to keep 14 of them. **Do not put
the Points files through a DOM parser.**
- `<Objects2>` is `Type:MX=n:SB=…` segments joined by `:OBJ=`. Split on `:OBJ=`
*first* — a naive `split(':')` shreds it. A single Trammel point carries six
types.