Compare commits
10 Commits
docs/spawn
...
cdea1aa7cd
| Author | SHA1 | Date | |
|---|---|---|---|
| cdea1aa7cd | |||
| 6ce60a82c3 | |||
| 70d49b7792 | |||
| ee0c146d7a | |||
| e3aabf9e3e | |||
| be9f5019fa | |||
| f715323aa0 | |||
| 8e857a9c8d | |||
| 64fb7edc3e | |||
| be7e1a69ce |
@@ -20,6 +20,9 @@ ci/ cross-cutting CI/quality notes
|
||||
| [HERO_EDITOR.md](website/HERO_EDITOR.md) | Hero canvas editor feature spec |
|
||||
| [WIKI_UPGRADE.md](website/WIKI_UPGRADE.md) | Wiki subsystem upgrade notes |
|
||||
| [SHARD_VISIBILITY.md](website/SHARD_VISIBILITY.md) | Who sees which shard data — the admin-configurable audience framework |
|
||||
| [SPAWN_ATLAS.md](website/SPAWN_ATLAS.md) | The bestiary / spawn atlas: what the shard contains, parsed from its own ServUO tree |
|
||||
| [CLILOCS.md](website/CLILOCS.md) | UO's id → name table: converting one from your client so items have names |
|
||||
| [MARKETPLACE.md](website/MARKETPLACE.md) | The player-vendor index: how it is gathered, what it costs, how to tune it |
|
||||
| [website-README.md](website/website-README.md) | Snapshot of the website repo's README (setup/run reference) |
|
||||
| [PROJECT_TREE.md](website/PROJECT_TREE.md) | Auto-generated snapshot of the repo's tracked file layout |
|
||||
|
||||
|
||||
@@ -381,6 +381,146 @@ Absent entirely if the shard runs `Bridge.RulesetEnabled=false` or an older plug
|
||||
This **supersedes the `world.systems` frame** sketched in [`PROTOCOL_2.md`](PROTOCOL_2.md) §10.4 and
|
||||
never implemented; the `systems` block above is what that asked for.
|
||||
|
||||
#### Points / loyalty leaderboards (Protocol 3.0)
|
||||
|
||||
ServUO carries ~25 separate point currencies — Queen's Loyalty, Void Pool, Casino, Clean Up Britannia,
|
||||
the nine city loyalties, Blackthorn, the Doom / Khaldun / Kotl treasure systems — every one a standing
|
||||
players accumulate over months, and none of them visible outside an in-game gump before 3.0.
|
||||
|
||||
A diff sweep (default 300 s), **one frame per system** rather than one large frame for all of them,
|
||||
matching `champ.update` / `guild.update`. A system is emitted only when its top N or its participant
|
||||
count actually changes.
|
||||
|
||||
| kind | fields | notes |
|
||||
|------|--------|-------|
|
||||
| `points.board` | `system`, `nameString`, `nameNumber`, `maxPoints`, `showOnGump`, `players`, `top[]` | One system's complete board — **never a delta**. The latest frame for a `system` replaces the previous one outright. `top[]` entries are `{rank, serial, name, points}`. |
|
||||
|
||||
`system` is the shard's own `PointsType` enum name (`QueensLoyalty`, `CleanUpBritannia`, …) and is the
|
||||
board's stable key. There is deliberately **no `points.remove`**: the set of systems is fixed at startup
|
||||
by `PointsSystem.Configure`, so a system cannot disappear at runtime — the same argument `city.update`
|
||||
makes for cities.
|
||||
|
||||
```json
|
||||
{"kind":"points.board","system":"QueensLoyalty",
|
||||
"nameString":"Queen's Loyalty","nameNumber":1114938,
|
||||
"maxPoints":15000,"showOnGump":true,"players":842,
|
||||
"top":[{"rank":1,"serial":"0x1A2B","name":"Darrow","points":29500},
|
||||
{"rank":2,"serial":"0x1A2C","name":"Mireille","points":21000}],
|
||||
"t":1752489280000}
|
||||
```
|
||||
|
||||
**Four things consumers get wrong.**
|
||||
|
||||
1. **`maxPoints` of `0` means UNCAPPED, not "zero points allowed".** ServUO's idiom for an uncapped
|
||||
system is `double.MaxValue` (`DespiseCrystals`, `ShameCrystals` and `VoidPool` all use it), which
|
||||
the plugin normalises to `0` rather than emitting a nonsense integer. On a real shard **most
|
||||
systems are uncapped**, so a UI that renders `points / maxPoints` must special-case this or it will
|
||||
divide by zero on the common path.
|
||||
2. **`nameString` is usually `null`.** The shard's `Name` is a `TextDefinition`, which may carry a
|
||||
literal *or* a cliloc id, and in practice most systems use the cliloc — so `nameNumber` is set and
|
||||
`nameString` is `null`. Resolve clilocs consumer-side; failing that, humanising the `system` key
|
||||
("CleanUpBritannia" → "Clean Up Britannia") reads better than showing a bare number. This is the
|
||||
same contract `titles.reward` already documents.
|
||||
3. **`players` counts players who actually hold points**, not the size of the system's table. Ten of
|
||||
the ~25 systems have `AutoAdd = true` and therefore keep a zero-point row for every character that
|
||||
has ever logged in, so the raw table size would report the shard's entire character census as that
|
||||
system's participants.
|
||||
4. **Entries carry `serial` and `name` only — never `acct` or `webId`.** A board is the widest-audience
|
||||
surface the bridge has, so the account name of every ranked player deliberately does not cross the
|
||||
wire; resolve serial → site user from your own link mirror if you need it.
|
||||
|
||||
Absent entirely if the shard runs `Bridge.PointsLeaderboardEnabled=false` or an older plugin. Render
|
||||
from `GET /points` (§6) on connect, then keep live with this event.
|
||||
|
||||
##### `char.profile` gains a `points` block
|
||||
|
||||
Read-model enrichment on the existing kind — there is **no** request kind for one character's points,
|
||||
the same precedent `titles` set in [`PROTOCOL_2.md`](PROTOCOL_2.md) §10.3:
|
||||
|
||||
```json
|
||||
"points":[{"system":"QueensLoyalty","nameString":"Queen's Loyalty","nameNumber":1114938,
|
||||
"points":29500,"maxPoints":15000}]
|
||||
```
|
||||
|
||||
Systems where the character has no entry, or an entry at zero, are **omitted** — otherwise every sheet
|
||||
would carry ~25 zeroes. `maxPoints` follows the same `0 == uncapped` rule as the board.
|
||||
|
||||
`rank` is **absent by default** and appears only when the shard runs `Bridge.PointsProfileRank=true`:
|
||||
a points lookup stops at the character's own row, but a rank must count every row that beats them, in
|
||||
every system, on every profile build. Derive rank from `points.board` instead for anyone in the top N.
|
||||
|
||||
#### Player-vendor marketplace (Protocol 3.0)
|
||||
|
||||
The shard-wide shop index: every player vendor's shop name, owner, location and priced inventory —
|
||||
the same set the in-game **Vendor Search** gump reads, published so a site can offer the same search
|
||||
from outside the game.
|
||||
|
||||
An **amortized round-robin diff sweep**, not a snapshot RPC, and the distinction is load-bearing:
|
||||
`rpc.rs::try_route` correlates a reply on the FIRST frame carrying a matching `reqId`, so a chunked
|
||||
reply sharing one `reqId` would deliver chunk 1 to the HTTP caller and leak chunks 2..N onto the
|
||||
broadcast feed. A whole-world snapshot could not fit in one frame inside the 10 s reply timeout
|
||||
either. The per-account `vendor.snapshot` RPC (§5) is unaffected and still serves the player portal.
|
||||
|
||||
Each tick inventories at most `Bridge.MarketSweepBatch` vendors (default 25) starting from a
|
||||
persistent cursor, so **per-tick cost is bounded independently of world size**; full coverage takes
|
||||
`ceil(vendors / batch) × MarketSweepSeconds`. A vendor is emitted only when its contents, prices,
|
||||
shop name or location actually change.
|
||||
|
||||
| kind | fields | notes |
|
||||
|------|--------|-------|
|
||||
| `vendor.listing` | `serial`, `shopName`, `ownerSerial`, `ownerName`, `location{}`, `count`, `total`, `truncated`, `items[]` | One vendor's complete shop — **never a delta**. The latest frame for a `serial` replaces the previous one outright. |
|
||||
| `vendor.listing.remove` | `serial` | The shop is gone from the index: dismissed, expired, or its owner switched off the in-game Vendor Search flag. |
|
||||
|
||||
```json
|
||||
{"kind":"vendor.listing","serial":"0x40001234",
|
||||
"shopName":"Darrow's Bargains","ownerSerial":"0x1A2B","ownerName":"Darrow",
|
||||
"location":{"map":"Trammel","x":1421,"y":1699,"z":0,
|
||||
"region":"Britain","house":"Darrow's Villa"},
|
||||
"count":2,"total":2,"truncated":false,
|
||||
"items":[{"serial":"0x40012ABC","itemId":3922,"hue":0,"amount":1,
|
||||
"price":25000,"name":null,"cliloc":1023721},
|
||||
{"serial":"0x40012ABD","itemId":7026,"hue":1157,"amount":3,
|
||||
"price":500,"name":"a shard sigil","cliloc":1041243}],
|
||||
"t":1752489280000}
|
||||
```
|
||||
|
||||
**Six things consumers get wrong.**
|
||||
|
||||
1. **`name` is `null` for nearly every item; `cliloc` is the real label.** Items carry a
|
||||
`LabelNumber`, not a name. The plugin deliberately never calls `VendorSearch.GetItemName`, which
|
||||
builds an `ObjectPropertyList`, serialises it and byte-parses the packet **per item** — a
|
||||
multi-hundred-millisecond stall across a full pass. (It would not work anyway: every current
|
||||
client ships its cliloc files compressed and ServUO's bundled `Ultima.StringList` cannot read
|
||||
them, so the in-game gump has the same gap.) Resolve clilocs consumer-side; a non-null `name` is a
|
||||
player-set literal and is strictly more specific, so **prefer it over the cliloc**.
|
||||
2. **`location` is one nested object, and it may be absent entirely.** It is nested so that a
|
||||
consumer gating vendor whereabouts gates one field rather than five that can drift apart — the
|
||||
website's `market.location` rule removes the whole object. Treat a missing `location` as "not
|
||||
published", not as an error.
|
||||
3. **`truncated` means the shop holds more than the frame carries.** `count` is what was published,
|
||||
`total` is what the shop actually holds, capped by `Bridge.MarketMaxListings` (default 250). A
|
||||
commodity reseller with thousands of stacked resources is real and an uncapped frame for one is
|
||||
measured in megabytes. Say "showing 250 of 3,104" rather than presenting a partial shop as
|
||||
complete.
|
||||
4. **`child: true` means the price buys the ENCLOSING CONTAINER.** ServUO prices a container as a
|
||||
unit and everything inside inherits that price with no `VendorItem` of its own; `DoSearch`
|
||||
surfaces the same flag. A UI that prints the container's price against each item inside it is
|
||||
lying about the shard.
|
||||
5. **Opted-out vendors are absent, and that is a privacy control.** `pv.VendorSearch` is the player's
|
||||
own in-game toggle and the sweep honours it — hide your vendor in game and it is hidden here too.
|
||||
The same goes for `Map.Internal` and a null backpack, matching `DoSearch`. Process
|
||||
`vendor.listing.remove` promptly: it is how a player *revoking* that consent reaches you.
|
||||
6. **Prices are inherently stale, by design.** The round-robin sweep means a shop can be a full cycle
|
||||
behind. Any UI over this must say how old the data may be — the website derives it from the oldest
|
||||
vendor row.
|
||||
|
||||
Entries carry `ownerSerial`/`ownerName` and **never `acct` or `webId`**, the same rule `points.board`
|
||||
follows. Absent entirely if the shard runs `Bridge.MarketEnabled=false` or an older plugin. Render
|
||||
from `GET /market` (§6) on connect, then keep live with these events — though note that a live
|
||||
firehose of whole vendor inventories is the largest stream the bridge produces, and a consumer that
|
||||
only needs a browsable index (as the website does) is better served by the REST read plus the
|
||||
periodic re-sweep.
|
||||
|
||||
---
|
||||
|
||||
## 5. REST — read queries
|
||||
@@ -420,7 +560,7 @@ Full character sheet: stats, all trained skills, worn equipment with flattened i
|
||||
Field notes:
|
||||
- `skills[].base` is trained value, `value` includes item/temp bonuses, `cap` is the cap. **Do not assume `base <= cap`** — GM characters can exceed it.
|
||||
- `equipment[].mods` is a flattened map of every non-zero AOS attribute on the item (weapon or armor). Empty `{}` for plain items.
|
||||
- Item names are usually **clilocs**, not strings: use `name` when present, otherwise resolve `cliloc` against a UO cliloc table on the site.
|
||||
- Item names are usually **clilocs**, not strings: use `name` when present, otherwise resolve `cliloc` against a UO cliloc table on the site. **Do not expect the shard to resolve them for you** — on any modern client ServUO's own `Ultima.StringList` cannot read the client's compressed cliloc files, so `VendorSearch.GetItemName` returns `item.Name` and the in-game Vendor Search gump has the same gap. Building that table is a consumer-side job; the website's is described in [`website/CLILOCS.md`](../website/CLILOCS.md).
|
||||
- `titles` (Protocol 2.0): `selected` is the index into `reward` currently displayed (`-1` if none). `fameKarma`/`skill` are computed display titles, omitted when the character has none. `reward` entries may be a **cliloc number as a string** or a literal string — resolve numeric ones against your cliloc table, same as item names.
|
||||
- Errors: unknown account → **404** `{"kind":"bridge.error","reason":"unknown account"}`; bad slot → **404**/**400** similarly.
|
||||
|
||||
@@ -740,6 +880,56 @@ worse than one that is briefly stale. Keep it current with the `world.ruleset` s
|
||||
`Bridge.RulesetEnabled=false`. That is a real answer distinct from a published ruleset, and worth
|
||||
rendering differently ("not published yet") rather than as an empty ruleset.
|
||||
|
||||
### Points / loyalty leaderboards (Protocol 3.0)
|
||||
|
||||
```
|
||||
GET /points
|
||||
→ { "boards": [ {"kind":"points.board","system":"QueensLoyalty","nameString":"Queen's Loyalty",
|
||||
"nameNumber":1114938,"maxPoints":15000,"showOnGump":true,"players":842,
|
||||
"top":[{"rank":1,"serial":"0x1A2B","name":"Darrow","points":29500}, ...],"t":...}, ... ] }
|
||||
|
||||
GET /points/{system} # e.g. /points/QueensLoyalty
|
||||
→ {"kind":"points.board","system":"QueensLoyalty", ... }
|
||||
```
|
||||
|
||||
Every system's latest board, or one by its `PointsType` name (§4 for the frame and its four gotchas).
|
||||
Served from the sidecar's projection, kept current by the `points.board` stream, ordered by display
|
||||
name. Survives a sidecar restart — which matters more here than for live state, since these are
|
||||
standings built over months and blanking them during a restart reads as data loss.
|
||||
|
||||
`GET /points/{system}` returns **404** for a system the shard has never published (an unknown name, or
|
||||
one excluded by `Bridge.PointsSystems`). That is distinct from a published board nobody has scored in
|
||||
yet, which is **200** with an empty `top[]` — and the two are worth rendering differently.
|
||||
|
||||
### Player-vendor marketplace (Protocol 3.0)
|
||||
|
||||
```
|
||||
GET /market?limit=200&offset=0
|
||||
→ { "vendors": [ {"kind":"vendor.listing","serial":"0x40001234",
|
||||
"shopName":"Darrow's Bargains","ownerSerial":"0x1A2B","ownerName":"Darrow",
|
||||
"location":{"map":"Trammel","x":1421,"y":1699,"z":0,
|
||||
"region":"Britain","house":"Darrow's Villa"},
|
||||
"count":2,"total":2,"truncated":false,"items":[ ... ],"t":...}, ... ],
|
||||
"total": 137, "limit": 200, "offset": 0 }
|
||||
```
|
||||
|
||||
Every vendor's latest shop, exactly as `vendor.listing` published it (§4 for the frame and its six
|
||||
gotchas). Served from the sidecar's projection, so it answers while the shard is down.
|
||||
|
||||
**This is the only PAGED read the sidecar serves**, because it is the only board that can be a whole
|
||||
world's inventory. `limit` is clamped to 1..1000 (default 200); `total` is returned so a caller knows
|
||||
when to stop rather than paging until it sees a short page, which would race a concurrent sweep.
|
||||
Ordering is by **serial**, not by shop name — a serial is stable while a shop name is renameable, so
|
||||
a rename mid-walk cannot make a vendor skip or repeat a page.
|
||||
|
||||
The route is `/market` and deliberately **not** `/vendors`: `/vendors/{account}` next door is the
|
||||
per-account RPC (§5), and two routes a prefix apart meaning "this player's shops" and "every shop on
|
||||
the shard" is a trap nobody wins.
|
||||
|
||||
Frames are served **verbatim**, owner names and coordinates included. That is not an oversight: the
|
||||
sidecar defines no audiences. Deciding who may see what is the consuming site's job — see
|
||||
[`v3.md`](v3.md) §3 for how the website does it.
|
||||
|
||||
---
|
||||
|
||||
## 7. Status codes
|
||||
|
||||
34
link/PLAN.md
34
link/PLAN.md
@@ -284,6 +284,15 @@ Counts in `hello` are a live snapshot taken on the Core thread, not a cached val
|
||||
|
||||
`Item.Name` is frequently `null`; the display name is `LabelNumber`, a cliloc id. **There is no `Data/Cliloc.enu` in this repo** — `BRIDGE_FINDINGS.md` §IV.4 is wrong about this. Cliloc data lives in the client install, which `DataPath` resolves to `D:\Games\Electronic Arts\Ultima Online Classic\`. Ship **both** `name` (when non-null) and `cliloc`, and resolve the number **on the website** against a cliloc map. That avoids a server-side dependency on the client directory.
|
||||
|
||||
**Update (3.0).** That recommendation held, and the reason it had to hold turned out to be stronger
|
||||
than "avoids a dependency": **ServUO cannot resolve clilocs either.** Every current client ships its
|
||||
`Cliloc.*` files compressed, and the bundled `Ultima.StringList` reads only the older plain layout —
|
||||
so `VendorSearch.StringList` is null and `VendorSearch.GetItemName` returns `item.Name` on any modern
|
||||
shard. The in-game Vendor Search gump has the same gap, which is why `vendor.listing` never calls it.
|
||||
Pushing name resolution to the plugin was never an option. See [`v3.md`](v3.md) §8.6 and
|
||||
`docs/website/CLILOCS.md` for how the site gets a table instead (the operator converts one from their
|
||||
own client, once).
|
||||
|
||||
---
|
||||
|
||||
## 8. Corrections to `BRIDGE_FINDINGS.md`
|
||||
@@ -318,10 +327,25 @@ Counts in `hello` are a live snapshot taken on the Core thread, not a cached val
|
||||
**Beyond 1.0.** Phases above are the 1.0 read/event plane. Protocol 2.0's phasing (provisioning +
|
||||
world-state boards) is [`PROTOCOL_2.md`](PROTOCOL_2.md) §13; Protocol 3.0's (visibility framework,
|
||||
shard content and standings) is [`v3.md`](v3.md) §9, which also tracks what has landed. Shipped from
|
||||
3.0 so far: **Part A** — the visibility framework — and **`world.ruleset`** ([`v3.md`](v3.md) §5),
|
||||
`BridgeRuleset.cs`, the first bridge stream that is neither an event subscription nor a sweep: it is
|
||||
3.0 so far: **Part A** — the visibility framework — **`world.ruleset`** ([`v3.md`](v3.md) §5),
|
||||
`BridgeRuleset.cs`, the first bridge stream that is neither an event subscription nor a sweep (it is
|
||||
emitted once per connect, like `server.hello`, because shard config changes only when an operator
|
||||
edits a file.
|
||||
edits a file) — the **spawn atlas** ([`v3.md`](v3.md) §6), which is website-only and touches no wire
|
||||
at all — and **`points.board`** ([`v3.md`](v3.md) §7), `BridgePoints.cs`, the loyalty/points
|
||||
leaderboards. `BridgePoints` is the widest read the bridge performs: ten of ServUO's ~25 point systems
|
||||
keep a row for every character ever created, so it selects the top N in a single bounded pass rather
|
||||
than sorting, and runs on a deliberately slow 300 s interval.
|
||||
|
||||
Also shipped: **`vendor.listing`** ([`v3.md`](v3.md) §8), `BridgeMarket.cs`, the shard-wide
|
||||
player-vendor index. It introduces the one sweep pattern the bridge did not previously have — an
|
||||
**amortized round-robin**. Every other sweep walks its whole collection per tick, which is fine for
|
||||
tens of houses or a fixed set of point systems and is not fine for a world of shops whose inventories
|
||||
recurse into containers. `BridgeMarket` inventories at most `MarketSweepBatch` vendors per tick from
|
||||
a persistent cursor, so the per-tick cost is bounded by the batch rather than by world size, and full
|
||||
coverage takes `ceil(vendors / batch) x MarketSweepSeconds`. Measured at **15.4 ms** for a cold tick
|
||||
of 25 vendors x 40 listings and **0.3 ms** in steady state (the per-vendor diff), on a shard of 209k
|
||||
items / 43k mobiles. It is also the first stream to honour a per-player privacy toggle: ServUO's own
|
||||
`PlayerVendor.VendorSearch` flag, so a shop hidden in game is hidden on the site.
|
||||
|
||||
### Config keys (`Config/Bridge.cfg`)
|
||||
|
||||
@@ -338,7 +362,9 @@ Read in `Configure()` via `Config.Get<T>("Bridge.<Key>", default)`. Key scope is
|
||||
|
||||
The set above is the 1.0 sample, not the current one — every later phase added keys (sweep intervals
|
||||
for each board, the town-crier/news caps, the admin write plane, account provisioning, and 3.0's
|
||||
`RulesetEnabled` / `PublicConnectAddress` / `RulesetIncludeSchedule`). **`servuo-plugins/overlay/Config/Bridge.cfg`
|
||||
`RulesetEnabled` / `PublicConnectAddress` / `RulesetIncludeSchedule`, and the `Points*` and `Market*`
|
||||
blocks).
|
||||
**`servuo-plugins/overlay/Config/Bridge.cfg`
|
||||
is the authoritative, commented list**; `BridgeConfig.cs` holds the defaults.
|
||||
|
||||
---
|
||||
|
||||
258
link/v3.md
258
link/v3.md
@@ -13,11 +13,16 @@ Each part is marked off here as it lands on `edge`. §9 carries the same state p
|
||||
|---|---|---|---|
|
||||
| 1 | **A** — visibility framework + actor-leak fix (§3) | ✅ **Done** | website [#109](https://gitea.whitlocktech.com/RunicGateway/website/pulls/109) + [#110](https://gitea.whitlocktech.com/RunicGateway/website/pulls/110), docs [#64](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/64) + [#65](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/65) |
|
||||
| 2 | **B/1** — `world.ruleset` (§5) | ✅ **Done** | servuo-plugins [#3](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/3), link [#17](https://gitea.whitlocktech.com/RunicGateway/link/pulls/17), website [#111](https://gitea.whitlocktech.com/RunicGateway/website/pulls/111), docs [#66](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/66) |
|
||||
| 3 | **C** — spawn atlas (§6) | 🟡 **Data pipeline done** | website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112) (parsers + CLI + tables); API/client PR next |
|
||||
| 4 | **B/2** — `points.board` (§7) | ⬜ Not started | — |
|
||||
| 5 | **B/3** — `vendor.listing` (§8) | ⬜ Not started | — |
|
||||
| 3 | **C** — spawn atlas (§6) | ✅ **Done** | website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112) (parsers + CLI + tables) + [#113](https://gitea.whitlocktech.com/RunicGateway/website/pulls/113) (API + pages + admin panel), docs [#67](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/67) + [#68](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/68) |
|
||||
| 4 | **B/2** — `points.board` (§7) | ✅ **Done** | servuo-plugins [#4](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/4), link [#18](https://gitea.whitlocktech.com/RunicGateway/link/pulls/18), website [#114](https://gitea.whitlocktech.com/RunicGateway/website/pulls/114), docs [#69](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/69) |
|
||||
| 5a | **B/3 dependency** — cliloc table (§8.6) | ✅ **Done** | website [#115](https://gitea.whitlocktech.com/RunicGateway/website/pulls/115), docs [#70](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/70) |
|
||||
| 5b | **B/3** — `vendor.listing` (§8) | 🟨 In review | servuo-plugins [#5](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/5), link [#19](https://gitea.whitlocktech.com/RunicGateway/link/pulls/19), website [#116](https://gitea.whitlocktech.com/RunicGateway/website/pulls/116), docs [#71](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/71) |
|
||||
| 6 | **Cutover** — `PROTOCOL_VERSION` 2→3 (§4) | ⬜ Not started | — |
|
||||
|
||||
Order 5 split in two once §8.6's cliloc dependency turned out to be a client-format problem rather
|
||||
than a parser (see §8.6). 5a is website-only and lands first so the marketplace ships with real item
|
||||
names; 5b is the four-repo wire change.
|
||||
|
||||
---
|
||||
|
||||
## 1. Why 3.0
|
||||
@@ -329,8 +334,10 @@ frame during verification.
|
||||
|
||||
**No plugin, no sidecar, no `Bridge.cfg` knob, no new kinds.** Not part of the v3 wire change.
|
||||
|
||||
> **Status:** data pipeline landed on `edge` — website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112)
|
||||
> (parsers, build/import CLI, tables, artifact). API + client pages are the second website PR.
|
||||
> **Status:** complete on `edge` — website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112)
|
||||
> (parsers, import CLI, tables) and [#113](https://gitea.whitlocktech.com/RunicGateway/website/pulls/113)
|
||||
> (the six public routes, the five admin ones, `/site/atlas` + `/site/atlas/:slug`, and the
|
||||
> Admin → Spawn Atlas panel).
|
||||
> Part C ships as **two** website PRs, not one: the parsing half is where the correctness risk
|
||||
> lives, and burying it under routes and React would have meant reviewing it in a 10k-line diff.
|
||||
> Full operator documentation: [`docs/website/SPAWN_ATLAS.md`](../website/SPAWN_ATLAS.md).
|
||||
@@ -471,9 +478,46 @@ region, 1,690 by landmark, 1,086 Wilderness).
|
||||
**One thing the design got exactly right:** the point-in-rect transform really is the reason to
|
||||
build this. "Where does a lizardman spawn?" answers *Shrines, Isamu-Jima, Yew* across three facets.
|
||||
|
||||
### 6.3 What the API/client half added
|
||||
|
||||
The second website PR built the six public routes, the five admin ones, `/site/atlas` +
|
||||
`/site/atlas/:slug`, and the Admin → Spawn Atlas panel. Three things it changed or established:
|
||||
|
||||
**1. Respawn delays were being read in the wrong unit — sometimes.** XmlSpawner writes
|
||||
`MinDelay`/`MaxDelay` in minutes and switches to seconds only when a delay does not divide into
|
||||
whole minutes, flagging that per record with `DelayInSec`
|
||||
(`XmlSpawner2.cs:7462-7480`, read back at `:6345-6358`). So a `5` means five *minutes* on one
|
||||
spawner and five *seconds* on the next, both plausible, and the pipeline stored the raw number.
|
||||
170 of 6,455 stock spawners are second-flagged — few enough to look like noise on a page and be
|
||||
believed. The parser now normalises to **seconds**, and the API and UI carry seconds throughout.
|
||||
*This is the class of bug §6.2 is a list of: the atlas still builds, it is just quietly wrong.*
|
||||
|
||||
**2. The hash gate needed a parser version, and this generalises.** Fixing the parse exposed that
|
||||
"has the tree changed?" is the wrong question on its own — an install whose maps never change would
|
||||
have kept serving the old readings forever, because the only thing compared was the tree.
|
||||
`spawnAtlasSource.PARSER_VERSION` is stored in `shard_atlas_meta` beside the source hashes, and a
|
||||
mismatch counts as drift. Any future parse correction lands on the next boot without an operator
|
||||
having to know it happened. **Bump it whenever the parser derives different data from identical
|
||||
files.**
|
||||
|
||||
**3. `points` is a count; `spawners` is the list.** The first cut of the detail route spread the
|
||||
creature row and then set `points` to the array of spawn points — the same key meaning a number on
|
||||
the search route and an array on the detail route. Renamed before it shipped, and worth recording
|
||||
because the two names are one letter apart in meaning and it reads as correct.
|
||||
|
||||
**On projection.** The `atlas` feature declares no sensitive fields, so `projectFeature` is a no-op
|
||||
on every one of these routes today. Every handler calls it anyway, per §3.6.1's rule — the point of
|
||||
the rule is that the *first* field that needs gating is covered by construction rather than by a
|
||||
retrofit nobody remembers to do.
|
||||
|
||||
---
|
||||
|
||||
## 7. Part B/2 — `points.board`
|
||||
## 7. Part B/2 — `points.board` ✅ Done
|
||||
|
||||
*Landed on `edge`: servuo-plugins [#4](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/4),
|
||||
link [#18](https://gitea.whitlocktech.com/RunicGateway/link/pulls/18), website [#114](https://gitea.whitlocktech.com/RunicGateway/website/pulls/114),
|
||||
docs [#69](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/69). Verified against the real ServUO tree
|
||||
per §11 — see §7.5 for what that run changed.*
|
||||
|
||||
Two deliverables: a diff sweep for the boards, and a `points` block folded into `char.profile` —
|
||||
the `PROTOCOL_2.md` §10.3 `titles` precedent (read-model enrichment, no new request kind).
|
||||
@@ -556,9 +600,46 @@ Client — NEW `routes/public/Leaderboards.jsx` at `/site/leaderboards`; a "Loya
|
||||
added to `components/CharacterSheet.jsx`, one edit serving both `PlayerCharacter.jsx` and
|
||||
`AdminCharacter.jsx`.
|
||||
|
||||
### 7.5 What the run against a real shard changed
|
||||
|
||||
The plan above was written from reading `PointsSystem.cs`. Booting the actual shard (ServUO 57.4, a
|
||||
43,011-mobile world) and letting one sweep run corrected four things — all of them invisible to a
|
||||
fake-shard test, because a fake shard emits whatever the spec says it should.
|
||||
|
||||
1. **`maxPoints` overflowed to `long.MinValue`.** `MaxPoints` is a `double`, and ServUO's idiom for an
|
||||
uncapped system is `double.MaxValue` — which `DespiseCrystals`, `ShameCrystals` and `VoidPool` all
|
||||
use. `(long)double.MaxValue` in C# is an **unchecked** conversion: it does not throw, it yields
|
||||
`long.MinValue`, and the first real sweep published
|
||||
`"maxPoints": -9223372036854775808` for three of the five live boards. Fixed with `Cap()` /
|
||||
`Score()` converters that normalise anything unrepresentable to `0`, which is now the wire's
|
||||
documented **"uncapped"** value. Worth stating plainly because it inverts the obvious reading:
|
||||
**on a real shard, `maxPoints: 0` is the common case, not an edge case**, so any UI dividing by it
|
||||
must special-case it.
|
||||
2. **`nameString` is usually `null`.** Most systems define their `Name` as a cliloc rather than a
|
||||
literal: four of the five boards on the live shard came back `nameString: null` with only
|
||||
`nameNumber` set. The humanise-the-`system`-key fallback is therefore the *primary* display path,
|
||||
not a defensive nicety, and both the leaderboards page and the character sheet lead with it.
|
||||
3. **`GetEntry`/`GetPoints` cannot be used in the read model.** `GetEntry(from, create: false)` still
|
||||
calls `AddEntry` when the system has `AutoAdd` (`PointsSystem.cs:207`) — it **mutates the world**.
|
||||
Ten of the ~25 systems have `AutoAdd = true`, so a profile built with the obvious accessor would
|
||||
have appended up to ten rows to the points save file every time anyone viewed a character sheet.
|
||||
`BridgeProfile.WritePoints` hand-rolls a read-only scan instead, and says so loudly.
|
||||
4. **`players` had to be redefined.** §7.2 called for "the entry count", but those same ten `AutoAdd`
|
||||
systems hold a zero-point row per character ever created — so the raw count reports the shard's
|
||||
whole census as one system's participants. It is now the number of players actually holding points,
|
||||
which is both the honest number and a strictly better diff signal (it moves when someone scores,
|
||||
not when someone logs in for the first time).
|
||||
|
||||
One deviation from the plan as written, for the same class of reason: §7.4 named the per-field
|
||||
visibility rule `characterName`, but `projectValue` matches on the **literal JSON key**, and the wire
|
||||
key is `name`. A rule under the descriptive name would have been silently inert — an admin tightening
|
||||
character names would have got no enforcement and no error, exactly the failure §3.6.1 records for the
|
||||
flattened `ownerAcct`. `FEATURES.leaderboards.fields` therefore keys on `name`, with a test that fails
|
||||
if it is renamed back.
|
||||
|
||||
---
|
||||
|
||||
## 8. Part B/3 — `vendor.listing`
|
||||
## 8. Part B/3 — `vendor.listing` 🟨 In review
|
||||
|
||||
### 8.1 It cannot be an RPC, and this is load-bearing
|
||||
|
||||
@@ -639,17 +720,86 @@ admin can turn the stream on. `uoLinkSocket` paginates `/market` on reconnect, b
|
||||
`/market/vendors/:serial`, behind `requireFeature('market')`. **Rate-limit it** — this is the first
|
||||
genuinely expensive public endpoint; `express-rate-limit` is already a dependency.
|
||||
|
||||
### 8.6 The open dependency — cliloc names
|
||||
### 8.6 The open dependency — cliloc names ✅ Resolved (shipped ahead of §8)
|
||||
|
||||
`CharacterSheet.jsx:14-15` already documents the gap ("without a cliloc table on the site we can only
|
||||
show literals") and renders equipment as `id {itemId}`. Search-by-name needs that table.
|
||||
`CharacterSheet.jsx:14-15` documented the gap ("without a cliloc table on the site we can only
|
||||
show literals") and rendered equipment as `id {itemId}`. Search-by-name needs that table.
|
||||
|
||||
- **Recommended:** `scripts/buildClilocs.js` reads the UO client's `Cliloc.enu` → committed
|
||||
`db/data/clilocs.json`; ingest denormalizes into `shard_vendor_items.display_name`. Same
|
||||
build-artifact pattern as §6, and it **also fixes the character sheet**.
|
||||
- **Fallback:** ship with item-art + price + region filters, and name search only over renamed items.
|
||||
**Resolved as its own website-only change, landed BEFORE the market so `/site/market` ships with real
|
||||
item names.** Full design and operator guide: [`docs/website/CLILOCS.md`](../website/CLILOCS.md).
|
||||
Ingest denormalizes the resolved name into `shard_vendor_items.display_name` as planned.
|
||||
|
||||
This decision is the reason §8 is sequenced last.
|
||||
Two things in the original recommendation above turned out to be wrong, and both are worth recording
|
||||
because the reasoning generalises.
|
||||
|
||||
**1. The committed `db/data/clilocs.json` artifact was dropped.** It predates the two Part C
|
||||
corrections (§6.1) and violates both: no committed snapshot of derived content, and nothing
|
||||
EA-derived ever shipped. UO's strings are EA's, exactly as the creature sprites are. Replaced with
|
||||
the §6 pattern instead — parse on every boot from an operator-configured path, hash-gated, output
|
||||
gitignored, `PARSER_VERSION` counted as drift.
|
||||
|
||||
**2. `scripts/buildClilocs.js reads the UO client's Cliloc.enu` is not possible, and the reason
|
||||
matters.** **Every current client ships its cliloc files COMPRESSED** — all eight `Cliloc.*` files
|
||||
open with a DWORD whose high byte is `0x8E`, the "Mythic" container. The plain layout (`02 00 00 00
|
||||
01 00`, then `{int32 number, byte flag, uint16 length, UTF-8}`) is what those files looked like
|
||||
*before* that change. Parsing a compressed file as plain does not fail cleanly: it yields ~19k
|
||||
"records" with negative ids, 1,722 distinct keys out of 19,508, one 62 KB "string", and a truncation
|
||||
somewhere in the middle.
|
||||
|
||||
Decompressing means porting an inverse-BWT coder with a 1 KB frequency header — a few hundred lines
|
||||
whose failure mode is plausible-looking garbage rather than an error. Two facts closed off the
|
||||
alternatives:
|
||||
|
||||
- **ServUO cannot read it either.** Its bundled `Ultima.StringList` implements only the plain layout,
|
||||
so on a modern client `VendorSearch.StringList` is null and `VendorSearch.GetItemName` returns
|
||||
`item.Name`. **The in-game Vendor Search gump has the same gap** — which also means §8.2's warning
|
||||
never to call `GetItemName` in the sweep costs us nothing we could otherwise have had.
|
||||
- The shard therefore cannot supply names on our behalf, so this could not be pushed to the plugin.
|
||||
|
||||
⇒ **the operator converts once, from their own client, and the site reads the result.** Accepted
|
||||
shapes are the plain binary layout and a `number<TAB|,|;>text` export; the site sniffs which.
|
||||
`server/tools/cliloc-export/` drives UOFiddler's `Ultima.dll` (the decompressor that already exists)
|
||||
and writes the plain form. A shard that never converts is fully supported — names render as ids,
|
||||
exactly as before.
|
||||
|
||||
**Shards edit items and add new ones**, and those carry ids no stock client table has — so this reads
|
||||
a **set** of sources, not one file, hash-gated together and re-read on every boot exactly as §6 reads
|
||||
the ServUO tree: a base (the converted client table) plus every overlay under `custom/`, later
|
||||
winning. Adding one custom item therefore never means re-exporting a 5 MB client file. Measured on
|
||||
the live shard for scale: its script tree references **16,434** cliloc ids and only **37** are absent
|
||||
from stock — tens against a 67k base, which is why an overlay and not a second table. `custom/` is the
|
||||
one convention here that is ours rather than the shard's, because **ServUO has no server-side notion
|
||||
of a custom cliloc**: they live in the patched client a shard distributes, and nothing in the tree
|
||||
declares them.
|
||||
|
||||
That set also brings back a hazard a single file did not have, and §8.6 answers it the way §6 does. A
|
||||
corrupt source fails the parse loudly, but a source that has **vanished** parses perfectly and imports
|
||||
a table quietly missing everything it contributed — an unmounted volume is indistinguishable from a
|
||||
deliberate deletion. So it is **staged, not applied** (`status: 'needsReview'`), reported by both the
|
||||
import and `status()`, and accepted with `{approve:true}`. It is a flag rather than §6's
|
||||
approve/reject pair because the atlas stores a pending decision *so that approving re-parses*; here
|
||||
nothing is stored, so re-reading at approval time is automatic.
|
||||
|
||||
Five traps found by building it, all recorded in `CLILOCS.md`:
|
||||
|
||||
- **`StringList.SaveStringList` RE-COMPRESSES on save.** It looks exactly like the export path and is
|
||||
not; its output is byte-identical to its compressed input, because its purpose is round-tripping a
|
||||
file back into the client.
|
||||
- **Trimming a text line before splitting silently drops half the table.** Roughly half of a real
|
||||
cliloc table is empty strings (ids the client reserves), exported as `1005008<TAB>`. Trimming eats
|
||||
the trailing separator, leaving a bare number that then looks like a header row — 55,994 of 123,490
|
||||
entries vanished, and the import still looked successful.
|
||||
- **`Number('')` is `0`, not `NaN`.** A line starting with a separator imports as a bogus cliloc 0
|
||||
unless the empty field is rejected explicitly.
|
||||
- **Tidying punctuation unconditionally corrupts real names.** Stripping leftover brackets is right
|
||||
after a placeholder is removed (`[~1_stuff~]` → nothing) and wrong otherwise: a shard's custom
|
||||
`"Runic Gateway Sigil (v2)"` rendered as `"(v2"`. Same shape as the `%` rule. **Found only by
|
||||
running a shard-style overlay through it** — every stock-table fixture passed.
|
||||
- **Source labels must be forward-slashed and root-relative**, or the same directory fingerprints
|
||||
differently on Windows and Linux and every boot looks like a change. The identical bug §6 records.
|
||||
|
||||
Blank entries are dropped at import (123,490 parsed → **67,496** stored), which also makes the binary
|
||||
and text paths converge on identical content.
|
||||
|
||||
### 8.7 Client
|
||||
|
||||
@@ -657,6 +807,75 @@ This decision is the reason §8 is sequenced last.
|
||||
driven by `staleAt` (the oldest `shard_vendors.updated_at`). The round-robin sweep means data is
|
||||
inherently up to one full cycle old, and the UI must say so.
|
||||
|
||||
Shipped with a second page, `routes/public/MarketVendor.jsx` at `/site/market/vendors/:serial` —
|
||||
where a search result points. It is the only surface that can render the two states the result list
|
||||
cannot: a `truncated` shop (*"showing 250 of 3,104 — this shop holds more than the shard
|
||||
publishes"*) and a `location` an admin has gated away, which is a real answer rather than an empty
|
||||
coordinate.
|
||||
|
||||
### 8.8 What the build changed
|
||||
|
||||
Four things the implementation settled differently from §8 as written, all of them found by building
|
||||
against the live shard.
|
||||
|
||||
**1. `location` is a nested object, not flat `map`/`x`/`y`/`region`.** §8.1's payload sketch had them
|
||||
flat, and it would have made `market.location` — a rule Part A pre-wired — **inert**, exactly like
|
||||
the `characterName` miss §7.5 records: `projectValue` matches literal JSON keys, so there is no
|
||||
`location` key for the rule to match. Flat keys would have needed five rules that could drift apart.
|
||||
Nesting makes one rule hide the facet, the coordinates, the region and the house together, on the
|
||||
live frame and the stored read model alike, because both now spell it the same way.
|
||||
|
||||
The other pre-wired rule, `market.ownerName`, checked out — it is a real key on the frame. Owner is
|
||||
written as flat `ownerSerial`/`ownerName` rather than through `BridgeJson.Actor`, which would add
|
||||
`acct` and `webId`; same argument `points.board` makes. `ownerSerial` was **added** to the
|
||||
configurable fields alongside `ownerName`, because an admin who hides the owner's name and leaves a
|
||||
serial every other board resolves back to that name has not hidden anything.
|
||||
|
||||
**2. The per-vendor diff signature is the full listing set, not §8.3's `count | Σ(serial ^ price)`.**
|
||||
That hash collides on the single most common change a shop makes: two items swapping prices, which
|
||||
is what re-pricing looks like. The signature is built over the same buffer the frame is written
|
||||
from, in the same order, so a match really does mean an identical frame.
|
||||
|
||||
**3. There is no `payload` column on `shard_vendors`.** §8.5 implied the board pattern (whole frame
|
||||
in JSON, columns hoisted for display). It does not apply here: the items ARE the searchable rows, so
|
||||
they are normalized into `shard_vendor_items` and there is nothing left worth duplicating. The
|
||||
sidecar keeps the whole blob, because outage resilience is its job and search is not.
|
||||
|
||||
**4. Sweep cost is reported, and a slow tick warns.** The batch cap is a *claim* about per-tick cost,
|
||||
and an operator tuning `MarketSweepBatch` was otherwise tuning blind. `[bridge status` now carries
|
||||
`lastMs`/`maxMs`, and a tick over 50 ms prints a rate-limited warning naming the knob.
|
||||
|
||||
Measured on the live shard (27 vendors × 40 listings, 209k items / 43k mobiles):
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| First tick — 25 vendors emitted cold | **15.4 ms** |
|
||||
| Second tick — the remaining 2 | **3.4 ms** |
|
||||
| Steady state — nothing changed | **0.3 ms** |
|
||||
| Website `/market` search over 1,040 listings | 1,040 total, names resolved |
|
||||
| Cliloc re-resolution pass over 1,040 rows | **50 ms** |
|
||||
|
||||
The diff is what makes the steady state ~free; the batch cap is what bounds the cold case. Note the
|
||||
arithmetic the warning exists for: at the default cap of 250 listings, a batch of 25 **full** shops
|
||||
is 6,250 items ≈ 95 ms — over budget. Real shops hold tens, which is why 25 is the default, but a
|
||||
shard of commodity resellers should lower the batch, and now it will be told to.
|
||||
|
||||
Two smaller things worth not rediscovering:
|
||||
|
||||
- **`BridgeJson.Escape` takes a NON-NULL string** — it dereferences `value.Length` immediately — and
|
||||
`BridgeJson.Str` writes its own `,"key":` prefix, so neither serves a value inside a hand-built
|
||||
object. Nearly everything this frame writes is legitimately null (an item's plain `Name` is null
|
||||
for almost every item; a vendor in the street has no house), so that is the common path, not an
|
||||
edge case. `BridgeMarket.Text()` is the two-line writer that was missing.
|
||||
- **The ServUO console writes in the OS code page**, so an em dash in a `Console.WriteLine` renders
|
||||
as `???` in the log an operator would paste into an issue. Bridge console output is ASCII.
|
||||
|
||||
Search-side, one thing the site had to fix rather than inherit: `%` and `_` in a user's query are
|
||||
**LIKE** metacharacters, not SQL ones, so parameterization does not neutralize them — a search for
|
||||
`%` would otherwise match every listing on the shard. `shardMarket.db.js` escapes them. (The atlas's
|
||||
`LIKE` searches predate this and have the same shape over a much smaller table; worth a follow-up,
|
||||
not a blocker here.)
|
||||
|
||||
---
|
||||
|
||||
## 9. Sequencing
|
||||
@@ -665,9 +884,10 @@ inherently up to one full cycle old, and the UI must say so.
|
||||
|---|---|---|---|---|
|
||||
| 1 | **A** — visibility framework + actor-leak fix | website, docs | none | ✅ Done |
|
||||
| 2 | **B/1** — `world.ruleset` (§5) | all four | new kind | ✅ Done |
|
||||
| 3 | **C** — spawn atlas (§6) | website, docs | none | 🟡 Pipeline done, API/client next |
|
||||
| 4 | **B/2** — `points.board` (§7) | all four | new kind + `char.profile` field | ⬜ |
|
||||
| 5 | **B/3** — `vendor.listing` (§8) | all four | new kinds | ⬜ |
|
||||
| 3 | **C** — spawn atlas (§6) | website, docs | none | ✅ Done |
|
||||
| 4 | **B/2** — `points.board` (§7) | all four | new kind + `char.profile` field | ✅ Done |
|
||||
| 5a | **B/3 dependency** — cliloc table (§8.6) | website, docs | none | ✅ Done |
|
||||
| 5b | **B/3** — `vendor.listing` (§8) | all four | new kinds | 🟨 In review |
|
||||
| 6 | **Cutover** — `PROTOCOL_VERSION` 2→3, `edge` → `main` | all four | the bump | ⬜ |
|
||||
|
||||
---
|
||||
@@ -685,7 +905,7 @@ inherently up to one full cycle old, and the UI must say so.
|
||||
in the security section.
|
||||
- NEW `website/SHARD_VISIBILITY.md` — admin-facing: what each feature exposes, what each rung means,
|
||||
what cannot be loosened.
|
||||
- NEW `website/SPAWN_ATLAS.md`, NEW `website/MARKETPLACE.md`.
|
||||
- NEW `website/SPAWN_ATLAS.md`, NEW `website/CLILOCS.md`, NEW `website/MARKETPLACE.md`.
|
||||
- `PROJECT_TREE.md` in each touched repo.
|
||||
- `npm run swagger` **and** `npm run routes:manifest` on every route-touching PR — both are committed
|
||||
artifacts, and `test/routeManifest.test.js` fails on drift.
|
||||
|
||||
@@ -101,6 +101,10 @@ server/
|
||||
deliberately not site-mode gated
|
||||
shard.router.js (14) /public/shard/* incl. the anonymous
|
||||
SSE stream; never site-mode gated
|
||||
atlas.router.js (6) /public/atlas/* — the spawn atlas.
|
||||
NOT under /shard: nothing here
|
||||
touches the sidecar, and unlike
|
||||
/shard it IS site-mode gated
|
||||
site.router.js (4) /settings /status /version /contact —
|
||||
the group-root singletons; declares no
|
||||
router-level middleware
|
||||
@@ -375,6 +379,78 @@ they are cheap to display — the same payload-plus-hoisted-columns shape `shard
|
||||
served as `null` rather than `{}`: "not published yet" and "published, everything off" are different
|
||||
answers and the page renders them differently.
|
||||
|
||||
### shard_points_boards — points / loyalty leaderboards (Protocol 3.0)
|
||||
|
||||
One row per point system, keyed by the shard's own `PointsType` name (`QueensLoyalty`,
|
||||
`CleanUpBritannia`, …). The shard carries ~25 of these, each a standing players build over months.
|
||||
Columns: `system` (PK), `name`, `name_cliloc`, `max_points`, `players`, `show_on_gump`, `payload` JSON
|
||||
(the whole `points.board` frame), `t`, `updated_at`.
|
||||
|
||||
**The top-N list stays inside `payload`** rather than being normalized into a `shard_points_entries`
|
||||
table. It is a fixed-size list (10 by default) that is only ever read whole — exactly like
|
||||
`shard_governors.candidates` — so normalizing buys nothing until something needs a per-character
|
||||
reverse lookup, and a character's own standings already ride inside `char.profile` instead.
|
||||
|
||||
Board state, not events: `points.board` is **not** in `LOGGED_KINDS`, for the same reason
|
||||
`guild.update` isn't. The shard emits a frame every time anyone's score moves a top ten, so logging
|
||||
would grow `shard_events` without bound for something whose only interesting value is its latest
|
||||
version. There is also **no delete path** — the shard's set of systems is fixed at startup, so there is
|
||||
no `points.remove` to mirror.
|
||||
|
||||
Two values carry non-obvious meanings, both set by the plugin and both documented in
|
||||
[`link/INTEGRATION.md`](../link/INTEGRATION.md) §4:
|
||||
|
||||
- **`max_points = 0` means uncapped**, and on a real shard that is the *common* case (ServUO's
|
||||
uncapped idiom is `double.MaxValue`, which the plugin normalises to 0). Anything rendering
|
||||
`points / max_points` must special-case it.
|
||||
- **`name` is usually NULL**, with `name_cliloc` set instead — most systems name themselves with a
|
||||
cliloc rather than a literal. Listing therefore orders by `COALESCE(name, system)`, so boards
|
||||
awaiting cliloc resolution sort by their own key rather than clumping together under NULL.
|
||||
|
||||
### shard_vendors / shard_vendor_items — the player-vendor marketplace (Protocol 3.0)
|
||||
|
||||
The shard-wide shop index, fed by `vendor.listing` / `vendor.listing.remove`. One row per player
|
||||
vendor and one per priced listing. Full operator detail in [`MARKETPLACE.md`](MARKETPLACE.md); the
|
||||
design is `docs/link/v3.md` §8.
|
||||
|
||||
| Table | Shape |
|
||||
|---|---|
|
||||
| `shard_vendors` | `serial` (PK), `shop_name`, `owner_serial`, `owner_name`, `map`/`x`/`y`/`z`, `region`, `house`, `item_count`, `item_total`, `truncated`, `t`, `updated_at`. Indexes on owner, map, region and `updated_at`. |
|
||||
| `shard_vendor_items` | `id` (PK), `vendor_serial`, `serial`, `item_id`, `hue`, `amount`, `price`, `name`, `cliloc`, `display_name`, `child`. Indexes on `vendor_serial`, `price`, `item_id`, `display_name`, and `(display_name, price)`. |
|
||||
|
||||
**Ingest is per-vendor and authoritative**: the frame is the whole shop, so ingest is
|
||||
delete-then-insert of that vendor's listings inside one transaction. All-or-nothing matters
|
||||
specifically because the two writes are "the shop" and "what is in it" — a failure between them
|
||||
leaves a shop advertising an inventory it no longer has, which is visibly wrong and indistinguishable
|
||||
from a genuinely empty shop. No foreign keys, consistent with every other `shard_*` table.
|
||||
|
||||
**There is deliberately no `payload` column**, unlike `shard_points_boards` directly above. The
|
||||
board's top-N is a fixed-size list read whole, so it lives in JSON; here the items *are* the
|
||||
searchable rows, so they are normalized and nothing is left worth duplicating. The sidecar keeps the
|
||||
whole blob — outage resilience is its job, search is ours.
|
||||
|
||||
Market state, not events: neither kind is in `LOGGED_KINDS`, and this is the strongest case of the
|
||||
three v3 kinds. One frame carries up to 250 listings and the sweep re-emits a shop on any price
|
||||
change, so logging would turn `shard_events` into a price history nobody reads.
|
||||
|
||||
Two columns carry non-obvious meanings:
|
||||
|
||||
- **`item_count` vs `item_total`.** `item_count` is what the frame published; `item_total` is what
|
||||
the shop actually holds. They differ when `truncated` — the shard caps listings per frame
|
||||
(`Bridge.MarketMaxListings`, 250 by default), and a commodity reseller with thousands of stacks
|
||||
genuinely exceeds it. Any UI must show both or it presents a partial shop as complete.
|
||||
- **`display_name` is denormalized at ingest**, resolved from the item's literal `name` (preferred —
|
||||
a player set it, so it is more specific) else its `cliloc` against `shard_clilocs`. Resolving at
|
||||
query time would put the cliloc table on the hot path and make search-by-name impossible. Because
|
||||
the shard's diff sweep will not re-send an unchanged shop just because the site learned what its
|
||||
items are called, **a cliloc import triggers a bulk re-resolution** of this column (after a boot
|
||||
import and after an admin import; ~50 ms per thousand rows, never throws).
|
||||
|
||||
`updated_at` is written explicitly on every upsert rather than left to `ON UPDATE CURRENT_TIMESTAMP`,
|
||||
which MariaDB does not fire when every column is written back unchanged. A shop re-published
|
||||
identically is still *freshly confirmed*, and without this the staleness banner would age a perfectly
|
||||
current shop forever.
|
||||
|
||||
### shard_feature_visibility — per-feature audience config (Protocol 3.0)
|
||||
|
||||
One row per shard feature: `feature` (PK), `enabled`, `audience` (a rung on the ladder in §6.5),
|
||||
@@ -409,7 +485,7 @@ when its maps are updated; the facet set is discovered from the tree, and the lo
|
||||
| `shard_regions` | `facet`, `name`, `type`, `priority`, `parent`, `rects` JSON |
|
||||
| `shard_landmarks` | `facet`, `name`, `grp`, `x`, `y`, `z` |
|
||||
| `shard_champion_spawns` | `slug` PK, `name`, `grp`, `type`, `random_type`, `facet`, `x`, `y`, `z`, `radius`, `label` |
|
||||
| `shard_atlas_meta` | Singleton (`id = 1`), `payload` JSON (counts + a sha256 per source file), `imported_at` |
|
||||
| `shard_atlas_meta` | Singleton (`id = 1`), `payload` JSON (counts, a sha256 per source file, `parserVersion`), `imported_at` |
|
||||
| `shard_atlas_pending` | Singleton (`id = 1`), `status` (`pending`/`rejected`), `payload` JSON, `detected_at` |
|
||||
|
||||
The first seven are **import-owned**: a refresh empties and reloads every one inside a single
|
||||
@@ -430,6 +506,11 @@ The boot refresh is **best-effort by contract**: no configured path, an unreadab
|
||||
file or a database error is caught and logged, and the site comes up serving whatever atlas it had.
|
||||
The tree path comes from the `spawn_atlas_servuo_path` setting, falling back to `SERVUO_PATH`.
|
||||
|
||||
**A refresh re-derives when the tree changed OR the parser did.** `spawnAtlasSource.PARSER_VERSION`
|
||||
is stored in `shard_atlas_meta` beside the source hashes and bumped whenever the parser produces
|
||||
different data from identical files. Hashing the tree alone would strand an install whose maps never
|
||||
change on whatever an older build derived — a corrected parse would ship and never reach the data.
|
||||
|
||||
Four column choices worth stating, because each one is a trap:
|
||||
|
||||
- **`spawn_range`, not `range`**, and **`grp`, not `group`** — both are reserved words.
|
||||
@@ -449,6 +530,64 @@ artwork: sprites live in the operator's own client `.mul`/`.uop` files and are t
|
||||
redistribute. An operator supplies art via a gitignored map plus images under the (already
|
||||
gitignored) `server/uploads/atlas/`. Text-only is the normal, supported state.
|
||||
|
||||
### shard_clilocs / shard_cliloc_meta — UO's localization table (Protocol 3.0)
|
||||
|
||||
Items on the wire carry a `LabelNumber`, not a name. The bridge has always sent it —
|
||||
`char.profile.equipment.cliloc`, reward titles as a cliloc number in string form, and one per
|
||||
marketplace listing — but with no table to resolve it against, the character sheet could only render
|
||||
`id 1023721` where the game renders "quarter staff".
|
||||
|
||||
| Table | Shape |
|
||||
|---|---|
|
||||
| `shard_clilocs` | `number` INT PK, `flag`, `text` TEXT |
|
||||
| `shard_cliloc_meta` | Singleton (`id = 1`), `payload` JSON (source file, sha256, count, `parserVersion`), `imported_at` |
|
||||
|
||||
Import-owned and all-or-nothing in one transaction, same contract as the atlas — including **`DELETE`,
|
||||
not `TRUNCATE`**, for the same reason.
|
||||
|
||||
**Sourced from files the operator supplies**, at a path from the `cliloc_client_path` setting falling
|
||||
back to `UO_CLIENT_PATH`. Nothing client-derived is committed: UO's strings are EA's, exactly as the
|
||||
creature sprites are. A shard with nothing configured is fully supported — names render as ids. Full
|
||||
design and operator guide: [`CLILOCS.md`](CLILOCS.md).
|
||||
|
||||
**It reads a SET of sources, not one file**, because shards edit items and add new ones and those
|
||||
carry cliloc ids no stock client table has. A base (the converted client table) plus every overlay
|
||||
under `custom/` are re-read on every boot and hash-gated **together**, exactly as the atlas re-reads
|
||||
`Regions.xml` + `Locations/*.xml` + `Spawns/*.xml` + `ChampionSpawns.xml`. Later sources win, so an
|
||||
overlay both adds ids and overrides stock ones, and adding one custom item never means re-exporting a
|
||||
5 MB client file. Scale, measured on the live shard: its script tree references 16,434 cliloc ids and
|
||||
only 37 are absent from stock — tens of entries against a 67k base, which is why this is an overlay
|
||||
and not a second table.
|
||||
|
||||
The conversion step is not avoidable: **every current client ships its cliloc files compressed**
|
||||
(first DWORD's high byte `0x8E`), and ServUO's own bundled `Ultima.StringList` cannot read that
|
||||
either — so the shard cannot supply names on our behalf. The plain layout and a delimited text export
|
||||
are both accepted, sniffed by header rather than extension.
|
||||
|
||||
Three decisions worth stating:
|
||||
|
||||
- **`text` is TEXT, not VARCHAR.** Long property descriptions reach 12 KB. The index that matters for
|
||||
marketplace search is the denormalized `shard_vendor_items.display_name`, not this table.
|
||||
- **Blank entries are dropped at import** — 123,490 parsed → **67,496** stored. Roughly half a cliloc
|
||||
table is empty strings for ids the client reserves and never uses; a row that resolves to no name is
|
||||
indistinguishable from no row at all, and dropping them makes the binary and text imports converge
|
||||
on identical content.
|
||||
- **Two refusals, one of them the atlas's.** A corrupt source fails the parse on a truncated record,
|
||||
so it is caught outright and leaves the previous table serving. But a source that has **vanished**
|
||||
parses perfectly and imports a table quietly missing everything it contributed — an unmounted volume
|
||||
and a deliberate deletion are indistinguishable from here, which is precisely the ambiguity the
|
||||
atlas stages a facet removal for. So it is escalated: `status: 'needsReview'`, nothing applied,
|
||||
`missingSources` reported by both the import and `status()`, and an admin accepts it with
|
||||
`{approve:true}`. That is a flag rather than the atlas's approve/reject pair because the atlas
|
||||
stores a pending decision so that approving **re-parses** the tree; here nothing is stored, so
|
||||
re-reading at approval time is automatic.
|
||||
|
||||
**Resolution is server-side and there is no public route.** The table is never served *as* a table:
|
||||
67k rows would dwarf any page using them, and the Android client consumes the same already-resolved
|
||||
JSON. `resolveMany()` returns only ids that resolved to something displayable — placeholders like
|
||||
`~1_val~` are stripped, since the bridge sends the id and never the property packet that carries the
|
||||
arguments — and it never throws, because a cliloc lookup is decoration on a character sheet.
|
||||
|
||||
---
|
||||
|
||||
## 4. API contract
|
||||
@@ -463,7 +602,7 @@ are authoritative, and they answer different questions:
|
||||
|
||||
| Artifact | Source of truth for | Generated by |
|
||||
|---|---|---|
|
||||
| `server/routes.manifest.json` — mirrored as [api-route-inventory.json](./api-route-inventory.json) | **What URLs exist.** 200 public routes + 2 on the internal listener, sorted, method + path only. | `npm run routes:manifest`, by walking the live Express stack |
|
||||
| `server/routes.manifest.json` — mirrored as [api-route-inventory.json](./api-route-inventory.json) | **What URLs exist.** 215 public routes + 2 on the internal listener, sorted, method + path only. | `npm run routes:manifest`, by walking the live Express stack |
|
||||
| `server/swagger/swagger-output.json` — served at `/api/docs` | **What each route means.** Parameters, bodies, response codes, security. | `npm run swagger`, from `#swagger.*` annotations |
|
||||
|
||||
The split is deliberate: Swagger is annotation-derived, so an unannotated route is invisible in it and
|
||||
@@ -648,7 +787,18 @@ from the per-route **siteMode** middleware (§5), never from an auth gate.
|
||||
| GET | `/wiki/:slug` | single page |
|
||||
| POST | `/contact` | (rate-limited) send mail via SMTP; if unconfigured, respond `{fallback:"mailto", email}` |
|
||||
| GET | `/shard/ruleset` | the shard's own published ruleset (Protocol 3.0 `world.ruleset`): expansion, which optional systems are on, skill/stat caps, account and house limits, champion scroll rules, the save/restart schedule. Served from `shard_ruleset`, so it renders while the shard is down; live via `world.ruleset` on `/shard/stream`. Behind `requireFeature('ruleset')`. **`null`** means the shard has never published one — a real answer, distinct from a published ruleset. `caps.skill` / `caps.totalSkill` are in **tenths** (1000 = 100.0). |
|
||||
| GET | `/shard/points` | every points/loyalty leaderboard the shard publishes (Protocol 3.0 `points.board`) — Queen's Loyalty, Void Pool, the nine city loyalties, Clean Up Britannia, … Served from `shard_points_boards`, so it renders while the shard is down; live via `points.board` on `/shard/stream`. Behind `requireFeature('leaderboards')`, ordered by display name. **`maxPoints: 0` means uncapped** (the common case), and `nameString` is usually `null` with `nameNumber` holding a cliloc — resolve client-side or humanise the `system` key. |
|
||||
| GET | `/shard/points/:system` | one board by the shard's `PointsType` name (e.g. `QueensLoyalty`); `:system` must match `/^[A-Za-z][A-Za-z0-9_]{0,47}$/` or **400** before any query runs. **404** = the shard has never published that system, which is distinct from a published board nobody has scored in yet (**200** with an empty `top`). |
|
||||
| GET | `/shard/market?q=&minPrice=&maxPrice=&itemId=&map=®ion=&sort=&limit=&offset=` | search the player-vendor marketplace (Protocol 3.0 `vendor.listing`). Returns **listings**, not vendors — "who sells X and for how much" is the question, and a vendor-shaped result would make every caller flatten the shops back out. Served from `shard_vendors` + `shard_vendor_items`, so it renders while the shard is down. Behind `requireFeature('market')` **and rate-limited** — the first genuinely expensive public read on the site (a `LIKE` scan plus a `COUNT` over what is typically the largest `shard_*` table, reachable with no session). `sort ∈ {price_asc, price_desc, recent}`. `q` matches the resolved display name **or** the item's literal name, with `%`/`_` escaped: they are `LIKE` metacharacters, not SQL ones, so parameterization alone would let `?q=%` match every listing on the shard. Every response repeats `staleAt` (the oldest vendor row) because the shard sweeps round-robin — a banner that ages with the results it labels, not one fetched once. |
|
||||
| GET | `/shard/market/meta` | index size, staleness (`staleAt`/`freshAt`) and which facets and regions actually hold vendors, so a client builds its filters without running a search it will discard. |
|
||||
| GET | `/shard/market/vendors/:serial` | one shop and its listings; `:serial` must match `/^0x[0-9A-Fa-f]{1,16}$/` or **400** before any query runs. **404** = a serial the index has never seen, which also covers a vendor since dismissed or hidden — to an anonymous caller those are the same answer, and distinguishing them would leak that a hidden vendor exists. `truncated` (with `total` exceeding `count`) means the shop holds more than the shard publishes per frame. |
|
||||
| GET | `/shard/features` | the shard features **this caller** may reach plus the audience rung they resolved to (§6.5), so a client hides nav it can't follow. Reports only what the caller can see — the list itself never discloses a gated feature. Consumed by the SPA header and (pending) the Android nav. |
|
||||
| GET | `/atlas/creatures?q=&facet=&limit=&offset=` | the bestiary, most numerous first, with an unpaginated `total`. Static content parsed from the shard's ServUO tree — **not** sidecar-backed, which is why the atlas sits outside `/shard`, and unlike `/shard/*` it **is** site-mode gated. Behind `requireFeature('atlas')`. `?facet=` is matched exactly and never validated against a list (no facet name exists in the code); the filter is an `EXISTS` over the points rather than a JSON path or `JSON_SEARCH` built from caller input, whose `%`/`_` wildcards would make `?facet=%` match everything. |
|
||||
| GET | `/atlas/creatures/:slug` | one creature: `places` (the point-in-rect aggregate — "lizardman → Shrines, Isamu-Jima, Yew"), `spawners` (the bounded raw list, with `spawnersTruncated`), `alsoHere`. **`points` is a COUNT and `spawners` is the LIST** — named apart so one key never means a number on one route and an array on another. `minDelay`/`maxDelay` are in **seconds**, normalised at parse time from the source's per-record minutes-or-seconds. 404 = no such creature in this atlas. |
|
||||
| GET | `/atlas/regions?facet=&q=` | named regions and the rectangles that placed each spawner |
|
||||
| GET | `/atlas/landmarks?facet=&q=` | points of interest, labelled by `group` ("Covetous", not "Level 1") |
|
||||
| GET | `/atlas/champions?facet=` | the **configured** altar roster. Not `/shard/champs`, which is the live board. |
|
||||
| GET | `/atlas/meta` | facets, counts and when the atlas was parsed. Game-world facts only — the ServUO path, source hashes and any pending refresh are operator detail and live on the admin route. |
|
||||
|
||||
Public content GETs pass through the **siteMode** gate (§5).
|
||||
|
||||
@@ -692,6 +842,13 @@ file a route sits in — that is the property the route manifest freezes.
|
||||
| GET | `/users/:id/trusted-devices` | list a user's active trusted devices (never tokens) |
|
||||
| DELETE | `/users/:id/trusted-devices` · `…/:deviceId` | revoke all / one of a user's trusted devices (logs `admin.trusted_device.revoke[_all]`) |
|
||||
| POST | `/users/:id/mfa/reset` | recover a locked-out user: disable TOTP + revoke all trusted devices + clear recovery codes (logs `admin.user.totp.reset`) |
|
||||
| GET | `/shard/atlas` | spawn-atlas status (`adminOnly`): the ServUO path, whether the tree is readable, whether it has drifted from what is loaded, counts, facets, and any refresh staged for review. The public `/atlas/meta` reports the game world only; the filesystem detail is here. |
|
||||
| POST | `/shard/atlas/import` | re-import without restarting; `{force}` ignores the hash gate. **An unreadable tree answers 200 with `status:"unavailable"`, not 500** — `refresh()` reports outcomes rather than throwing (the boot path must never be blocked by a bad tree) and that contract is preserved at the API. |
|
||||
| POST | `/shard/atlas/approve` · `/shard/atlas/reject` | answer a refresh staged because it would REMOVE a facet. Approving **re-parses** the tree, so what lands matches it at approval time; rejecting is remembered against those source hashes so it does not re-prompt every restart. 404 when nothing is staged. |
|
||||
| PUT | `/shard/atlas/path` | point the atlas at a different tree (persisted as `spawn_atlas_servuo_path`, which wins over `SERVUO_PATH`). Blank clears it. Deliberately **does not import** — moving the mount and reloading the world are separate decisions — and returns fresh status so the panel can offer the import next. |
|
||||
| GET | `/shard/clilocs` | cliloc-table status (`adminOnly`): every source found now (base first, then `custom/` overlays in merge order), what each contributed at the last import, readability, drift across the set, the entry count, and `missingSources`. `configured:false` is a supported state — item names then render as ids. No public counterpart: the table is never served *as* a table. |
|
||||
| POST | `/shard/clilocs/import` | reload after a client patch or an overlay edit; `{force}` ignores the hash gate, `{approve}` accepts a **vanished** source (refused by default — see the table notes above). **A missing path — or the likely mistake of pointing at the client's own COMPRESSED `Cliloc.enu` — answers 200 with `status:"unavailable"` and a `code`, not 500.** `COMPRESSED` is called out by name: a 500 would say only "something broke", and the operator needs to be told which file to convert. |
|
||||
| PUT | `/shard/clilocs/path` | point the site at a different cliloc base file or directory (persisted as `cliloc_client_path`, which wins over `UO_CLIENT_PATH`). Overlays are read from `custom/` beside it either way. Blank clears it. Deliberately **does not import**, same reasoning as the atlas path. |
|
||||
|
||||
Every admin write logs to `activity_log`.
|
||||
|
||||
@@ -725,7 +882,11 @@ who"; `activity_log` provides the history feed.
|
||||
- **Cookie**: `httpOnly`, `sameSite=Lax`, `path=/`, and **`secure` decided per-request** (`COOKIE_SECURE=auto` → `secure: req.secure`).
|
||||
- **Trusted-device MFA.** A second, separate httpOnly cookie (`rg_trust`, default 30d) — opaque, sha256-hashed server-side in `trusted_devices` — lets a browser/app **skip the TOTP step** (never the password) on future logins. It is a server-side, per-row-revocable record (never a JWT claim), so the stateless session JWT is unchanged and trust stays revocable. It only ever gates the **second factor**; it deliberately outlives logout, and is cleared on untrust / password change / password reset / TOTP disable. **Recovery codes** (bcrypt, single-use) are the 2FA-lockout fallback. All admin trusted-device/MFA actions and the self actions (`auth.login.trusted_device`, `account.trusted_device.*`, `account.recovery_code*`, `admin.trusted_device.*`, `admin.user.totp.reset`) are audit-logged. See `docs/website/TRUSTED_DEVICES_MFA.md`. This is the key to dual access: the cookie is `Secure` when reached through Pangolin (HTTPS, `X-Forwarded-Proto: https`) but **not** `Secure` when reached directly over the LAN IP on plain HTTP — so login works in both. `COOKIE_SECURE=true|false` can force it. Requires `trust proxy` (below). `localhost:5173` (Vite) and `localhost:3000` are same-site, so the cookie flows in dev too.
|
||||
- **bcrypt** hashing (cost 10+); plaintext passwords never stored, logged, or returned.
|
||||
- **Rate limiting** (`express-rate-limit`) on `/auth/login` and `/public/contact`.
|
||||
- **Rate limiting** (`express-rate-limit`) on `/auth/login`, `/public/contact`, and — the only limited
|
||||
*read* — `/public/shard/market` and `/public/shard/market/vendors/:serial` (60/min/IP). Every other
|
||||
public read is an indexed lookup of bounded size; the marketplace search is a `LIKE` scan plus a
|
||||
`COUNT` over the largest `shard_*` table, anonymous by default, so it is the one public GET that is
|
||||
worth money to serve.
|
||||
- **Validation** (`express-validator`) on all writes; centralized error handler.
|
||||
- **helmet** with a Content-Security-Policy tuned for the built React SPA. The policies now live in
|
||||
**`server/src/config/csp.js`** (`app.js` only wires them up):
|
||||
|
||||
296
website/CLILOCS.md
Normal file
296
website/CLILOCS.md
Normal file
@@ -0,0 +1,296 @@
|
||||
# Cliloc table (item and title names)
|
||||
|
||||
**Status:** Complete on `edge` — website [#115](https://gitea.whitlocktech.com/RunicGateway/website/pulls/115), docs [#70](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/70).
|
||||
**Design:** [`docs/link/v3.md` §8.6](../link/v3.md) — Protocol 3.0, the dependency Part B/3 was sequenced behind.
|
||||
|
||||
A "cliloc" is UO's localization table: an integer id mapped to a display string.
|
||||
**Items on the wire carry a `LabelNumber`, not a name.** The bridge has always
|
||||
sent that number — `char.profile.equipment` has a `cliloc` field, reward titles
|
||||
arrive as a cliloc number in string form, and every marketplace listing carries
|
||||
one — but the site had no table to look it up in, so a character sheet could only
|
||||
render `id 1023721` where the game renders **"quarter staff"**.
|
||||
|
||||
The number was never the missing piece. The table was.
|
||||
|
||||
## Why the operator has to convert the file
|
||||
|
||||
This is the awkward part, and it is not avoidable:
|
||||
|
||||
**Every current UO client ships its cliloc files compressed.** All eight
|
||||
`Cliloc.*` files in a modern client (`chs`, `cht`, `deu`, `enu`, `esp`, `fra`,
|
||||
`jpn`, `kor`) begin with a DWORD whose high byte is `0x8E` — the "Mythic"
|
||||
compressed container. The plain layout this site parses is what those files
|
||||
looked like *before* that change.
|
||||
|
||||
Decompressing it means an inverse-BWT coder with a frequency header — a few
|
||||
hundred lines of bit-level work whose failure mode is plausible-looking garbage
|
||||
rather than an error. The site has no business carrying that at runtime.
|
||||
|
||||
Two facts make the alternatives worse, not better:
|
||||
|
||||
- **ServUO cannot read it either.** Its bundled `Ultima.StringList` implements
|
||||
only the plain layout, so on a modern client `VendorSearch.StringList` is null
|
||||
and `VendorSearch.GetItemName` returns `item.Name` — usually nothing. The
|
||||
shard cannot supply names on our behalf; the in-game Vendor Search gump has the
|
||||
same gap.
|
||||
- **Nothing client-derived may be committed.** UO's strings are EA's. The repo
|
||||
ships no string table for the same reason it ships no artwork and no map
|
||||
snapshot — see [`SPAWN_ATLAS.md`](SPAWN_ATLAS.md).
|
||||
|
||||
So the conversion happens **once, on the operator's machine, against their own
|
||||
client**, and the site reads the result from a path it is given. A shard that
|
||||
never does this is in a fully supported state: names render as ids, exactly as
|
||||
they did before the table existed.
|
||||
|
||||
## Converting
|
||||
|
||||
Either format below is accepted; the site sniffs which one it was handed.
|
||||
|
||||
| Format | Fidelity | Notes |
|
||||
|---|---|---|
|
||||
| **Plain binary** (recommended) | Exact | 6-byte header, then `{int32 number, byte flag, uint16 length, UTF-8}` records |
|
||||
| Delimited text | Loses leading/trailing whitespace | `number<TAB\|,\|;>text` per line; a header row, blank lines and `#` comments are ignored |
|
||||
|
||||
The whitespace caveat is real but cosmetic: ~1,300 of the 123,490 entries in a
|
||||
stock `Cliloc.enu` are label prefixes like `"max = "` whose trailing space is
|
||||
meaningful when the client concatenates a value onto them. Nothing on this site
|
||||
concatenates, and every consumer passes through `displayText()`, which trims.
|
||||
|
||||
### Using the bundled tool
|
||||
|
||||
`server/tools/cliloc-export/` is a small .NET console app that drives
|
||||
[UOFiddler](https://github.com/polserver/UOFiddler)'s `Ultima.dll` — the
|
||||
decompressor that already exists and is already maintained — and writes the plain
|
||||
format. It loads that DLL **reflectively** so it compiles against any SDK, and it
|
||||
writes the records by hand because UOFiddler's own `SaveStringList` *re-compresses*
|
||||
on save (its purpose is round-tripping a file back into the client, so its output
|
||||
is byte-identical to its input — a trap worth knowing about).
|
||||
|
||||
```bash
|
||||
cd website/server/tools/cliloc-export
|
||||
dotnet build -c Release
|
||||
|
||||
# binary (recommended)
|
||||
dotnet run -- "<UOFiddler>/Ultima.dll" "<UO client>/Cliloc.enu" /srv/uo-data/clilocs.plain
|
||||
|
||||
# or tab-delimited
|
||||
dotnet run -- "<UOFiddler>/Ultima.dll" "<UO client>/Cliloc.enu" /srv/uo-data/clilocs.tsv --tsv
|
||||
```
|
||||
|
||||
A UOFiddler GUI export works equally well — anything producing one of the two
|
||||
shapes above is fine.
|
||||
|
||||
## Shard-added and shard-edited items
|
||||
|
||||
**Shards edit items and add new ones**, and those carry cliloc ids no stock
|
||||
client table has. The table is therefore built from a **set** of sources, all
|
||||
re-read on every boot and hash-gated together — the same shape as the spawn
|
||||
atlas, which reads `Regions.xml` + `Locations/*.xml` + `Spawns/*.xml` +
|
||||
`ChampionSpawns.xml` and merges them:
|
||||
|
||||
```
|
||||
<cliloc path>/
|
||||
clilocs.plain ← base: the converted client table
|
||||
custom/
|
||||
01-uomysticmoon.tsv ← overlays: shard additions and overrides
|
||||
02-events.tsv
|
||||
```
|
||||
|
||||
Overlays use the same delimited-text format, are read in **sorted order**, and
|
||||
**later sources win** — so an overlay both *adds* ids the client never had and
|
||||
*overrides* stock ones the shard has re-purposed. Any `.tsv`, `.csv`, `.txt`,
|
||||
`.enu` or `.plain` file in `custom/` is picked up; anything else (a `README.md`,
|
||||
say) is ignored.
|
||||
|
||||
Adding, editing or removing any overlay counts as drift, so a new custom item
|
||||
needs only a file edit and a restart — or the admin panel's Import button.
|
||||
**Adding one item never means re-exporting a 5 MB client file.**
|
||||
|
||||
The import result reports what each source contributed, which is how you confirm
|
||||
an overlay took effect — `overrode: 0` on a file meant to re-label stock items
|
||||
says it did not:
|
||||
|
||||
```json
|
||||
"sources": [
|
||||
{ "label": "clilocs.plain", "kind": "base", "entries": 123490, "added": 123490, "overrode": 0 },
|
||||
{ "label": "custom/uomysticmoon.tsv", "kind": "custom", "entries": 2, "added": 1, "overrode": 1 }
|
||||
]
|
||||
```
|
||||
|
||||
**Why a convention rather than discovery.** Everywhere else this pipeline follows
|
||||
the shard's own files, but **ServUO has no server-side notion of a custom
|
||||
cliloc** — they live in the patched client a shard distributes to its players,
|
||||
and nothing in the tree declares them. There is nothing to discover, so `custom/`
|
||||
is the one thing here that is our convention rather than the shard's. (An
|
||||
operator who *does* patch their client cliloc needs no overlay at all: convert
|
||||
the patched file and their edits are simply in the base.)
|
||||
|
||||
Measured on the live shard for scale: its script tree references **16,434** cliloc
|
||||
ids and only **37** are absent from the stock client table — tens of entries
|
||||
against a 67k base, which is what makes an overlay the right shape rather than a
|
||||
second full table.
|
||||
|
||||
## Configuring the path
|
||||
|
||||
Two ways to point at the sources, the setting winning over the environment:
|
||||
|
||||
| Source | Notes |
|
||||
|---|---|
|
||||
| `cliloc_client_path` setting | Admin-editable (Admin → Shard); takes effect on the next refresh without a redeploy |
|
||||
| `UO_CLIENT_PATH` env var | The deploy-time default, since the path usually describes a mount the deployment sets up |
|
||||
|
||||
The value may be **the base file itself or a directory to search**, because both
|
||||
are natural answers to "where is it". Overlays are read from a `custom/`
|
||||
directory beside the base **either way** — pointing at a file does not forfeit
|
||||
them.
|
||||
|
||||
A directory is searched case-insensitively (the client writes `Cliloc.enu` on
|
||||
Windows; the site usually runs on Linux) for, in order: `clilocs.tsv`,
|
||||
`clilocs.csv`, `clilocs.plain`, `cliloc.plain`, `cliloc.plain.enu`,
|
||||
`cliloc.enu.plain`, `clilocs.txt`, `cliloc.enu`.
|
||||
|
||||
That ordering puts explicitly-converted names first on purpose. Pointing the
|
||||
setting straight at an unconverted client directory finds `cliloc.enu`, which is
|
||||
compressed — and the site says so by name rather than failing obscurely:
|
||||
|
||||
```
|
||||
status: unavailable
|
||||
code: COMPRESSED
|
||||
reason: This is a compressed (Mythic-format) cliloc file, which the site cannot
|
||||
read. Convert it to the plain format first — see docs/website/CLILOCS.md.
|
||||
```
|
||||
|
||||
## Refresh contract
|
||||
|
||||
Identical in shape to the spawn atlas, and for the same reasons:
|
||||
|
||||
- **It never blocks startup.** No path, an unreadable file, a wrong-format file,
|
||||
a database error — all caught and logged. The site comes up either way.
|
||||
- **Hash-gated.** The boot path hashes the file and skips the parse entirely when
|
||||
it matches what is loaded, which is every restart that did not follow a client
|
||||
patch. Measured on a stock table: **14 ms** for the no-op, **663 ms** for a full
|
||||
parse and replace.
|
||||
- **A `PARSER_VERSION` bump also counts as drift**, so a corrected parse reaches
|
||||
an install whose client never patches.
|
||||
|
||||
### Two ways a refresh is refused
|
||||
|
||||
**A corrupt file** — the realistic failure for any single source — makes the
|
||||
parser fail on a truncated record rather than yield a plausible-but-short table,
|
||||
so it is caught outright. Verified: a file truncated to half its length reports
|
||||
|
||||
```
|
||||
code: TRUNCATED
|
||||
reason: Truncated record header at byte 2486759 (74909 entries read)
|
||||
```
|
||||
|
||||
and the rows already loaded are untouched. A malformed overlay names the file it
|
||||
came from (`custom/broken.tsv: No cliloc entries found…`), because "which of my
|
||||
six overlay files is broken" is otherwise a guessing game.
|
||||
|
||||
**A source that has VANISHED** is the hazard a single file did not have. It
|
||||
parses perfectly and imports a table quietly missing everything that file
|
||||
contributed — and an unmounted volume looks exactly like a deliberate deletion
|
||||
from here. This is the same ambiguity the atlas stages a facet removal for, so it
|
||||
is escalated rather than applied:
|
||||
|
||||
```
|
||||
status: needsReview
|
||||
reason: 1 previously-loaded cliloc source(s) are missing;
|
||||
the existing table is unchanged
|
||||
missingSources: ["custom/uomysticmoon.tsv"]
|
||||
```
|
||||
|
||||
`status()` reports `missingSources` too, so the panel can show it before anyone
|
||||
clicks Import. An admin accepts it by re-running the import with
|
||||
`{ "approve": true }`.
|
||||
|
||||
**Why that is a flag and not the atlas's approve/reject pair.** The atlas stores
|
||||
a pending decision in its own table so that approving *re-parses the tree*, which
|
||||
is what keeps a multi-megabyte blob out of the database and makes the applied
|
||||
result match the tree at approval time. Here nothing is stored, so re-reading at
|
||||
approval time is automatic — the decision is a single boolean on the import an
|
||||
admin was already going to run.
|
||||
|
||||
## What gets stored
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Parsed from a stock `Cliloc.enu` | **123,490** entries |
|
||||
| Of those, empty strings | **55,994** (ids the client reserves and never uses) |
|
||||
| Stored in `shard_clilocs` | **67,496** |
|
||||
|
||||
Blank entries are dropped at import. A row resolving to no name is
|
||||
indistinguishable from no row at all to every caller, and dropping them makes the
|
||||
binary and text imports converge on **identical** content — the binary format
|
||||
carries the blanks explicitly and a text export may or may not, depending on the
|
||||
tool. Verified: both formats import to the same 67,496 rows with the same keys.
|
||||
|
||||
`text` is `TEXT`, not `VARCHAR`: the long property descriptions reach 12 KB, and
|
||||
silently truncating them would be worse than storing them. The index that matters
|
||||
for marketplace search is on the denormalized `shard_vendor_items.display_name`,
|
||||
not here.
|
||||
|
||||
## How names are applied
|
||||
|
||||
**Resolution happens server-side.** The table is never served *as* a table and
|
||||
there is no public route for it. Two reasons: 67k rows would dwarf any page that
|
||||
used them, and the Android client consumes the same JSON and would otherwise need
|
||||
its own copy.
|
||||
|
||||
`resolveMany()` takes a batch of ids and returns a `Map` holding only those that
|
||||
resolved to something displayable, so "no such id" and "id with no usable name"
|
||||
collapse into one branch at the call site. It never throws — a cliloc lookup is
|
||||
decoration on someone's character sheet, and a database blip must not fail the
|
||||
sheet. A capped in-process cache fronts it; measured cold **4.2 ms**, warm
|
||||
**0.015 ms**.
|
||||
|
||||
### `displayText()`
|
||||
|
||||
Cliloc strings interpolate arguments the client pulls from an item's property
|
||||
list — `~1_val~`, `~2_NAME~`. **We never have those**: the bridge sends the id,
|
||||
not the packet. So a name carrying them is reduced to what is actually knowable.
|
||||
|
||||
| Raw | Displayed |
|
||||
|---|---|
|
||||
| `quarter staff` | `quarter staff` |
|
||||
| `cold damage ~1_val~%` | `cold damage` |
|
||||
| `[~1_stuff~]` | *(nothing — the whole string was the argument)* |
|
||||
| `50%` | `50%` |
|
||||
| `Runic Gateway Sigil (v2)` | `Runic Gateway Sigil (v2)` |
|
||||
|
||||
**Punctuation is only tidied when a placeholder was actually removed.** The
|
||||
trailing `%` in row two is the unit belonging to the number we never had, and the
|
||||
brackets in row three only ever wrapped the argument — but a string with no
|
||||
placeholder has no such debris, and trimming it anyway corrupts real names. Rows
|
||||
four and five are the ones that caught it: a shard's custom
|
||||
`"Runic Gateway Sigil (v2)"` rendered as `"(v2"` while the bracket trim was
|
||||
unconditional.
|
||||
|
||||
### Consumers
|
||||
|
||||
- **Character sheet equipment.** `enrichCharProfile` attaches `clilocName` to each
|
||||
item. A player-given `name` always wins — "Bob's lucky axe" must not be
|
||||
relabelled "hatchet" — and the client re-states that precedence.
|
||||
- **Reward titles.** `titles.rewardResolved` is a parallel array with the numeric
|
||||
entries turned into words (`null` where nothing resolved). The sheet used to
|
||||
*skip* numeric reward titles entirely, having no way to render them.
|
||||
- **Marketplace listings** (Protocol 3.0 §8) denormalize the resolved name into
|
||||
`shard_vendor_items.display_name` so search can index it.
|
||||
|
||||
## Admin surface
|
||||
|
||||
All admin-only, alongside the atlas under Admin → Shard:
|
||||
|
||||
| Route | Purpose |
|
||||
|---|---|
|
||||
| `GET /api/v1/admin/shard/clilocs` | Sources found, what each contributed at the last import, readability, drift, entry count, `missingSources` |
|
||||
| `POST /api/v1/admin/shard/clilocs/import` | Reload after a client patch or an overlay edit; `{ "force": true }` reimports an unchanged set, `{ "approve": true }` accepts a vanished source |
|
||||
| `PUT /api/v1/admin/shard/clilocs/path` | Set the path; blank disables resolution |
|
||||
|
||||
A refresh **result is not an exception**: a missing file, or the likely mistake of
|
||||
pointing at the client's own compressed `Cliloc.enu`, answers `200` with
|
||||
`status: "unavailable"` and a reason. A `500` would say only "something broke";
|
||||
the operator needs to be told which file to convert. Setting the path
|
||||
deliberately does **not** import as a side effect — the response carries the
|
||||
refreshed status so the panel can offer that as the next step.
|
||||
167
website/MARKETPLACE.md
Normal file
167
website/MARKETPLACE.md
Normal file
@@ -0,0 +1,167 @@
|
||||
# Marketplace — the player-vendor index
|
||||
|
||||
**Status:** On `edge` — servuo-plugins [#5](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/5), link [#19](https://gitea.whitlocktech.com/RunicGateway/link/pulls/19), website [#116](https://gitea.whitlocktech.com/RunicGateway/website/pulls/116).
|
||||
**Design:** [`docs/link/v3.md` §8](../link/v3.md) — Protocol 3.0 Part B/3.
|
||||
**Depends on:** [`CLILOCS.md`](CLILOCS.md) — without a cliloc table, listings render as item ids.
|
||||
|
||||
The marketplace is a searchable index of every player vendor on the shard: what
|
||||
each shop is selling, for how much, and where it is standing. It is the same set
|
||||
the in-game **Vendor Search** gump reads, offered from outside the game — so a
|
||||
player can find the vanquishing kryss they want before logging in, and someone
|
||||
who does not play at all can see that the economy exists.
|
||||
|
||||
Page: `/site/market`, plus `/site/market/vendors/:serial` for one shop.
|
||||
|
||||
## Three things the pages must say out loud
|
||||
|
||||
Everything below follows from how the data is gathered, and each has a visible
|
||||
consequence the UI is required to surface.
|
||||
|
||||
**1. The prices are not live.** The shard sweeps vendors **round-robin** — at
|
||||
most `Bridge.MarketSweepBatch` shops per tick — so a given shop can be a full
|
||||
cycle behind. The page carries a *"prices last refreshed N minutes ago"* banner
|
||||
driven by the **oldest** vendor row, not the newest: the one stale shop is the
|
||||
one that wastes somebody's trip.
|
||||
|
||||
**2. A shop can be truncated.** `Bridge.MarketMaxListings` (250 by default) caps
|
||||
how many listings one frame carries. A commodity reseller with thousands of
|
||||
stacked resources is a real thing, and an uncapped frame for one is measured in
|
||||
megabytes. Over the cap the shop reports `truncated`, and the vendor page says
|
||||
*"showing 250 of 3,104 — this shop holds more than the shard publishes"* rather
|
||||
than presenting a partial shop as complete.
|
||||
|
||||
**3. An item may have no name.** Items on the wire carry a cliloc id, not a name.
|
||||
On a shard whose operator has not converted a cliloc table
|
||||
([`CLILOCS.md`](CLILOCS.md)) the honest render is the item id — never an invented
|
||||
label, which would be indistinguishable from a real one.
|
||||
|
||||
## Privacy: the player's own toggle wins
|
||||
|
||||
Only vendors whose owner left the in-game **Vendor Search** flag ON are ever sent
|
||||
to the site. A player who hides their shop in game is hidden here too, and no
|
||||
admin setting overrides that. When they hide one that was already indexed, the
|
||||
shard emits `vendor.listing.remove` and the row is deleted — so revoking consent
|
||||
takes effect, it does not merely stop refreshing.
|
||||
|
||||
Shop name, owner character name and location default to **Everyone**, because the
|
||||
stock Vendor Search gump already shows exactly that set to any player in game.
|
||||
They remain admin-configurable; see [`SHARD_VISIBILITY.md`](SHARD_VISIBILITY.md).
|
||||
Account names and website user ids never cross the wire at all.
|
||||
|
||||
## How it is put together
|
||||
|
||||
```
|
||||
ServUO uo-link sidecar website
|
||||
────── ─────────────── ───────
|
||||
BridgeMarket.cs vendors table shard_vendors
|
||||
round-robin sweep ──────► (whole frame blob) ──────► shard_vendor_items
|
||||
per-vendor diff GET /market (paged) + display_name
|
||||
vendor.listing resolved at ingest
|
||||
vendor.listing.remove
|
||||
```
|
||||
|
||||
**The shard side** walks at most `MarketSweepBatch` vendors per tick from a
|
||||
persistent cursor, diffs each against what it last published, and emits a whole
|
||||
frame for any shop that moved. Per-tick cost is therefore bounded by the batch,
|
||||
not by how many vendors the world holds — full coverage takes
|
||||
`ceil(vendors / batch) × MarketSweepSeconds`.
|
||||
|
||||
**The sidecar** stores each frame whole and serves `GET /market`, its only paged
|
||||
read. It normalizes nothing and defines no audiences: it is a dumb forwarder, and
|
||||
search is the website's job.
|
||||
|
||||
**The website** splits each frame into a vendor row and its listings, replacing
|
||||
that vendor's whole listing set inside one transaction (the frame is
|
||||
authoritative for that vendor, never a delta). Item names are resolved against
|
||||
the cliloc table **on the way in** and stored denormalized, which is what makes
|
||||
search-by-name possible and keeps the cliloc table off the hot path.
|
||||
|
||||
## Operating it
|
||||
|
||||
Everything is in `Config/Bridge.cfg` on the shard. There is nothing to configure
|
||||
on the website.
|
||||
|
||||
| Setting | Default | What it does |
|
||||
|---|---|---|
|
||||
| `MarketEnabled` | `true` | Master switch. Off publishes nothing; the page shows an empty index. |
|
||||
| `MarketSweepSeconds` | `60` | Tick interval. |
|
||||
| `MarketSweepBatch` | `25` | Vendors inventoried per tick. Clamped 1..500. |
|
||||
| `MarketMaxListings` | `250` | Per-shop listing cap, after which `truncated`. Clamped 1..5000. |
|
||||
|
||||
**Faster coverage vs. per-tick cost.** Lowering `MarketSweepSeconds` or raising
|
||||
`MarketSweepBatch` both refresh the index sooner and both cost more per tick.
|
||||
The expensive part is the item walk, which recurses into every container a vendor
|
||||
is selling — so a shard of big shops should raise the interval rather than the
|
||||
batch.
|
||||
|
||||
`[bridge status` reports the sweep, including `lastMs` and `maxMs`:
|
||||
|
||||
```
|
||||
market(enabled=True sweeps=42 scanned=108 emitted=27 removed=0 skipped=0
|
||||
truncated=0 tracked=27 vendors=27 cursor=2 batch=25 lastMs=0.31 maxMs=15.40)
|
||||
```
|
||||
|
||||
A tick over **50 ms** prints a rate-limited warning naming the knob:
|
||||
|
||||
```
|
||||
[Bridge] market sweep took 82.4 ms (budget 50 ms) - lower Bridge.MarketSweepBatch (now 25) if this persists
|
||||
```
|
||||
|
||||
Measured on a shard with 27 vendors × 40 listings (209k items, 43k mobiles):
|
||||
**15.4 ms** for the first cold tick of 25 vendors, **0.3 ms** in steady state —
|
||||
the diff is what makes an unchanged world nearly free. Note the arithmetic: 25
|
||||
*full* shops at the 250-listing cap is 6,250 items ≈ 95 ms, over budget. Real
|
||||
shops hold tens, which is why 25 is the default and why the warning exists.
|
||||
|
||||
`[bridge sweepnow` runs one tick immediately; `[bridge reload` re-reads the
|
||||
settings above without a restart.
|
||||
|
||||
## Names arriving late
|
||||
|
||||
Item names come from the cliloc table, and the market sweep will **not** re-send
|
||||
an unchanged shop just because the site learned what its items are called. So a
|
||||
cliloc import triggers a bulk re-resolution of every stored listing — otherwise
|
||||
an operator who configures clilocs after the first sweep would see item ids until
|
||||
every shop happened to change on its own. It runs after a boot import and after
|
||||
an admin import, takes ~50 ms per thousand listings, and never throws: a failure
|
||||
leaves names exactly as they were.
|
||||
|
||||
## API
|
||||
|
||||
All under `/api/v1/public/shard`, gated by the `market` feature and
|
||||
**rate-limited** — these are the first genuinely expensive public reads on the
|
||||
site (a `LIKE` scan plus a `COUNT` over what is typically the largest `shard_*`
|
||||
table, reachable with no session).
|
||||
|
||||
| Route | What |
|
||||
|---|---|
|
||||
| `GET /market` | Search. Returns **listings**, not vendors — "who sells X and for how much" is the question. `?q=&minPrice=&maxPrice=&itemId=&map=®ion=&sort=&limit=&offset=`, `sort ∈ {price_asc, price_desc, recent}`. |
|
||||
| `GET /market/meta` | Index size, staleness, and which facets and regions actually hold vendors — so a client builds its filters without running a search it will discard. |
|
||||
| `GET /market/vendors/:serial` | One shop and its listings. **404** for a serial the index has never seen, which also covers a vendor since dismissed or hidden — to an anonymous caller those are the same answer. |
|
||||
|
||||
`q` matches the resolved display name **or** the item's own literal name, because
|
||||
an item with a player-set name (most of what is worth searching for on a
|
||||
player-run shard) may carry a generic cliloc. `%` and `_` in a query are escaped:
|
||||
they are `LIKE` metacharacters, not SQL ones, so parameterization alone would let
|
||||
a search for `%` match every listing on the shard.
|
||||
|
||||
Full schemas are in the OpenAPI spec (`ShardMarketPage`, `ShardMarketVendor`,
|
||||
`ShardMarketMeta`, `ShardMarketListing`, `ShardMarketLocation`).
|
||||
|
||||
## Tables
|
||||
|
||||
`shard_vendors` (one row per shop) and `shard_vendor_items` (one row per priced
|
||||
listing). Both are ingest-owned; nothing else writes to them. No foreign keys,
|
||||
in keeping with every other `shard_*` table — the ingest transaction is what
|
||||
keeps them consistent, and an FK would turn a malformed frame into a failed write
|
||||
rather than a dropped row.
|
||||
|
||||
There is deliberately **no `payload` column** on `shard_vendors`, unlike the
|
||||
points board next door. The board's top-N is a fixed-size list read whole, so it
|
||||
lives in JSON; here the items *are* the searchable rows, so they are normalized
|
||||
and there is nothing left worth duplicating. The sidecar keeps the whole blob,
|
||||
because outage resilience is its job.
|
||||
|
||||
`shard_vendor_items.display_name` is denormalized and indexed (alone, and
|
||||
composite with `price` for "cheapest matching X"). See "Names arriving late"
|
||||
above for how it is kept current.
|
||||
@@ -60,12 +60,32 @@ board while holding back one column. See the table in §3.
|
||||
| **Shard rules** | Skill/stat caps, house limits, vet rewards, the ruleset | Everyone | Connect address → Everyone |
|
||||
| **Spawn atlas** | Bestiary and spawn locations (static content) | Everyone | — |
|
||||
| **Leaderboards** | Point and loyalty standings | Everyone | Character names → Everyone |
|
||||
| **Marketplace** | The shard-wide player-vendor index | Everyone, **live updates off** | Vendor owner name → Everyone · Location → Everyone |
|
||||
| **Marketplace** | The shard-wide player-vendor index | Everyone, **live updates off** | Vendor owner name → Everyone · Vendor owner character id → Everyone · In-game location → Everyone |
|
||||
|
||||
**Why the marketplace ships with live updates off.** A live feed of every vendor's full inventory
|
||||
would be the single largest thing the site sends. No page needs it — the marketplace is a search over
|
||||
stored data with a “prices last refreshed N minutes ago” stamp. Turn it on only if you want it.
|
||||
|
||||
**Why the marketplace's fields default to Everyone.** A vendor's shop name, its owner's character
|
||||
name and where it is standing are *already* visible to every player in game: the stock Vendor Search
|
||||
gump surfaces exactly that set to anyone who opens it. Publishing them on the site is not a new
|
||||
disclosure. They stay configurable because a shard may still prefer to keep its economy behind a
|
||||
login — and because "already public in game" is a judgement about your shard, not ours.
|
||||
|
||||
**Location is one setting covering four things.** Hiding it removes the facet, the coordinates, the
|
||||
region *and* the house name together. That is deliberate: those are four ways of saying the same
|
||||
thing, and a setting that hid the coordinates while publishing the house name would not have hidden
|
||||
anything.
|
||||
|
||||
**Hiding the owner name also hides the owner character id.** They are separate settings so you can
|
||||
be explicit, but leaving the id published while hiding the name achieves nothing — the leaderboards
|
||||
and guild boards resolve that same id back to a character name. Set both.
|
||||
|
||||
**What hiding a vendor cannot do.** Only vendors whose owner left the in-game *Vendor Search* flag ON
|
||||
are ever sent to the site, so a player who hides their shop in game is hidden here too — and no
|
||||
setting on this page can override that. It works the other way as well: these settings control who
|
||||
sees the index, not whether players can find each other's shops in game.
|
||||
|
||||
**Why house owner/price default to Staff.** The public Houses page has always been a "where are the
|
||||
falling houses" board — location only. Owner and price are the staff view. That split is preserved.
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Spawn atlas
|
||||
|
||||
**Status:** Data pipeline landed on `edge` (website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112)); API and client pages follow in a second PR.
|
||||
**Status:** Complete on `edge` — data pipeline in website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112), API + pages in website [#113](https://gitea.whitlocktech.com/RunicGateway/website/pulls/113).
|
||||
**Design:** [`docs/link/v3.md` §6](../link/v3.md) — Protocol 3.0 Part C.
|
||||
|
||||
The spawn atlas is a browsable catalogue of what the shard *contains*: which
|
||||
@@ -91,7 +91,7 @@ A rejection is remembered against those exact source hashes, so a declined
|
||||
refresh does not re-prompt on every restart. Change the tree and the hashes
|
||||
differ, which asks again.
|
||||
|
||||
From the admin panel (second PR), or from the CLI:
|
||||
From **Admin → Spawn Atlas**, or from the CLI:
|
||||
|
||||
```bash
|
||||
cd website/server
|
||||
@@ -259,3 +259,96 @@ Parsing notes:
|
||||
- `<Objects2>` is `Type:MX=n:SB=…` segments joined by `:OBJ=`. Split on `:OBJ=`
|
||||
*first* — a naive `split(':')` shreds it. A single Trammel point carries six
|
||||
types.
|
||||
- **Respawn delays are stored in two different units, per record.** XmlSpawner
|
||||
writes `MinDelay`/`MaxDelay` in minutes, and switches to seconds only when a
|
||||
spawner's delay does not divide into whole minutes — flagging that with
|
||||
`DelayInSec` on the same record. A `5` therefore means five *minutes* on one
|
||||
spawner and five *seconds* on the next, and both are plausible respawn times,
|
||||
so a reader assuming either unit is silently wrong about the other. Stock
|
||||
ServUO 57.4 has ~170 second-flagged spawners out of 6,455. The parser
|
||||
normalises everything to **seconds**; the API and UI carry seconds throughout.
|
||||
|
||||
### The parser version
|
||||
|
||||
`spawnAtlasSource.js` exports `PARSER_VERSION`, stored in `shard_atlas_meta`
|
||||
alongside the source hashes and bumped whenever the parser derives **different
|
||||
data from identical files** — a fixed misreading, a new field, a changed unit.
|
||||
|
||||
A refresh re-derives when the tree changed **or** the parser did. Hashing the
|
||||
tree alone would be a trap: an install whose maps never change would keep serving
|
||||
whatever an older build derived, indefinitely, and a deploy that corrects the
|
||||
parse would never reach the data. A version mismatch counts as drift, so the
|
||||
correction lands on the next boot without an operator having to know it happened.
|
||||
|
||||
## The API
|
||||
|
||||
Everything is served from MariaDB. Nothing on this path touches the sidecar, so
|
||||
the pages stay complete while the shard is down — which is why the routes sit at
|
||||
`/api/v1/public/atlas` and **not** under `/public/shard`, where a prefix means
|
||||
"sidecar-dependent". Unlike `/shard/*`, they *are* `siteMode`-gated, like
|
||||
`/posts` and `/wiki`: a bestiary is site content and follows site content's rules.
|
||||
|
||||
Every route carries `requireFeature('atlas')` — **404** when an admin has
|
||||
disabled the feature (its pages must not reveal that it exists) and **403** when
|
||||
the caller sits below its configured audience. The default is `anonymous`, so the
|
||||
gates are inert until an admin changes something. Responses are field-projected
|
||||
like every other shard read; `atlas` declares no sensitive fields today, and the
|
||||
projection call is there so the first one that does is covered by construction
|
||||
rather than by a retrofit ([`v3.md` §3.6.1](../link/v3.md)).
|
||||
|
||||
| Route | Answers |
|
||||
|---|---|
|
||||
| `GET /atlas/creatures?q=&facet=&limit=&offset=` | The bestiary, most numerous first, paginated with an unpaginated `total` |
|
||||
| `GET /atlas/creatures/:slug?facet=&points=` | One creature: `places`, `spawners`, `alsoHere` |
|
||||
| `GET /atlas/regions?facet=&q=` | Named regions and their rectangles |
|
||||
| `GET /atlas/landmarks?facet=&q=` | Points of interest, labelled by `group` |
|
||||
| `GET /atlas/champions?facet=` | The configured altar roster |
|
||||
| `GET /atlas/meta` | Facets, counts and when the atlas was parsed |
|
||||
|
||||
Two shapes worth knowing:
|
||||
|
||||
- **`places` is the aggregate the atlas exists for.** "Lizardman → Shrines,
|
||||
Isamu-Jima, Yew", grouped in SQL rather than by summing 6,455 point rows in
|
||||
Node. `spawners` is the raw list underneath it, bounded, with
|
||||
`spawnersTruncated` saying when it was cut.
|
||||
- **`points` is a COUNT, `spawners` is the LIST.** The two are named apart
|
||||
deliberately: the same key meaning a number on the search route and an array on
|
||||
the detail route is the kind of thing a client only discovers in production.
|
||||
|
||||
`GET /atlas/meta` reports the **game world only**. The ServUO path, the per-file
|
||||
hashes and any pending refresh describe the operator's filesystem, and live on
|
||||
the admin route instead.
|
||||
|
||||
A facet is never validated against a list — nothing in the codebase names one.
|
||||
`?facet=` is length-bounded and matched exactly, so an unknown name returns an
|
||||
empty result rather than an error. The filter is an `EXISTS` over the points and
|
||||
deliberately not a JSON path or `JSON_SEARCH` built from caller input: that
|
||||
function treats `%` and `_` as wildcards, which would make `?facet=%` match
|
||||
everything.
|
||||
|
||||
## The admin panel
|
||||
|
||||
**Admin → Spawn Atlas** (`/admin/shard-atlas`, admin-only — it reads a path on
|
||||
the server's filesystem and replaces every atlas table, which is closer to a
|
||||
deploy action than to moderation).
|
||||
|
||||
| Route | Does |
|
||||
|---|---|
|
||||
| `GET /admin/shard/atlas` | Status: path, readable, drift, counts, facets, pending |
|
||||
| `POST /admin/shard/atlas/import` | Import now; `{ force: true }` ignores the hash gate |
|
||||
| `POST /admin/shard/atlas/approve` | Apply a staged refresh, facet loss and all |
|
||||
| `POST /admin/shard/atlas/reject` | Keep the current atlas; remember the decision |
|
||||
| `PUT /admin/shard/atlas/path` | Point the atlas at a different tree |
|
||||
|
||||
Three behaviours that are deliberate:
|
||||
|
||||
- **An unreadable tree is a 200, not a 500.** `refresh()` reports outcomes rather
|
||||
than throwing, because the boot path must never be stopped by a bad tree, and
|
||||
that contract is preserved at the API. The panel says *"The tree could not be
|
||||
read: …"*; a 500 would say only that something broke.
|
||||
- **Setting the path does not import.** Moving the mount and reloading the world
|
||||
are separate decisions, and an operator fixing a typo should not have a
|
||||
multi-thousand-row replace happen under them. The response carries fresh status
|
||||
so the panel can offer the import as the next step.
|
||||
- **Every action is written to the admin activity log** (`shard.atlas.import` /
|
||||
`.approve` / `.reject` / `.path`).
|
||||
|
||||
@@ -257,6 +257,26 @@
|
||||
"method": "GET",
|
||||
"path": "/api/v1/admin/shard/accounts"
|
||||
},
|
||||
{
|
||||
"method": "GET",
|
||||
"path": "/api/v1/admin/shard/atlas"
|
||||
},
|
||||
{
|
||||
"method": "POST",
|
||||
"path": "/api/v1/admin/shard/atlas/approve"
|
||||
},
|
||||
{
|
||||
"method": "POST",
|
||||
"path": "/api/v1/admin/shard/atlas/import"
|
||||
},
|
||||
{
|
||||
"method": "PUT",
|
||||
"path": "/api/v1/admin/shard/atlas/path"
|
||||
},
|
||||
{
|
||||
"method": "POST",
|
||||
"path": "/api/v1/admin/shard/atlas/reject"
|
||||
},
|
||||
{
|
||||
"method": "GET",
|
||||
"path": "/api/v1/admin/shard/audit"
|
||||
@@ -313,6 +333,14 @@
|
||||
"method": "GET",
|
||||
"path": "/api/v1/admin/shard/vendors/:account"
|
||||
},
|
||||
{
|
||||
"method": "GET",
|
||||
"path": "/api/v1/admin/shard/visibility"
|
||||
},
|
||||
{
|
||||
"method": "PUT",
|
||||
"path": "/api/v1/admin/shard/visibility"
|
||||
},
|
||||
{
|
||||
"method": "PUT",
|
||||
"path": "/api/v1/admin/site-mode"
|
||||
@@ -705,6 +733,30 @@
|
||||
"method": "GET",
|
||||
"path": "/api/v1/player/shard/vendors/:account"
|
||||
},
|
||||
{
|
||||
"method": "GET",
|
||||
"path": "/api/v1/public/atlas/champions"
|
||||
},
|
||||
{
|
||||
"method": "GET",
|
||||
"path": "/api/v1/public/atlas/creatures"
|
||||
},
|
||||
{
|
||||
"method": "GET",
|
||||
"path": "/api/v1/public/atlas/creatures/:slug"
|
||||
},
|
||||
{
|
||||
"method": "GET",
|
||||
"path": "/api/v1/public/atlas/landmarks"
|
||||
},
|
||||
{
|
||||
"method": "GET",
|
||||
"path": "/api/v1/public/atlas/meta"
|
||||
},
|
||||
{
|
||||
"method": "GET",
|
||||
"path": "/api/v1/public/atlas/regions"
|
||||
},
|
||||
{
|
||||
"method": "POST",
|
||||
"path": "/api/v1/public/contact"
|
||||
@@ -737,6 +789,10 @@
|
||||
"method": "GET",
|
||||
"path": "/api/v1/public/shard/economy"
|
||||
},
|
||||
{
|
||||
"method": "GET",
|
||||
"path": "/api/v1/public/shard/features"
|
||||
},
|
||||
{
|
||||
"method": "GET",
|
||||
"path": "/api/v1/public/shard/feed"
|
||||
@@ -769,6 +825,10 @@
|
||||
"method": "GET",
|
||||
"path": "/api/v1/public/shard/presence"
|
||||
},
|
||||
{
|
||||
"method": "GET",
|
||||
"path": "/api/v1/public/shard/ruleset"
|
||||
},
|
||||
{
|
||||
"method": "GET",
|
||||
"path": "/api/v1/public/shard/status"
|
||||
|
||||
Reference in New Issue
Block a user