docs/modules/uo/SCHEMA.md

# module-uo — the tables it owns

The 27 tables `module-uo` creates and owns, and the reasoning behind their shapes. **26 `shard_*`
plus `uo_link_config`**, all created by the idempotent `server/db/schema.sql` fragment core replays
after its own schema on every boot, and all dropped by `server/db/purge.sql` when an operator purges
the module's data.

This page moved out of [`BACKEND_DESIGN.md`](../../website/BACKEND_DESIGN.md) §3 when Phase 4 closed
([`MODULE_SYSTEM.md`](../../website/MODULE_SYSTEM.md) §2.7.2): core's schema reference describes
core's tables, and these are not core's. The text is unchanged — where it says "the shard", it means
the Ultima Online shard this module fronts. Core's own module bookkeeping table,
`installed_modules`, stays in `BACKEND_DESIGN.md` where it belongs.

The `shard_` and `uo_link_` prefixes are **grandfathered**
([`MODULE_API.md`](../../website/MODULE_API.md) §6.5). A new module's tables are prefixed with its
own id; these predate the rule, and they are read by the shipped Android app, so renaming them would
be a data migration plus a client break.

**Not every table has a section here.** The pre-3.0 ingest tables (`shard_events`, `shard_online`,
`shard_economy`, `shard_houses`, `shard_champs`, `shard_guilds`, `shard_governors`,
`shard_governor_terms`, `shard_presence`, `shard_pages`, `shard_account_links`, `uo_link_config`)
are described where their wire frames are — [`link/PLAN.md`](../../link/PLAN.md) §5 and
[`link/INTEGRATION.md`](../../link/INTEGRATION.md). What follows is everything that carried a
design note worth keeping.

---

## shard_ruleset  — the shard's published ruleset (Protocol 3.0)

Singleton row (`id = 1`, CHECK-constrained) holding the latest `world.ruleset` frame: `rev`,
`expansion`, `payload` JSON (the whole frame), `t`, `updated_at`. The shard re-emits the complete
ruleset on every sidecar connect, so this is an **overwrite, not an append** — and the kind is
deliberately **not** in `LOGGED_KINDS`, since logging it would put a duplicate row in `shard_events`
on every reconnect while `server.hello` already marks each of those.

The frame is stored whole rather than normalized into columns: it is a flat description of server
config that is read as one page, so splitting it up would mean a schema change every time the shard
grows a new block. `rev` (the shard's FNV-1a of the body) and `expansion` are hoisted only because
they are cheap to display — the same payload-plus-hoisted-columns shape `shard_champs` uses.

**No row means the shard has never published one** (an older plugin, or `Bridge.RulesetEnabled=false`),
served as `null` rather than `{}`: "not published yet" and "published, everything off" are different
answers and the page renders them differently.

## shard_points_boards  — points / loyalty leaderboards (Protocol 3.0)

One row per point system, keyed by the shard's own `PointsType` name (`QueensLoyalty`,
`CleanUpBritannia`, …). The shard carries ~25 of these, each a standing players build over months.
Columns: `system` (PK), `name`, `name_cliloc`, `max_points`, `players`, `show_on_gump`, `payload` JSON
(the whole `points.board` frame), `t`, `updated_at`.

**The top-N list stays inside `payload`** rather than being normalized into a `shard_points_entries`
table. It is a fixed-size list (10 by default) that is only ever read whole — exactly like
`shard_governors.candidates` — so normalizing buys nothing until something needs a per-character
reverse lookup, and a character's own standings already ride inside `char.profile` instead.

Board state, not events: `points.board` is **not** in `LOGGED_KINDS`, for the same reason
`guild.update` isn't. The shard emits a frame every time anyone's score moves a top ten, so logging
would grow `shard_events` without bound for something whose only interesting value is its latest
version. There is also **no delete path** — the shard's set of systems is fixed at startup, so there is
no `points.remove` to mirror.

Two values carry non-obvious meanings, both set by the plugin and both documented in
[`link/INTEGRATION.md`](../../link/INTEGRATION.md) §4:

- **`max_points = 0` means uncapped**, and on a real shard that is the *common* case (ServUO's
  uncapped idiom is `double.MaxValue`, which the plugin normalises to 0). Anything rendering
  `points / max_points` must special-case it.
- **`name` is usually NULL**, with `name_cliloc` set instead — most systems name themselves with a
  cliloc rather than a literal. Listing therefore orders by `COALESCE(name, system)`, so boards
  awaiting cliloc resolution sort by their own key rather than clumping together under NULL.

## shard_vendors / shard_vendor_items — the player-vendor marketplace (Protocol 3.0)

The shard-wide shop index, fed by `vendor.listing` / `vendor.listing.remove`. One row per player
vendor and one per priced listing. Full operator detail in [`MARKETPLACE.md`](../../website/MARKETPLACE.md); the
design is `docs/link/v3.md` §8.

| Table | Shape |
|---|---|
| `shard_vendors` | `serial` (PK), `shop_name`, `owner_serial`, `owner_name`, `map`/`x`/`y`/`z`, `region`, `house`, `item_count`, `item_total`, `truncated`, `t`, `updated_at`. Indexes on owner, map, region and `updated_at`. |
| `shard_vendor_items` | `id` (PK), `vendor_serial`, `serial`, `item_id`, `hue`, `amount`, `price`, `name`, `cliloc`, `display_name`, `child`. Indexes on `vendor_serial`, `price`, `item_id`, `display_name`, and `(display_name, price)`. |

**Ingest is per-vendor and authoritative**: the frame is the whole shop, so ingest is
delete-then-insert of that vendor's listings inside one transaction. All-or-nothing matters
specifically because the two writes are "the shop" and "what is in it" — a failure between them
leaves a shop advertising an inventory it no longer has, which is visibly wrong and indistinguishable
from a genuinely empty shop. No foreign keys, consistent with every other `shard_*` table.

**There is deliberately no `payload` column**, unlike `shard_points_boards` directly above. The
board's top-N is a fixed-size list read whole, so it lives in JSON; here the items *are* the
searchable rows, so they are normalized and nothing is left worth duplicating. The sidecar keeps the
whole blob — outage resilience is its job, search is ours.

Market state, not events: neither kind is in `LOGGED_KINDS`, and this is the strongest case of the
three v3 kinds. One frame carries up to 250 listings and the sweep re-emits a shop on any price
change, so logging would turn `shard_events` into a price history nobody reads.

Two columns carry non-obvious meanings:

- **`item_count` vs `item_total`.** `item_count` is what the frame published; `item_total` is what
  the shop actually holds. They differ when `truncated` — the shard caps listings per frame
  (`Bridge.MarketMaxListings`, 250 by default), and a commodity reseller with thousands of stacks
  genuinely exceeds it. Any UI must show both or it presents a partial shop as complete.
- **`display_name` is denormalized at ingest**, resolved from the item's literal `name` (preferred —
  a player set it, so it is more specific) else its `cliloc` against `shard_clilocs`. Resolving at
  query time would put the cliloc table on the hot path and make search-by-name impossible. Because
  the shard's diff sweep will not re-send an unchanged shop just because the site learned what its
  items are called, **a cliloc import triggers a bulk re-resolution** of this column (after a boot
  import and after an admin import; ~50 ms per thousand rows, never throws).

`updated_at` is written explicitly on every upsert rather than left to `ON UPDATE CURRENT_TIMESTAMP`,
which MariaDB does not fire when every column is written back unchanged. A shop re-published
identically is still *freshly confirmed*, and without this the staleness banner would age a perfectly
current shop forever.

## shard_feature_visibility  — per-feature audience config (Protocol 3.0)

One row per shard feature: `feature` (PK), `enabled`, `audience` (a rung on the ladder in [`API.md`](API.md)),
`stream` (whether the feature's kinds fan out over SSE at all), `field_rules` JSON (`{field: rung}`
for the sensitive fields only), `updated_by`, `updated_at`.

**An absent row means "use the compiled default", and the compiled defaults reproduce pre-3.0
behavior — so an empty table is a no-op and there is nothing to seed.** Stored rows are merged over
the defaults on read, which is also where the invariants are re-applied: a row naming an unknown
feature is ignored (a stale row must not resurrect a removed feature), an invalid rung falls back to
the default rather than failing open, and a rule touching a locked field (`acct` / `webId`) is
discarded. See [`API.md`](API.md).

## shard_spawn_* / shard_regions / shard_landmarks / shard_champion_spawns / shard_atlas_meta — the spawn atlas (Protocol 3.0)

Static shard **content**, not live shard state. Nothing here comes from the sidecar: the atlas is
derived from the shard's own ServUO tree, re-read on **every server boot** and hash-gated so an
unchanged tree costs one read pass and no write. Nothing is precomputed and committed — a shard's
maps change over its life, and a snapshot in the repo would silently drift from the world players
actually see. These tables stay populated whether the shard is up or not. Full operator detail in
[`SPAWN_ATLAS.md`](../../website/SPAWN_ATLAS.md); the design is `docs/link/v3.md` §6.

**No facet name appears anywhere in the code.** A shard may add facets, replace them, or rename them
when its maps are updated; the facet set is discovered from the tree, and the loose spellings in
`Data/Locations` are matched against it rather than looked up in a table.

| Table | Key columns |
|---|---|
| `shard_spawn_creatures` | `slug` PK, `name`, `total`, `points`, `facets` JSON, `art` NULL |
| `shard_spawn_points` | `id` PK, `facet`, `name`, `x`, `y`, `width`, `height`, `spawn_range`, `max_count`, `min_delay`, `max_delay`, `tod_start/end/mode`, `region`, `landmark`, `label` |
| `shard_spawn_point_types` | `(point_id, slug)` PK, `max_count` |
| `shard_regions` | `facet`, `name`, `type`, `priority`, `parent`, `rects` JSON |
| `shard_landmarks` | `facet`, `name`, `grp`, `x`, `y`, `z` |
| `shard_champion_spawns` | `slug` PK, `name`, `grp`, `type`, `random_type`, `facet`, `x`, `y`, `z`, `radius`, `label` |
| `shard_atlas_meta` | Singleton (`id = 1`), `payload` JSON (counts, a sha256 per source file, `parserVersion`), `imported_at` |
| `shard_atlas_pending` | Singleton (`id = 1`), `status` (`pending`/`rejected`), `payload` JSON, `detected_at` |

The first seven are **import-owned**: a refresh empties and reloads every one inside a single
transaction, so a failed reload leaves the previous atlas intact rather than a half-loaded world.
Nothing else writes to them, and nothing holds a foreign key to them — no FKs at all, consistent with
every other `shard_*` table.

**`shard_atlas_pending` is the security-relevant one.** A refresh that would REMOVE a facet is never
applied automatically: facet loss is indistinguishable at boot from a half-copied or mid-update tree,
so it is staged here for an admin to approve or reject, and **startup is never blocked by it**. Only
the decision is stored — source hashes plus the facet diff, a few KB — and approving re-parses the
tree, so a multi-megabyte blob never lands in the database and what gets applied matches the tree at
approval time. A rejection is remembered against those exact hashes so a declined refresh does not
re-prompt on every restart. Everything else (new facets, renamed regions, changed spawns) applies
immediately, since none of it can destroy data an operator would miss.

The boot refresh is **best-effort by contract**: no configured path, an unreadable mount, a malformed
file or a database error is caught and logged, and the site comes up serving whatever atlas it had.
The tree path comes from the `spawn_atlas_servuo_path` setting, falling back to `SERVUO_PATH`.

**A refresh re-derives when the tree changed OR the parser did.** `spawnAtlasSource.PARSER_VERSION`
is stored in `shard_atlas_meta` beside the source hashes and bumped whenever the parser produces
different data from identical files. Hashing the tree alone would strand an install whose maps never
change on whatever an older build derived — a corrected parse would ship and never reach the data.

Four column choices worth stating, because each one is a trap:

- **`spawn_range`, not `range`**, and **`grp`, not `group`** — both are reserved words.
- **`DELETE`, not `TRUNCATE`.** `TRUNCATE` is DDL in MariaDB and implicitly commits, which would
  defeat the all-or-nothing reload. At ~7k rows the difference does not matter.
- **Point ids are assigned explicitly**, not left to `AUTO_INCREMENT`: the `shard_spawn_point_types`
  rows need to know them, and `conn.batch()` reports no usable `insertId` for a multi-row insert.
- **Plain `INDEX` on `name`, deliberately not `FULLTEXT`.** ~800 creature rows makes a `LIKE` scan
  free, and FULLTEXT's minimum token length would break searches for names like "orc".

`shard_champion_spawns` is the *configured* altar roster ("there is an Unholy Terror altar in
Deceit"). The live `champ.update` feed in `shard_champs` is the separate answer to "it is on level 3
right now". Both exist; they are not the same data.

**`shard_spawn_creatures.art` is DERIVED, never written by the atlas import itself** — see
`shard_assets` below. The project still ships no creature artwork: sprites live in the operator's own
client `.mul`/`.uop` files and are theirs, not ours to redistribute. What changed in Protocol 8 is
who extracts them: the shard does, from its own client, over the bridge. An operator's hand-drawn map
plus images under the (already gitignored) `server/uploads/atlas/` still wins over anything imported.
Text-only is still the normal, supported state — an install with no shard link never imports one, and
even a complete import leaves two thirds of the playable ghost and gargoyle bodies without art.

## shard_assets / shard_creature_bodies / shard_asset_meta — the Asset Bridge (Protocol 8)

Creature artwork read from the shard's own UO client ([`link/v8.md`](../../link/v8.md) §6, §8, §12).

| Table | Shape |
|---|---|
| `shard_assets` | `asset_key` VARCHAR PK (§5's key, e.g. `body/34/a0`, `body/820/a23`, `static/3922/h33`, `land/3`), `family`, `sha256`, `bytes`, `width`, `height`, `body`, `action`, `direction`, `file`, `catalog`, `imported_at` |
| `shard_creature_bodies` | `slug` PK, `type_name` (the ServUO class name asked), `body` nullable, `status`, `resolved_at` |
| `shard_asset_meta` | Singleton (`id = 1`), `payload` JSON (catalogue id, extractor version, source fingerprint, counts), `imported_at` |

**These are the one part of this schema that is deliberately NOT import-owned**, and the reason is
the atlas tables sitting directly above them. `replaceAtlas` empties and refills
`shard_spawn_creatures` on every refresh, and a refresh runs on every boot; an imported filename
stored on that row would be destroyed by an ordinary re-parse of the ServUO tree, with the next asset
Update finding the client files unchanged, reporting "nothing to do", and never restoring it. So the
assets live out here, upserted per key, and `replaceAtlas` re-derives `art` from them on the way
past — `{ ...derived, ...operatorMap }`, which is the one place "the operator's map wins" is
enforced.

Four details that are load-bearing rather than incidental:

- **`file` is a filename under the uploads directory, never a path**, and it is content-addressed
  (`uo-body-34-a0-<sha8>.png`). A stable name overwritten in place would leave every browser and CDN
  serving the previous client's sprite from cache, with the row perfectly correct.
- **The `art` derivation joins on the catalogue key**, `a.asset_key = CONCAT('body/', b.body, '/a0')`,
  not `a.body = b.body`. The simpler join is correct today and stops being correct the moment deeper
  animation keys (`body/400/a2/f0`) arrive, at which point one slug matches dozens of rows.
- **`shard_creature_bodies` IS replaced whole**, unlike `shard_assets`: it is derived from the atlas's
  creature list, so a slug that has left the atlas has no meaning, and the pass that rebuilds it is a
  shard round trip rather than a file transfer.
- **`status` keeps the negative answers** — `unknown` (the spawn files name a type this shard's
  scripts do not define, which is real drift), `notCreature` (a spawn entry for an item or
  decoration, a permanent answer), `failed`. Without them the next pass asks again, and each name
  costs a real constructor on the shard's Core thread.

`shard_spawn_creatures.name` already holds the ServUO **class name** — the atlas build picks the
winning spelling of the spawn type token rather than inventing a display label — which is why the
body pass needs no extra column to ask its question.

### Item and land art: the same table, a different shape of use (phase 5)

The creature catalogue is a **set**: one manifest walk covers every key, so one stored fingerprint in
`shard_asset_meta` describes all of them and an Update is a hash diff. Item art has no set — the
client addresses 49,152 item graphics times 3,000 hues — so those rows arrive one at a time, because
something on this site named the key.

Three consequences in this schema:

- **`catalog` is per row**, and it is what phase 5 added. It records the shard's art catalogue id (a
  hash of `artLegacyMUL.uop`/`art.mul`, `hues.mul`, `tiledata.mul`, `verdata.mul` and its extractor
  version), so staleness is a column comparison rather than a manifest diff. A client patch changes
  it; a restart does not. NULL means "written before the column existed", which is stale by the same
  test and costs one re-fetch. The body import fills it too, so one column answers the question
  everywhere.
- **`shard_asset_meta` stays the body catalogue's alone.** A warm pass writing there would tell the
  body import that a client it never looked at is unchanged, and the creature catalogue would quietly
  stop updating.
- **`family` is now load-bearing**, not decoration: `body`, `static` and `land` rows share the table
  and have different lifetimes. The admin status counts them separately for the same reason — there
  is no "how many are there" to compare `static` against, so the only honest number is how many the
  site has been asked for and holds.

Pictures land in `server/uploads/items/` (gitignored like `uploads/atlas/`), content-addressed the
same way — `uo-static-3922-h33-<sha8>.png`. A key the shard has no art for writes **no row at all**:
an empty row would make it "held" and it would never be asked again, including after the operator
patches in the graphic that was missing.

## shard_clilocs / shard_cliloc_meta — UO's localization table (Protocol 3.0)

Items on the wire carry a `LabelNumber`, not a name. The bridge has always sent it —
`char.profile.equipment.cliloc`, reward titles as a cliloc number in string form, and one per
marketplace listing — but with no table to resolve it against, the character sheet could only render
`id 1023721` where the game renders "quarter staff".

| Table | Shape |
|---|---|
| `shard_clilocs` | `number` INT PK, `flag`, `text` TEXT |
| `shard_cliloc_meta` | Singleton (`id = 1`), `payload` JSON (`source` — `bridge` or `file` — the base's fingerprint under `base`, the overlay `hashes`, per-source counts, `parserVersion`), `imported_at` |

Import-owned and all-or-nothing in one transaction, same contract as the atlas — including **`DELETE`,
not `TRUNCATE`**, for the same reason.

**The base table comes from the SHARD** on any install with uo-link configured (Protocol 8, phase 2):
it reads its own client's compressed `Cliloc.enu` and serves the table paged over the bridge, so
nothing is converted and nothing is copied to the web host. Without a shard link it falls back to a
converted file on disk at a path from the `cliloc_client_path` setting, then `UO_CLIENT_PATH` — the
pre-protocol-8 pipeline, deprecated rather than removed so an existing install keeps working.

**Overlays are always the filesystem's**, either way: ServUO has no server-side notion of a custom
cliloc, so `custom/` is the only place shard-added ids exist and there is nothing on the shard to ask
for. That gap is in the *game*, not in this pipeline.

Nothing client-derived is committed: UO's strings are EA's, exactly as the creature sprites are. A
shard with nothing configured is fully supported — names render as ids. Full design and operator
guide: [`CLILOCS.md`](../../website/CLILOCS.md).

**It reads a SET of sources, not one file**, because shards edit items and add new ones and those
carry cliloc ids no stock client table has. A base (from the shard, or a converted file) plus every
overlay under `custom/` are hash-gated **together**, exactly as the atlas re-reads
`Regions.xml` + `Locations/*.xml` + `Spawns/*.xml` + `ChampionSpawns.xml`. Later sources win, so an
overlay both adds ids and overrides stock ones, and adding one custom item never means re-exporting a
5 MB client file. Scale, measured on the live shard: its script tree references 16,434 cliloc ids and
only 37 are absent from stock — tens of entries against a 67k base, which is why this is an overlay
and not a second table.

**The conversion step used to be unavoidable, and is not any more.** Every current client ships its
cliloc files compressed (first DWORD's high byte `0x8E`) and ServUO's own bundled
`Ultima.StringList` cannot read that either — which is why, for two protocol versions, the operator
had to install UOFiddler, build a converter against its `Ultima.dll` and copy a 5 MB file to the web
host. Protocol 8 phase 2 ported the Mythic decompressor into the overlay, so **the shard reads its
own client and supplies the names**. The file half survives only as the fallback above, where the
plain layout and a delimited text export are both accepted, sniffed by header rather than extension.

Two consequences for what this table holds. `shard_cliloc_meta.payload` carries the base's
fingerprint under `meta.base` (the shard's file size, mtime, hash and `EXTRACTOR_VERSION`) separately
from the overlay hashes, because on the bridge the old `clilocs.plain` label is *supposed* to
disappear and a single hash map would read that upgrade as a vanished source. And **boot does not
import on the bridge path**: a file could be re-hashed locally on every restart, but asking the shard
would put a sidecar round trip in the boot sequence for a table that changes only when an operator
patches their client. Importing is an admin action.

Three decisions worth stating:

- **`text` is TEXT, not VARCHAR.** Long property descriptions reach 12 KB. The index that matters for
  marketplace search is the denormalized `shard_vendor_items.display_name`, not this table.
- **Blank entries are dropped at import** — 123,490 parsed → **67,496** stored. Roughly half a cliloc
  table is empty strings for ids the client reserves and never uses; a row that resolves to no name is
  indistinguishable from no row at all, and dropping them makes the binary and text imports converge
  on identical content.
- **Two refusals, one of them the atlas's.** A corrupt source fails the parse on a truncated record,
  so it is caught outright and leaves the previous table serving. But a source that has **vanished**
  parses perfectly and imports a table quietly missing everything it contributed — an unmounted volume
  and a deliberate deletion are indistinguishable from here, which is precisely the ambiguity the
  atlas stages a facet removal for. So it is escalated: `status: 'needsReview'`, nothing applied,
  `missingSources` reported by both the import and `status()`, and an admin accepts it with
  `{approve:true}`. That is a flag rather than the atlas's approve/reject pair because the atlas
  stores a pending decision so that approving **re-parses** the tree; here nothing is stored, so
  re-reading at approval time is automatic.

**Resolution is server-side and there is no public route.** The table is never served *as* a table:
67k rows would dwarf any page using them, and the Android client consumes the same already-resolved
JSON. `resolveMany()` returns only ids that resolved to something displayable — placeholders like
`~1_val~` are stripped, since the bridge sends the id and never the property packet that carries the
arguments — and it never throws, because a cliloc lookup is decoration on a character sheet.