feat(cliloc): import the table from the shard, not from a file someone converted (Phase 2)

The base cliloc table now comes over the bridge. `clilocBridge.js` walks
`GET /cliloc` page by page and the model merges the `custom/` overlays over it —
overlays stay on disk because ServUO has no server-side notion of a custom
cliloc, so there is nothing on the shard to ask for.

**The shard wins whenever uo-link is configured and enabled**, with no mode
setting: there is no version of "which source?" an operator benefits from
answering. A file on disk remains the source only where there is no shard link,
plus a one-off explicit `path` — deprecated, not removed, and unchanged.

**Boot no longer imports on the bridge.** The file path could hash 5 MB locally
and skip in 14 ms; a shard round trip in the boot sequence would be spent
answering "no" on every restart but the one after a client patch — and patching a
client is an operator action, so importing became one. Admin → Shard → Import.
Whatever table is loaded keeps serving until then.

Three checks in the walk, each for a way a shard can hand back a table that looks
complete:

  * only `cut: 'end'` finishes it — a short page can equally be a spent budget,
    and a truncated table renders some items named and some not, which is exactly
    what NO table looks like;
  * the cursor must advance, or the walk stops rather than spinning;
  * every page echoes the source's size and mtime, so a client patched mid-import
    is refused outright rather than stitched from two files.

**The base is exempt from the vanished-source rule**, which is an upgrade detail
rather than a preference: an install that used the file pipeline carries its base
file's label in the stored fingerprint, and on the bridge that label is *supposed*
to disappear. Counting it as vanished would demand an approval for a change the
upgrade itself made. Overlays keep the rule in full.

**The protocol pin moves 7 → 8** — the third declaration site, and the one
nothing enforces. Phase 1 moved the sidecar and the overlay together because the
installer refuses a mismatched bundle; this one has to be moved by hand, in the
phase that first calls a protocol-8 route. The schema block above it is the
record of what forgetting costs: two phases of every REST call answered 409.

Verified against a live shard, sidecar and site: 12 pages, 67,496 rows imported
in 1.68 s, the operator's three-row overlay overriding stock strings on top of
it, and the next import correctly `unchanged`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4

server/utils/clilocBridge.js

@@ -0,0 +1,306 @@
+// Cliloc table — the SHARD source (docs/link/v8.md §9, protocol 8 phase 2).
+//
+// `clilocSource.js` is the filesystem half of this story and predates it. This is
+// the half that replaces the part of it nobody enjoyed: until protocol 8 the base
+// table reached the site because an operator installed UOFiddler, built a
+// converter against its `Ultima.dll`, ran it over their client's compressed
+// `Cliloc.enu` and copied a five-megabyte file to the web host — every time they
+// patched their client.
+//
+// The shard has always had those files (a ServUO server cannot boot without a UO
+// client) and, as of phase 2, has the decompressor too. So the base table now
+// arrives over the same request/reply path as every other shard read, and the
+// operator installs nothing.
+//
+// **What is NOT here.** Overlays. Shard-added items carry cliloc ids no client
+// table has, ServUO has no server-side notion of a custom cliloc, and there is
+// therefore nothing on the shard to ask for. `custom/` stays a directory the site
+// reads (`clilocSource.readOverlays`), and the model merges it OVER whatever
+// arrives here. That division is the whole of CLILOCS.md §Shard-added items and
+// it is unchanged by this file.
+//
+// ── Why this walks pages instead of asking for a table ────────────────────
+//
+// The sidecar's reply timeout is 10 s and its inbound line cap is 1 MiB, so a
+// five-megabyte table cannot be one answer. The shard cuts pages at a 512 KiB
+// byte budget and hands back a cursor; this walks them. A stock English table is
+// about eleven pages.
+//
+// Three properties of that envelope are load-bearing and each has a check below:
+//
+//   - **Only `cut: 'end'` means finished.** A short page can equally mean the
+//     budget was spent (`budget`) or the family stopped at its own limit
+//     (`limit`). Treating a short page as the end would import a truncated table,
+//     which is indistinguishable downstream from a complete one — some items
+//     named, some not, exactly what "no table at all" looks like.
+//   - **The cursor must advance.** A shard that answered the same cursor forever
+//     would spin this loop until the request timeout with nothing to show.
+//   - **The file must not change underneath the walk.** Every page echoes the
+//     source's size and mtime; an operator patching their client mid-import would
+//     otherwise produce one table stitched from two, with no error anywhere.
+
+// Required as a namespace, not destructured: a test that stubs the sidecar
+// replaces these on the module object, and a destructured copy taken at load
+// time would keep calling the real one.
+const uoLinkClient = require('./uoLinkClient')
+const log = require('../core').logger('cliloc-bridge')
+
+/** The client file the base table comes from, as `assets.sources` names it. */
+const SOURCE_FILE = 'cliloc.enu'
+
+const DEFAULT_LANGUAGE = 'enu'
+
+// Bounds on the walk. Neither is expected to be reached — a stock table is ~11
+// pages and ~67k rows — and both exist so that a shard answering nonsense costs a
+// bounded amount of time rather than an unbounded amount of memory.
+const MAX_PAGES = 200
+const MAX_ROWS = 500000
+
+// 425 is the ordinary answer during an import, not an error: the shard's asset
+// plane serves one request at a time on purpose, because its outbound queue is
+// bounded in lines rather than bytes. So a page that comes back busy is retried
+// with a short backoff rather than failing the import.
+const BUSY_RETRIES = 5
+const BUSY_BACKOFF_MS = [200, 400, 800, 1600, 3200]
+
+class ClilocBridgeError extends Error {
+  constructor(message, code) {
+    super(message)
+    this.name = 'ClilocBridgeError'
+    this.code = code
+  }
+}
+
+const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms))
+
+/**
+ * Map a sidecar response onto one of this module's codes.
+ *
+ * The statuses are the ones `respond_assets` produces, and the distinction that
+ * matters most to an operator is 403 vs 404: "you have not switched this on" and
+ * "your client does not have that file" are different jobs, and both are things
+ * they can fix.
+ */
+function describeFailure(res, what) {
+  const reason = res?.data?.reason || res?.error || `sidecar responded ${res?.status}`
+
+  switch (res?.status) {
+    case 403:
+      return new ClilocBridgeError(
+        `The shard is refusing to serve client assets (Bridge.AssetsEnabled is off): ${reason}`,
+        'DISABLED',
+      )
+    case 404:
+      return new ClilocBridgeError(`The shard has no ${what}: ${reason}`, 'NO_SOURCE')
+    case 409:
+      return new ClilocBridgeError(
+        `The sidecar refused the protocol version this build declares: ${reason}`,
+        'PROTOCOL',
+      )
+    case 422:
+      return new ClilocBridgeError(`The shard could not read its own ${what}: ${reason}`, 'UNREADABLE')
+    case 425:
+      return new ClilocBridgeError(
+        'The shard is busy serving another asset request and stayed busy',
+        'BUSY',
+      )
+    case 503:
+    case 504:
+      return new ClilocBridgeError(`The shard did not answer: ${reason}`, 'SHARD_DOWN')
+    default:
+      return new ClilocBridgeError(reason, 'UNAVAILABLE')
+  }
+}
+
+/**
+ * Stage 1: the fingerprint of the shard's own cliloc file.
+ *
+ * Returns `{ file, size, mtime, sha256, extractorVersion, hashing, complete }`.
+ *
+ * `sha256` may be **null** — the shard reports hashes only once it has computed
+ * them off the request path, because hashing the client files it also serves
+ * (343 MB of art and animation) cannot fit inside a 10 s reply. A null hash means
+ * "not yet", never "changed", and `sameSource` below compares (size, mtime) in
+ * that case, which is the same gate the shard itself uses.
+ */
+async function fingerprint() {
+  const res = await uoLinkClient.getAssetSources()
+  if (!res.ok) throw describeFailure(res, 'client file manifest')
+
+  const files = Array.isArray(res.data?.files) ? res.data.files : []
+  const entry = files.find((f) => String(f?.name || '').toLowerCase() === SOURCE_FILE)
+
+  if (!entry) {
+    throw new ClilocBridgeError(
+      `The shard's UO client has no ${SOURCE_FILE} (it reported ${files.length} client file(s))`,
+      'NO_SOURCE',
+    )
+  }
+
+  return {
+    kind: 'bridge',
+    file: entry.name,
+    path: entry.path ?? null,
+    size: Number(entry.size) || 0,
+    mtime: Number(entry.mtime) || 0,
+    sha256: entry.sha256 ?? null,
+    extractorVersion: Number(res.data?.extractorVersion) || 0,
+    hashing: Boolean(res.data?.hashing),
+    complete: Boolean(res.data?.complete),
+  }
+}
+
+/**
+ * True when two fingerprints describe the same client file.
+ *
+ * Hash first when both sides have one, because a hash is the only thing that
+ * catches a file rewritten with the same length and timestamp. Falls back to
+ * (size, mtime) when either side's hash is missing, which is the case on the
+ * first poll after a shard restart and the reason `hashing` exists at all.
+ */
+function sameSource(a, b) {
+  if (!a || !b) return false
+  if (a.extractorVersion !== b.extractorVersion) return false
+  if (a.sha256 && b.sha256) return a.sha256 === b.sha256
+  return a.size === b.size && a.mtime === b.mtime && a.size > 0
+}
+
+/** One page, with the 425 backoff. */
+async function fetchPage({ lang, cursor }) {
+  for (let attempt = 0; ; attempt++) {
+    const res = await uoLinkClient.getClilocTable({ lang, cursor })
+    if (res.ok) return res.data
+
+    if (res.status === 425 && attempt < BUSY_RETRIES) {
+      await sleep(BUSY_BACKOFF_MS[Math.min(attempt, BUSY_BACKOFF_MS.length - 1)])
+      continue
+    }
+
+    throw describeFailure(res, `cliloc.${lang}`)
+  }
+}
+
+/**
+ * Walk the whole table.
+ *
+ * Returns `{ entries, source }` where `entries` is `[{ number, flag, text }]` in
+ * the shape `clilocParse` produces, so the merge in `shardClilocs.model` does not
+ * care which source an entry came from.
+ *
+ * Blanks are already gone: the shard drops the ~56,000 empty strings a stock
+ * table carries before they reach the wire, since the site would drop them at
+ * import anyway. Nothing downstream changes — `db.replaceAll` still filters, and
+ * still would if a source ever sent one.
+ */
+async function readCliloc({ lang = DEFAULT_LANGUAGE } = {}) {
+  const started = Date.now()
+  const entries = []
+
+  let cursor = null
+  let pages = 0
+  let first = null
+  let finished = false
+  let total = null
+
+  while (pages < MAX_PAGES) {
+    const page = await fetchPage({ lang, cursor })
+    pages++
+
+    if (!page || !Array.isArray(page.rows)) {
+      throw new ClilocBridgeError('The shard sent a cliloc page with no rows array', 'MALFORMED')
+    }
+
+    if (first === null) {
+      first = { size: Number(page.size) || 0, mtime: Number(page.mtime) || 0 }
+      total = Number.isFinite(Number(page.total)) ? Number(page.total) : null
+    } else if (Number(page.size) !== first.size || Number(page.mtime) !== first.mtime) {
+      // The client was patched (or a different one mounted) between two pages.
+      // Refusing is the only honest answer: half of what we hold is from a file
+      // that no longer exists, and nothing later can tell which half.
+      throw new ClilocBridgeError(
+        'The shard\'s cliloc file changed while it was being read; nothing was imported',
+        'SOURCE_CHANGED',
+      )
+    }
+
+    for (const row of page.rows) {
+      const number = Number(row?.n)
+      if (!Number.isInteger(number)) continue
+      entries.push({ number, flag: Number(row?.f) || 0, text: String(row?.t ?? '') })
+    }
+
+    if (entries.length > MAX_ROWS) {
+      throw new ClilocBridgeError(
+        `The shard sent more than ${MAX_ROWS} cliloc rows; refusing to keep reading`,
+        'TOO_LARGE',
+      )
+    }
+
+    if (!page.more) {
+      // `cut` is the field that says WHY a page was the last one, and only one of
+      // its values means the table ended. A shard that stopped for its own limit
+      // has not finished, and importing what arrived would silently drop the tail.
+      if (page.cut !== 'end') {
+        throw new ClilocBridgeError(
+          `The shard stopped sending cliloc rows after ${entries.length} (cut: ${page.cut || 'unknown'})`,
+          'INCOMPLETE',
+        )
+      }
+      finished = true
+      break
+    }
+
+    if (!page.cursor || page.cursor === cursor) {
+      // Either would loop forever: no cursor to advance with, or the same one
+      // back again.
+      throw new ClilocBridgeError(
+        `The shard asked for another cliloc page without advancing its cursor (${page.cursor || 'none'})`,
+        'STUCK',
+      )
+    }
+
+    cursor = page.cursor
+  }
+
+  if (!finished) {
+    throw new ClilocBridgeError(
+      `The cliloc table did not end within ${MAX_PAGES} pages; nothing was imported`,
+      'TOO_LARGE',
+    )
+  }
+
+  log.info('cliloc table read from the shard', {
+    lang,
+    entries: entries.length,
+    pages,
+    ms: Date.now() - started,
+  })
+
+  return {
+    entries,
+    source: {
+      kind: 'bridge',
+      lang,
+      file: `cliloc.${lang}`,
+      size: first?.size ?? 0,
+      mtime: first?.mtime ?? 0,
+      pages,
+      // What the shard said it holds, kept beside what actually arrived. They
+      // agree or the walk is wrong, and an operator seeing them disagree in the
+      // panel learns more than a single number would tell them.
+      reported: total,
+      received: entries.length,
+    },
+  }
+}
+
+module.exports = {
+  ClilocBridgeError,
+  SOURCE_FILE,
+  DEFAULT_LANGUAGE,
+  MAX_PAGES,
+  MAX_ROWS,
+  fingerprint,
+  sameSource,
+  readCliloc,
+}

server/utils/clilocSource.js

@@ -6,6 +6,24 @@
 //   - the server, which refreshes the table on boot (`shardClilocs.model.js`)
 //   - the admin panel, which can force a reimport without a restart
 //
+// ── What protocol 8 took away, and what it left ───────────────────────────
+//
+// The BASE table no longer comes from here on a shard that has uo-link
+// configured: `clilocBridge.js` asks the shard for it, because the shard has the
+// operator's client files already and, since phase 2, the decompressor to read
+// them (docs/link/v8.md §9). Nobody converts a file by hand any more.
+//
+// Two things keep this module alive rather than deleting it:
+//
+//   - **Overlays.** Shard-added items carry cliloc ids no client table has, and
+//     ServUO has no server-side notion of a custom cliloc — there is nothing on
+//     the shard to ask for. `custom/` is still a directory the site reads, and
+//     `readOverlays` below is the entry point the bridge path uses.
+//   - **Installs with no shard link**, and development. A site that has never
+//     configured uo-link can still be pointed at a converted file; that path is
+//     deprecated, not removed, and it stays the whole of this module's base-table
+//     behaviour.
+//
 // The files are the OPERATOR'S (see docs/website/CLILOCS.md). Nothing derived
 // from them is committed: the repo holds no string table, exactly as it holds no
 // map snapshot and no artwork. That rule is why this module reads a configured
@@ -194,6 +212,63 @@ function readSources(configured) {
   return { root, files }
 }
 
+/**
+ * Read the OVERLAY files only, with no base table.
+ *
+ * The bridge path needs exactly this: the base arrives from the shard and the
+ * `custom/` directory beside the configured path still has to be merged over it.
+ * `readSources` cannot answer it, because resolving a base is the first thing it
+ * does and there may not be one — an operator on the bridge is entitled to point
+ * this setting at a directory that holds nothing but `custom/`.
+ *
+ * **Never throws.** A path that is blank, missing or unreadable is reported as a
+ * `problem` string and an empty file list, because none of those may stop a base
+ * table that arrived perfectly well from being imported. The model decides what
+ * to do about it — and it has a real decision to make, since an overlay that was
+ * loaded last time and is missing now is the vanished-source hazard, not a
+ * config typo.
+ */
+function readOverlays(configured) {
+  const target = String(configured ?? '').trim()
+  if (target === '') return { root: null, files: [], problem: null }
+
+  let root
+  try {
+    const stat = fs.statSync(target)
+    root = stat.isFile() ? path.dirname(target) : target
+  } catch {
+    return { root: null, files: [], problem: `Cliloc path does not exist: ${target}` }
+  }
+
+  let overlays
+  try {
+    overlays = listCustom(root)
+  } catch (err) {
+    return { root, files: [], problem: err.message }
+  }
+
+  const files = []
+  for (const file of overlays) {
+    let buffer
+    try {
+      buffer = fs.readFileSync(file)
+    } catch {
+      return { root, files: [], problem: `Cliloc overlay is not readable: ${file}` }
+    }
+    files.push({
+      label: path.relative(root, file).split(path.sep).join('/'),
+      kind: 'custom',
+      file,
+      buffer,
+      sha256: sha256(buffer),
+      bytes: buffer.length,
+      compressed: isCompressedCliloc(buffer),
+    })
+  }
+
+  return { root, files, problem: null }
+}
+
 /**
  * A fingerprint of every source: `{ "<label>": "<sha256>" }`, plus the base's
  * details for the admin panel.
@@ -244,6 +319,25 @@ function missingSources(current, loaded) {
   return Object.keys(loaded).filter((label) => !Object.hasOwn(current, label))
 }
 
+/**
+ * The same question asked of OVERLAYS only.
+ *
+ * Needed because the base table moved to the bridge. An install upgraded from the
+ * file pipeline carries a base label (`clilocs.plain`, say) in its loaded
+ * fingerprint, and that label is *supposed* to disappear when the base starts
+ * arriving from the shard — reporting it as a vanished source would make every
+ * first import after the upgrade demand an approval for a change the upgrade
+ * itself made. Overlay labels are the ones whose absence is genuinely ambiguous,
+ * and they are exactly the labels under `custom/`.
+ */
+function missingOverlays(current, loaded) {
+  if (!loaded) return []
+  const prefix = `${CUSTOM_DIR}/`
+  return Object.keys(loaded).filter(
+    (label) => label.startsWith(prefix) && !Object.hasOwn(current, label),
+  )
+}
+
 /**
  * Read and parse every source, merged into one entry list.
  *
@@ -309,8 +403,10 @@ module.exports = {
   resolveBase,
   listCustom,
   readSources,
+  readOverlays,
   hashSources,
   sameSources,
   missingSources,
+  missingOverlays,
   readCliloc,
 }

server/utils/uoLinkClient.js

@@ -178,6 +178,37 @@ const getPointsBoard = (system) => call(`/points/${encodeURIComponent(system)}`)
 const getMarket = ({ limit = 200, offset = 0 } = {}) =>
   call(`/market?limit=${encodeURIComponent(limit)}&offset=${encodeURIComponent(offset)}`)
 
+// ── Protocol 8: the Asset Bridge (docs/link/v8.md) ────────────────────────
+//
+// The shard reads the operator's own UO client files and hands the results over
+// this link, which is why nobody has to install UOFiddler any more.
+
+// Stage 1 of the import gate: what those client files currently ARE — size, mtime
+// and content hash of each, plus the version of the shard's extractor that would
+// read them. No pixels and no strings cross on this call; its whole job is to let
+// the site decide that nothing has changed and stop, which is the normal case on
+// every restart.
+//
+// `sha256` comes back NULL for a file the shard has not hashed yet (anim.mul is
+// 195 MB and hashing it cannot fit in a reply), with `hashing: true` alongside.
+// That is "ask again in a moment", not "the file changed".
+const getAssetSources = () => call('/assets/sources')
+
+// The cliloc table out of the shard's own client, PAGED: each reply carries `rows`
+// plus `more` / `cursor` / `cut`, and the caller echoes the cursor back until a
+// reply says `more: false`. Only `cut: 'end'` means the table is finished — a short
+// page can equally mean the byte budget was spent.
+//
+// `clilocBridge.js` is the thing that walks it; nothing else should call this
+// directly, because a half-walked table is worse than none.
+const getClilocTable = ({ lang, cursor } = {}) => {
+  const params = new URLSearchParams()
+  if (lang) params.set('lang', lang)
+  if (cursor) params.set('cursor', cursor)
+  const qs = params.toString()
+  return call(`/cliloc${qs ? `?${qs}` : ''}`)
+}
+
 // ── Commands ──────────────────────────────────────────────────────────────
 const confirmLink = (code, websiteUserId) =>
   call('/link/confirm', { method: 'POST', body: { code, websiteUserId: String(websiteUserId) } })
@@ -398,6 +429,8 @@ module.exports = {
   getPoints,
   getPointsBoard,
   getMarket,
+  getAssetSources,
+  getClilocTable,
   confirmLink,
   linkLookup,
   createAccount,