feat(cliloc): import the table from the shard, not from a file someone converted (Phase 2)

The base cliloc table now comes over the bridge. `clilocBridge.js` walks
`GET /cliloc` page by page and the model merges the `custom/` overlays over it —
overlays stay on disk because ServUO has no server-side notion of a custom
cliloc, so there is nothing on the shard to ask for.

**The shard wins whenever uo-link is configured and enabled**, with no mode
setting: there is no version of "which source?" an operator benefits from
answering. A file on disk remains the source only where there is no shard link,
plus a one-off explicit `path` — deprecated, not removed, and unchanged.

**Boot no longer imports on the bridge.** The file path could hash 5 MB locally
and skip in 14 ms; a shard round trip in the boot sequence would be spent
answering "no" on every restart but the one after a client patch — and patching a
client is an operator action, so importing became one. Admin → Shard → Import.
Whatever table is loaded keeps serving until then.

Three checks in the walk, each for a way a shard can hand back a table that looks
complete:

  * only `cut: 'end'` finishes it — a short page can equally be a spent budget,
    and a truncated table renders some items named and some not, which is exactly
    what NO table looks like;
  * the cursor must advance, or the walk stops rather than spinning;
  * every page echoes the source's size and mtime, so a client patched mid-import
    is refused outright rather than stitched from two files.

**The base is exempt from the vanished-source rule**, which is an upgrade detail
rather than a preference: an install that used the file pipeline carries its base
file's label in the stored fingerprint, and on the bridge that label is *supposed*
to disappear. Counting it as vanished would demand an approval for a change the
upgrade itself made. Overlays keep the rule in full.

**The protocol pin moves 7 → 8** — the third declaration site, and the one
nothing enforces. Phase 1 moved the sidecar and the overlay together because the
installer refuses a mismatched bundle; this one has to be moved by hand, in the
phase that first calls a protocol-8 route. The schema block above it is the
record of what forgetting costs: two phases of every REST call answered 409.

Verified against a live shard, sidecar and site: 12 pages, 67,496 rows imported
in 1.68 s, the operator's three-row overlay overriding stock strings on top of
it, and the next import correctly `unchanged`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4

server/model/shardClilocs/shardClilocs.model.js

@@ -1,6 +1,6 @@
 const db = require('./shardClilocs.db')
 const { settings } = require('../../core')
-const { displayText } = require('../../utils/clilocParse')
+const { displayText, parseCliloc } = require('../../utils/clilocParse')
 const {
   ClilocFormatError,
   ClilocSourceError,
@@ -8,8 +8,12 @@ const {
   hashSources,
   sameSources,
   missingSources,
+  missingOverlays,
+  readOverlays,
   readCliloc,
 } = require('../../utils/clilocSource')
+const bridge = require('../../utils/clilocBridge')
+const uoLinkConfig = require('../uoLinkConfig/uoLinkConfig.model')
 const log = require('../../core').logger('shardClilocs')
 
 // The cliloc table — UO's id → display-string map, refreshed from a file the
@@ -29,11 +33,35 @@ const log = require('../../core').logger('shardClilocs')
 //   2. **Nothing client-derived is committed.** The table is built from the
 //      operator's own file at a configured path. The repo ships no strings.
 //
-// The table is built from a SET of sources — the converted client table plus
-// every operator-maintained overlay beside it — because shards edit items and
-// add new ones, and those carry cliloc ids no stock client table has. All of
-// them are re-read on every boot and hash-gated together, so adding one custom
-// item never means re-exporting a 5 MB client file. Later sources win.
+// The table is built from a SET of sources — a base table plus every
+// operator-maintained overlay beside it — because shards edit items and add new
+// ones, and those carry cliloc ids no stock client table has. Later sources win,
+// so an overlay both adds ids the client never had and overrides stock ones.
+//
+// ── Where the base comes from (protocol 8, docs/link/v8.md §9) ─────────
+//
+// **The shard**, on any install with uo-link configured. It has the operator's
+// client files already — a ServUO server cannot boot without them — and since
+// phase 2 it has the decompressor too, so `GET /cliloc` returns the table and
+// nobody installs UOFiddler or copies a 5 MB file anywhere.
+//
+// **A file on disk** otherwise. That is the pipeline this replaces, kept for
+// installs with no shard link and for development, and deprecated rather than
+// removed: an operator who has one keeps working, and an operator who has a shard
+// never builds one. Passing an explicit `path` to `refresh()` still selects it,
+// which is the escape hatch for "import from this file, this once".
+//
+// **Overlays are always the filesystem's**, either way. There is nothing on the
+// shard to ask for: ServUO has no server-side notion of a custom cliloc, so the
+// `custom/` directory is the only place those ids exist.
+//
+// ── What that changed about WHEN this runs ───────────────────────
+//
+// Boot no longer imports on the shard path. The file path could hash 5 MB locally
+// on every restart and skip; the shard path would mean a sidecar round trip in the
+// boot sequence, for a table that changes when an operator patches their client —
+// an event they know about and we do not. So on the bridge, importing is an admin
+// action (Admin → Shard), and boot leaves whatever is loaded serving.
 //
 // That set is also why this has the atlas's escalation, in a lighter form. A
 // single corrupt file fails the parse loudly, but a source that has simply
@@ -96,8 +124,224 @@ const currentParser = (meta) => meta?.parserVersion === PARSER_VERSION
  *
  * `force` skips the hash check (an admin asking for a reimport). `approve`
  * additionally accepts a vanished source.
+ *
+ * Which SOURCE it reads is decided here and nowhere else: the shard when uo-link
+ * is configured and enabled, a file otherwise, and always a file when the caller
+ * named one.
  */
 async function refresh({ force = false, approve = false, path: pathOverride = '' } = {}) {
+  const override = String(pathOverride ?? '').trim()
+
+  if (override === '' && (await shardLinked())) {
+    return refreshFromShard({ force, approve })
+  }
+
+  return refreshFromFile({ force, approve, path: override })
+}
+
+/**
+ * Is there a shard to ask?
+ *
+ * Both halves matter. `baseUrl` alone is an install that has been configured and
+ * then switched off, and calling it would spend a 12 s timeout to learn what the
+ * row already says. Never throws: an unreadable config means "no shard", and the
+ * file path is a working answer.
+ */
+async function shardLinked() {
+  try {
+    const config = await uoLinkConfig.getSafe()
+    return Boolean(config?.enabled && config?.baseUrl)
+  } catch {
+    return false
+  }
+}
+
+/**
+ * Merge parsed sources in order, later winning.
+ *
+ * Shared by both paths, because the merge is the same question whichever end the
+ * base arrived from: what did each source contribute, and what did it override.
+ * The per-source breakdown is for the admin panel — an operator who adds an
+ * overlay wants to see it took effect, and `overrode: 0` on a file meant to
+ * re-label stock items says it did not.
+ */
+function mergeSources(groups) {
+  const merged = new Map()
+  const sources = []
+
+  for (const group of groups) {
+    let added = 0
+    let overrode = 0
+
+    for (const entry of group.entries) {
+      if (!Number.isInteger(entry.number)) continue
+      if (merged.has(entry.number)) overrode++
+      else added++
+      merged.set(entry.number, entry)
+    }
+
+    sources.push({
+      label: group.label,
+      kind: group.kind,
+      entries: group.entries.length,
+      added,
+      overrode,
+    })
+  }
+
+  return { entries: [...merged.values()], sources }
+}
+
+/** Overlay hashes as a `{ label: sha256 }` map, in merge order. */
+function overlayHashes(files) {
+  const hashes = {}
+  for (const file of files) hashes[file.label] = file.sha256
+  return hashes
+}
+
+/** The overlay half of a stored fingerprint — everything under `custom/`. */
+function onlyOverlays(hashes) {
+  if (!hashes) return null
+  const out = {}
+  for (const [label, sha] of Object.entries(hashes)) {
+    if (label.startsWith('custom/')) out[label] = sha
+  }
+  return out
+}
+
+/**
+ * Import with the shard as the base source.
+ *
+ * The gate is two-part, and neither part is something the shard can answer for
+ * us: has the client file changed (size/mtime/sha256, plus the shard's own
+ * `EXTRACTOR_VERSION`), and has any overlay beside the configured path changed.
+ * Either is drift; neither is the normal case.
+ */
+async function refreshFromShard({ force = false, approve = false } = {}) {
+  let fingerprint
+
+  try {
+    fingerprint = await bridge.fingerprint()
+  } catch (err) {
+    if (err instanceof bridge.ClilocBridgeError) {
+      return { status: 'unavailable', source: 'bridge', reason: err.message, code: err.code }
+    }
+    return { status: 'failed', source: 'bridge', reason: err.message }
+  }
+
+  const configured = await getClientPath()
+  const overlays = readOverlays(configured)
+  const hashes = overlayHashes(overlays.files)
+
+  const meta = await db.getMeta().catch(() => null)
+
+  if (
+    !force &&
+    bridge.sameSource(fingerprint, meta?.base) &&
+    sameSources(hashes, onlyOverlays(meta?.hashes)) &&
+    currentParser(meta)
+  ) {
+    return {
+      status: 'unchanged',
+      source: 'bridge',
+      file: fingerprint.file,
+      count: meta.count ?? null,
+      customCount: overlays.files.length,
+      hashing: fingerprint.hashing,
+    }
+  }
+
+  // An overlay that was loaded last time and is not there now is refused rather
+  // than applied — an unmounted volume and a deliberate deletion look identical
+  // from here, and the wrong guess silently drops every name that file gave.
+  // The BASE is deliberately not part of this question: an install upgraded from
+  // the file pipeline is *supposed* to stop having one.
+  const gone = missingOverlays(hashes, meta?.hashes)
+
+  if (gone.length > 0 && !approve) {
+    return {
+      status: 'needsReview',
+      source: 'bridge',
+      reason: `${gone.length} previously-loaded cliloc overlay(s) are missing; the existing table is unchanged`,
+      missingSources: gone,
+      file: fingerprint.file,
+    }
+  }
+
+  let base
+
+  try {
+    base = await bridge.readCliloc({ lang: bridge.DEFAULT_LANGUAGE })
+  } catch (err) {
+    if (err instanceof bridge.ClilocBridgeError) {
+      return { status: 'unavailable', source: 'bridge', reason: err.message, code: err.code }
+    }
+    return { status: 'failed', source: 'bridge', reason: err.message }
+  }
+
+  const groups = [{ label: base.source.file, kind: 'shard', entries: base.entries }]
+
+  for (const file of overlays.files) {
+    try {
+      groups.push({ label: file.label, kind: 'custom', entries: parseCliloc(file.buffer) })
+    } catch (err) {
+      if (err instanceof ClilocFormatError) {
+        // Named, because "which of my six overlay files is malformed" is
+        // otherwise a guessing game.
+        return {
+          status: 'unavailable',
+          source: 'bridge',
+          reason: `${file.label}: ${err.message}`,
+          code: err.code,
+        }
+      }
+      return { status: 'failed', source: 'bridge', reason: err.message }
+    }
+  }
+
+  const merged = mergeSources(groups)
+
+  try {
+    const applied = await db.replaceAll(merged.entries, {
+      source: 'bridge',
+      base: fingerprint,
+      hashes,
+      parserVersion: PARSER_VERSION,
+      sources: merged.sources,
+      file: base.source.file,
+      bytes: fingerprint.size,
+    })
+
+    invalidate()
+
+    return {
+      status: 'imported',
+      source: 'bridge',
+      file: base.source.file,
+      count: applied.count,
+      parsed: merged.entries.length,
+      blank: applied.blank,
+      pages: base.source.pages,
+      sources: merged.sources,
+      // The shard says how many rows it holds; this is how many arrived. They
+      // agree, or the walk is wrong in a way no count on its own would show.
+      reported: base.source.reported,
+      received: base.source.received,
+      overlayProblem: overlays.problem ?? undefined,
+      acceptedMissing: gone.length > 0 ? gone : undefined,
+    }
+  } catch (err) {
+    return { status: 'failed', source: 'bridge', reason: err.message }
+  }
+}
+
+/**
+ * Import from a converted file on disk — the pre-protocol-8 pipeline, unchanged.
+ *
+ * Deprecated but supported: an install with no shard link has no other way to get
+ * a table, and development without a running ServUO is the same case.
+ */
+async function refreshFromFile({ force = false, approve = false, path: pathOverride = '' } = {}) {
   // An explicit override wins outright — a one-off "use this file", which must
   // not be silently overruled by the configured path the way an env default is.
   const configured = pathOverride.trim() !== '' ? pathOverride.trim() : await getClientPath()
@@ -153,7 +397,10 @@ async function refresh({ force = false, approve = false, path: pathOverride = ''
   }
 
   try {
-    const applied = await db.replaceAll(parsed.entries, parsed.source)
+    // `source: 'file'` is what lets the NEXT refresh — and `status()` — tell a
+    // table built from a converted file from one built over the bridge. Without
+    // it an install that gains a shard link looks like it already imported.
+    const applied = await db.replaceAll(parsed.entries, { ...parsed.source, source: 'file' })
     invalidate()
     return {
       status: 'imported',
@@ -177,9 +424,22 @@ async function refresh({ force = false, approve = false, path: pathOverride = ''
 /**
  * Boot hook. Best-effort by contract: it logs and returns, never throws, so a
  * missing or malformed cliloc file can never stop the site coming up.
+ *
+ * **On the bridge it imports nothing**, deliberately. The file path can hash a
+ * local 5 MB file on every restart and skip in 14 ms; asking the shard would put
+ * a sidecar round trip in the boot sequence to answer a question whose answer is
+ * "no" every time except after a client patch — which is an operator action, and
+ * therefore something an operator can press a button for. Whatever table is
+ * loaded keeps serving, which is exactly what happens today when a restart finds
+ * nothing changed.
  */
 async function refreshOnBoot() {
   try {
+    if (await shardLinked()) {
+      log.info('cliloc table comes from the shard; import is admin-triggered (Admin → Shard)')
+      return { status: 'skipped', source: 'bridge', reason: 'the shard is the cliloc source' }
+    }
+
     const result = await refresh()
     switch (result.status) {
       case 'imported':
@@ -220,8 +480,18 @@ async function refreshOnBoot() {
   }
 }
 
-/** Everything the admin panel needs to describe cliloc state. */
+/**
+ * Everything the admin panel needs to describe cliloc state.
+ *
+ * Two shapes, one per source, sharing every field a panel actually renders
+ * (`count`, `drift`, `problem`, `sources`, `missingSources`, `importedAt`). What
+ * differs is what `file` means and what a problem with it looks like: on the
+ * bridge it is the shard's own client file and the problems are transport ones,
+ * on disk it is a path an operator typed.
+ */
 async function status({ path: pathOverride = '' } = {}) {
+  if (pathOverride.trim() === '' && (await shardLinked())) return shardStatus()
+
   const configured = pathOverride.trim() !== '' ? pathOverride.trim() : await getClientPath()
   const meta = await db.getMeta().catch(() => null)
   const loaded = await db.count().catch(() => 0)
@@ -280,6 +550,72 @@ async function status({ path: pathOverride = '' } = {}) {
   }
 }
 
+/**
+ * Status when the shard is the source.
+ *
+ * The one thing worth knowing here that the file path has no equivalent of:
+ * `hashing`. The shard reports a null `sha256` for a client file it has not
+ * hashed yet — hashing the 343 MB of art and animation it also serves cannot fit
+ * in a 10 s reply, so it happens on its own thread — and a null hash means "ask
+ * again", never "changed". Drift falls back to (size, mtime) meanwhile, which is
+ * the same gate the shard itself applies, so an operator is never blocked from
+ * importing by a hash that has not landed.
+ */
+async function shardStatus() {
+  const configured = await getClientPath()
+  const meta = await db.getMeta().catch(() => null)
+  const loaded = await db.count().catch(() => 0)
+
+  const overlays = readOverlays(configured)
+  const hashes = overlayHashes(overlays.files)
+
+  let fingerprint = null
+  let problem = overlays.problem ?? null
+  let code = null
+
+  try {
+    fingerprint = await bridge.fingerprint()
+  } catch (err) {
+    problem = err.message
+    code = err.code ?? null
+  }
+
+  const drift = fingerprint
+    ? !bridge.sameSource(fingerprint, meta?.base) ||
+      !sameSources(hashes, onlyOverlays(meta?.hashes)) ||
+      !currentParser(meta)
+    : null
+
+  return {
+    source: 'bridge',
+    configured: true,
+    // The overlay directory, which is all the path setting still selects on this
+    // source. Reported so a panel can say where `custom/` is being read from.
+    path: configured,
+    file: fingerprint?.file ?? bridge.SOURCE_FILE,
+    fileReadable: Boolean(fingerprint),
+    problem,
+    code,
+    drift,
+    count: loaded,
+    shard: fingerprint
+      ? {
+          size: fingerprint.size,
+          mtime: fingerprint.mtime,
+          sha256: fingerprint.sha256,
+          extractorVersion: fingerprint.extractorVersion,
+          hashing: fingerprint.hashing,
+          complete: fingerprint.complete,
+        }
+      : null,
+    sources: Object.keys(hashes),
+    loadedSources: meta?.sources ?? null,
+    missingSources: missingOverlays(hashes, meta?.hashes),
+    importedAt: meta?.importedAt ?? null,
+    sourceBytes: meta?.base?.size ?? meta?.bytes ?? null,
+  }
+}
+
 // ── Lookup ─────────────────────────────────────────────────────────────────
 //
 // Resolution happens SERVER-SIDE, not in the browser. Two reasons: the table is
@@ -359,6 +695,7 @@ module.exports = {
   SETTING_KEY,
   getClientPath,
   setClientPath,
+  shardLinked,
   refresh,
   refreshOnBoot,
   status,

server/model/uoLinkConfig/uoLinkConfig.model.js

@@ -11,8 +11,11 @@ const { secretBox } = require('../../core')
 // Only used before an admin has saved anything — the stored row wins once it exists,
 // and UOLINK_PROTOCOL still overrides for an operator running an older sidecar.
 //
-// This says 7 because this build speaks protocol 7: the idempotency key and the
-// participation ledger (6), and the world verbs plus the targeted lease planes (7).
+// This says 8 because this build speaks protocol 8: the idempotency key and the
+// participation ledger (6), the world verbs plus the targeted lease planes (7), and
+// the Asset Bridge (8) -- of which this module is the first consumer, importing the
+// cliloc table over `GET /cliloc` instead of reading a file an operator converted by
+// hand (docs/link/v8.md §9).
 //
 // It said 4 before 5, and 3 for a while after protocol 4 shipped — which is the bug this
 // constant was introduced to fix. A FRESH install pinned 3, the sidecar answered
@@ -32,7 +35,7 @@ const { secretBox } = require('../../core')
 // (`PROTOCOL_VERSION`) and the overlay's in `servuo-plugins/overlay.toml`; the thing that
 // actually pairs them is the installer's bundle check, at deploy time. So bumping this in
 // the same change as the emitters is still the discipline, and no test here replaces it.
-const DEFAULT_PROTOCOL = Number(process.env.UOLINK_PROTOCOL) || 7
+const DEFAULT_PROTOCOL = Number(process.env.UOLINK_PROTOCOL) || 8
 
 function toSafe(row) {
   if (!row) {