feat(assets): creature artwork from the shard's own client (Phase 3)

Until now the only way a creature got a picture on this site was for an
operator to open UOFiddler on a desktop, export sprites by hand, copy them to
the web host and write a spawnAtlas.art.json naming each one. Almost nobody
did, so shard_spawn_creatures.art was NULL on every install.

The shard has had those files the whole time. Admin -> Shard -> Import now
walks its asset manifest, fetches only the sprites whose hash changed, writes
them under uploads/atlas/, asks the shard for a body id per atlas creature
(§8: it CONSTRUCTS the creature and reads Body.BodyID, which is the only thing
that is right for a shard's own custom creatures) and points each creature at
its picture. On a stock client that is 787 portraits, about a megabyte.

**The one thing v8.md §12 got wrong, and it is not cosmetic.** It says
`shard_spawn_creatures.art` "starts being filled by the import". That table is
emptied and refilled by replaceAtlas on EVERY atlas refresh, and a refresh runs
on every boot -- so a filename stored there would be destroyed by an ordinary
re-parse of the ServUO tree, with the next Update finding the client files
unchanged, reporting "nothing to do", and never restoring it. Nothing would
report a fault; the pictures would just be gone.

So the assets and the body map live in their own tables outside that blast
radius, and applyAtlas re-derives `art` on the way past as
`{ ...derived, ...operatorMap }` -- which is also the one place "the operator's
own artwork wins" is enforced, on every rebuild rather than only at import.

Smaller decisions worth not rediscovering:

- The derivation joins on the catalogue KEY, not on the body id. The simpler
  join is correct today and stops being correct the moment phase 6 adds
  body/400/a2/f0, at which point one slug matches dozens of rows.
- Filenames are content-addressed. A stable name overwritten in place leaves
  every browser and CDN serving the previous client's sprite, with the database
  row perfectly correct.
- An unchanged key whose FILE is missing is fetched again. The row and the disk
  can disagree (a wiped uploads volume, a restore from a dump), and a broken
  image on a creature page is worse than one re-fetched sprite.
- A key the shard cannot render is not a failure. Two thirds of the playable
  ghost and gargoyle bodies have no art on a stock client, and an import that
  reported eight failures every time would teach an operator to ignore the panel.
- A key that VANISHED from the manifest needs review before anything changes:
  an unmounted client volume and a deliberate downgrade look identical here.

23 new tests; 674 server and 42 client tests pass. The SQL was also run against
a real MariaDB, which is what proved the CONCAT join and the singleton CHECK.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4

server/test/assetBridge.test.js

@@ -0,0 +1,368 @@
+const { test } = require('node:test')
+const assert = require('node:assert/strict')
+
+const uoLinkClient = require('../utils/uoLinkClient')
+const bridge = require('../utils/assetBridge')
+
+// The three walks over the asset plane, driven against a stubbed sidecar client
+// (docs/link/v8.md §5, §6, §8 — protocol 8, phase 3).
+//
+// Two families of failure are asserted here and they are not the same shape.
+//
+// **The envelope failures** are ways the shard can be wrong that leave this side
+// holding a catalogue it believes is complete. They are invisible downstream: a
+// catalogue missing its last three hundred bodies renders as a site where some
+// creatures have pictures and some do not, which is exactly what NO catalogue
+// looks like. Each corresponds to a field §3.4 puts on the wire specifically so
+// this side can tell the difference.
+//
+// **The absence failures** are the opposite mistake, and phase 3's more likely
+// one: treating a body this client has no art for as an error. Two thirds of the
+// playable ghost and gargoyle bodies are in that state on a stock client, and an
+// import that failed — or even warned loudly — on them would teach an operator to
+// ignore the panel.
+
+const saved = {}
+
+function stub({ sources, manifest = [], fetch = [], bodies = [] } = {}) {
+  saved.getAssetSources = uoLinkClient.getAssetSources
+  saved.getAssetManifest = uoLinkClient.getAssetManifest
+  saved.fetchAssets = uoLinkClient.fetchAssets
+  saved.resolveBodies = uoLinkClient.resolveBodies
+
+  const calls = { manifest: [], fetch: [], bodies: [] }
+
+  uoLinkClient.getAssetSources = async () => sources
+  uoLinkClient.getAssetManifest = async ({ family, cursor } = {}) => {
+    calls.manifest.push({ family: family ?? null, cursor: cursor ?? null })
+    const next = manifest.shift()
+    if (!next) throw new Error('the walk asked for more manifest pages than the test supplied')
+    return next
+  }
+  uoLinkClient.fetchAssets = async ({ keys, catalog, cursor } = {}) => {
+    calls.fetch.push({ keys, catalog: catalog ?? null, cursor: cursor ?? null })
+    const next = fetch.shift()
+    if (!next) throw new Error('the walk asked for more fetch pages than the test supplied')
+    return next
+  }
+  uoLinkClient.resolveBodies = async (types) => {
+    calls.bodies.push(types)
+    const next = bodies.shift()
+    if (!next) throw new Error('the walk asked for more body chunks than the test supplied')
+    return next
+  }
+
+  return calls
+}
+
+function restore() {
+  for (const [name, fn] of Object.entries(saved)) {
+    if (fn) uoLinkClient[name] = fn
+  }
+}
+
+const ok = (data) => ({ ok: true, status: 200, data })
+const fail = (status, data) => ({ ok: false, status, data })
+
+const CATALOG = 'a3f9c21d4b8e0771'
+
+const manifestPage = (rows, extra = {}) =>
+  ok({
+    kind: 'assets.manifest.ok',
+    family: 'body',
+    catalog: CATALOG,
+    extractorVersion: 1,
+    playerBodies: [400, 401, 402, 403],
+    scanned: rows.length,
+    rows,
+    more: false,
+    cut: 'end',
+    ...extra,
+  })
+
+const fetchPage = (rows, extra = {}) =>
+  ok({
+    kind: 'assets.fetch.ok',
+    family: 'body',
+    catalog: CATALOG,
+    rows,
+    more: false,
+    cut: 'end',
+    ...extra,
+  })
+
+const row = (body, sha = 'aa') => ({
+  key: `body/${body}/a0`,
+  sha256: sha,
+  bytes: 900,
+  width: 24,
+  height: 63,
+  body,
+  direction: 1,
+})
+
+const png = Buffer.from([0x89, 0x50, 0x4e, 0x47]).toString('base64')
+
+const sourcesReply = (extra = {}) =>
+  ok({
+    kind: 'assets.sources.ok',
+    extractorVersion: 1,
+    imaging: { ok: true },
+    hashing: false,
+    complete: true,
+    files: [
+      { name: 'anim.idx', size: 10, mtime: 1, sha256: 'a' },
+      { name: 'anim.mul', size: 20, mtime: 2, sha256: 'b' },
+      { name: 'body.def', size: 30, mtime: 3, sha256: 'c' },
+      // Not a source this family reads: `art.mul` decides item pictures, not
+      // creature ones, and folding it in would make every item-art change look
+      // like a reason to re-import the whole body catalogue.
+      { name: 'art.mul', size: 148000000, mtime: 4, sha256: 'd' },
+    ],
+    ...extra,
+  })
+
+// ── the source gate (§6 stage 1) ──────────────────────────────────────────
+
+test('the source fingerprint keeps only the files the body catalogue reads', async (t) => {
+  stub({ sources: sourcesReply() })
+  t.after(restore)
+
+  const fingerprint = await bridge.sourceFingerprint()
+
+  assert.deepEqual(Object.keys(fingerprint.files).sort(), ['anim.idx', 'anim.mul', 'body.def'])
+  assert.equal(fingerprint.extractorVersion, 1)
+})
+
+test('a bumped extractor version is drift even when every client file is identical', () => {
+  const files = { 'anim.mul': { size: 1, mtime: 2, sha256: 'x' } }
+
+  assert.equal(
+    bridge.sameSources({ files, extractorVersion: 1 }, { files, extractorVersion: 1 }),
+    true,
+  )
+  // §7: a corrected frame offset changes every derived byte while every source
+  // file stays byte-identical. If this returned true the fix would never reach
+  // an install whose client never moves.
+  assert.equal(
+    bridge.sameSources({ files, extractorVersion: 2 }, { files, extractorVersion: 1 }),
+    false,
+  )
+})
+
+test('a client that GAINED an anim file is drift, not a match', () => {
+  const before = { files: { 'anim.mul': { size: 1, mtime: 2, sha256: 'x' } }, extractorVersion: 1 }
+  const after = {
+    files: {
+      'anim.mul': { size: 1, mtime: 2, sha256: 'x' },
+      // A client that grows an anim5.mul is a client whose gargoyles suddenly
+      // resolve. Comparing only the files present in both would call that
+      // unchanged and never import them.
+      'anim5.mul': { size: 9, mtime: 9, sha256: 'y' },
+    },
+    extractorVersion: 1,
+  }
+
+  assert.equal(bridge.sameSources(before, after), false)
+})
+
+test('a null hash falls back to size and mtime rather than reading as changed', () => {
+  // The shard hashes 195 MB anim files off the request path, so a null sha256 is
+  // "not computed yet". Treating it as a difference would re-import the whole
+  // catalogue on every restart until the background pass finished.
+  const a = { files: { 'anim.mul': { size: 5, mtime: 7, sha256: null } }, extractorVersion: 1 }
+  const b = { files: { 'anim.mul': { size: 5, mtime: 7, sha256: 'later' } }, extractorVersion: 1 }
+
+  assert.equal(bridge.sameSources(a, b), true)
+})
+
+// ── the manifest walk (§6 stage 2) ────────────────────────────────────────
+
+test('the manifest walks every page and stops only on cut: end', async (t) => {
+  const calls = stub({
+    manifest: [
+      manifestPage([row(12), row(34)], { more: true, cursor: 'b:34', cut: 'limit' }),
+      manifestPage([row(400)]),
+    ],
+  })
+  t.after(restore)
+
+  const result = await bridge.readManifest()
+
+  assert.equal(result.rows.length, 3)
+  assert.equal(result.catalog, CATALOG)
+  assert.deepEqual(result.playerBodies, [400, 401, 402, 403])
+  assert.deepEqual(
+    calls.manifest.map((c) => c.cursor),
+    [null, 'b:34'],
+  )
+})
+
+test('a short page that did not end the catalogue is refused', async (t) => {
+  // `cut: 'limit'` with `more: false` is the shard saying it stopped for its own
+  // reason. Importing what arrived would silently drop every body after it, and
+  // the result is indistinguishable from a client with fewer creatures.
+  stub({ manifest: [manifestPage([row(12)], { more: false, cut: 'limit' })] })
+  t.after(restore)
+
+  await assert.rejects(() => bridge.readManifest(), /stopped sending assets/)
+})
+
+test('a cursor that does not advance is refused rather than looped on', async (t) => {
+  stub({
+    manifest: [
+      manifestPage([row(12)], { more: true, cursor: 'b:12', cut: 'budget' }),
+      manifestPage([row(13)], { more: true, cursor: 'b:12', cut: 'budget' }),
+    ],
+  })
+  t.after(restore)
+
+  await assert.rejects(() => bridge.readManifest(), /without advancing its cursor/)
+})
+
+test('the client files changing mid-walk aborts the whole import', async (t) => {
+  // The catalogue id is derived from the client files themselves, so a change
+  // between two pages means half of what we hold describes files that no longer
+  // exist — and nothing later can tell which half.
+  stub({
+    manifest: [
+      manifestPage([row(12)], { more: true, cursor: 'b:12', cut: 'limit' }),
+      manifestPage([row(34)], { catalog: 'something-else' }),
+    ],
+  })
+  t.after(restore)
+
+  await assert.rejects(() => bridge.readManifest(), /changed while the manifest was being read/)
+})
+
+// ── the fetch (§5) ────────────────────────────────────────────────────────
+
+test('a fetch passes the catalogue id and decodes the PNG', async (t) => {
+  const calls = stub({
+    fetch: [fetchPage([{ key: 'body/12/a0', status: 'ok', sha256: 'aa', bytes: 4, width: 24, height: 63, body: 12, direction: 1, png }])],
+  })
+  t.after(restore)
+
+  const { assets } = await bridge.fetchAssets({ keys: ['body/12/a0'], catalog: CATALOG })
+
+  assert.equal(calls.fetch[0].catalog, CATALOG)
+  assert.equal(assets.get('body/12/a0').png.length, 4)
+  assert.equal(assets.get('body/12/a0').width, 24)
+})
+
+test('an absent asset is a counted row, not a failed fetch', async (t) => {
+  // The whole reason this is not an error: two thirds of the playable ghost and
+  // gargoyle bodies have no art on a stock client (§5.2), and an import that
+  // failed on them could never succeed.
+  stub({
+    fetch: [
+      fetchPage([
+        { key: 'body/12/a0', status: 'ok', sha256: 'aa', bytes: 4, png },
+        { key: 'body/666/a0', status: 'absent' },
+        { key: 'body/400/a2/f3', status: 'unsupported' },
+      ]),
+    ],
+  })
+  t.after(restore)
+
+  const { assets, missing } = await bridge.fetchAssets({
+    keys: ['body/12/a0', 'body/666/a0', 'body/400/a2/f3'],
+    catalog: CATALOG,
+  })
+
+  assert.equal(assets.size, 1)
+  // Counted apart, because they mean different things: `absent` is a gap in the
+  // operator's client and `unsupported` is a bug on this side.
+  assert.equal(missing.absent, 1)
+  assert.equal(missing.unsupported, 1)
+})
+
+test('a busy shard is retried rather than failing the walk', async (t) => {
+  saved.fetchAssets = uoLinkClient.fetchAssets
+  t.after(restore)
+
+  let attempts = 0
+
+  uoLinkClient.fetchAssets = async () => {
+    attempts++
+    if (attempts < 3) return fail(425, { reason: 'busy' })
+    return fetchPage([{ key: 'body/12/a0', status: 'ok', sha256: 'aa', bytes: 4, png }])
+  }
+
+  const { assets } = await bridge.fetchAssets({ keys: ['body/12/a0'], catalog: CATALOG })
+
+  assert.equal(attempts, 3)
+  assert.equal(assets.size, 1)
+})
+
+test('a shard host with no libgdiplus is named, not reported as a dead shard', async (t) => {
+  saved.getAssetManifest = uoLinkClient.getAssetManifest
+  t.after(restore)
+
+  uoLinkClient.getAssetManifest = async () =>
+    fail(503, { reason: "this shard host cannot render images - Mono's System.Drawing needs libgdiplus" })
+
+  await assert.rejects(
+    () => bridge.readManifest(),
+    (err) => err.code === 'NO_IMAGING',
+  )
+})
+
+// ── the body pass (§8) ────────────────────────────────────────────────────
+
+test('body resolution chunks to the shard cap and records every outcome', async (t) => {
+  const creatures = []
+
+  for (let i = 0; i < bridge.BODY_CHUNK + 5; i++) {
+    creatures.push({ slug: `c-${i}`, name: `Creature${i}` })
+  }
+
+  const reply = (types) =>
+    ok({
+      kind: 'assets.bodies.ok',
+      rows: types.map((type, i) => (i === 0 ? { type, status: 'unknown' } : { type, status: 'ok', body: 100 + i })),
+      more: false,
+      cut: 'end',
+    })
+
+  const calls = stub({ bodies: [] })
+  t.after(restore)
+
+  uoLinkClient.resolveBodies = async (types) => {
+    calls.bodies.push(types)
+    return reply(types)
+  }
+
+  const rows = await bridge.resolveBodies({ creatures })
+
+  // Two chunks, and neither over the cap: the shard REFUSES an over-long list
+  // rather than truncating it, so a chunk size above its cap does not degrade —
+  // every request fails.
+  assert.equal(calls.bodies.length, 2)
+  assert.ok(calls.bodies.every((chunk) => chunk.length <= bridge.BODY_CHUNK))
+
+  assert.equal(rows.length, creatures.length)
+  // The negative answers are kept. Without them the next pass asks again, and
+  // the pass costs a real constructor per name on the shard's Core thread.
+  assert.equal(rows.filter((r) => r.status === 'unknown').length, 2)
+})
+
+test('two slugs sharing a class name are asked once and both get the answer', async (t) => {
+  const calls = stub({
+    bodies: [
+      ok({ kind: 'assets.bodies.ok', rows: [{ type: 'GiantSpider', status: 'ok', body: 28 }], more: false, cut: 'end' }),
+    ],
+  })
+  t.after(restore)
+
+  const rows = await bridge.resolveBodies({
+    creatures: [
+      { slug: 'giant-spider', name: 'GiantSpider' },
+      { slug: 'giantspider', name: 'GiantSpider' },
+    ],
+  })
+
+  assert.deepEqual(calls.bodies[0], ['GiantSpider'])
+  assert.equal(rows.length, 2)
+  assert.ok(rows.every((r) => r.body === 28))
+})