feat(atlas): the spawn atlas reads the shard, not the shard's filesystem (Phase 7)
All checks were successful
PR Checks / server-tests (pull_request) Successful in 23s
PR Checks / frozen-manifest (pull_request) Successful in 1m8s
PR Checks / client-build (pull_request) Successful in 7m59s

`spawnAtlasSource.js` gains a second backend behind its existing interface
(docs/link/v8.md 10). Where a shard is linked and enabled the tree arrives over
the sidecar; where there is none, a local ServUO tree is read exactly as before.
An explicit --servuo path is an instruction and overrules both.

The parsers do not move. spawnAtlasParse.js is still pure, still fs-free and
still CI-covered without a ServUO tree anywhere near it; `buildFromFiles` is now
where the parse starts, and both readers feed it the same shape.

treeBridge.js walks the manifest and then the chunks. Three of its checks are
not decoration -- each is a way this ends in a tree that LOOKS imported, and XML
is forgiving enough that a mis-assembled spawn file parses cleanly and simply
has fewer spawns in it:

  - every chunk re-declares its address and carries the hash of its own
    uncompressed bytes, and chunks are placed by declared index rather than
    arrival order
  - the whole file is hashed after reassembly against its manifest row
  - the catalog must not move mid-walk, or the import is refused rather than
    stitched out of two trees

Boot does not call the shard. The same answer 17.7 gave the cliloc table, and
the same reasoning: a local tree hashes in ~120 ms and skips, while a round trip
in the boot sequence would answer "no" on every restart that did not follow a
map edit. Editing spawn files is an operator action, so importing is one --
Admin -> Spawn Atlas -> Import. What that costs is real and is said out loud in
the panel, the CLI and the log: an install on the bridge has NO automatic
refresh at all.

Two things the live walk found that the unit tests could not:

  - PARSER_VERSION 4 -> 5. The parse is order-sensitive in one place -- the
    decoration index keeps the FIRST item id it sees for a type -- and the two
    readers agreed on a stock tree by coincidence, since the filesystem reader
    walks each directory with localeCompare while the shard sorts whole relative
    paths. buildFromFiles now sorts by label, ordinally, once, whatever order
    the files arrived in. Identical input, a different answer for a handful of
    types: exactly what the version number exists to push through the hash gate.
    The parity test asserted deepEqual, which ignores key order; it now asserts
    serialised equality too.
  - The source fingerprint is taken over RAW BYTES at both ends. Hashing decoded
    text hashes a UTF-8 re-encoding -- identical for valid UTF-8, different for a
    file that is not, because an undecodable byte becomes U+FFFD and never comes
    back. One Latin-1 character in a creature name would have made the drift gate
    report a change on every import, forever, with the tree untouched.

A 200 from assets.sources also stopped meaning "the client files are on offer":
a shard may now serve its configuration tree while declining to serve its UO
client. Both client-file readers check `assetsEnabled` and say DISABLED, instead
of reading an empty file list as "your client has no cliloc.enu" and sending an
operator to their client install for a setting that lives on their shard.

Measured end to end against a live shard and the real sidecar: 141 files,
11.9 MB, 158 chunks, 3 pages, 1.33 MB on the wire, 512 ms; every file
byte-identical to disk; and the atlas built over the bridge identical to the one
built off it -- 6,455 points, 800 creatures, 387 regions, 558 landmarks,
25 champions, 309 decoration types.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4
This commit is contained in:
2026-09-14 02:00:43 -05:00
parent 679762b643
commit d4d5989926
13 changed files with 1432 additions and 73 deletions

View File

@@ -205,7 +205,7 @@ test('a short page that did not end the catalogue is refused', async (t) => {
stub({ manifest: [manifestPage([row(12)], { more: false, cut: 'limit' })] })
t.after(restore)
await assert.rejects(() => bridge.readManifest(), /stopped sending assets/)
await assert.rejects(() => bridge.readManifest(), /stopped sending asset rows/)
})
test('a cursor that does not advance is refused rather than looped on', async (t) => {

View File

@@ -0,0 +1,156 @@
// Which atlas source runs, and what boot does with the answer
// (docs/link/v8.md §10, §17.7; docs/website/SPAWN_ATLAS.md).
//
// The model is the only place that decides between a local ServUO tree and the
// shard bridge, so these drive it with the shard, the database and the
// filesystem all stubbed. Nothing here reaches a real sidecar or a real tree.
//
// The rule under test is the one the cliloc pipeline settled first and this
// inherits: the bridge wins whenever uo-link is configured and enabled, a local
// path is what a site with no shard link uses, and an explicit path is an
// instruction that overrules both.
const { test } = require('node:test')
const assert = require('node:assert/strict')
const atlas = require('../model/shardAtlas/shardAtlas.model')
const db = require('../model/shardAtlas/shardAtlas.db')
const source = require('../utils/spawnAtlasSource')
const uoLinkConfig = require('../model/uoLinkConfig/uoLinkConfig.model')
const { ctx } = require('./_setup')
const saved = {
getMeta: db.getMeta,
getPending: db.getPending,
getFacets: db.getFacets,
hashFrom: source.hashFrom,
buildFrom: source.buildFrom,
getSafe: uoLinkConfig.getSafe,
settingsGet: ctx.settings.get,
}
function restore() {
db.getMeta = saved.getMeta
db.getPending = saved.getPending
db.getFacets = saved.getFacets
source.hashFrom = saved.hashFrom
source.buildFrom = saved.buildFrom
uoLinkConfig.getSafe = saved.getSafe
ctx.settings.get = saved.settingsGet
}
/** Whatever source the model chose, captured rather than read. */
function rig({ linked = true, treePath = '', meta = null } = {}) {
const asked = { hash: [], build: [] }
uoLinkConfig.getSafe = async () => ({
enabled: linked,
baseUrl: linked ? 'http://127.0.0.1:8099' : null,
})
ctx.settings.get = async () => treePath
db.getMeta = async () => meta
db.getPending = async () => null
db.getFacets = async () => []
source.hashFrom = async (descriptor) => {
asked.hash.push(descriptor)
return { 'Data/Regions.xml': 'aa' }
}
source.buildFrom = async (descriptor) => {
asked.build.push(descriptor)
throw new Error('the test stops before a build')
}
return asked
}
test('a linked shard is the atlas source, and the configured path is not consulted', async () => {
const asked = rig({ linked: true, treePath: '/srv/servuo' })
const status = await atlas.status()
assert.equal(status.source, 'bridge')
assert.equal(status.path, 'the shard bridge')
assert.equal(status.configured, true)
assert.deepEqual(asked.hash[0], { kind: 'bridge', root: '' })
restore()
})
test('with no shard linked the configured tree is the source', async () => {
const asked = rig({ linked: false, treePath: '/srv/servuo' })
const status = await atlas.status()
assert.equal(status.source, 'fs')
assert.equal(status.path, '/srv/servuo')
assert.deepEqual(asked.hash[0], { kind: 'fs', root: '/srv/servuo' })
restore()
})
test('an explicit path overrules the bridge — it is an instruction, not a default', async () => {
const asked = rig({ linked: true, treePath: '/srv/servuo' })
await atlas.status({ path: '/tmp/other-tree' })
assert.deepEqual(asked.hash[0], { kind: 'fs', root: '/tmp/other-tree' })
restore()
})
test('no shard and no path is "nothing configured", not an error', async () => {
rig({ linked: false, treePath: '' })
const status = await atlas.status()
assert.equal(status.configured, false)
const result = await atlas.refresh()
assert.equal(result.status, 'skipped')
restore()
})
test('boot does not call the shard; it says where the import lives instead', async () => {
// §17.7's rule, and the reason it is not free: an install whose atlas comes
// over the bridge has NO automatic refresh at all, so the skip has to be
// deliberate and visible rather than a path that quietly does nothing.
const asked = rig({ linked: true, treePath: '/srv/servuo' })
const result = await atlas.refreshOnBoot()
assert.equal(result.status, 'skipped')
assert.equal(result.source, 'bridge')
assert.equal(asked.hash.length, 0, 'boot made no shard call at all')
assert.equal(asked.build.length, 0)
restore()
})
test('boot still refreshes by itself from a local tree', async () => {
const asked = rig({ linked: false, treePath: '/srv/servuo' })
await atlas.refreshOnBoot()
assert.deepEqual(asked.hash[0], { kind: 'fs', root: '/srv/servuo' })
restore()
})
test('a source that cannot be read is reported, with which end could not read it', async () => {
rig({ linked: true, treePath: '' })
source.hashFrom = async () => {
const { TreeBridgeError } = require('../utils/treeBridge')
throw new TreeBridgeError('the shard is not serving its tree', 'DISABLED')
}
const result = await atlas.refresh()
assert.equal(result.status, 'unavailable')
assert.equal(result.source, 'bridge')
assert.equal(result.code, 'DISABLED')
restore()
})

View File

@@ -0,0 +1,413 @@
const fs = require('fs')
const os = require('os')
const path = require('path')
const zlib = require('zlib')
const crypto = require('crypto')
const { test, after } = require('node:test')
const assert = require('node:assert/strict')
// Installs the `ctx` core would have handed over — treeBridge takes a logger
// from it at call time, so a test that skips this dies on the first log line.
require('./_setup')
const uoLinkClient = require('../utils/uoLinkClient')
const treeBridge = require('../utils/treeBridge')
const { buildFrom, readFrom } = require('../utils/spawnAtlasSource')
// The atlas source walk over the bridge (docs/link/v8.md §10 — protocol 8,
// phase 7), driven against a stub that behaves the way `BridgeTree.cs` does.
//
// The test that matters most is the LAST one: the same synthetic tree, read off
// a disk and read over the bridge, must produce a byte-identical atlas. Every
// other test here is one specific way a walk can end in something that LOOKS
// imported — which is the failure mode this whole family is shaped around, since
// XML is forgiving enough that a tree reassembled wrong still parses and simply
// has fewer spawns in it.
const saved = {}
function restore() {
for (const [name, fn] of Object.entries(saved)) {
if (fn) uoLinkClient[name] = fn
}
}
after(restore)
const ok = (data) => ({ ok: true, status: 200, data })
const fail = (status, data) => ({ ok: false, status, data })
const sha = (buf) => crypto.createHash('sha256').update(buf).digest('hex')
// ── A stub shard ───────────────────────────────────────────────────────────
//
// Chunks and gzips exactly as the overlay does, so the reader under test is
// exercised against the wire shape rather than against a convenience.
function serveTree(files, { chunkBytes = 64, catalog = 'cafebabe12345678', tweak = {} } = {}) {
saved.getAssetManifest = saved.getAssetManifest ?? uoLinkClient.getAssetManifest
saved.fetchAssets = saved.fetchAssets ?? uoLinkClient.fetchAssets
const chunksOf = (bytes) => Math.max(1, Math.ceil(bytes.length / chunkBytes))
const rows = files.map(([label, bytes]) => ({
key: `tree/${label}`,
label,
bytes: bytes.length,
mtime: 1700000000000,
chunks: chunksOf(bytes),
sha256: sha(bytes),
}))
const byLabel = new Map(files)
const calls = { manifest: 0, fetch: 0 }
uoLinkClient.getAssetManifest = async ({ family, cursor } = {}) => {
calls.manifest++
assert.equal(family, 'tree', 'the walk must name its family')
assert.equal(cursor ?? null, null, 'this stub answers in one page')
if (tweak.manifestReply) return tweak.manifestReply(rows, catalog)
return ok({
kind: 'assets.manifest.ok',
family: 'tree',
catalog,
chunkBytes,
total: rows.length,
rows,
more: false,
cut: 'end',
})
}
uoLinkClient.fetchAssets = async ({ keys, catalog: asked } = {}) => {
calls.fetch++
assert.equal(asked, catalog, 'a fetch must assert the catalog it was listed under')
const out = []
for (const key of keys) {
const slash = key.lastIndexOf('/')
const label = key.slice('tree/'.length, slash)
const chunk = Number(key.slice(slash + 2))
const bytes = byLabel.get(label)
if (!bytes) {
out.push({ key, status: 'absent', reason: 'no such file' })
continue
}
const raw = bytes.subarray(chunk * chunkBytes, (chunk + 1) * chunkBytes)
out.push({
key,
status: 'ok',
label,
chunk,
chunks: chunksOf(bytes),
offset: chunk * chunkBytes,
bytes: raw.length,
sha256: sha(raw),
gzip: zlib.gzipSync(raw).toString('base64'),
})
}
if (tweak.fetchRows) tweak.fetchRows(out)
return ok({
kind: 'assets.fetch.ok',
family: 'tree',
catalog,
rows: out,
more: false,
cut: 'end',
...(tweak.fetchEnvelope || {}),
})
}
return calls
}
const FILES = [
['Data/Regions.xml', Buffer.from('<ServerRegions><Region /></ServerRegions>', 'utf8')],
['Spawns/Sosaria.xml', Buffer.from('<Spawns>' + 'x'.repeat(400) + '</Spawns>', 'utf8')],
]
// ── The walk ───────────────────────────────────────────────────────────────
test('a chunked, gzipped tree reassembles to the exact bytes the shard holds', async () => {
const calls = serveTree(FILES)
const { files } = await treeBridge.readSources()
assert.equal(files.length, 2)
assert.equal(calls.manifest, 1, 'one manifest call')
for (const [label, bytes] of FILES) {
const got = files.find((f) => f.label === label)
assert.ok(got, `${label} came back`)
assert.equal(got.text, bytes.toString('utf8'))
assert.equal(got.bytes, bytes.length)
assert.equal(got.sha256, sha(bytes))
}
restore()
})
test('chunks are placed by their declared index, not by the order they arrive in', async () => {
// The rows come back in the order they were asked for today. A reader that
// appended them would agree with this test until the day something reorders a
// page — and then produce a file that still parses and is quietly wrong.
serveTree(FILES, { tweak: { fetchRows: (rows) => rows.reverse() } })
const { files } = await treeBridge.readSources()
const spawns = files.find((f) => f.label === 'Spawns/Sosaria.xml')
assert.equal(spawns.text, FILES[1][1].toString('utf8'))
restore()
})
test('a chunk the shard refuses fails the import rather than shortening a file', async () => {
serveTree(FILES, {
tweak: {
fetchRows: (rows) => {
rows[rows.length - 1] = { key: rows[rows.length - 1].key, status: 'absent', reason: 'gone' }
},
},
})
await assert.rejects(() => treeBridge.readSources(), /refused .*absent: gone/)
restore()
})
test('a missing chunk is named, with which one and out of how many', async () => {
serveTree(FILES, { tweak: { fetchRows: (rows) => rows.splice(2, 1) } })
await assert.rejects(() => treeBridge.readSources(), /missing chunk 1 of/)
restore()
})
test('a chunk that does not match its own hash is refused', async () => {
serveTree(FILES, {
tweak: {
fetchRows: (rows) => {
rows[1].gzip = zlib.gzipSync(Buffer.from('not what was hashed')).toString('base64')
rows[1].bytes = 19
},
},
})
await assert.rejects(() => treeBridge.readSources(), /does not match its own hash/)
restore()
})
test('a file whose reassembly does not match its manifest hash is refused', async () => {
// Every chunk is individually honest and the whole is not — which is what a
// dropped or duplicated chunk looks like from here.
serveTree(FILES, {
tweak: {
manifestReply: (rows, catalog) =>
ok({
kind: 'assets.manifest.ok',
family: 'tree',
catalog,
chunkBytes: 64,
total: rows.length,
rows: rows.map((r) => ({ ...r, sha256: r.sha256.replace(/^./, '0') })),
more: false,
cut: 'end',
}),
},
})
await assert.rejects(() => treeBridge.readSources(), /does not match the hash its manifest row carried/)
restore()
})
test('a tree that moves mid-read is refused rather than stitched together', async () => {
serveTree(FILES, { tweak: { fetchEnvelope: { catalog: 'deadbeefdeadbeef' } } })
await assert.rejects(() => treeBridge.readSources(), {
code: 'SOURCE_CHANGED',
})
restore()
})
test('a short page that did not end the walk is refused', async () => {
serveTree(FILES, { tweak: { fetchEnvelope: { more: false, cut: 'budget' } } })
await assert.rejects(() => treeBridge.readSources(), { code: 'INCOMPLETE' })
restore()
})
test('403 names the tree switch, not the asset switch', async () => {
// The two consents are different settings with different fixes, and sending an
// operator to Bridge.AssetsEnabled when the answer is Bridge.TreeEnabled costs
// them an afternoon.
saved.getAssetManifest = saved.getAssetManifest ?? uoLinkClient.getAssetManifest
uoLinkClient.getAssetManifest = async () => fail(403, { reason: 'not served' })
await assert.rejects(() => treeBridge.readSources(), {
code: 'DISABLED',
message: /Bridge\.TreeEnabled/,
})
restore()
})
test('the manifest alone answers the drift gate, with no file bytes at all', async () => {
const calls = serveTree(FILES)
const listing = await treeBridge.manifest()
const fingerprint = treeBridge.fingerprintOf(listing.files)
assert.equal(calls.fetch, 0, 'nothing was fetched to answer "has anything changed"')
assert.deepEqual(Object.keys(fingerprint).sort(), [
'Data/Regions.xml',
'Spawns/Sosaria.xml',
])
assert.equal(fingerprint['Spawns/Sosaria.xml'], sha(FILES[1][1]))
restore()
})
test('a zero-byte file crosses as one chunk carrying a real gzip stream', async () => {
// Stock ServUO 57.4 ships TWO empty decoration files, so this is the ordinary
// case rather than an edge one — and it is the case .NET gets wrong on its own:
// `GZipStream` writes the gzip header lazily, so zero bytes in produces zero
// bytes out, which is not a gzip stream at all. The overlay answers with a
// literal empty member; a reader that accepted an empty payload instead would
// have hidden the bug rather than caught it.
const empty = Buffer.alloc(0)
serveTree([
['Data/Regions.xml', Buffer.from('<ServerRegions />', 'utf8')],
['Data/Decoration/nothing.cfg', empty],
])
const { files } = await treeBridge.readSources()
const blank = files.find((f) => f.label === 'Data/Decoration/nothing.cfg')
assert.equal(blank.bytes, 0)
assert.equal(blank.text, '')
assert.equal(blank.sha256, sha(empty))
restore()
})
test('an empty payload for a chunk is refused, whatever the row declares', async () => {
serveTree(FILES, { tweak: { fetchRows: (rows) => { rows[0].gzip = '' } } })
await assert.rejects(() => treeBridge.readSources(), /Could not decompress/)
restore()
})
test('a tree-only shard tells the client-file readers so, rather than looking empty', async () => {
// Phase 7 opened `assets.sources` to a shard that serves ONLY its configuration
// tree, so a 200 from it stopped meaning "the client files are on offer". Both
// client-file readers have to say DISABLED rather than read the empty file list
// as "your UO client has no cliloc.enu", which sends an operator to their client
// install for a setting that lives on their shard.
const clilocBridge = require('../utils/clilocBridge')
const assetBridge2 = require('../utils/assetBridge')
saved.getAssetSources = saved.getAssetSources ?? uoLinkClient.getAssetSources
uoLinkClient.getAssetSources = async () =>
ok({
kind: 'assets.sources.ok',
extractorVersion: 3,
assetsEnabled: false,
treeEnabled: true,
imaging: { ok: true },
families: ['tree'],
files: [],
more: false,
cut: 'end',
complete: true,
})
await assert.rejects(() => clilocBridge.fingerprint(), {
code: 'DISABLED',
message: /Bridge\.AssetsEnabled/,
})
await assert.rejects(() => assetBridge2.sourceFingerprint(), {
code: 'DISABLED',
message: /Bridge\.AssetsEnabled/,
})
restore()
})
// ── The parity test ────────────────────────────────────────────────────────
test('the same tree read off a disk and read over the bridge builds the same atlas', async () => {
const root = fs.mkdtempSync(path.join(os.tmpdir(), 'atlas-parity-'))
const tree = [
[
'Data/Regions.xml',
'<?xml version="1.0"?><ServerRegions>'
+ '<region type="Region"><name>Yew</name><map>Sosaria</map>'
+ '<rect x="100" y="100" width="200" height="200" /></region>'
+ '</ServerRegions>',
],
[
'Data/Locations/Sosaria.xml',
'<?xml version="1.0"?><locations><location><name>Yew Bank</name>'
+ '<x>150</x><y>150</y><z>0</z></location></locations>',
],
[
'Spawns/Sosaria.xml',
'<?xml version="1.0"?><Spawns>'
+ Array.from({ length: 40 }, (_, i) =>
`<Spawn Name="s${i}" X="${120 + i}" Y="${130 + i}" Map="Sosaria" Count="3" `
+ 'Running="True" MinDelay="00:05:00" MaxDelay="00:10:00" SpawnRange="5" '
+ 'HomeRange="5"><Object>Lizardman</Object></Spawn>').join('')
+ '</Spawns>',
],
['Config/ChampionSpawns.xml', '<?xml version="1.0"?><champions />'],
['Data/Decoration/top.cfg', 'Brazier 0x0E31\n100 100 0\n'],
['Data/Decoration/Deep/nested.cfg', 'LargeCrate 0x0E3C\n500 500 0\n'],
]
for (const [label, text] of tree) {
const file = path.join(root, label.replace(/\//g, path.sep))
fs.mkdirSync(path.dirname(file), { recursive: true })
fs.writeFileSync(file, text, 'utf8')
}
const fromDisk = await buildFrom({ kind: 'fs', root })
// A chunk size small enough that the spawn file alone is dozens of chunks,
// because a one-chunk-per-file test proves nothing about reassembly.
serveTree(tree.map(([label, text]) => [label, Buffer.from(text, 'utf8')]), { chunkBytes: 37 })
const fromBridge = await buildFrom({ kind: 'bridge' })
// `generatedAt` is a timestamp and the only field that legitimately differs.
delete fromDisk.meta.generatedAt
delete fromBridge.meta.generatedAt
// **Serialised, not deepEqual.** `deepEqual` ignores object key order, and key
// order is precisely what differed between the two readers on a real tree —
// which a live walk caught and this test, written first, did not.
assert.equal(JSON.stringify(fromBridge), JSON.stringify(fromDisk))
assert.deepEqual(fromBridge, fromDisk)
// And the source fingerprints agree, which is what makes switching backends on
// an existing install NOT look like a change to the drift gate.
assert.deepEqual(fromBridge.meta.source, fromDisk.meta.source)
restore()
fs.rmSync(root, { recursive: true, force: true })
})
test('readFrom hands both backends back in one shape', async () => {
serveTree(FILES)
const bridged = await readFrom({ kind: 'bridge' })
assert.deepEqual(Object.keys(bridged), ['files'])
assert.deepEqual(Object.keys(bridged.files[0]).sort(), ['bytes', 'label', 'sha256', 'text'])
restore()
})