feat(assets): item pictures on the marketplace and the character sheet (Phase 5)
Both places this site already knew an item's (ItemID, hue) and could only print it as text now show the picture, hued the way the client would draw it. The shard does the hueing: whether a hue repaints every pixel or only the grey ones is a flag in `tiledata.mul`, which a browser has no way to read. **Ingest warms; the route only serves** (org lead, 2026-09-11). A page never waits on the shard and never causes a fetch -- it renders what is stored and leaves out what is not, which is the state every install was in before this phase. Fetching happens behind that, on a timer, from the keys the site's own rows name. The alternative, fetching on first request, was rejected on one number: the shard's asset plane serves ONE request at a time, so a URL that fetched would let any visitor walk 49,152 ids times 3,000 hues through that slot and park an operator's own import behind it. The wanted set is DERIVED (`SELECT DISTINCT item_id, hue`) rather than queued, so it is self-healing: a restart loses nothing, and a key stops being wanted the moment the vendor row naming it is deleted. The in-memory hint set on top is only for the character sheet, which is fetched live from the shard and stored nowhere -- nothing on disk would ever name those keys. Staleness without a manifest (§7): every row records the shard's `catalog` id, a hash of the files that decide its bytes. A client patch changes it and a restart does not, so "is this out of date?" is a per-row question -- and pictures nobody looks at any more are simply never re-fetched, which is why this is lazy rather than a sweep. `shard_asset_meta` is deliberately NOT written here: it is the body catalogue's singleton, and a warm pass touching it would tell the body import that a client it never looked at is unchanged. A key the shard has no art for writes no row at all. An empty row would make the key held and it would never be asked again -- including after the operator patches in the graphic that was missing. `assets.sources` now reports which families an overlay serves, so an overlay older than phase 5 is one reported state with a sentence naming the fix, instead of a refusal per pass forever with no picture ever appearing. 688 server tests pass (14 new); client builds; the frozen manifest regenerates with one added route, all documented, no core URL moved. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4
This commit is contained in:
@@ -33,7 +33,7 @@ async function batched(conn, sql, rows) {
|
||||
/** Every asset row we hold, as a Map of key → row. */
|
||||
async function allAssets() {
|
||||
const rows = await query(
|
||||
'SELECT asset_key, family, sha256, bytes, width, height, body, direction, file FROM shard_assets',
|
||||
'SELECT asset_key, family, sha256, bytes, width, height, body, direction, file, catalog FROM shard_assets',
|
||||
)
|
||||
|
||||
const map = new Map()
|
||||
@@ -49,6 +49,7 @@ async function allAssets() {
|
||||
body: row.body === null ? null : Number(row.body),
|
||||
direction: row.direction === null ? null : Number(row.direction),
|
||||
file: row.file || null,
|
||||
catalog: row.catalog || null,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -82,16 +83,17 @@ async function saveAssets(rows, meta) {
|
||||
r.body ?? null,
|
||||
r.direction ?? null,
|
||||
r.file ?? null,
|
||||
r.catalog ?? meta?.catalog ?? null,
|
||||
])
|
||||
|
||||
await batched(
|
||||
conn,
|
||||
'INSERT INTO shard_assets (asset_key, family, sha256, bytes, width, height, body, direction, file) ' +
|
||||
'VALUES (?,?,?,?,?,?,?,?,?) ' +
|
||||
'INSERT INTO shard_assets (asset_key, family, sha256, bytes, width, height, body, direction, file, catalog) ' +
|
||||
'VALUES (?,?,?,?,?,?,?,?,?,?) ' +
|
||||
'ON DUPLICATE KEY UPDATE family = VALUES(family), sha256 = VALUES(sha256), ' +
|
||||
'bytes = VALUES(bytes), width = VALUES(width), height = VALUES(height), ' +
|
||||
'body = VALUES(body), direction = VALUES(direction), file = VALUES(file), ' +
|
||||
'imported_at = CURRENT_TIMESTAMP',
|
||||
'catalog = VALUES(catalog), imported_at = CURRENT_TIMESTAMP',
|
||||
values,
|
||||
)
|
||||
|
||||
@@ -114,6 +116,74 @@ async function saveAssets(rows, meta) {
|
||||
}
|
||||
}
|
||||
|
||||
// ── the on-demand side (§11, phase 5) ──────────────────────────────────────
|
||||
|
||||
/**
|
||||
* The pictures we hold for an explicit list of keys, as a Map of key → filename.
|
||||
*
|
||||
* This is the read on the hot path — every marketplace page and every character
|
||||
* sheet runs it — so it is one statement over the primary key and it returns only
|
||||
* what it is asked for. It deliberately does NOT check staleness: a page renders
|
||||
* the picture it has, and deciding whether that picture is out of date is the warm
|
||||
* pass's job, off the request.
|
||||
*/
|
||||
async function filesForKeys(keys) {
|
||||
const list = [...new Set(keys.filter((k) => typeof k === 'string' && k !== ''))]
|
||||
|
||||
if (list.length === 0) return new Map()
|
||||
|
||||
const rows = await query(
|
||||
`SELECT asset_key, file FROM shard_assets WHERE file IS NOT NULL AND asset_key IN (${list
|
||||
.map(() => '?')
|
||||
.join(',')})`,
|
||||
list,
|
||||
)
|
||||
|
||||
const map = new Map()
|
||||
|
||||
for (const row of rows) map.set(row.asset_key, row.file)
|
||||
|
||||
return map
|
||||
}
|
||||
|
||||
/**
|
||||
* Which of these keys we already hold under the shard's CURRENT catalogue.
|
||||
*
|
||||
* The warm pass subtracts this from what it wants, so everything it does not
|
||||
* return gets fetched: a key we have never seen, and a key whose row was written
|
||||
* against a catalogue the shard has since moved past (§7 — an operator patched
|
||||
* their client). A row with no file is not held either, because the database and
|
||||
* the uploads volume can disagree and a broken image is worse than a re-fetch.
|
||||
*/
|
||||
async function freshKeys(keys, catalog) {
|
||||
const list = [...new Set(keys.filter((k) => typeof k === 'string' && k !== ''))]
|
||||
|
||||
if (list.length === 0) return new Set()
|
||||
|
||||
const rows = await query(
|
||||
`SELECT asset_key FROM shard_assets WHERE file IS NOT NULL AND catalog <=> ? ` +
|
||||
`AND asset_key IN (${list.map(() => '?').join(',')})`,
|
||||
[catalog ?? null, ...list],
|
||||
)
|
||||
|
||||
return new Set(rows.map((r) => r.asset_key))
|
||||
}
|
||||
|
||||
/** Counts for the admin surface, split by family. */
|
||||
async function countByFamily() {
|
||||
const rows = await query(
|
||||
'SELECT family, COUNT(*) AS total, SUM(file IS NOT NULL) AS stored FROM shard_assets GROUP BY family',
|
||||
)
|
||||
|
||||
const out = {}
|
||||
|
||||
for (const row of rows) {
|
||||
out[row.family] = { total: Number(row.total) || 0, stored: Number(row.stored) || 0 }
|
||||
}
|
||||
|
||||
return out
|
||||
}
|
||||
|
||||
async function getMeta() {
|
||||
const rows = await query('SELECT payload, imported_at FROM shard_asset_meta WHERE id = 1')
|
||||
if (rows.length === 0) return null
|
||||
@@ -220,4 +290,7 @@ module.exports = {
|
||||
allBodies,
|
||||
countBodies,
|
||||
artBySlug,
|
||||
filesForKeys,
|
||||
freshKeys,
|
||||
countByFamily,
|
||||
}
|
||||
|
||||
@@ -456,6 +456,7 @@ async function getStatus() {
|
||||
const counts = await db.countAssets().catch(() => ({ total: 0, stored: 0 }))
|
||||
const bodies = await db.countBodies().catch(() => ({ total: 0, resolved: 0 }))
|
||||
const meta = await db.getMeta().catch(() => null)
|
||||
const families = await db.countByFamily().catch(() => ({}))
|
||||
|
||||
const status = {
|
||||
loaded: {
|
||||
@@ -466,6 +467,12 @@ async function getStatus() {
|
||||
catalog: meta?.catalog ?? null,
|
||||
extractorVersion: meta?.extractorVersion ?? null,
|
||||
importedAt: meta?.importedAt ?? null,
|
||||
// Item and land pictures, counted separately because they are a different
|
||||
// KIND of thing (§11, phase 5): no manifest, no set, and no "how many are
|
||||
// there" to compare against. `items` is how many the site has been asked
|
||||
// for and holds, which is the only number that means anything here.
|
||||
items: families.static?.stored ?? 0,
|
||||
land: families.land?.stored ?? 0,
|
||||
},
|
||||
shard: null,
|
||||
drift: null,
|
||||
@@ -485,6 +492,10 @@ async function getStatus() {
|
||||
hashing: sources.hashing,
|
||||
complete: sources.complete,
|
||||
imaging: sources.imaging,
|
||||
// Which §5 families this overlay serves. A phase-3 or phase-4 overlay says
|
||||
// `['body']`, which is what an admin panel needs in order to say "update
|
||||
// your plugin" rather than showing an item-art pipeline that cannot work.
|
||||
families: sources.families,
|
||||
}
|
||||
|
||||
status.drift = meta ? !bridge.sameSources(sources, meta.sources) : true
|
||||
|
||||
510
server/model/shardAssets/shardItemArt.model.js
Normal file
510
server/model/shardAssets/shardItemArt.model.js
Normal file
@@ -0,0 +1,510 @@
|
||||
const fs = require('fs')
|
||||
const path = require('path')
|
||||
|
||||
const db = require('./shardAssets.db')
|
||||
const core = require('../../core')
|
||||
const bridge = require('../../utils/assetBridge')
|
||||
const uoLinkConfig = require('../uoLinkConfig/uoLinkConfig.model')
|
||||
const log = require('../../core').logger('shardItemArt')
|
||||
|
||||
// Item and land pictures, fetched because something on this site names them
|
||||
// (docs/link/v8.md §5, §11 — protocol 8, phase 5).
|
||||
//
|
||||
// ── Why this is not the body catalogue with a different prefix ─────────
|
||||
//
|
||||
// The bestiary wants every creature, so phase 3 imports a SET: walk a manifest,
|
||||
// diff the hashes, fetch what moved. That works because the set is 1,022 rows
|
||||
// and one megabyte.
|
||||
//
|
||||
// This side has no set. The shard's client addresses 49,152 item graphics and
|
||||
// has art for 39,189 of them; multiply by three thousand hues and there is
|
||||
// nothing to enumerate, no manifest worth building and nothing worth importing
|
||||
// ahead of time. What there IS, at any moment, is a few hundred keys that the
|
||||
// site's own rows actually name: the items on a vendor, the things a character
|
||||
// is wearing. That is the working set, and it is what this fetches.
|
||||
//
|
||||
// ── Who is allowed to make the shard do work ──────────────────────────
|
||||
//
|
||||
// **Ingest warms; the route only serves** (org lead, 2026-09-11). A page never
|
||||
// waits on the shard and never causes a fetch: it renders the pictures already
|
||||
// on disk and leaves out the ones that are not, which is exactly the state every
|
||||
// install was in before this phase and which every surface already handles.
|
||||
// Fetching happens behind that, from the keys the site has stored.
|
||||
//
|
||||
// The alternative — fetch on the first request for a key — was rejected on one
|
||||
// number. The shard's asset plane serves **one request at a time** by design
|
||||
// (§3.2), so any anonymous visitor able to name a key could walk 49,152 ids
|
||||
// times 3,000 hues through that single slot and keep an operator's own import
|
||||
// waiting behind it, from a URL with nothing to authenticate. Warming from the
|
||||
// site's own data has no such surface: the ceiling is the number of distinct
|
||||
// (item, hue) pairs the shard itself has told us about.
|
||||
//
|
||||
// ── Why the wanted set is DERIVED and not a queue ─────────────────────
|
||||
//
|
||||
// A queue table would need writing on the ingest path, draining, retrying,
|
||||
// pruning and reconciling after a restart. The same answer falls out of a
|
||||
// `SELECT DISTINCT` over the rows that name the items — which is self-healing by
|
||||
// construction: a key lost to a restart comes back the next time the pass runs,
|
||||
// and a key for a vendor that has gone stops being wanted the moment its row is
|
||||
// deleted. The in-memory set below is an optimisation on top of that, never the
|
||||
// record: it exists so a picture seen on a LIVE character sheet — which is
|
||||
// fetched from the shard per request and stored nowhere — is not forgotten.
|
||||
//
|
||||
// ── Staleness, without a manifest (§7) ────────────────────────────────
|
||||
//
|
||||
// Every fetched row records the shard's `catalog` id, which is a hash of the
|
||||
// files that decide the bytes. A client patch changes it, a restart does not. So
|
||||
// "is this picture out of date?" is a per-row comparison rather than a manifest
|
||||
// diff, and the answer costs nothing for the pictures nobody is looking at any
|
||||
// more: they are simply never re-fetched.
|
||||
|
||||
/** Where item and land pictures land, under core's upload directory. */
|
||||
const ART_SUBDIR = 'items'
|
||||
|
||||
/**
|
||||
* How many keys one warm pass will fetch.
|
||||
*
|
||||
* A bound rather than a target. The shard serves one asset request at a time, so
|
||||
* a pass that asked for everything at once would hold that slot for as long as it
|
||||
* took — against an operator who might be trying to run an import. Passes are
|
||||
* cheap and repeat; a backlog drains over several of them and nothing waits.
|
||||
*/
|
||||
const WARM_BATCH = 400
|
||||
|
||||
/**
|
||||
* How many live-observed keys are remembered between passes.
|
||||
*
|
||||
* Bounded because this is a set fed by page views. It is an optimisation over the
|
||||
* derived set, so dropping from it costs a picture appearing one pass later, and
|
||||
* never a picture that is lost.
|
||||
*/
|
||||
const SEEN_CAP = 5000
|
||||
|
||||
const seen = new Set()
|
||||
|
||||
// ── keys (§5) ──────────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* The one place an item key is spelled.
|
||||
*
|
||||
* Hue 0 means "not hued" on the wire, so it produces the plain key rather than a
|
||||
* `/h0` one — the shard refuses `/h0` outright for the same reason, and the two
|
||||
* agreeing is what stops the same PNG being stored twice under two names.
|
||||
*/
|
||||
function staticKey(itemId, hue = 0) {
|
||||
const id = Number(itemId)
|
||||
|
||||
if (!Number.isInteger(id) || id < 0) return null
|
||||
|
||||
const h = Number(hue)
|
||||
|
||||
return Number.isInteger(h) && h > 0 ? `static/${id}/h${h}` : `static/${id}`
|
||||
}
|
||||
|
||||
function landKey(tileId) {
|
||||
const id = Number(tileId)
|
||||
|
||||
return Number.isInteger(id) && id >= 0 && id < 0x4000 ? `land/${id}` : null
|
||||
}
|
||||
|
||||
function artDir() {
|
||||
return path.join(core.uploads.UPLOAD_DIR, ART_SUBDIR)
|
||||
}
|
||||
|
||||
/**
|
||||
* Content-addressed, exactly as the body catalogue's names are and for the same
|
||||
* reason: a stable name overwritten in place leaves every browser and CDN serving
|
||||
* last month's client's sprite while the database row stays perfectly correct.
|
||||
*/
|
||||
function fileNameFor(key, sha256) {
|
||||
const stem = key.replace(/[^a-zA-Z0-9]+/g, '-').replace(/^-+|-+$/g, '')
|
||||
return `uo-${stem}-${String(sha256).slice(0, 8)}.png`
|
||||
}
|
||||
|
||||
// ── noticing ───────────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Remember that something on this site showed these (itemId, hue) pairs.
|
||||
*
|
||||
* Called from the market ingest and from the character sheet, and deliberately
|
||||
* synchronous and allocation-light: it is on a request path and a page must never
|
||||
* pay for a picture it is not going to get anyway.
|
||||
*/
|
||||
function notice(items) {
|
||||
if (!Array.isArray(items)) return 0
|
||||
|
||||
let added = 0
|
||||
|
||||
for (const item of items) {
|
||||
const key = staticKey(item?.itemId ?? item?.item_id, item?.hue)
|
||||
|
||||
if (!key || seen.has(key)) continue
|
||||
|
||||
// Oldest-first, and only when full. The derived set is the record; this is a
|
||||
// cache of hints, so forgetting one costs a pass, not a picture.
|
||||
if (seen.size >= SEEN_CAP) seen.delete(seen.values().next().value)
|
||||
|
||||
seen.add(key)
|
||||
added++
|
||||
}
|
||||
|
||||
return added
|
||||
}
|
||||
|
||||
/** For tests and the admin surface: how many hints are waiting. */
|
||||
function noticedCount() {
|
||||
return seen.size
|
||||
}
|
||||
|
||||
// ── serving ────────────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Attach `art` to rows that name an item, in place, and notice what is missing.
|
||||
*
|
||||
* `art` is a FILENAME under `uploads/items/`, never a path or a URL — the same
|
||||
* shape `shard_spawn_creatures.art` uses, so the client builds one URL the same
|
||||
* way everywhere and the API never hard-codes a mount point.
|
||||
*
|
||||
* A row with no stored picture gets `art: null` rather than being changed in any
|
||||
* other way. That is a first-class state: it is what every row looked like before
|
||||
* this phase, every surface renders it, and it is what an item this client has no
|
||||
* art for looks like permanently.
|
||||
*/
|
||||
async function decorate(rows, { itemIdField = 'itemId', hueField = 'hue' } = {}) {
|
||||
const list = Array.isArray(rows) ? rows.filter((r) => r && typeof r === 'object') : []
|
||||
|
||||
if (list.length === 0) return list
|
||||
|
||||
const keys = list.map((row) => staticKey(row[itemIdField], row[hueField]))
|
||||
|
||||
let files = new Map()
|
||||
|
||||
try {
|
||||
files = await db.filesForKeys(keys.filter(Boolean))
|
||||
} catch (err) {
|
||||
// Decoration, not the page. A picture lookup that fails must not fail a
|
||||
// marketplace search.
|
||||
log.warn('could not read item art', { error: err.message })
|
||||
return list
|
||||
}
|
||||
|
||||
for (let i = 0; i < list.length; i++) {
|
||||
list[i].art = (keys[i] && files.get(keys[i])) || null
|
||||
}
|
||||
|
||||
// Everything this page WANTED is worth warming, whether or not we had it: the
|
||||
// ones we had may be stale, and the ones we did not are the point.
|
||||
notice(list.map((row) => ({ itemId: row[itemIdField], hue: row[hueField] })))
|
||||
|
||||
return list
|
||||
}
|
||||
|
||||
// ── warming ────────────────────────────────────────────────────────────────
|
||||
|
||||
async function shardLinked() {
|
||||
try {
|
||||
const config = await uoLinkConfig.getSafe()
|
||||
return Boolean(config?.enabled && config?.baseUrl)
|
||||
} catch {
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Every item key the site's own rows name, newest-priced first.
|
||||
*
|
||||
* `shard_vendor_items` is the only stored table that carries (item_id, hue)
|
||||
* today. The character sheet's equipment is fetched live from the shard per
|
||||
* request and stored nowhere, which is precisely what the in-memory hint set is
|
||||
* for.
|
||||
*/
|
||||
async function wantedKeys() {
|
||||
const keys = []
|
||||
|
||||
try {
|
||||
const rows = await core.query(
|
||||
'SELECT DISTINCT item_id, hue FROM shard_vendor_items WHERE item_id > 0 LIMIT 20000',
|
||||
)
|
||||
|
||||
for (const row of rows) {
|
||||
const key = staticKey(row.item_id, row.hue)
|
||||
if (key) keys.push(key)
|
||||
}
|
||||
} catch (err) {
|
||||
log.warn('could not read the marketplace for item art', { error: err.message })
|
||||
}
|
||||
|
||||
// Hints last, so a backlog of stored rows is never starved by page traffic.
|
||||
for (const key of seen) keys.push(key)
|
||||
|
||||
return [...new Set(keys)]
|
||||
}
|
||||
|
||||
function writePicture(key, sha256, png) {
|
||||
const name = fileNameFor(key, sha256)
|
||||
|
||||
try {
|
||||
fs.mkdirSync(artDir(), { recursive: true })
|
||||
fs.writeFileSync(path.join(artDir(), name), png)
|
||||
return name
|
||||
} catch (err) {
|
||||
log.warn('could not write an item picture', { key, error: err.message })
|
||||
return null
|
||||
}
|
||||
}
|
||||
|
||||
function removePicture(name) {
|
||||
if (!name) return
|
||||
|
||||
try {
|
||||
fs.unlinkSync(path.join(artDir(), name))
|
||||
} catch {
|
||||
// Already gone, or never written. A warm pass must not fail because a file it
|
||||
// was tidying up was tidied already.
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* One warm pass: fetch the wanted keys we do not already hold, and store them.
|
||||
*
|
||||
* Returns a result rather than throwing, with the same vocabulary the body import
|
||||
* uses — `skipped`, `unavailable`, `unchanged`, `imported`, `failed` — so the
|
||||
* admin surface reports one set of words for both halves of this protocol.
|
||||
*
|
||||
* `limit` bounds one pass. `force` re-fetches keys we hold, which is how an
|
||||
* operator recovers from a wiped uploads volume without waiting for a client
|
||||
* patch to invalidate every row.
|
||||
*/
|
||||
async function warm({ limit = WARM_BATCH, force = false } = {}) {
|
||||
if (!(await shardLinked())) {
|
||||
return { status: 'skipped', reason: 'uo-link is not configured, so there is no shard to ask' }
|
||||
}
|
||||
|
||||
let sources
|
||||
|
||||
try {
|
||||
sources = await bridge.sourceFingerprint()
|
||||
} catch (err) {
|
||||
return failure(err, 'client file manifest')
|
||||
}
|
||||
|
||||
if (sources.imaging && sources.imaging.ok === false) {
|
||||
return {
|
||||
status: 'unavailable',
|
||||
code: 'NO_IMAGING',
|
||||
reason: sources.imaging.reason || 'the shard host cannot render images',
|
||||
}
|
||||
}
|
||||
|
||||
// A phase-3 or phase-4 overlay serves bodies and nothing else. Asking it for a
|
||||
// static is refused per request, which would be a warn on every pass forever —
|
||||
// so it is checked once, here, and reported as the ordinary state it is.
|
||||
// Defensive default rather than a trusted field: an older sidecar, an older
|
||||
// overlay or a stubbed fingerprint can all leave it off, and `['body']` is the
|
||||
// truthful reading of its absence (§6 — the families field arrived in phase 5).
|
||||
const families = Array.isArray(sources.families) ? sources.families : ['body']
|
||||
|
||||
if (!families.includes('static')) {
|
||||
return {
|
||||
status: 'unavailable',
|
||||
code: 'UNSUPPORTED',
|
||||
reason:
|
||||
"this shard's overlay does not serve item art; it offers " +
|
||||
`${families.join(', ')}. Update the plugin overlay to get it.`,
|
||||
}
|
||||
}
|
||||
|
||||
const wanted = await wantedKeys()
|
||||
|
||||
if (wanted.length === 0) {
|
||||
return { status: 'unchanged', wanted: 0, fetched: 0, written: 0 }
|
||||
}
|
||||
|
||||
// The catalogue is learned from the first reply rather than asked for, so this
|
||||
// pass cannot be the thing that decides what is stale. `catalog: null` on the
|
||||
// request means "whatever you have"; the mid-walk guard in `fetchAssets` is what
|
||||
// catches a client that moves underneath it.
|
||||
let held = new Set()
|
||||
|
||||
if (!force) {
|
||||
const current = await currentCatalog()
|
||||
|
||||
try {
|
||||
held = await db.freshKeys(wanted, current)
|
||||
} catch (err) {
|
||||
return { status: 'failed', reason: err.message }
|
||||
}
|
||||
}
|
||||
|
||||
const todo = wanted.filter((key) => !held.has(key)).slice(0, Math.max(1, limit))
|
||||
|
||||
if (todo.length === 0) {
|
||||
forget(wanted)
|
||||
return { status: 'unchanged', wanted: wanted.length, held: held.size, fetched: 0, written: 0 }
|
||||
}
|
||||
|
||||
let fetched
|
||||
|
||||
try {
|
||||
fetched = await bridge.fetchAssets({ keys: todo })
|
||||
} catch (err) {
|
||||
return failure(err, 'item art')
|
||||
}
|
||||
|
||||
// Only the filenames, and only for the keys in hand: the old file is removed
|
||||
// when a key's hash moves, so `uploads/items/` tracks the working set instead of
|
||||
// accumulating one file per client patch forever.
|
||||
const existing = await db.filesForKeys(todo).catch(() => new Map())
|
||||
const rows = []
|
||||
let written = 0
|
||||
|
||||
for (const key of todo) {
|
||||
const got = fetched.assets.get(key)
|
||||
|
||||
// A key the shard has no art for is not a failure and not a row: writing an
|
||||
// empty row would make it "held" and stop it ever being asked again, which is
|
||||
// wrong the moment an operator patches in the missing graphic.
|
||||
if (!got) continue
|
||||
|
||||
const name = writePicture(key, got.sha256, got.png)
|
||||
|
||||
if (!name) continue
|
||||
|
||||
written++
|
||||
|
||||
const before = existing.get(key)
|
||||
|
||||
if (before && before !== name) removePicture(before)
|
||||
|
||||
rows.push({
|
||||
key,
|
||||
family: key.startsWith('land/') ? 'land' : 'static',
|
||||
sha256: got.sha256,
|
||||
bytes: got.bytes,
|
||||
width: got.width,
|
||||
height: got.height,
|
||||
body: null,
|
||||
direction: null,
|
||||
file: name,
|
||||
catalog: fetched.catalog,
|
||||
})
|
||||
}
|
||||
|
||||
if (rows.length > 0) {
|
||||
try {
|
||||
// No meta: `shard_asset_meta` is the BODY catalogue's singleton — what an
|
||||
// Update compares a manifest against — and this family has no manifest. A
|
||||
// warm pass writing there would tell the body import that a client it never
|
||||
// looked at is unchanged.
|
||||
await db.saveAssets(rows, null)
|
||||
} catch (err) {
|
||||
return { status: 'failed', reason: err.message }
|
||||
}
|
||||
}
|
||||
|
||||
forget(todo)
|
||||
|
||||
const result = {
|
||||
status: 'imported',
|
||||
catalog: fetched.catalog,
|
||||
wanted: wanted.length,
|
||||
held: held.size,
|
||||
asked: todo.length,
|
||||
fetched: fetched.assets.size,
|
||||
written,
|
||||
absent: fetched.missing.absent,
|
||||
unsupported: fetched.missing.unsupported,
|
||||
remaining: Math.max(0, wanted.length - held.size - todo.length),
|
||||
}
|
||||
|
||||
log.info('item art warmed', result)
|
||||
|
||||
return result
|
||||
}
|
||||
|
||||
/** Drop hints a pass has dealt with, so the set does not grow without bound. */
|
||||
function forget(keys) {
|
||||
for (const key of keys) seen.delete(key)
|
||||
}
|
||||
|
||||
/**
|
||||
* The catalogue id the shard would answer under right now.
|
||||
*
|
||||
* Read from a one-key probe rather than from a dedicated command: the shard puts
|
||||
* `catalog` on every fetch reply, so the cheapest honest way to ask is to fetch
|
||||
* something. `static/0` is the smallest such question and its answer is thrown
|
||||
* away — what is wanted is the id beside it.
|
||||
*
|
||||
* A shard that cannot answer returns null, and null compares unequal to every
|
||||
* stored catalogue, so the pass falls back to "everything is stale" — which costs
|
||||
* a re-fetch and never serves a wrong picture. That is the right way round.
|
||||
*/
|
||||
async function currentCatalog() {
|
||||
try {
|
||||
const probe = await bridge.fetchAssets({ keys: ['static/0'] })
|
||||
return probe.catalog ?? null
|
||||
} catch (err) {
|
||||
log.warn('could not read the shard art catalogue', { error: err.message })
|
||||
return null
|
||||
}
|
||||
}
|
||||
|
||||
function failure(err, what) {
|
||||
if (err instanceof bridge.AssetBridgeError) {
|
||||
return { status: 'unavailable', code: err.code, reason: err.message }
|
||||
}
|
||||
|
||||
log.warn(`item art failed reading the ${what}`, { error: err.message })
|
||||
|
||||
return { status: 'failed', reason: err.message }
|
||||
}
|
||||
|
||||
// ── the background pass ────────────────────────────────────────────────────
|
||||
|
||||
let timer = null
|
||||
|
||||
/**
|
||||
* Run a warm pass every few minutes, forever, while the process lives.
|
||||
*
|
||||
* Deliberately a plain interval and not a debounce on ingest. A market sweep
|
||||
* delivers dozens of `vendor.listing` frames in a burst and debouncing each of
|
||||
* them would either fire once per frame or need its own state machine; a pass is
|
||||
* cheap when there is nothing to do (one `SELECT DISTINCT` and one probe) and the
|
||||
* work it exists for is not urgent — a picture appearing a few minutes after the
|
||||
* listing that wants it is invisible to everyone.
|
||||
*
|
||||
* `unref()` so this never holds the process open at shutdown.
|
||||
*/
|
||||
function startWarming({ everyMs = 5 * 60 * 1000 } = {}) {
|
||||
if (timer) return
|
||||
|
||||
timer = setInterval(() => {
|
||||
warm().catch((err) => log.warn('item art warm pass failed', { error: err.message }))
|
||||
}, everyMs)
|
||||
|
||||
if (typeof timer.unref === 'function') timer.unref()
|
||||
}
|
||||
|
||||
function stopWarming() {
|
||||
if (!timer) return
|
||||
|
||||
clearInterval(timer)
|
||||
timer = null
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
ART_SUBDIR,
|
||||
WARM_BATCH,
|
||||
artDir,
|
||||
fileNameFor,
|
||||
staticKey,
|
||||
landKey,
|
||||
notice,
|
||||
noticedCount,
|
||||
decorate,
|
||||
wantedKeys,
|
||||
warm,
|
||||
currentCatalog,
|
||||
startWarming,
|
||||
stopWarming,
|
||||
}
|
||||
Reference in New Issue
Block a user