From 0ce92152a1da6ffd4eff7e32ce0c12d1764d91bb Mon Sep 17 00:00:00 2001 From: wtclaude Date: Thu, 10 Sep 2026 02:55:37 -0500 Subject: [PATCH 01/11] =?UTF-8?q?feat(asset-bridge):=20phase=200=20spike?= =?UTF-8?q?=20=E2=80=94=20the=20decoders,=20from=20inside=20a=20live=20sha?= =?UTF-8?q?rd?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit docs/link/v8.md §16 phase 0. §4 chose to CALL ServUO's vendored `Ultima` rather than reimplement it, on the evidence of a PowerShell probe against a stock client — neither the process nor the client the extractor will run in. This runs the same decoders from inside a running ServUO 57.4 against a client broken in 21 catalogued ways, and it found more than a crash. Adds, all under tools/ and therefore never deployed: * BridgeAssetProbe.cs — the sweep, plus BridgeAssetValidator, a prototype of the validate-before-calling response chosen for §4.2's residual risk. Runs off the Core thread, snapshots Race.AllRaces on it, and writes the id it is ABOUT to touch to a checkpoint file before every call. * BridgeMythicCliloc.cs — the §9 Mythic cliloc reader in net48 C#, ported from UOFiddler (Beerware) with every file-derived index bounds-checked. Phase 2 promotes this into overlay/. * patch_client.ps1 — builds the patched client in five tiers. Hashes every file it touches in the SOURCE before and after and aborts on a change. * an `assetprobe` verb on BridgeRigDriver, so stock and patched can be run against one boot rather than two shard processes. The findings are written up in tools/scaffolding/README.md. The four that change what phase 1 has to build: * FileIndex's UOP constructor ends `MulPath = uopPath`, so artLegacyMUL.uop wins outright and art.mul/artidx.mul are never opened on a current client. A validator bounding offsets against art.mul is not approximate, it is nonsense — the first run refused 34,299 good statics on that mistake, and every refusal looked like a real finding. * 22,102 WRONG PICTURES on a stock, unmodified client. Empty UOP index slots read `lookup 0, length 0`; Seek treats that as a hit, and LoadStatic decodes zero bytes into a shared buffer it reuses, only ever grows, and fills from a Read whose return value is discarded — so the id renders the previously-decoded asset. The mul path does not do this (artidx stores -1), which is why the earlier probe counted 32,766 of them as "ok". A bulk import that trusted the library would have written 22,102 duplicate images under ids that have no art. * The validator caught all 8 record-level defects — 7 of which the library rendered without raising anything, including a verdata lookup past verdata.mul's own end (Verdata.Seek is bounds-checked nowhere) and an 8000x8000 bitmap allocated from two bytes in a file. It refused NOTHING across 49,151 statics and 16,384 land tiles on the stock client, which is the number that makes the boundary defensible. * §4.1's crash reproduces in-process: one Ultima.Gumps.GetGump(2) and the ServUO process disappeared — no catch reached, no console line, the checkpoint file the only record. "Nothing calls Ultima.Gumps" is now an earned safety rule. §9 is proven: 123,490 entries in 218 ms, byte-identical to UOFiddler's own output, with no UOFiddler installed and nothing copied to a server. Not covered, and named as phase 1 work: the animation path has no validator at all, and the patched wolf decoded something else in silence. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4 --- tools/patch_client.ps1 | 580 ++++++++++ tools/scaffolding/BridgeAssetProbe.cs | 1346 +++++++++++++++++++++++ tools/scaffolding/BridgeMythicCliloc.cs | 532 +++++++++ tools/scaffolding/BridgeRigDriver.cs | 8 + tools/scaffolding/README.md | 128 ++- 5 files changed, 2593 insertions(+), 1 deletion(-) create mode 100644 tools/patch_client.ps1 create mode 100644 tools/scaffolding/BridgeAssetProbe.cs create mode 100644 tools/scaffolding/BridgeMythicCliloc.cs diff --git a/tools/patch_client.ps1 b/tools/patch_client.ps1 new file mode 100644 index 0000000..d1baf90 --- /dev/null +++ b/tools/patch_client.ps1 @@ -0,0 +1,580 @@ +<# +.SYNOPSIS + Builds a deliberately patched UO client for the Asset Bridge phase 0 spike. + +.DESCRIPTION + docs/link/v8.md section 16 phase 0 drives ServUO's vendored `Ultima` decoders "over a deliberately + patched client". Stock clients are not the interesting case: they are the case the library was + written against, and the whole reason phase 0 exists is that section 4 chose to call code that can + take the shard down if it is wrong. A shard operator's client is patched -- custom art, a + verdata.mul, a hand-edited Bodyconv.def -- and that is what has to be survived. + + This copies a client and then breaks the copy in four deliberate, catalogued ways. It NEVER + writes to the source: every file it patches is hashed before and after, and a changed source + hash aborts the run. + + Each defect is recorded in `patched-client.manifest.json` next to the copy, so the probe's + report can be read against what was actually done rather than against a memory of it. The + manifest is the answer to "is a nonzero REFUSED-BUT-DECODED count a bug or the point?". + +.PARAMETER Source + The client to copy. Defaults to this machine's. + +.PARAMETER Dest + Where to build the patched copy. Needs ~3.5 GB. + +.PARAMETER Tiers + Which defects to apply. Default: all four. + + verdata Author a verdata.mul, which this client does not have. Ultima consults Verdata on + EVERY art and anim lookup (Art's FileIndex is built with verdata file id 4, + Animations' with 6), so on a client with no verdata.mul that entire branch is + dead code that has never been exercised -- the largest untested surface in the + library we are about to depend on. Includes one legitimate patch and one whose + lookup points past verdata.mul's own end, because `FileIndex.Seek` bounds-checks + the mul and does not bounds-check verdata. + + customart Fill unused artidx.mul slots with real records appended to art.mul, the way a + custom-art shard does. Tests that our out-of-range accounting comes from the + file rather than from a constant someone wrote down. + + corrupt Rewrite index entries and record headers into the shapes that reading Art.cs + says are reachable: a lookup past EOF, a record that starts inside the file and + ends outside it, a length too small for a header, absurd dimensions, a row table + pointing outside its own record, and a land tile shorter than the fixed 2,024 + bytes LoadLand always reads. + + bodyconv Add Bodyconv.def lines pointing bodies at an anim file that holds nothing, and at + an index in another file that holds something unrelated -- the gargoyle-666 spider + case, reproduced on purpose. Proves the extractor takes BodyConverter.Convert's + answer and stops (v8.md section 4.3). + + nouop Move artLegacyMUL.uop aside, so art is read from art.mul/artidx.mul. + + This is not cosmetic and it is not optional if you want the customart or corrupt + tiers to mean anything. FileIndex's UOP constructor ends with a bare + `MulPath = uopPath`: when artLegacyMUL.uop is present it wins OUTRIGHT and + art.mul / artidx.mul are never opened. Every index-level defect below writes to + files the library does not read on a modern client, so without this tier those + two tiers are inert while still reporting that they applied. + + It is also a real configuration in its own right: plenty of shards run mul-only + clients, and a custom-art shard that adds graphics to art.mul while the UOP is + still there gets nothing at all -- an operator trap worth knowing about. + +.PARAMETER SkipCopy + Re-patch an existing copy without re-copying 3.5 GB. Only safe on a copy this script made and + has not patched yet -- patching twice compounds the defects and invalidates the manifest. + +.EXAMPLE + .\tools\patch_client.ps1 -Dest D:\uo-patched-client + +.NOTES + Test scaffolding. Never deployed. The copy contains EA's client art -- like every other + extraction in this project it stays on the machine that made it and is never committed. +#> + +[CmdletBinding()] +param( + [string] $Source = 'D:\Games\Electronic Arts\Ultima Online Classic', + [Parameter(Mandatory = $true)] + [string] $Dest, + [ValidateSet('verdata', 'customart', 'corrupt', 'bodyconv', 'nouop')] + [string[]] $Tiers = @('nouop', 'verdata', 'customart', 'corrupt', 'bodyconv'), + [switch] $SkipCopy, + [switch] $Force +) + +$ErrorActionPreference = 'Stop' + +# Files this script may write to in the copy. Anything not on this list is a bug in the script, +# and the source-hash check at the end is what proves it. +$PatchTargets = @('artidx.mul', 'art.mul', 'verdata.mul', 'Bodyconv.def', 'artLegacyMUL.uop') + +# -- Little-endian helpers (BitConverter is fine, but the intent reads better named) ---------- + +function Read-Int32LE([byte[]] $Bytes, [int] $Offset) { + return [BitConverter]::ToInt32($Bytes, $Offset) +} + +function Write-Int32LE([byte[]] $Bytes, [int] $Offset, [int] $Value) { + [Array]::Copy([BitConverter]::GetBytes([int] $Value), 0, $Bytes, $Offset, 4) +} + +function Get-ArtEntry([byte[]] $Idx, [int] $Index) { + $at = $Index * 12 + return [pscustomobject]@{ + Index = $Index + Lookup = Read-Int32LE $Idx $at + Length = Read-Int32LE $Idx ($at + 4) + Extra = Read-Int32LE $Idx ($at + 8) + } +} + +function Set-ArtEntry([byte[]] $Idx, [int] $Index, [int] $Lookup, [int] $Length, [int] $Extra) { + $at = $Index * 12 + Write-Int32LE $Idx $at $Lookup + Write-Int32LE $Idx ($at + 4) $Length + Write-Int32LE $Idx ($at + 8) $Extra +} + +# The defect catalogue. Every mutation appends to this, and it is written out as the manifest. +$script:Defects = New-Object System.Collections.ArrayList + +function Add-Defect([string] $Tier, [string] $Key, [string] $What, [string] $Expect) { + [void] $script:Defects.Add([pscustomobject]@{ + tier = $Tier + key = $Key + what = $What + expect = $Expect + }) + Write-Host (" {0,-22} {1}" -f $Key, $What) +} + +# -- Preflight -------------------------------------------------------------------------------- + +if (-not (Test-Path -LiteralPath $Source)) { + throw "source client not found: $Source" +} + +$sourceFull = (Resolve-Path -LiteralPath $Source).Path + +if (Test-Path -LiteralPath $Dest) { + $destFull = (Resolve-Path -LiteralPath $Dest).Path + if ($destFull -eq $sourceFull) { + throw "Dest is the source client. Refusing -- this script destroys what it points at." + } + if (-not $SkipCopy -and -not $Force) { + throw "$Dest already exists. Pass -Force to overwrite it, or -SkipCopy to patch it in place." + } +} + +Write-Host "source: $sourceFull" +Write-Host "dest: $Dest" +Write-Host "tiers: $($Tiers -join ', ')" +Write-Host "" + +# Hash the source files we are about to touch, so "it never writes to the source" is checked and +# not merely asserted. +$before = @{} +foreach ($name in $PatchTargets) { + $path = Join-Path $sourceFull $name + if (Test-Path -LiteralPath $path) { + $before[$name] = (Get-FileHash -LiteralPath $path -Algorithm SHA256).Hash + } +} + +# -- Copy ------------------------------------------------------------------------------------- + +if ($SkipCopy) { + Write-Host "skipping copy (-SkipCopy)" + if (-not (Test-Path -LiteralPath $Dest)) { throw "-SkipCopy but $Dest does not exist" } +} else { + Write-Host "copying (this is ~3.5 GB; a few minutes)..." + # /MIR so a -Force re-run starts clean rather than merging into an already-patched tree. + # /NJH /NJS /NDL /NFL keep robocopy's output to the errors. + $null = robocopy $sourceFull $Dest /MIR /R:1 /W:1 /NJH /NJS /NDL /NFL /NP + # Robocopy exit codes below 8 are success; 8 and above are real failures. + if ($LASTEXITCODE -ge 8) { throw "robocopy failed with exit code $LASTEXITCODE" } + # Robocopy's "1 = files were copied" would otherwise become this script's exit code and read + # as a failure to anything checking it. + $global:LASTEXITCODE = 0 + Write-Host "copied." +} + +Write-Host "" + +$destFull = (Resolve-Path -LiteralPath $Dest).Path + +$artIdxPath = Join-Path $destFull 'artidx.mul' +$artMulPath = Join-Path $destFull 'art.mul' + +if (-not (Test-Path -LiteralPath $artIdxPath)) { throw "no artidx.mul in the copy" } + +$idx = [System.IO.File]::ReadAllBytes($artIdxPath) +$entryCount = [int] ($idx.Length / 12) +$artMulLength = (Get-Item -LiteralPath $artMulPath).Length + +Write-Host ("artidx.mul holds {0:N0} entries; art.mul is {1:N0} bytes" -f $entryCount, $artMulLength) +Write-Host "" + +# -- Tier: nouop ------------------------------------------------------------------------------ + +$uopPath = Join-Path $destFull 'artLegacyMUL.uop' +$uopPresent = Test-Path -LiteralPath $uopPath + +if ($Tiers -contains 'nouop') { + Write-Host "tier nouop" + + if (-not $uopPresent) { + Write-Host " no artLegacyMUL.uop in the copy -- already a mul-only client" + } else { + Move-Item -LiteralPath $uopPath -Destination "$uopPath.disabled" -Force + $uopPresent = $false + Add-Defect 'nouop' 'artLegacyMUL.uop' 'moved aside so art is read from art.mul/artidx.mul' ` + 'every index-level defect below becomes reachable; without this they are inert' + } + + Write-Host "" +} elseif ($uopPresent -and (($Tiers -contains 'corrupt') -or ($Tiers -contains 'customart'))) { + Write-Host " WARNING: artLegacyMUL.uop is present and the nouop tier was not selected." + Write-Host " FileIndex prefers the UOP outright, so the corrupt and customart tiers" + Write-Host " will write to files the library never opens. Add -Tiers nouop." + Write-Host "" +} + +# Static ids are offset by 0x4000 in the index; land tiles occupy 0..0x3FFF. +$StaticBase = 0x4000 + +# Find donor records to copy and victims to corrupt: real, modestly sized statics, so the defects +# are applied to entries that genuinely work today. Picking arbitrary ids risks landing on slots +# that are already empty, where a "defect" would prove nothing. +$donors = New-Object System.Collections.ArrayList +for ($id = 0x1000; $id -lt 0x3000 -and $donors.Count -lt 24; $id++) { + $e = Get-ArtEntry $idx ($id + $StaticBase) + if ($e.Lookup -ge 0 -and $e.Length -gt 200 -and $e.Length -lt 4000 -and ($e.Lookup + $e.Length) -le $artMulLength) { + [void] $donors.Add([pscustomobject]@{ Id = $id; Entry = $e }) + } +} + +if ($donors.Count -lt 12) { throw "found only $($donors.Count) usable donor statics -- the copy looks wrong" } + +Write-Host "using donor statics: $(($donors | Select-Object -First 12 | ForEach-Object { $_.Id }) -join ', ')" +Write-Host "" + +$idxDirty = $false + +# -- Tier: customart -------------------------------------------------------------------------- + +if ($Tiers -contains 'customart') { + Write-Host "tier customart" + + # A custom-art client does not fill spare slots -- artidx.mul is exactly sized (62,692 + # entries here, not one to spare), so adding art means GROWING the index. `Art` builds its + # FileIndex with length 0x10000, so there is room for 2,844 more ids before the library stops + # looking, and the stock ceiling turns out to be nothing more than the size of a file. + $idxCeiling = 0x10000 + + if ($entryCount -ge $idxCeiling) { + Write-Host " artidx.mul is already at the 0x10000 ceiling -- skipping tier" + } else { + $addCount = 8 + $grown = New-Object byte[] (($entryCount + $addCount) * 12) + [Array]::Copy($idx, 0, $grown, 0, $idx.Length) + $idx = $grown + + # Read every donor record BEFORE opening the append handle. Append mode takes an + # exclusive lock, so reading the same file while appending to it fails outright. + $buffers = @() + $reader = [System.IO.File]::Open($artMulPath, 'Open', 'Read', 'ReadWrite') + + try { + for ($n = 0; $n -lt $addCount; $n++) { + $donor = $donors[$n] + $buffer = New-Object byte[] $donor.Entry.Length + [void] $reader.Seek($donor.Entry.Lookup, 'Begin') + [void] $reader.Read($buffer, 0, $buffer.Length) + $buffers += , $buffer + } + } finally { $reader.Dispose() } + + $appendAt = $artMulLength + $stream = [System.IO.File]::Open($artMulPath, 'Append', 'Write') + + try { + for ($n = 0; $n -lt $addCount; $n++) { + $buffer = $buffers[$n] + $stream.Write($buffer, 0, $buffer.Length) + + $slot = $entryCount + $n + $newId = $slot - $StaticBase + Set-ArtEntry $idx $slot $appendAt $buffer.Length $donors[$n].Entry.Extra + $appendAt += $buffer.Length + + Add-Defect 'customart' "static/$newId" ` + "custom art appended past the stock ceiling (a copy of static/$($donors[$n].Id))" ` + 'decodes cleanly; proves the ceiling is read from the file, not from a constant' + } + } finally { $stream.Dispose() } + + $entryCount += $addCount + $idxDirty = $true + } + + Write-Host "" +} + +# -- Tier: corrupt ---------------------------------------------------------------------------- + +if ($Tiers -contains 'corrupt') { + Write-Host "tier corrupt" + + $artMulLength = (Get-Item -LiteralPath $artMulPath).Length + $v = 8 # donors 0..7 may have been consumed by customart as sources; they are unmodified + + # 1. A lookup past the end of art.mul. FileIndex.Seek DOES check this one + # (`Stream.Length < e.lookup`), so the library and the validator should agree. + $victim = $donors[$v++].Id + Set-ArtEntry $idx ($victim + $StaticBase) ([int]($artMulLength + 4096)) 512 0 + Add-Defect 'corrupt' "static/$victim" 'lookup 4 KB past the end of art.mul' ` + 'refused by the validator; Seek also catches this one, so no picture' + + # 2. A record that STARTS inside the file and ENDS outside it. This is the gap: Seek checks + # the start and never the end, stream.Read returns short, the decoders ignore the count, + # and m_StreamBuffer still holds the PREVIOUS asset. The expected outcome is a picture of + # something else entirely, reported as a success by every count in the library. + $victim = $donors[$v++].Id + Set-ArtEntry $idx ($victim + $StaticBase) ([int]($artMulLength - 64)) 8192 0 + Add-Defect 'corrupt' "static/$victim" 'record starts 64 bytes before EOF and declares 8,192' ` + 'REFUSED BUT DECODED -- the stale-buffer wrong picture' + + # 3. A length too small to hold even the 8-byte header. + $victim = $donors[$v++].Id + $donorEntry = $donors[$v - 1].Entry + Set-ArtEntry $idx ($victim + $StaticBase) $donorEntry.Lookup 4 0 + Add-Defect 'corrupt' "static/$victim" 'declared length 4 -- smaller than the static header' ` + 'refused by the validator' + + # 4/5/6 rewrite the record BODY, so they need their own bytes rather than an index edit. + # Appended to art.mul and pointed at, which leaves the donor's real record intact. + $stream = [System.IO.File]::Open($artMulPath, 'Append', 'Write') + try { + $appendAt = (Get-Item -LiteralPath $artMulPath).Length + + # 4. Absurd dimensions. LoadStatic allocates new Bitmap(width, height) straight from two + # ushorts in the file. 8000x8000 is ~128 MB -- survivable, and the point is made; the + # same field can ask for 65535x65535, which is 8 GB from a two-byte edit. + $victim = $donors[$v++].Id + $rec = New-Object byte[] 2048 + [Array]::Copy([BitConverter]::GetBytes([uint16] 8000), 0, $rec, 4, 2) + [Array]::Copy([BitConverter]::GetBytes([uint16] 8000), 0, $rec, 6, 2) + $stream.Write($rec, 0, $rec.Length) + Set-ArtEntry $idx ($victim + $StaticBase) $appendAt $rec.Length 0 + $appendAt += $rec.Length + Add-Defect 'corrupt' "static/$victim" 'header declares 8000x8000 (a ~128 MB allocation from two bytes)' ` + 'refused by the validator; the library would allocate it' + + # 5. A row-lookup table pointing outside the record. This is what LoadStatic's unbounded + # read cursor was written to walk off the end of. + $victim = $donors[$v++].Id + $rec = New-Object byte[] 512 + [Array]::Copy([BitConverter]::GetBytes([uint16] 32), 0, $rec, 4, 2) # width + [Array]::Copy([BitConverter]::GetBytes([uint16] 32), 0, $rec, 6, 2) # height + for ($row = 0; $row -lt 32; $row++) { + # Each row's offset is added to (height + 4); 60000 puts every row far outside. + [Array]::Copy([BitConverter]::GetBytes([uint16] 60000), 0, $rec, (8 + $row * 2), 2) + } + $stream.Write($rec, 0, $rec.Length) + Set-ArtEntry $idx ($victim + $StaticBase) $appendAt $rec.Length 0 + $appendAt += $rec.Length + Add-Defect 'corrupt' "static/$victim" 'row table points 60,000 words outside a 512-byte record' ` + 'refused by the validator; the library reads adjacent heap' + + # 6. A well-formed row table whose run length overruns the record. + $victim = $donors[$v++].Id + $rec = New-Object byte[] 256 + [Array]::Copy([BitConverter]::GetBytes([uint16] 16), 0, $rec, 4, 2) + [Array]::Copy([BitConverter]::GetBytes([uint16] 2), 0, $rec, 6, 2) + [Array]::Copy([BitConverter]::GetBytes([uint16] 0), 0, $rec, 8, 2) # row 0 offset + [Array]::Copy([BitConverter]::GetBytes([uint16] 0), 0, $rec, 10, 2) # row 1 offset + $runAt = (2 + 4) * 2 # (height + 4) words + [Array]::Copy([BitConverter]::GetBytes([uint16] 0), 0, $rec, $runAt, 2) # xOffset + [Array]::Copy([BitConverter]::GetBytes([uint16] 16), 0, $rec, ($runAt + 2), 2) # xRun, but + # the record has nowhere near 16 pixels left after this point. + $stream.Write($rec, 0, $rec.Length) + Set-ArtEntry $idx ($victim + $StaticBase) $appendAt 20 0 + $appendAt += $rec.Length + Add-Defect 'corrupt' "static/$victim" 'a 16-pixel run declared in a 20-byte record' ` + 'refused by the validator' + } finally { $stream.Dispose() } + + # 7. A land tile shorter than the 2,024 bytes LoadLand reads unconditionally. + $landVictim = 0x0100 + $landEntry = Get-ArtEntry $idx $landVictim + if ($landEntry.Lookup -ge 0 -and $landEntry.Length -gt 0) { + Set-ArtEntry $idx $landVictim $landEntry.Lookup 512 0 + Add-Defect 'corrupt' "land/$landVictim" 'land record declared 512 bytes; LoadLand always reads 2,024' ` + 'refused by the validator; the library reads past the buffer' + } + + $idxDirty = $true + Write-Host "" +} + +if ($idxDirty) { + [System.IO.File]::WriteAllBytes($artIdxPath, $idx) + Write-Host "wrote artidx.mul" + Write-Host "" +} + +# -- Tier: verdata ---------------------------------------------------------------------------- + +if ($Tiers -contains 'verdata') { + Write-Host "tier verdata" + + # Layout: int32 count, then count * 5 int32 (file, index, lookup, length, extra), then the + # payloads. `lookup` is an absolute offset into this file. + $entries = New-Object System.Collections.ArrayList + $payloads = New-Object System.Collections.ArrayList + + $donorA = $donors[$donors.Count - 1] + $donorB = $donors[$donors.Count - 2] + + $artSource = [System.IO.File]::Open($artMulPath, 'Open', 'Read', 'ReadWrite') + try { + $bufferA = New-Object byte[] $donorA.Entry.Length + [void] $artSource.Seek($donorA.Entry.Lookup, 'Begin') + [void] $artSource.Read($bufferA, 0, $bufferA.Length) + } finally { $artSource.Dispose() } + + # The victims: ids whose art will now come from verdata.mul rather than art.mul. + $legitVictim = $donors[$donors.Count - 3].Id + $pastEofVictim = $donors[$donors.Count - 4].Id + + # A legitimate patch -- the branch working as designed. Without this the tier only proves the + # failure case, and "verdata is broken" and "verdata is never reached" look identical. + [void] $payloads.Add($bufferA) + [void] $entries.Add([pscustomobject]@{ + File = 4; Index = ($legitVictim + $StaticBase); Length = $bufferA.Length; Extra = $donorA.Entry.Extra + PayloadIndex = 0; PastEof = $false + }) + + # The failure case. FileIndex.Seek bounds-checks the mul stream and calls Verdata.Seek with no + # check at all; seeking a FileStream past EOF is legal, the read returns nothing, and the + # shared decode buffer still holds the previous asset. + [void] $entries.Add([pscustomobject]@{ + File = 4; Index = ($pastEofVictim + $StaticBase); Length = 900; Extra = 0 + PayloadIndex = -1; PastEof = $true + }) + + # An anim patch, so the tier covers the other file the verdata branch serves. anim.mul is + # verdata file 6; for body < 200 the record index is body*110 + action*5 + direction. + $animBody = 34 # wolf -- decodes on this client, so a patch to it is observable + $animIndex = ($animBody * 110) + (0 * 5) + 1 + [void] $entries.Add([pscustomobject]@{ + File = 6; Index = $animIndex; Length = 700; Extra = 0 + PayloadIndex = -1; PastEof = $true + }) + + $headerSize = 4 + ($entries.Count * 20) + $offset = $headerSize + foreach ($entry in $entries) { + if ($entry.PayloadIndex -ge 0) { + $entry | Add-Member -NotePropertyName Lookup -NotePropertyValue $offset -Force + $offset += $payloads[$entry.PayloadIndex].Length + } + } + + $totalSize = $offset + + # Past-EOF lookups are resolved last, because "past the end" is only meaningful once the end + # is known. + foreach ($entry in $entries) { + if ($entry.PastEof) { + $entry | Add-Member -NotePropertyName Lookup -NotePropertyValue ($totalSize + 8192) -Force + } + } + + $verdata = New-Object byte[] $totalSize + Write-Int32LE $verdata 0 $entries.Count + + $at = 4 + foreach ($entry in $entries) { + Write-Int32LE $verdata $at $entry.File + Write-Int32LE $verdata ($at + 4) $entry.Index + Write-Int32LE $verdata ($at + 8) $entry.Lookup + Write-Int32LE $verdata ($at + 12) $entry.Length + Write-Int32LE $verdata ($at + 16) $entry.Extra + $at += 20 + } + + foreach ($entry in $entries) { + if ($entry.PayloadIndex -ge 0) { + $payload = $payloads[$entry.PayloadIndex] + [Array]::Copy($payload, 0, $verdata, $entry.Lookup, $payload.Length) + } + } + + [System.IO.File]::WriteAllBytes((Join-Path $destFull 'verdata.mul'), $verdata) + + Add-Defect 'verdata' "static/$legitVictim" ` + "legitimately patched to static/$($donorA.Id)'s art via verdata.mul" ` + 'decodes; the picture must CHANGE, which is how we know the branch ran' + Add-Defect 'verdata' "static/$pastEofVictim" ` + 'verdata entry whose lookup is 8 KB past the end of verdata.mul' ` + 'REFUSED BUT DECODED -- Verdata.Seek is not bounds-checked' + Add-Defect 'verdata' "body/$animBody" ` + "anim.mul record $animIndex patched to a verdata offset past EOF" ` + 'the wolf must not silently become another creature' + + Write-Host (" wrote verdata.mul: {0} entries, {1:N0} bytes" -f $entries.Count, $totalSize) + Write-Host "" +} + +# -- Tier: bodyconv --------------------------------------------------------------------------- + +if ($Tiers -contains 'bodyconv') { + Write-Host "tier bodyconv" + + $bodyconvPath = Join-Path $destFull 'Bodyconv.def' + + if (-not (Test-Path -LiteralPath $bodyconvPath)) { + Write-Host " no Bodyconv.def in the copy -- skipping tier" + } else { + # Columns are tab-separated: original, anim2, anim3, anim4, anim5. -1 means "not in that + # file". BodyConverter.Convert returns the file type of the FIRST column that is not -1, + # and the extractor must take that answer and stop. + $lines = @( + "", + "# Asset Bridge phase 0 -- deliberate defects (tools/patch_client.ps1)", + "1900`t-1`t-1`t-1`t60000", + "1901`t666`t-1`t-1`t-1" + ) + + Add-Content -LiteralPath $bodyconvPath -Value ($lines -join "`r`n") -Encoding ASCII + + Add-Defect 'bodyconv' 'body/1900' 'mapped to anim5 index 60,000, which does not exist' ` + 'reports nothing -- and must NOT fall back to another anim file' + Add-Defect 'bodyconv' 'body/1901' 'mapped to anim2 index 666, where something unrelated lives' ` + 'decodes a picture of the WRONG creature -- the spider case, on purpose' + } + + Write-Host "" +} + +# -- The source must be untouched ------------------------------------------------------------- + +$tampered = @() +foreach ($name in $before.Keys) { + $path = Join-Path $sourceFull $name + $now = (Get-FileHash -LiteralPath $path -Algorithm SHA256).Hash + if ($now -ne $before[$name]) { $tampered += $name } +} + +if ($tampered.Count -gt 0) { + throw "THE SOURCE CLIENT WAS MODIFIED: $($tampered -join ', '). Restore it from the installer before doing anything else." +} + +Write-Host "source client verified unchanged ($($before.Count) files hashed before and after)" + +# -- Manifest --------------------------------------------------------------------------------- + +$manifest = [pscustomobject]@{ + built = (Get-Date).ToUniversalTime().ToString('u') + source = $sourceFull + dest = $destFull + tiers = $Tiers + defects = @($script:Defects) +} + +$manifestPath = Join-Path $destFull 'patched-client.manifest.json' +$manifest | ConvertTo-Json -Depth 5 | Set-Content -LiteralPath $manifestPath -Encoding utf8 + +Write-Host "" +Write-Host ("{0} deliberate defects; manifest at {1}" -f $script:Defects.Count, $manifestPath) +Write-Host "" +Write-Host "Point the probe at it by adding to the shard's Config/Bridge.cfg:" +Write-Host "" +Write-Host " AssetProbeClient=$destFull" +Write-Host "" +Write-Host "then, in game or from the rig driver: [assetprobe all patched" diff --git a/tools/scaffolding/BridgeAssetProbe.cs b/tools/scaffolding/BridgeAssetProbe.cs new file mode 100644 index 0000000..6f0bfb9 --- /dev/null +++ b/tools/scaffolding/BridgeAssetProbe.cs @@ -0,0 +1,1346 @@ +using System; +using System.Collections.Generic; +using System.Drawing; +using System.Globalization; +using System.IO; +using System.Text; +using System.Threading; + +using Server.Commands; + +using Ultima; + +namespace Server.Custom +{ + /// + /// **Asset Bridge phase 0 — the spike** (docs/link/v8.md §16). + /// + /// §4 decided to call ServUO's own vendored Ultima decoders rather than reimplement + /// them. That decision rests on a probe run from **PowerShell** against a **stock** client, + /// and neither of those is the environment the extractor will actually live in. This runs + /// the same decoders from **inside a running ServUO**, against a **deliberately patched** + /// client, and its whole job is to find a fault on a path we call before eight phases are + /// built on top of one. + /// + /// What "a fault" means here is wider than a crash, and the wider half is the dangerous + /// half. `Ultima`'s decoders take their bounds from the files they are reading, so a + /// malformed record does not usually throw — it produces **a confident, wrong picture**. + /// Four such shapes are known from reading the source and each has its own counter below: + /// + /// * LoadStatic walks bindata[count++] with no bound on count. The + /// two guards in that loop bound the *write* into the bitmap and not the *read* out of + /// the record, so a record whose row table points outside itself reads adjacent heap. + /// * stream.Read(m_StreamBuffer, 0, length) ignores its return value, and + /// m_StreamBuffer is reused and only ever grown. A short read therefore decodes + /// **the previous asset's bytes** under this asset's id. + /// * LoadLand reads a fixed 2,024 bytes whatever length says. + /// * Art.GetLegalItemID returns **0** for an out-of-range id, so GetStatic + /// of a nonexistent id can hand back item 0's picture instead of nothing. + /// + /// So every sweep here compares two answers: what says + /// about the index entry *before* the call, and what Ultima does *with* it. The + /// interesting cell is not the error count. It is **REFUSED-BUT-DECODED** — a record the + /// validator rejects and the library cheerfully returns a bitmap for. Those are the wrong + /// pictures, and they are invisible to any success count. + /// + /// **Nothing here calls Ultima.Gumps**, which is a safety rule and not a preference + /// (v8.md §4.1) — with one deliberate exception, the opt-in gump section, whose + /// entire purpose is to reproduce the process-killing access violation from inside ServUO + /// so the rule has evidence behind it. It is off by default and it **takes the shard down**. + /// + /// Test scaffolding. Never deployed; deploy.ps1 copies only overlay/. + /// In game / from BridgeRigDriver: [assetprobe [section] [stock|patched]. + /// Flag: AssetProbeOnStart. Build the patched client with + /// tools/patch_client.ps1. + /// + public static class BridgeAssetProbe + { + // Where the run writes. The checkpoint is the point of the whole arrangement: some of + // these faults are corrupted-state exceptions that no catch block sees, so the last id + // written to disk is the only evidence of where the process died. + private static readonly string OutputDir = Path.Combine(Core.BaseDirectory, "Logs", "AssetProbe"); + + private static string _checkpointPath; + private static StreamWriter _report; + private static readonly object _sync = new object(); + private static bool _running; + + // Snapshot of the player-character body ids, taken on the Core thread (§5.2). Not + // hardcoded: RaceDefinitions.cs passes the gargoyle's ghost bodies in the opposite order + // to the other races, and a shard that calls RegisterRace adds ids no table of ours holds. + private static List _playerBodies; + + private struct PlayerBody + { + public string Race; + public string Slot; + public int Body; + + public PlayerBody(string race, string slot, int body) + { + Race = race; + Slot = slot; + Body = body; + } + } + + public static void Initialize() + { + CommandSystem.Register("assetprobe", AccessLevel.Administrator, Probe_OnCommand); + + if (Config.Get("Bridge.AssetProbeOnStart", false)) + EventSink.ServerStarted += () => Timer.DelayCall(TimeSpan.FromSeconds(10.0), () => Begin(null, "all", null)); + } + + [Usage("assetprobe [all|paths|statics|land|bodies|players|cliloc|gump] [stock|patched]")] + [Description("Drives ServUO's vendored Ultima decoders against a client and reports every fault and every wrong picture.")] + private static void Probe_OnCommand(CommandEventArgs e) + { + var section = e.Arguments.Length > 0 ? e.Arguments[0].ToLowerInvariant() : "all"; + var which = e.Arguments.Length > 1 ? e.Arguments[1].ToLowerInvariant() : null; + + Begin(e.Mobile, section, which); + } + + // ── Entry ──────────────────────────────────────────────────────────────────────────── + + /// + /// Reads what only the Core thread may read, then hands the sweep to a background + /// thread. That split is not tidiness — it is the §8 threading shape this protocol + /// introduces, rehearsed here: the decode must run OFF the Core thread (a 66,000-id + /// sweep would freeze the shard), and the world reads it depends on must run ON it. + /// + public static void Begin(Mobile from, string section, string which) + { + lock (_sync) + { + if (_running) + { + Tell(from, "already running — one sweep at a time, so the checkpoint means something"); + return; + } + + _running = true; + } + + // Core-thread reads first. + _playerBodies = ReadPlayerBodies(); + + var thread = new Thread(() => Run(from, section, which)); + thread.IsBackground = true; + thread.Name = "BridgeAssetProbe"; + thread.Priority = ThreadPriority.BelowNormal; + thread.Start(); + + Tell(from, "started on a background thread — output in Logs/AssetProbe"); + } + + /// §5.2: ask the shard which bodies are player characters; never hardcode them. + private static List ReadPlayerBodies() + { + var list = new List(); + + foreach (var race in Race.AllRaces) + { + if (race == null) + continue; + + list.Add(new PlayerBody(race.Name, "male", race.MaleBody)); + list.Add(new PlayerBody(race.Name, "female", race.FemaleBody)); + list.Add(new PlayerBody(race.Name, "male ghost", race.MaleGhostBody)); + list.Add(new PlayerBody(race.Name, "female ghost", race.FemaleGhostBody)); + } + + return list; + } + + private static void Run(Mobile from, string section, string which) + { + try + { + Directory.CreateDirectory(OutputDir); + + var stamp = DateTime.UtcNow.ToString("yyyyMMdd-HHmmss", CultureInfo.InvariantCulture); + _checkpointPath = Path.Combine(OutputDir, "checkpoint.txt"); + _report = new StreamWriter(Path.Combine(OutputDir, "report-" + stamp + ".txt"), false, new UTF8Encoding(false)); + _report.AutoFlush = true; + + Say("Asset Bridge phase 0 probe — " + DateTime.UtcNow.ToString("u", CultureInfo.InvariantCulture)); + Say("section=" + section + " client=" + (which ?? "(config default)")); + Say(""); + + // Caching off, or a full sweep holds ~49,000 live Bitmaps. That is not only memory: + // every Bitmap is a GDI object and Windows caps a process at 10,000 of them, so a + // cached sweep fails partway through for a reason that has nothing to do with the + // files being read. Each bitmap below is disposed as soon as it is measured. + Files.CacheData = false; + + if (!PointAtClient(which)) + return; + + bool all = section == "all"; + + if (all || section == "paths") SectionPaths(); + if (all || section == "statics") SectionStatics(); + if (all || section == "land") SectionLand(); + if (all || section == "bodies") SectionBodies(); + if (all || section == "players") SectionPlayers(); + if (all || section == "cliloc") SectionCliloc(); + + // Never part of "all". This one is expected to kill the process. + if (section == "gump") SectionGump(); + + Checkpoint("done", 0); + Say(""); + Say("complete."); + } + catch (Exception e) + { + Say("PROBE FAILED: " + e); + } + finally + { + if (_report != null) + { + _report.Dispose(); + _report = null; + } + + lock (_sync) + _running = false; + } + } + + // ── Pointing Ultima at a client ────────────────────────────────────────────────────── + + /// + /// Re-points Ultima.Files at the stock or the patched client and reloads every + /// index built from it. + /// + /// **Not Files.SetMulPath(string).** That overload keeps any entry already + /// holding an absolute path — its own comment reads // absolut dir ignore — and + /// it *writes* absolute paths. So it works exactly once: the second call, and every + /// call that would switch back, is a silent no-op, and the probe would report a run + /// against the patched client while reading the stock one. The two-argument overload + /// writes the key directly and is the only one that can be called twice. + /// + private static bool PointAtClient(string which) + { + var patched = Config.Get("Bridge.AssetProbeClient", (string)null); + + if (which == null) + which = string.IsNullOrEmpty(patched) ? "stock" : "patched"; + + string root; + + if (which == "patched") + { + if (string.IsNullOrEmpty(patched)) + { + Say("no Bridge.AssetProbeClient configured — build one with tools/patch_client.ps1"); + return false; + } + + root = patched; + } + else + { + // What the shard itself resolved at boot: the §1 premise, read rather than assumed. + root = Core.DataDirectories.Count > 0 ? Core.DataDirectories[0] : Files.Directory; + } + + if (string.IsNullOrEmpty(root) || !Directory.Exists(root)) + { + Say("client directory does not exist: " + (root ?? "(null)")); + return false; + } + + foreach (var file in InterestingFiles) + { + var full = Path.Combine(root, file); + // An absent file must resolve to nothing, NOT fall through to the stock client — + // otherwise a patched tree missing a file quietly borrows the real one and the + // run proves nothing. + Files.SetMulPath(File.Exists(full) ? full : string.Empty, file); + } + + Verdata.Initialize(); + Art.Reload(); + Animations.Reload(); + BodyConverter.Initialize(); + Hues.Initialize(); + + Say("pointed at the " + which + " client: " + root); + Say("verdata patches loaded: " + (Verdata.Patches == null ? 0 : Verdata.Patches.Length)); + Say(""); + + return true; + } + + /// + /// Every key this protocol reads, lowercase because that is how Files.MulPath is + /// keyed. Gump keys are absent on purpose (§4.1). + /// + private static readonly string[] InterestingFiles = + { + "anim.idx", "anim.mul", "anim2.idx", "anim2.mul", "anim3.idx", "anim3.mul", + "anim4.idx", "anim4.mul", "anim5.idx", "anim5.mul", + "art.mul", "artidx.mul", "artlegacymul.uop", + "body.def", "bodyconv.def", "hues.mul", "verdata.mul", "cliloc.enu" + }; + + // ── paths ──────────────────────────────────────────────────────────────────────────── + + /// + /// §1's premise, verified from inside the shard rather than argued: the client files + /// are already here, and the shard already knows where. + /// + private static void SectionPaths() + { + Head("paths"); + + Say("Core.DataDirectories (" + Core.DataDirectories.Count + "):"); + + foreach (var dir in Core.DataDirectories) + Say(" " + dir); + + Say("Ultima.Files.Directory: " + (Files.Directory ?? "(null)")); + Say(""); + + foreach (var file in InterestingFiles) + { + var path = Files.GetFilePath(file); + + if (path == null) + { + Say(string.Format(" {0,-22} ABSENT", file)); + continue; + } + + var info = new FileInfo(path); + Say(string.Format( + " {0,-22} {1,14:N0} bytes {2}", file, info.Length, + info.LastWriteTimeUtc.ToString("u", CultureInfo.InvariantCulture))); + } + + Say(""); + } + + // ── statics ────────────────────────────────────────────────────────────────────────── + + private static void SectionStatics() + { + Head("statics — Art.GetStatic"); + + var index = BridgeAssetValidator.OpenArtIndex(); + + if (index == null) + { + Say("no art index — artidx.mul/art.mul did not resolve"); + return; + } + + var dataPath = BridgeAssetValidator.ArtDataPath(); + long mulLength = BridgeAssetValidator.MulLength(dataPath); + long verdataLength = BridgeAssetValidator.MulLength(Files.GetFilePath("verdata.mul")); + + Say(" index offsets are into: " + dataPath); + + var tally = new Tally(); + int max = Config.Get("Bridge.AssetProbeMaxStatic", 0xFFFF); + + using (var reader = new BridgeAssetValidator.RecordReader( + dataPath, Files.GetFilePath("verdata.mul"))) + { + for (int id = 0; id <= max; id++) + { + Checkpoint("statics", id); + + // The validator's verdict, taken from the index entry BEFORE the library is + // asked. 0x4000 is the static offset Art applies internally. + string reason; + var verdict = BridgeAssetValidator.CheckEntry(index, id + 0x4000, mulLength, verdataLength, out reason); + + // The entry can be well-formed and the record inside it still hostile, so a + // passing entry gets its row table walked before the library sees the id. + // This is the check with teeth — LoadStatic's read cursor is unbounded. + if (verdict == BridgeAssetValidator.Verdict.Ok) + { + string deepReason; + + if (!reader.StaticSane(index, id + 0x4000, out deepReason)) + { + verdict = BridgeAssetValidator.Verdict.Refused; + reason = deepReason; + } + } + + // checkmaxid:false deliberately. With it true, GetLegalItemID maps an + // out-of-range id to 0 and the call returns ITEM 0's picture — the + // out-of-range answer would be a real bitmap of the wrong thing, which is + // precisely the confusion being counted. + Bitmap bmp = null; + string thrown = null; + + try + { + bmp = Art.GetStatic(id, false); + } + catch (Exception e) + { + thrown = e.GetType().Name + ": " + e.Message; + } + + Record(tally, verdict, reason, bmp != null, thrown, "static/" + id); + + if (bmp != null) + bmp.Dispose(); + } + } + + tally.Report("statics 0.." + max); + } + + // ── land ───────────────────────────────────────────────────────────────────────────── + + private static void SectionLand() + { + Head("land — Art.GetLand"); + + var index = BridgeAssetValidator.OpenArtIndex(); + + if (index == null) + { + Say("no art index"); + return; + } + + long mulLength = BridgeAssetValidator.MulLength(BridgeAssetValidator.ArtDataPath()); + long verdataLength = BridgeAssetValidator.MulLength(Files.GetFilePath("verdata.mul")); + + var tally = new Tally(); + + for (int id = 0; id < 0x4000; id++) + { + Checkpoint("land", id); + + string reason; + var verdict = BridgeAssetValidator.CheckEntry(index, id, mulLength, verdataLength, out reason); + + // LoadLand reads a fixed 2,024 bytes whatever the record says, so a short record + // reads past the buffer. The validator's land rule is the only thing standing + // between that and an out-of-bounds read. + if (verdict == BridgeAssetValidator.Verdict.Ok) + { + string landReason; + + if (!BridgeAssetValidator.LandLengthSane(index, id, out landReason)) + { + verdict = BridgeAssetValidator.Verdict.Refused; + reason = landReason; + } + } + + Bitmap bmp = null; + string thrown = null; + + try + { + bmp = Art.GetLand(id); + } + catch (Exception e) + { + thrown = e.GetType().Name + ": " + e.Message; + } + + Record(tally, verdict, reason, bmp != null, thrown, "land/" + id); + + if (bmp != null) + bmp.Dispose(); + } + + tally.Report("land 0..16383"); + } + + // ── bodies ─────────────────────────────────────────────────────────────────────────── + + /// + /// Sweeps every body id, taking BodyConverter.Convert's answer and stopping + /// there. + /// + /// **It must never ask the other anim files when that answer yields nothing** (v8.md + /// §4.3). Doing so does not find missing art: gargoyle 666 maps to anim5, where + /// this client has nothing, and asking anim2 for index 666 returns 175 + /// decodable frames of a giant spider. Every one of those reads reports success, and + /// nothing downstream can tell. A "0 rows" outcome is the correct answer. + /// + /// So the sweep records the file type each body resolved to and whether that file + /// answered — and never a second opinion. + /// + private static void SectionBodies() + { + Head("bodies — Animations.GetAnimation, one direction, first frame"); + + int direction = Config.Get("Bridge.AssetProbeCreatureDirection", 1); + int decoded = 0, empty = 0, faulted = 0; + var byFileType = new int[8]; + var faults = new List(); + + for (int body = 0; body < 2048; body++) + { + Checkpoint("bodies", body); + + int translated = body; + int fileType; + + try + { + fileType = BodyConverter.Convert(ref translated); + } + catch (Exception e) + { + faulted++; + faults.Add("body " + body + " BodyConverter.Convert: " + e.GetType().Name + ": " + e.Message); + continue; + } + + if (fileType >= 0 && fileType < byFileType.Length) + byFileType[fileType]++; + + try + { + int hue = 0; + var frames = Animations.GetAnimation(body, 0, direction, ref hue, false, true); + + if (frames != null && frames.Length > 0 && frames[0] != null && frames[0].Bitmap != null) + { + decoded++; + frames[0].Bitmap.Dispose(); + } + else + { + empty++; + } + } + catch (Exception e) + { + faulted++; + + if (faults.Count < 40) + faults.Add("body " + body + " (fileType " + fileType + "): " + e.GetType().Name + ": " + e.Message); + } + } + + Say("direction " + direction + " (creature default — §5.1)"); + Say(string.Format(" decoded {0} empty {1} FAULTED {2}", decoded, empty, faulted)); + Say(" by file type: " + string.Join(", ", FileTypeCounts(byFileType))); + + if (faults.Count > 0) + { + Say(""); + Say(" faults:"); + + foreach (var f in faults) + Say(" " + f); + } + + Say(""); + } + + private static string[] FileTypeCounts(int[] byFileType) + { + var parts = new List(); + + for (int i = 0; i < byFileType.Length; i++) + { + if (byFileType[i] > 0) + parts.Add(i + "=" + byFileType[i]); + } + + return parts.ToArray(); + } + + // ── players ────────────────────────────────────────────────────────────────────────── + + /// + /// The twelve (on stock 57.4) player-character bodies, each at direction 0 — head-on, + /// because a character is a portrait and should look at you (§5.1). + /// + /// Six of them are expected to report nothing on the legacy path: both human ghosts and + /// every gargoyle body are UOP-only. **That is the measurement, not a failure** — it is + /// what phase 4's UOP reader exists for, and a probe that flagged it red would teach an + /// operator to ignore the panel. + /// + private static void SectionPlayers() + { + Head("player bodies — Race.AllRaces, direction 0"); + + if (_playerBodies == null || _playerBodies.Count == 0) + { + Say("no races registered (was the Core-thread snapshot taken?)"); + return; + } + + int direction = Config.Get("Bridge.AssetProbePlayerDirection", 0); + int decoded = 0, absent = 0; + + foreach (var pb in _playerBodies) + { + Checkpoint("players", pb.Body); + + int translated = pb.Body; + int fileType = BodyConverter.Convert(ref translated); + + string outcome; + + try + { + int hue = 0; + var frames = Animations.GetAnimation(pb.Body, 0, direction, ref hue, false, true); + + if (frames != null && frames.Length > 0 && frames[0] != null && frames[0].Bitmap != null) + { + var bmp = frames[0].Bitmap; + outcome = "decoded " + bmp.Width + "x" + bmp.Height; + bmp.Dispose(); + decoded++; + } + else + { + outcome = "no art on the legacy path (UOP-only — phase 4)"; + absent++; + } + } + catch (Exception e) + { + outcome = "FAULTED " + e.GetType().Name + ": " + e.Message; + } + + Say(string.Format(" {0,-10} {1,-14} body {2,-5} fileType {3,-3} {4}", + pb.Race, pb.Slot, pb.Body, fileType, outcome)); + } + + Say(""); + Say(string.Format(" {0} decoded, {1} absent, of {2}", decoded, absent, _playerBodies.Count)); + Say(""); + } + + // ── cliloc ─────────────────────────────────────────────────────────────────────────── + + /// + /// Runs against the client's own Cliloc.enu and, when + /// a reference is configured, diffs it against UOFiddler's output entry by entry. + /// + /// The reference is what makes this a test rather than a demonstration. A decompressor + /// that is subtly wrong still produces a plausible table — mostly-right strings with a + /// few mangled ones is the expected shape of a bug in an inverse-BWT coder, and a row + /// count alone would pass it. Produce the reference with + /// website/server/tools/cliloc-export --tsv. + /// + private static void SectionCliloc() + { + Head("cliloc — the ported Mythic reader (§9)"); + + var path = Files.GetFilePath("cliloc.enu"); + + if (path == null) + { + Say("cliloc.enu did not resolve"); + return; + } + + Checkpoint("cliloc", 0); + + var started = DateTime.UtcNow; + + List entries; + string warning, error; + + if (!BridgeMythicCliloc.TryLoadFile(path, out entries, out warning, out error)) + { + Say("FAILED: " + error); + return; + } + + var elapsed = DateTime.UtcNow - started; + + int blank = 0; + + foreach (var entry in entries) + { + if (string.IsNullOrEmpty(entry.Text)) + blank++; + } + + Say(string.Format(" {0:N0} entries in {1:N0} ms ({2:N0} blank, {3:N0} would be stored)", + entries.Count, elapsed.TotalMilliseconds, blank, entries.Count - blank)); + + if (warning != null) + Say(" WARNING: " + warning); + + var reference = Config.Get("Bridge.AssetProbeClilocRef", (string)null); + + if (string.IsNullOrEmpty(reference)) + { + Say(" no Bridge.AssetProbeClilocRef set — row count only, which proves nothing about the strings"); + Say(""); + return; + } + + CompareToReference(entries, reference); + } + + /// + /// Diffs against a UOFiddler-produced tab-separated table. The comparison is + /// deliberately two-sided: an id we produced and it did not is as much a defect as a + /// mismatched string, and only checking the ids we happen to hold would hide a table + /// that stopped early. + /// + private static void CompareToReference(List entries, string reference) + { + if (!File.Exists(reference)) + { + Say(" reference not found: " + reference); + return; + } + + var theirs = new Dictionary(); + + foreach (var line in File.ReadAllLines(reference)) + { + var tab = line.IndexOf('\t'); + + if (tab <= 0) + continue; + + int number; + + if (!int.TryParse(line.Substring(0, tab), NumberStyles.Integer, CultureInfo.InvariantCulture, out number)) + continue; + + theirs[number] = line.Substring(tab + 1); + } + + var ours = new Dictionary(); + + foreach (var entry in entries) + ours[entry.Number] = entry.Text; + + int matched = 0, differed = 0, onlyOurs = 0, onlyTheirs = 0; + var examples = new List(); + + foreach (var pair in ours) + { + string theirText; + + if (!theirs.TryGetValue(pair.Key, out theirText)) + { + onlyOurs++; + continue; + } + + // The reference is written by a tool that collapses tabs and newlines to spaces, + // so compare on the same footing rather than reporting whitespace as a defect. + if (Flatten(pair.Value) == theirText) + { + matched++; + } + else + { + differed++; + + if (examples.Count < 10) + { + examples.Add(" #" + pair.Key + + "\n ours: " + Truncate(Flatten(pair.Value)) + + "\n theirs: " + Truncate(theirText)); + } + } + } + + foreach (var key in theirs.Keys) + { + if (!ours.ContainsKey(key)) + onlyTheirs++; + } + + Say(string.Format(" vs UOFiddler: {0:N0} identical, {1:N0} differ, {2:N0} only ours, {3:N0} only theirs", + matched, differed, onlyOurs, onlyTheirs)); + + if (differed == 0 && onlyOurs == 0 && onlyTheirs == 0) + Say(" IDENTICAL — the port reproduces UOFiddler's table exactly"); + + foreach (var example in examples) + Say(example); + + Say(""); + } + + private static string Flatten(string s) + { + return s.Replace('\t', ' ').Replace('\r', ' ').Replace('\n', ' '); + } + + private static string Truncate(string s) + { + return s.Length <= 90 ? s : s.Substring(0, 90) + "…"; + } + + // ── gump: the deliberate crash ─────────────────────────────────────────────────────── + + /// + /// Reproduces §4.1's access violation **from inside a running ServUO**, which is the + /// only place the claim actually matters. `Gumps` is the one decoder that builds its + /// `FileIndex` with hasExtra: true, and FileIndex.cs's own comment says + /// that branch exists for gumpartlegacy.uop. + /// + /// AccessViolationException is a corrupted-state exception and .NET Framework + /// 4.8 does not deliver it to an ordinary catch, so **this takes the shard down** and + /// there is no in-process defence. That is the finding, and the reason "nothing calls + /// Ultima.Gumps" is a safety rule rather than a scoping preference. Never part + /// of "all"; never run on anything but a rig. + /// + private static void SectionGump() + { + Head("gump — DELIBERATE CRASH (§4.1)"); + Say(" This is expected to kill the process. Nothing in Protocol 8 calls Ultima.Gumps."); + Say(" If the shard survives this section, that is itself the finding — record it."); + + Checkpoint("gump", 2); + + try + { + var bmp = Ultima.Gumps.GetGump(2); + Say(" SURVIVED: GetGump(2) returned " + (bmp == null ? "null" : bmp.Width + "x" + bmp.Height)); + + if (bmp != null) + bmp.Dispose(); + } + catch (Exception e) + { + Say(" caught (so it was not a corrupted-state exception): " + e.GetType().Name + ": " + e.Message); + } + + Say(""); + } + + // ── Tally ──────────────────────────────────────────────────────────────────────────── + + /// + /// The four counts that matter, and one of them is the point of the whole probe. + /// + /// RefusedButDecoded is a record the validator rejects and the library returned + /// a picture for anyway. On a stock client that number should be zero. On a patched + /// one it is the population of wrong pictures — the failure this protocol most needs + /// to avoid, because it raises no error anywhere and no success count can see it. + /// + private sealed class Tally + { + public int Ok; // validator passed, decoded + public int Absent; // validator says nothing there, library agreed + public int AbsentButDecoded; // NOTHING is there, and the library returned a picture + public int Refused; // validator refused, library also returned nothing + public int RefusedButDecoded; // validator refused, library returned a picture anyway + public int OkButNothing; // validator passed, library returned nothing + public int Threw; // the library threw + public readonly List Examples = new List(); + + public void Report(string label) + { + Say(string.Format(" {0}:", label)); + Say(string.Format(" ok {0:N0} absent {1:N0} refused {2:N0} threw {3:N0}", Ok, Absent, Refused, Threw)); + Say(string.Format(" validator passed but nothing decoded: {0:N0}", OkButNothing)); + Say(string.Format(" WRONG PICTURES, empty record: {0:N0}", AbsentButDecoded)); + Say(string.Format(" WRONG PICTURES, bad record: {0:N0}", RefusedButDecoded)); + + if (Examples.Count > 0) + { + Say(" examples:"); + + foreach (var example in Examples) + Say(" " + example); + } + + Say(""); + } + } + + private static void Record(Tally tally, BridgeAssetValidator.Verdict verdict, string reason, bool decoded, string thrown, string key) + { + if (thrown != null) + { + tally.Threw++; + + if (tally.Examples.Count < 20) + tally.Examples.Add(key + " THREW " + thrown + (reason == null ? "" : " [validator: " + reason + "]")); + + return; + } + + switch (verdict) + { + case BridgeAssetValidator.Verdict.Ok: + if (decoded) + tally.Ok++; + else + tally.OkButNothing++; + + break; + + case BridgeAssetValidator.Verdict.Absent: + // An empty record that still yields a bitmap is not a disagreement about + // strictness. It is the shared-buffer defect: LoadStatic reuses + // m_StreamBuffer, only ever grows it, and discards stream.Read's return, so + // a zero-length record decodes whatever the PREVIOUS asset left behind. + if (decoded) + { + tally.AbsentButDecoded++; + + if (tally.Examples.Count < 20) + tally.Examples.Add(key + " has no record (" + reason + ") — the library returned a picture"); + } + else + { + tally.Absent++; + } + + break; + + case BridgeAssetValidator.Verdict.Refused: + if (decoded) + { + tally.RefusedButDecoded++; + + if (tally.Examples.Count < 20) + tally.Examples.Add(key + " REFUSED (" + reason + ") — library returned a picture anyway"); + } + else + { + tally.Refused++; + } + + break; + } + } + + // ── Output ─────────────────────────────────────────────────────────────────────────── + + /// + /// Writes the id the probe is **about to** touch, then flushes. + /// + /// Learned the expensive way from the PowerShell probes and it matters more here: an + /// access violation is not catchable and does not unwind, so nothing in this file runs + /// after one. The last line in this file is the only evidence of which id killed the + /// shard. + /// + private static void Checkpoint(string section, int id) + { + try + { + File.WriteAllText(_checkpointPath, section + " " + id + " @ " + + DateTime.UtcNow.ToString("u", CultureInfo.InvariantCulture) + Environment.NewLine); + } + catch + { + // A checkpoint that cannot be written must not stop the sweep. + } + } + + private static void Head(string title) + { + Say("── " + title + " " + new string('─', Math.Max(0, 70 - title.Length))); + } + + private static void Say(string text) + { + Console.WriteLine("[assetprobe] {0}", text); + + var report = _report; + + if (report != null) + { + try + { + report.WriteLine(text); + } + catch + { + // Reporting must never be the thing that fails the run. + } + } + } + + private static void Tell(Mobile to, string text) + { + Console.WriteLine("[assetprobe] {0}", text); + + if (to != null) + to.SendMessage(text); + } + } + + /// + /// **Validate before calling** — the response the org lead chose for §4.2's residual risk, + /// prototyped here so phase 1 adopts it with measurements rather than on faith. + /// + /// The principle: `Ultima`'s decoders take their bounds from the file they are reading, so + /// the extractor must decide whether a record is worth handing over *before* handing it + /// over. Every check below is against the index entry and the record header — cheap, and + /// enough to turn an uncatchable corrupted-state exception into a skipped asset. + /// + /// It cannot be complete and does not claim to be. It closes the shapes that reading the + /// source showed are reachable; the probe's REFUSED-BUT-DECODED count is what says whether + /// the boundary is drawn in the right place. + /// + /// Promoted into the overlay in phase 1. + /// + public static class BridgeAssetValidator + { + public enum Verdict + { + /// Nothing at this id, and the index says so honestly. + Absent, + + /// The entry is self-consistent and inside its file. + Ok, + + /// The entry claims something the file cannot support. Do not decode it. + Refused + } + + /// Land tiles decode a fixed 44×44 diamond: 2 × (2+4+…+44) ushorts. + public const int LandRecordBytes = 2024; + + /// + /// A ceiling on decoded art dimensions. `LoadStatic` allocates + /// new Bitmap(width, height) straight from two ushorts in the record, so a + /// corrupt header asks for up to 65535×65535 — an 8 GB allocation, from a file. Real + /// art is a couple of hundred pixels at most. + /// + public const int MaxArtDimension = 1024; + + /// + /// Builds our own index over the same files, with the same constructor arguments + /// Art uses — including hasExtra: false, which is the whole reason the + /// art path is safe where the gump path is not (§4.1). + /// + public static FileIndex OpenArtIndex() + { + if (ArtDataPath() == null) + return null; + + return new FileIndex("Artidx.mul", "Art.mul", "artLegacyMUL.uop", 0x10000, 4, ".tga", 0x13FDC, false); + } + + /// + /// The file an art index entry's lookup is an offset **into** — which is not + /// art.mul on any current client. + /// + /// This cost a whole probe run to learn and it is the single most important thing + /// phase 1 must not get wrong. FileIndex's UOP constructor ends with a bare + /// MulPath = uopPath: **when artLegacyMUL.uop exists it wins outright**, + /// and art.mul / artidx.mul are never opened at all. A validator that + /// bounds offsets against art.mul while the index holds UOP offsets is not + /// merely approximate, it is nonsense — the first run of this probe refused 34,299 + /// perfectly good statics for "declaring 10533x2085" because it was reading UOP + /// offsets into the wrong file. + /// + /// So the resolution order here mirrors FileIndex's exactly, and anything that + /// needs the bytes behind an entry must ask this rather than assume. + /// + public static string ArtDataPath() + { + var uop = Files.GetFilePath("artlegacymul.uop"); + + if (uop != null) + return uop; + + return Files.GetFilePath("art.mul"); + } + + public static long MulLength(string path) + { + if (path == null) + return 0; + + try + { + return new FileInfo(path).Length; + } + catch + { + return 0; + } + } + + /// + /// Judges one index entry. + /// + /// The check FileIndex.Seek is missing is the last one: it tests + /// Stream.Length < e.lookup — that the record *starts* inside the file — and + /// never that it *ends* inside it. A record that begins two bytes before EOF and + /// declares a length of 4,000 passes, and stream.Read then returns a short count + /// that the decoders discard, leaving the previous asset's bytes in the shared buffer. + /// + public static Verdict CheckEntry(FileIndex index, int at, long mulLength, long verdataLength, out string reason) + { + reason = null; + + if (index == null || index.Index == null || at < 0 || at >= index.Index.Length) + { + reason = "index " + at + " out of range"; + return Verdict.Absent; + } + + Entry3D e = index.Index[at]; + + if (e.lookup < 0) + { + reason = "lookup " + e.lookup; + return Verdict.Absent; + } + + bool patched = (e.length & (1 << 31)) != 0; + int length = e.length & 0x7FFFFFFF; + + if (!patched && e.length < 0) + { + reason = "length " + e.length; + return Verdict.Absent; + } + + if (length == 0) + { + reason = "lookup " + e.lookup + ", length 0"; + return Verdict.Absent; + } + + long ceiling = patched ? verdataLength : mulLength; + + if (ceiling <= 0) + { + reason = (patched ? "verdata.mul" : "the art data file") + " has no length"; + return Verdict.Refused; + } + + if (e.lookup >= ceiling) + { + reason = "lookup " + e.lookup + " past the end of " + + (patched ? "verdata.mul" : "the mul") + " (" + ceiling + ")"; + return Verdict.Refused; + } + + // The missing check. A short read is silent, and its consequence is the PREVIOUS + // asset's picture served under this id. + if (e.lookup + (long)length > ceiling) + { + reason = "record runs " + (e.lookup + (long)length - ceiling) + " bytes past the end of " + + (patched ? "verdata.mul" : "the mul"); + return Verdict.Refused; + } + + return Verdict.Ok; + } + + /// + /// `LoadLand` reads 2,024 bytes regardless of the declared length, so a shorter record + /// reads past the end of a buffer sized from that length. + /// + public static bool LandLengthSane(FileIndex index, int at, out string reason) + { + reason = null; + + if (index == null || index.Index == null || at < 0 || at >= index.Index.Length) + return true; + + int length = index.Index[at].length & 0x7FFFFFFF; + + if (length > 0 && length < LandRecordBytes) + { + reason = "land record is " + length + " bytes; LoadLand always reads " + LandRecordBytes; + return false; + } + + return true; + } + + /// + /// Walks a static record's own row table the way LoadStatic will, and refuses + /// it if that walk would read outside the record. + /// + /// This is the check with teeth. LoadStatic's inner loop guards the write into + /// the bitmap (xOffset > delta, xOffset + xRun > delta) and does + /// nothing at all about the read cursor, which advances until it happens to find a + /// zero pair — potentially far outside a pinned array. Simulating the same walk with + /// a bound is the cheapest way to know whether handing the id over is safe. + /// + public static bool StaticRecordSane(byte[] record, int length, out string reason) + { + reason = null; + + if (length < 8) + { + reason = "record is " + length + " bytes; a static header needs 8"; + return false; + } + + int words = length / 2; + int width = ReadUInt16(record, 4); + int height = ReadUInt16(record, 6); + + // LoadStatic returns null for these rather than misbehaving, so it is not a refusal. + if (width <= 0 || height <= 0) + return true; + + if (width > MaxArtDimension || height > MaxArtDimension) + { + reason = "declares " + width + "x" + height + ", past the " + MaxArtDimension + "px ceiling"; + return false; + } + + // The row-lookup table: height ushorts starting at word 4. + if (4 + height > words) + { + reason = "row table (" + height + " entries) does not fit in a " + length + "-byte record"; + return false; + } + + int start = height + 4; + + for (int y = 0; y < height; y++) + { + int cursor = start + ReadUInt16(record, (4 + y) * 2); + + while (true) + { + // Two ushorts for the run header, and they must both be inside the record. + if (cursor < 0 || cursor + 1 >= words) + { + reason = "row " + y + " reads at word " + cursor + ", past the record's " + words; + return false; + } + + int xOffset = ReadUInt16(record, cursor * 2); + int xRun = ReadUInt16(record, (cursor + 1) * 2); + cursor += 2; + + if (xOffset + xRun == 0) + break; + + // LoadStatic stops the row here, so the read cursor stops with it. + if (xOffset > width || xOffset + xRun > width) + break; + + if (cursor + xRun > words) + { + reason = "row " + y + " declares a " + xRun + "-pixel run running past the record"; + return false; + } + + cursor += xRun; + } + } + + return true; + } + + private static int ReadUInt16(byte[] b, int at) + { + return b[at] | (b[at + 1] << 8); + } + + /// + /// Reads a record's actual bytes so can walk it. + /// + /// Holds its own handles rather than borrowing the library's, because FileIndex + /// hands out the stream it decodes from and moving that stream's position underneath + /// the decoder would be its own bug. Opened FileShare.ReadWrite to match how + /// FileIndex opens the same files. + /// + public sealed class RecordReader : IDisposable + { + private readonly FileStream _mul; + private readonly FileStream _verdata; + private byte[] _scratch = new byte[64 * 1024]; + + public RecordReader(string mulPath, string verdataPath) + { + _mul = Open(mulPath); + _verdata = Open(verdataPath); + } + + private static FileStream Open(string path) + { + if (path == null || !File.Exists(path)) + return null; + + try + { + return new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite); + } + catch + { + return null; + } + } + + /// + /// True when the record at is safe to hand to + /// Art.GetStatic. A record that cannot be read at all is reported sane — + /// has already judged the entry, and this must not + /// invent a second reason to refuse. + /// + public bool StaticSane(FileIndex index, int at, out string reason) + { + reason = null; + + if (index == null || index.Index == null || at < 0 || at >= index.Index.Length) + return true; + + Entry3D e = index.Index[at]; + bool patched = (e.length & (1 << 31)) != 0; + int length = e.length & 0x7FFFFFFF; + + var stream = patched ? _verdata : _mul; + + if (stream == null || length <= 0 || e.lookup < 0) + return true; + + if (_scratch.Length < length) + _scratch = new byte[length]; + + int read; + + try + { + stream.Seek(e.lookup, SeekOrigin.Begin); + read = stream.Read(_scratch, 0, length); + } + catch (Exception ex) + { + reason = "cannot read the record: " + ex.GetType().Name; + return false; + } + + // The short read the decoders discard. Refusing here is the whole point: the + // library would decode whatever the shared buffer happened to hold. + if (read < length) + { + reason = "short read — " + read + " of " + length + " bytes available"; + return false; + } + + return StaticRecordSane(_scratch, length, out reason); + } + + public void Dispose() + { + if (_mul != null) + _mul.Dispose(); + + if (_verdata != null) + _verdata.Dispose(); + } + } + } +} diff --git a/tools/scaffolding/BridgeMythicCliloc.cs b/tools/scaffolding/BridgeMythicCliloc.cs new file mode 100644 index 0000000..c77974d --- /dev/null +++ b/tools/scaffolding/BridgeMythicCliloc.cs @@ -0,0 +1,532 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Text; + +namespace Server.Custom +{ + /// + /// A reader for the **Mythic compressed** cliloc container, in plain .NET Framework 4.8 C#. + /// + /// This is the Asset Bridge's §9 decoder — the ONE decoder Protocol 8 writes rather than + /// calls (docs/link/v8.md §4, §9). ServUO's bundled Ultima.StringList implements only + /// the plain layout and throws Non-negative number required on every modern client's + /// file, which is also why the shard's own VendorSearch.GetItemName is already inert. + /// + /// **Provenance.** Ported from UOFiddler's Ultima/Helpers/MythicDecompress.cs, + /// MoveToFront.cs and StringList.TryParse (polserver/UOFiddler). UOFiddler is + /// released under the **Beerware** licence, so carrying its algorithm into this + /// GPL-3.0-or-later tree is clean — see v8.md §9. + /// + /// **What the port had to change**, and why the differences are not cosmetic: + /// + /// * UOFiddler targets net10.0 and its implementation is written in Span<T>, + /// stackalloc, ArrayPool and BinaryPrimitives. ServUO compiles the + /// overlay against net48 with no package feed, so all of that becomes plain arrays. + /// * Every read of the compressed payload is **bounds-checked here and is not there**. + /// Upstream indexes input[m + 1024] and input[firstVal + 1024] with + /// offsets derived from the file's own frequency header, inside a + /// try { } catch (Exception) { return false; }. That is adequate for a desktop + /// tool and is not adequate for us: this runs inside a live shard, and a corrupt or + /// hostile Cliloc.enu must produce a refusal, not an exception unwinding through the + /// bridge. Every such index is tested before use and returns false instead. + /// + /// Phase 0 uses this from to prove the port reproduces + /// UOFiddler's own output exactly. **Phase 2 promotes this file into + /// overlay/Scripts/Custom/Bridge/** — it lives in scaffolding only for as long as it + /// is a spike. + /// + public static class BridgeMythicCliloc + { + /// The first DWORD of a compressed file is the decompressed length, XORed with this. + private const uint HeaderXorKey = 0x8E2C9A3D; + + /// 256 little-endian int32 symbol frequencies precede the coded payload. + private const int FrequencyHeaderSize = 1024; + + /// One decoded cliloc row. Mirrors Ultima.StringEntry's three fields. + public struct Entry + { + public int Number; + public byte Flag; + public string Text; + + public Entry(int number, byte flag, string text) + { + Number = number; + Flag = flag; + Text = text; + } + } + + // ── Container detection ────────────────────────────────────────────────────────────── + + /// + /// True when the file looks like the Mythic container. The marker is the high byte of + /// the first DWORD being 0x8E — which is not a magic number in the file so much + /// as a consequence of : a plausible decompressed length is + /// small enough that its top byte is zero, so the XOR leaves 0x8E showing. + /// + public static bool LooksCompressed(byte[] buffer) + { + return buffer != null && buffer.Length >= 4 && buffer[3] == 0x8E; + } + + // ── The public entry point ─────────────────────────────────────────────────────────── + + /// + /// Reads a cliloc file, compressed or plain, and returns its entries. + /// + /// Tries the layout the header suggests first and the other one second — the same + /// fallback UOFiddler performs, and the reason an already-converted file passes + /// straight through. is non-null when a layout parsed + /// *partially*: that is the case a caller must surface rather than swallow, because a + /// quietly short table is the failure mode the website's importer refuses. + /// + public static bool TryLoadFile(string path, out List entries, out string warning, out string error) + { + entries = null; + warning = null; + error = null; + + byte[] buffer; + + try + { + buffer = File.ReadAllBytes(path); + } + catch (Exception e) + { + error = "cannot read " + path + ": " + e.Message; + return false; + } + + return TryLoad(buffer, out entries, out warning, out error); + } + + /// Reads an in-memory cliloc file. See . + public static bool TryLoad(byte[] buffer, out List entries, out string warning, out string error) + { + entries = null; + warning = null; + error = null; + + bool compressedFirst = LooksCompressed(buffer); + + List primary; + string primaryError; + bool primaryComplete; + + if (TryParse(buffer, compressedFirst, out primary, out primaryComplete, out primaryError) && primaryComplete) + { + entries = primary; + return true; + } + + List fallback; + string fallbackError; + bool fallbackComplete; + + if (TryParse(buffer, !compressedFirst, out fallback, out fallbackComplete, out fallbackError) && fallbackComplete) + { + entries = fallback; + return true; + } + + // Neither layout parsed to the end. Take whichever salvaged more rows and say so. + int primaryCount = primary == null ? 0 : primary.Count; + int fallbackCount = fallback == null ? 0 : fallback.Count; + + if (primaryCount == 0 && fallbackCount == 0) + { + error = "as " + Label(compressedFirst) + ": " + primaryError + + "; as " + Label(!compressedFirst) + ": " + fallbackError; + return false; + } + + if (primaryCount >= fallbackCount) + { + entries = primary; + warning = "parsed partially as " + Label(compressedFirst) + ": " + primaryError + + " (" + primaryCount + " entries salvaged)"; + } + else + { + entries = fallback; + warning = "parsed partially as " + Label(!compressedFirst) + ": " + fallbackError + + " (" + fallbackCount + " entries salvaged)"; + } + + return true; + } + + private static string Label(bool compressed) + { + return compressed ? "compressed" : "uncompressed"; + } + + // ── Record layout ──────────────────────────────────────────────────────────────────── + + /// + /// Walks the plain record layout: a 4-byte and a 2-byte header, then repeating + /// [int32 number][byte flag][uint16 length][length bytes of UTF-8]. + /// + /// distinguishes "parsed to the end of the file" from + /// "stopped early but salvaged rows", which is the distinction the caller needs and + /// an exception would destroy. + /// + private static bool TryParse(byte[] buffer, bool decompress, out List entries, out bool complete, out string error) + { + entries = new List(); + complete = false; + error = null; + + byte[] data; + + if (decompress) + { + if (!TryDecompress(buffer, out data, out error)) + return false; + } + else + { + data = buffer; + } + + if (data.Length < 6) + { + error = "file is " + data.Length + " bytes, smaller than the 6-byte header"; + return false; + } + + int cursor = 6; // int32 version marker + int16 language marker + int lastNumber = -1; + + while (cursor < data.Length) + { + int entryStart = cursor; + int remaining = data.Length - cursor; + + if (remaining < 7) + { + error = "unexpected " + remaining + " trailing byte(s) at 0x" + entryStart.ToString("X") + + " after entry #" + lastNumber + "; an entry header needs 7"; + return true; + } + + int number = ReadInt32(data, cursor); + byte flag = data[cursor + 4]; + // Deliberately UNSIGNED. Read as Int16, a string of 32768 bytes or more comes back + // negative and corrupts every record after it. + int length = data[cursor + 5] | (data[cursor + 6] << 8); + cursor += 7; + + if (length > data.Length - cursor) + { + error = "entry #" + number + " at 0x" + entryStart.ToString("X") + " declares length " + + length + " but only " + (data.Length - cursor) + " byte(s) remain (parsed " + + entries.Count + " so far)"; + return true; + } + + string text; + + try + { + text = Encoding.UTF8.GetString(data, cursor, length); + } + catch (Exception e) + { + error = "entry #" + number + " at 0x" + entryStart.ToString("X") + " has " + length + + " body bytes that are not valid UTF-8: " + e.Message; + return true; + } + + cursor += length; + + entries.Add(new Entry(number, flag, text)); + lastNumber = number; + } + + complete = true; + return true; + } + + // ── Mythic stage 1: the XOR header and the move-to-front code ──────────────────────── + + /// + /// Reads the obfuscated decompressed length from the first DWORD. Public so a caller + /// can size a buffer before committing to the decode. + /// + public static uint PeekDecompressedLength(byte[] source) + { + if (source == null || source.Length < 4) + return 0; + + return ReadUInt32(source, 0) ^ HeaderXorKey; + } + + /// + /// Decompresses the Mythic container: strip the 4-byte length header, undo the + /// move-to-front coding, then run stage 2. + /// + public static bool TryDecompress(byte[] source, out byte[] output, out string error) + { + output = null; + error = null; + + if (source == null || source.Length < 4) + { + error = "payload shorter than the 4-byte length header"; + return false; + } + + uint dataLength = ReadUInt32(source, 0) ^ HeaderXorKey; + + // A wrong guess about the container makes this astronomically large, which is the + // cheapest possible rejection and must happen before any allocation. + if (dataLength == 0 || dataLength > int.MaxValue) + { + error = "implausible decompressed length " + dataLength + " — not the compressed layout"; + return false; + } + + var mtf = new byte[source.Length - 4]; + MoveToFrontDecode(source, 4, mtf); + + var destination = new byte[(int)dataLength]; + int written; + + if (!TryInternalDecompress(mtf, destination, out written, out error)) + return false; + + if (written != (int)dataLength) + { + error = "decompressed " + written + " bytes, header declared " + dataLength; + return false; + } + + output = destination; + return true; + } + + /// + /// Move-to-front decode. Each input byte is an index into a 256-symbol table; the + /// symbol found there is emitted and moved to the front. + /// + private static void MoveToFrontDecode(byte[] input, int offset, byte[] output) + { + var symbols = new byte[256]; + + for (int i = 0; i < 256; i++) + symbols[i] = (byte)i; + + for (int i = 0; i < output.Length; i++) + { + int index = input[offset + i]; + byte symbol = symbols[index]; + output[i] = symbol; + + for (int j = index; j > 0; j--) + symbols[j] = symbols[j - 1]; + + symbols[0] = symbol; + } + } + + // ── Mythic stage 2 ─────────────────────────────────────────────────────────────────── + + /// + /// Turns the MTF-decoded payload back into the original bytes. + /// + /// The payload is a 1024-byte frequency header (256 little-endian int32 symbol counts) + /// followed by the coded stream. The counts partition the stream into one run per + /// symbol; cursor[] holds each run's read position and limit[] its end, + /// and the walk emits a symbol, advances that symbol's run, and re-orders the symbol + /// table by the index it reads. + /// + /// Every index derived from file content is checked. Upstream's equivalent is wrapped + /// in a blanket catch; here a malformed file is a false with a reason. + /// + private static bool TryInternalDecompress(byte[] input, byte[] destination, out int written, out string error) + { + written = 0; + error = null; + + if (input.Length < FrequencyHeaderSize) + { + error = "payload (" + input.Length + " bytes) is smaller than the 1024-byte frequency header"; + return false; + } + + var counts = new int[256]; // symbol → number of occurrences + var cursor = new int[256]; // symbol → next unread position in its run + var limit = new int[256]; // symbol → one past the end of its run + + int sum = 0; + + for (int i = 0; i < 256; i++) + { + counts[i] = ReadInt32(input, i * 4); + + if (counts[i] < 0) + { + error = "frequency header declares a negative count for symbol " + i; + return false; + } + + sum += counts[i]; + + if (sum < 0) + { + error = "frequency header sums past int range at symbol " + i; + return false; + } + } + + if (sum == 0) + { + written = 0; + return true; + } + + if (destination.Length < sum) + { + error = "destination holds " + destination.Length + " bytes, payload needs " + sum; + return false; + } + + int nonZeroCount = 0; + + for (int i = 0; i < 256; i++) + { + if (counts[i] != 0) + nonZeroCount++; + } + + // The coded stream must be long enough to hold one index per emitted byte. + if (input.Length - FrequencyHeaderSize < sum) + { + error = "coded stream holds " + (input.Length - FrequencyHeaderSize) + " bytes, frequency header claims " + sum; + return false; + } + + var order = new byte[256]; + FrequencyOrder(counts, order); + + var symbolTable = new byte[256]; + + for (int i = 0; i < 256; i++) + symbolTable[i] = (byte)i; + + for (int i = 0, m = 0; i < nonZeroCount; ++i) + { + byte symbol = order[i]; + + // m indexes the coded stream and comes from the file's own counts. + if (m < 0 || m >= input.Length - FrequencyHeaderSize) + { + error = "run table for symbol " + symbol + " starts at " + m + ", past the coded stream"; + return false; + } + + symbolTable[input[m + FrequencyHeaderSize]] = symbol; + cursor[symbol] = m + 1; + m += counts[symbol]; + limit[symbol] = m; + } + + byte val = symbolTable[0]; + int count = 0; + int liveSymbols = nonZeroCount; + + do + { + destination[count] = val; + + if (cursor[val] < limit[val]) + { + int at = cursor[val] + FrequencyHeaderSize; + + if (at < FrequencyHeaderSize || at >= input.Length) + { + error = "run for symbol " + val + " reads at " + at + ", past the " + input.Length + "-byte payload"; + return false; + } + + byte index = input[at]; + cursor[val]++; + + if (index != 0) + { + ShiftLeft(symbolTable, index); + symbolTable[index] = val; + val = symbolTable[0]; + } + } + else if (liveSymbols-- > 0) + { + ShiftLeft(symbolTable, liveSymbols); + val = symbolTable[0]; + } + + count++; + } + while (count < sum); + + written = sum; + return true; + } + + /// + /// Orders symbols by descending frequency: repeatedly take the largest remaining count + /// and record its symbol. Ties go to the lower symbol, because the scan keeps the first + /// strictly-greater value — matching upstream, and the tie-break is load-bearing. + /// + private static void FrequencyOrder(int[] counts, byte[] output) + { + var tmp = new int[256]; + Array.Copy(counts, tmp, 256); + + for (int i = 0; i < 256; i++) + { + int best = 0; + byte index = 0; + + for (int j = 0; j < 256; j++) + { + if (tmp[j] > best) + { + index = (byte)j; + best = tmp[j]; + } + } + + if (best == 0) + break; + + output[i] = index; + tmp[index] = 0; + } + } + + /// Shifts [1..element] down one slot, dropping element 0. + private static void ShiftLeft(byte[] input, int element) + { + for (int i = 0; i < element; ++i) + input[i] = input[i + 1]; + } + + // ── Little-endian readers (BinaryPrimitives is not available on net48) ─────────────── + + private static int ReadInt32(byte[] b, int at) + { + return b[at] | (b[at + 1] << 8) | (b[at + 2] << 16) | (b[at + 3] << 24); + } + + private static uint ReadUInt32(byte[] b, int at) + { + return (uint)(b[at] | (b[at + 1] << 8) | (b[at + 2] << 16) | (b[at + 3] << 24)); + } + } +} diff --git a/tools/scaffolding/BridgeRigDriver.cs b/tools/scaffolding/BridgeRigDriver.cs index 8bb61c1..116efc1 100644 --- a/tools/scaffolding/BridgeRigDriver.cs +++ b/tools/scaffolding/BridgeRigDriver.cs @@ -155,6 +155,14 @@ namespace Server.Custom case "partprobe": BridgeParticipationProbe.Run(null, Arg(parts, 1), Int(Arg(parts, 2)), Int(Arg(parts, 3))); break; + // Asset Bridge phase 0. Here for the same reason as partprobe, and for one more: + // the point of that spike is comparing the STOCK client's answers with a patched + // client's, and `AssetProbeOnStart` can only ever run whichever one the config + // names. Driving it from here runs both against a single boot, so a difference + // between them cannot be a difference between two shard processes. + case "assetprobe": + BridgeAssetProbe.Begin(null, Arg(parts, 1) ?? "all", Arg(parts, 2)); + break; // Phase 12a. `world.despawn` answering `gone` rather than `removed` is the // path a player takes every time they kill an event creature, and it is the one // outcome the rig cannot reach by asking the bridge: every bridge verb that diff --git a/tools/scaffolding/README.md b/tools/scaffolding/README.md index ed1d281..95a79cc 100644 --- a/tools/scaffolding/README.md +++ b/tools/scaffolding/README.md @@ -14,10 +14,12 @@ These two scripts produced the measured budget in [PLAN.md](https://gitea.whitlo | `BridgeCrierProbe.cs` | `Scripts/Custom/BridgeCrierProbe.cs` | Logs the global town-crier entry list every 3s so `towncrier.add` / `remove` can be seen landing in game state. Flag: `CrierProbeOnStart`. | | `BridgeVendorSaleProbe.cs` | `Scripts/Custom/BridgeVendorSaleProbe.cs` | Fires `PlayerVendorSale` (Phase 7) with real seeded-vendor data so `vendor.sale` can be verified without a live buy. Requires the Phase 7 patches applied. Flag: `VendorSaleProbeOnStart`. | | `BridgeDemoDress.cs` | `Scripts/Custom/BridgeDemoDress.cs` | Renames a seeded world so it is presentable in a screenshot: shop signs, vendor and character names, house signs. Also stages a few condemned houses back into IDOC, and sets a known password on `seed_000` so a character can be logged in. Flags: `DemoDressOnStart`, `DemoDressPassword`. In game: `[demodress`. | -| `BridgeRigDriver.cs` | `Scripts/Custom/BridgeRigDriver.cs` | Drives the shard from OUTSIDE the game, one verb per line in `Config/rigcmd.txt`, which the driver polls and truncates. Written for the engagement Phase 11b acceptance walk, where each step's assertion is what happened BETWEEN two steps, so the steps have to be separated by the observer rather than by a hard-coded delay -- and ServUO's console takes a fixed verb set (`Scripts/Misc/ConsoleCommands.cs`), so `[p5probe` cannot be typed at a headless shard at all. Verbs: `decaylist`, `decay`, `vendorlist`, `vendorfunds`, `citylist`, `governor`, `election`, `activate`, `password`, `configset`, `configread`, `partprobe`, `save`, `shutdown`. Flag: `RigDriverEnabled`. `configset` exists because **`Config.Get` is written by exactly ONE caller in the whole of ServUO 57.4** (`Server/ScriptCompiler.cs`): no in-game command, gump or console verb writes a config key, so on a stock shard a GM cannot drift a configuration lease even deliberately, and a lease's compare-and-set restore would have no way to be proved. `configread` reads a key back through `Config.Get` long after every type initialiser has run, which is how a key that TOOK is told from one that only appeared to. **Sets passwords, writes live config and mutates the world.** | +| `BridgeRigDriver.cs` | `Scripts/Custom/BridgeRigDriver.cs` | Drives the shard from OUTSIDE the game, one verb per line in `Config/rigcmd.txt`, which the driver polls and truncates. Written for the engagement Phase 11b acceptance walk, where each step's assertion is what happened BETWEEN two steps, so the steps have to be separated by the observer rather than by a hard-coded delay -- and ServUO's console takes a fixed verb set (`Scripts/Misc/ConsoleCommands.cs`), so `[p5probe` cannot be typed at a headless shard at all. Verbs: `decaylist`, `decay`, `vendorlist`, `vendorfunds`, `citylist`, `governor`, `election`, `activate`, `password`, `configset`, `configread`, `partprobe`, `assetprobe`, `save`, `shutdown`. Flag: `RigDriverEnabled`. `configset` exists because **`Config.Get` is written by exactly ONE caller in the whole of ServUO 57.4** (`Server/ScriptCompiler.cs`): no in-game command, gump or console verb writes a config key, so on a stock shard a GM cannot drift a configuration lease even deliberately, and a lease's compare-and-set restore would have no way to be proved. `configread` reads a key back through `Config.Get` long after every type initialiser has run, which is how a key that TOOK is told from one that only appeared to. **Sets passwords, writes live config and mutates the world.** | | `BridgeProtocol5Probe.cs` | `Scripts/Custom/BridgeProtocol5Probe.cs` | Drives all three Protocol 5 enrichments so their frames can be observed: walks one house Fairly -> Greatly -> IDOC (the PAIR is the assertion -- `estimatedCollapse` must appear only on the IDOC frame), reports each player vendor's fee state straight off the `PlayerVendor` so the emitted `fees` block can be checked against the shard's own numbers, and fires `EventSink.AccountLogin`. Flags: `Protocol5ProbeOnStart`, `Protocol5ProbeAccount`, `Protocol5ProbePassword`. In game: `[p5probe`. **Sets a password on the named account.** | | `BridgeProtocol6Probe.cs` | `Scripts/Custom/BridgeProtocol6Probe.cs` | Spawns a real champion boss through the shard's own `SpawnChampion()`, waits two champ sweeps so the boss is attributed to its altar, registers unequal damage from two seeded players and kills it -- so `champ.boss.killed` can be observed with a real damage table. **The wait is the assertion**: without it the kill still emits, but with no `serial`/`type`/`level`, which is the documented fallback rather than the case being tested. The altar is placed inside a NAMED region on purpose (see below). Flag: `Protocol6ProbeOnStart`. In game: `[p6probe`. **Spawns and kills a champion boss; rig only.** Protocol 6's other half, the idempotency key, needs no probe -- it is driven from outside with two identical POSTs to the sidecar. | | `BridgeParticipationProbe.cs` | `Scripts/Custom/BridgeParticipationProbe.cs` | Produces real kill credit inside a participation area with no game client: moves two player mobiles to the venue, spawns a creature there, damages it unequally from both and kills it. **Presence is the half it cannot drive** -- the sweep credits players with a live `NetState`, which is the correct test and not one a probe should loosen, so presence accrual needs a real login. In game: `[partprobe `; from a headless rig, through `BridgeRigDriver`'s `partprobe` verb (the two ship together for that reason). **Moves players and spawns and kills a creature; rig only.** | +| `BridgeAssetProbe.cs` | `Scripts/Custom/BridgeAssetProbe.cs` | **Asset Bridge phase 0** (docs/link/v8.md §16). Drives ServUO's vendored `Ultima` decoders from inside a running shard against a deliberately patched client, and compares every answer with what a pre-flight validator says about the index entry *before* the call. The interesting column is not the error count, it is **WRONG PICTURES** -- records the validator rejects and the library renders anyway. Sweeps statics, land, all 2,048 bodies, the player-character bodies from `Race.AllRaces`, and the ported Mythic cliloc reader against UOFiddler's own output. In game / from `BridgeRigDriver`: `[assetprobe [section] [stock|patched]`. Flag: `AssetProbeOnStart`. **Its `gump` section deliberately kills the shard** and is never part of `all`. | +| `BridgeMythicCliloc.cs` | `Scripts/Custom/BridgeMythicCliloc.cs` | The §9 reader for the **Mythic compressed** cliloc container -- the one decoder Protocol 8 writes rather than calls. Ported from UOFiddler (Beerware) into net48 C# with every file-derived index bounds-checked, which upstream's blanket `catch` does not do. Reproduces UOFiddler's 123,490-entry table exactly. **Phase 2 promotes this file into `overlay/`**; it is scaffolding only for as long as it is a spike. | ## Deploy overwrites Bridge.cfg @@ -189,3 +191,127 @@ value that lies, printed next to a frame that disagrees with it. Emitting from a named region is therefore the test. At a dungeon altar the field is legitimately absent and the probe proves nothing about it. + +## What phase 0 found + +`BridgeAssetProbe` exists because [v8.md](https://gitea.whitlocktech.com/RunicGateway/docs/src/branch/main/link/v8.md) §4 chose to **call** ServUO's vendored `Ultima` rather than reimplement it, and the evidence for that choice was a PowerShell probe against a stock client — neither the process nor the client the extractor will actually run in. These are its results, from inside a running ServUO 57.4 against this machine's client, and against a copy broken in 21 catalogued ways by `tools/patch_client.ps1`. + +### The UOP wins outright, and it took a whole run to notice + +`FileIndex`'s UOP constructor ends with a bare `MulPath = uopPath`. **When `artLegacyMUL.uop` is present it wins, and `art.mul` / `artidx.mul` are never opened at all.** Every current client ships the UOP, so: + +- A validator that bounds an index offset against `art.mul` while the index holds UOP offsets is not approximate, it is nonsense. The first run of this probe refused **34,299 perfectly good statics** for "declaring 10533x2085" — and every one of those refusals looked like a real finding. `BridgeAssetValidator.ArtDataPath()` now mirrors `FileIndex`'s own resolution order, and phase 1 must too. +- A custom-art shard that adds graphics to `art.mul` while the UOP is still in place **gets nothing**, silently. That is an operator trap rather than a bug in this protocol, but the extractor is where it will be noticed. +- The `corrupt` and `customart` tiers of `patch_client.ps1` therefore need its `nouop` tier to mean anything at all. Without it they report that they applied, and change nothing. + +### 22,102 wrong pictures on a stock, unmodified client + +The counts that matter, `assetprobe all stock`: + +``` +statics 0..65535 ok 39,189 WRONG PICTURES (empty record) 9,962 threw 16,385 +land 0..16383 ok 4,244 WRONG PICTURES (empty record) 12,140 +``` + +Those 22,102 ids have an index entry of `lookup 0, length 0` — **no record at all**. `FileIndex.Seek` treats that as a hit (it rejects `lookup < 0` and `length < 0`, and zero is neither), hands back the stream, and `LoadStatic` decodes `length` = 0 bytes into `m_StreamBuffer` — which is **reused, only ever grown, and filled by a `stream.Read` whose return value is discarded**. So the id renders whatever the previously-decoded asset left in the buffer. + +**It is specific to the UOP path.** Run the same sweep against the mul path and those ids come back empty and honest, because `artidx.mul` stores `-1` for an absent record while unmapped UOP slots are simply zeroed structs. That is also why the earlier PowerShell probe counted 32,766 of these as "ok": they decode, they raise nothing, and no success count can tell them from art. + +A bulk import that trusted the library would have written 22,102 duplicate images into the site under ids that have no art. This one measurement is the argument for validate-before-calling. + +### Every deliberate defect was caught by the validator and rendered by the library + +`assetprobe all patched`, against the 21-defect client: + +``` +statics ok 39,190 absent 9,954 refused 1 WRONG PICTURES (bad record) 6 threw 16,385 +land ok 4,243 absent 12,140 WRONG PICTURES (bad record) 1 +``` + +| id | the defect | what the library did | +|---|---|---| +| `static/4104` | lookup 4 KB past the end of `art.mul` | returns nothing — `Seek` does check the record's **start** | +| `static/4105` | starts 64 bytes before EOF, declares 8,192 | **renders the previous asset** — `Seek` never checks the record's **end** | +| `static/4108` | declared length 4, smaller than the header | renders something | +| `static/4109` | header declares 8000x8000 | **allocates it** — a ~128 MB bitmap from two bytes in a file, and the same field can ask for 65535×65535 | +| `static/4111` | row table points 60,000 words outside a 512-byte record | renders — `LoadStatic`'s two guards bound the *write* into the bitmap, and nothing bounds the *read* | +| `static/4112` | a 16-pixel run declared in a 20-byte record | renders | +| `static/4131` | verdata entry whose lookup is past verdata.mul's own end | renders — **`Verdata.Seek` has no bounds check whatsoever** | +| `land/256` | 512-byte land record | renders — `LoadLand` reads a fixed 2,024 bytes whatever the length says | + +Seven of the eight produce a confident, wrong picture and raise nothing anywhere. + +The validator refused all eight, and refused **nothing** on the stock client across 49,151 statics and 16,384 land tiles. That second number is the one that matters: a checker that refuses real art is worse than no checker, so "zero false refusals on a clean client" is what makes validate-before-calling more than a hopeful phrase. + +The eight `customart` ids appended past the stock ceiling all decode cleanly, which is that tier's whole point — the ceiling is a property of a file, not a constant anyone should write down. + +### Two more ways to get a wrong answer out of an id that has no art + +- **`Art.GetStatic(id, false)` throws `IndexOutOfRangeException` for `id >= 49,152`** rather than returning null — 16,385 of them in a full sweep. +- **`Art.GetStatic(id)` with the default `checkmaxid: true` is worse**: `GetLegalItemID` maps an out-of-range id to **0**, so the call returns **item 0's picture**. An exception is recoverable; a picture of the wrong item is not even detectable. + +So the extractor takes its id ceiling from the index it opened, and passes `checkmaxid: false` so an overrun is loud rather than plausible. + +### The gump crash reproduces in-process, and nothing catches it + +`assetprobe gump` called `Ultima.Gumps.GetGump(2)` once. **The ServUO process disappeared** — no exception line in the report, no `catch` reached, no shutdown, nothing in the console. The report ends mid-section, and `checkpoint.txt` reading `gump 2` is the entire record of what happened. That is exactly why the checkpoint is written *before* the call and flushed. + +`AccessViolationException` is a corrupted-state exception and .NET Framework 4.8 does not deliver it to ordinary handlers, so **there is no in-process defence** — on a live shard this is a crash with players on it. "Nothing calls `Ultima.Gumps`" is a safety rule, and phase 0's job was to make sure that sentence had been earned rather than assumed. It has. + +### The cliloc port is byte-identical to UOFiddler + +``` +123,490 entries in 218 ms (55,986 blank, 67,504 would be stored) +vs UOFiddler: 123,490 identical, 0 differ, 0 only ours, 0 only theirs +``` + +§9 is proven: the shard can produce the whole table with no UOFiddler installed, no `dotnet build`, and no 5 MB file copied to a server. + +The reference is what makes this a test rather than a demonstration. A subtly wrong inverse-BWT coder still produces a plausible table — mostly-right strings with a few mangled ones is the *expected* shape of a bug in this algorithm, and a row count alone would sail past it. + +Note the blank count is **55,986**, not the 55,994 recorded from the manual pipeline. The difference is eight whitespace-only entries, blank to a `trim()` and not to `IsNullOrEmpty` — a definition rather than a defect, but exactly the sort of eight-row drift that gets investigated as one. + +### What phase 0 did not cover, and phase 1 must + +**The animation path has no validator.** The patched client's verdata entry for body 34 points past verdata.mul's end and the wolf still "decoded" — counted among the 1,144 successes, silently rendering something else, with nothing in the report to say so. `GetAnimation` also allocates `new int[frameCount]` straight from a file-supplied int. Everything above about statics applies here and none of it is implemented yet. + +The deliberate `Bodyconv.def` mis-mappings (bodies 1900 and 1901) produced **nothing** rather than a wrong creature on this client, so they did not reproduce the spider. The gargoyle rows remain the real evidence for the never-sweep-file-types rule: 666, 667, 694 and 695 report nothing, and nothing is the correct answer. + +### Reference: the rest of the run + +``` +bodies 0..2047, direction 1 decoded 1,144 empty 904 faulted 0 + by file type: 1=1222, 2=140, 3=244, 4=150, 5=292 + +player bodies (Race.AllRaces, direction 0) 6 decoded, 6 absent, of 12 + Human 400 / 401 decode; ghosts 402 / 403 absent + Elf 605 / 606 / 607 / 608 all decode + Gargoyle 666 / 667 / 694 / 695 all absent +``` + +Two details worth keeping. The body counts reproduce the PowerShell probe **exactly**, from a different process against the same files, which is what makes the two runs comparable at all. And the gargoyle *ghost* bodies resolve to file type **1**, not 5 like the living gargoyle bodies — so "the gargoyle is an anim5 problem" is not quite the shape of it. + +## Building the patched client + +```powershell +.\tools\patch_client.ps1 -Dest D:\uo-patched-client +``` + +Copies a client (~3.5 GB) and breaks the copy in five catalogued tiers — `nouop`, `verdata`, `customart`, `corrupt`, `bodyconv`. **It never writes to the source**: every file it touches is hashed in the source before and after, and a changed hash aborts the run. Each defect is recorded in `patched-client.manifest.json` beside the copy, which is what makes a nonzero WRONG PICTURES count readable as "the tier worked" instead of "something broke". + +Then point the shard at it and drive the probe: + +```ini +RigDriverEnabled=true +AssetProbeClient=D:\uo-patched-client +AssetProbeClilocRef= +``` + +``` +assetprobe all stock # the baseline: the validator must refuse nothing here +assetprobe all patched # the experiment +``` + +Run both against **one boot**, through `rigcmd.txt`, so a difference between them cannot be a difference between two shard processes. Without `AssetProbeClilocRef` the cliloc section reports a row count, which proves nothing about the strings. + +**The copy is EA's client art.** It stays on the machine that made it, exactly like every other extraction in this project, and is never committed. -- 2.49.1 From c89e818dbf28b92701fe46a21776ee04272d4b47 Mon Sep 17 00:00:00 2001 From: wtclaude Date: Thu, 10 Sep 2026 08:32:18 -0500 Subject: [PATCH 02/11] feat(asset-bridge): the transport, and the 357 wrong pictures it found MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Asset Bridge phase 1 (docs/link/v8.md §16). Sidecar half: RunicGateway/link#41. Docs half: RunicGateway/docs#236. The transport for protocol 8, plus phase 0's validator promoted into the overlay and extended to animations — which is where the interesting part is. ## 357 of the 1,144 "decodable" bodies are wrong pictures, on a STOCK client Phase 0 measured the art path and left the animation half unbuilt. It has the same defect, and it is worse: `GetAnimation` decodes through `new MemoryStream(m_StreamBuffer, false)` — the whole shared buffer, not the `length` bytes just read into it — so a truncated or absent record does not even hit end-of-stream. It sails on into the previous animation's bytes. Measured directly, because no count could tell: | Decode body 320 (`lookup 22638982, length 0`) straight after… | Comes back | |---|---| | body 12, the dragon | the dragon, 176x167, identical hash | | body 34, the wolf | the wolf's dimensions, 35x34 | | body 400, the human male | the human, 27x63, identical hash | The catalogue is **787 bodies, not 1,144**. Importing the other 357 would have written duplicate creature portraits into the site showing whichever body the walk decoded before them. The record walk refused **0** real bodies on the stock client — the false-refusal measurement §4.5 says the boundary depends on. ## And four of the twelve player bodies, not six §5.2 listed the elf ghosts (607, 608) as decoding. Their index entry is `length 0`; what came back was the elf female at her exact dimensions, because 606 is what the walk decoded immediately before. Confirmed the same way — 607 after the dragon is the dragon. Phase 4's UOP decoder now covers eight ids rather than six. ## What is here - **`overlay/Scripts/Custom/Bridge/BridgeAssets.cs`** — the plane. Accepts on the Core thread, hands off to a dedicated asset worker, returns immediately. Three rules, all answering a specific failure: - **one slot**, second request answered `bridge.busy` (425). `Emit`'s queue is bounded in *lines*, so 10,000 queued 200 KB replies is 2 GB of shard memory; the bound that holds is flow control, on the side where the memory is. - **byte budgets** (`AssetBatchBytes`, 512 KiB) under the sidecar's new 1 MiB cap. The factor of two is load-bearing: a page always admits its first item, so it may overshoot by one, and the headroom is what makes that land on the wire. - **replies, never events** — no `reqId`, no answer. An uncorrelated frame is an event by definition, and §3.1 is why none of this may be one. - **`PageBuilder`** — one paging envelope (`more`/`cursor`/`cut`) for all five families that will page, defined before the first one needs it. `cut` matters: "short page" has three meanings and only `end` means finished. - **`assets.sources`** — stage 1 of the import gate, its first user. - **`BridgeAssetValidator.cs`** — promoted from `tools/`, plus `ResolveAnimation` (the never-sweep-file-types rule as code, with no loop and no fallback), `AnimationRecordSane` and the frame walk. - **`EXTRACTOR_VERSION`**, **`overlay.toml` protocol 7 → 8**, `AssetsEnabled`. ## Hashing had to come off the request path §6's gate is (size, mtime) first, hash only when those differ. The first call has nothing cached, so that still means hashing 1.06 GB — inside the sidecar's 10 s reply timeout it does not fit. So hashes are computed on their own thread (deliberately not the single-slot worker, which would answer every status poll `bridge.busy` for the whole pass) and the reply carries `hashing`/`complete`. Measured on the real rig: first call instant with `sha256: null`, second call **44 ms** with every hash present. ## Verified on the wire, not just compiled Real ServUO 57.4 + the real sidecar + the real client. `GET /assets/sources` → 200, `X-UOLink-Version: 8`, `imaging: {ok: true}`, and §4.6's diagnostic firing on a live client: `artDataFile: artlegacymul.uop`, with `art.mul` and `artidx.mul` both carrying `shadowedBy`. Live events kept flowing through the new capped reader with no warnings. Not exercised live: the disabled-plane 403 and the busy 425 (both unit-tested on the sidecar side; the shard halves are a config read and a lock). - [x] AI-assisted — Claude Code (Opus 5) Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4 --- overlay.toml | 5 +- overlay/Config/Bridge.cfg | 12 + .../Custom/Bridge/BridgeAssetValidator.cs | 751 ++++++++++++++++++ overlay/Scripts/Custom/Bridge/BridgeAssets.cs | 747 +++++++++++++++++ overlay/Scripts/Custom/Bridge/BridgeBoot.cs | 1 + overlay/Scripts/Custom/Bridge/BridgeConfig.cs | 24 + tools/scaffolding/BridgeAssetProbe.cs | 571 ++++--------- 7 files changed, 1683 insertions(+), 428 deletions(-) create mode 100644 overlay/Scripts/Custom/Bridge/BridgeAssetValidator.cs create mode 100644 overlay/Scripts/Custom/Bridge/BridgeAssets.cs diff --git a/overlay.toml b/overlay.toml index 7bc1fcc..5643385 100644 --- a/overlay.toml +++ b/overlay.toml @@ -23,8 +23,9 @@ # manual duty: when the protocol changes, bump it here in the same PR that # changes the emitters, exactly as link bumps PROTOCOL_VERSION. # -# Current: 6 — see docs/link/v6.md (idempotency keys on inbound commands, champ.boss.killed). -protocol = 7 +# Current: 8 — see docs/link/v8.md (the Asset Bridge: client assets over the loopback link +# instead of a converter on somebody's desktop). +protocol = 8 # ── ServUO compatibility ───────────────────────────────────────────────────── # diff --git a/overlay/Config/Bridge.cfg b/overlay/Config/Bridge.cfg index 5da1626..5be83c3 100644 --- a/overlay/Config/Bridge.cfg +++ b/overlay/Config/Bridge.cfg @@ -295,6 +295,18 @@ EventsMaxGrantStack=1000 # would land at a moment nobody chose. Set to 0 to allow a save at any time. EventsMinSaveIntervalSec=300 +# The asset plane (docs/link/v8.md, protocol 8). Its own switch, deliberately: turning +# this on is consenting to the website reading this host's UO CLIENT FILES -- art, +# animations, the string table -- over the link. Nothing on this plane writes anything. +AssetsEnabled=true + +# The largest reply the asset plane will build, in encoded bytes. Not an item count: +# the ceiling it lives inside is the sidecar's 1 MiB inbound line cap, and base64 adds +# 33% to every payload. Clamped to [64 KiB, 512 KiB] -- half the wire cap, so that a +# single oversized item (always admitted, or its family could never make progress) +# still fits. +AssetBatchBytes=524288 + # The test scaffolding in tools/scaffolding/ reads its own flags from this file # (SeedOnStart, CensusOnStart, ProbeOnStart). They are absent here on purpose: # Config.Get returns the default of false when a key is missing, so a deployed diff --git a/overlay/Scripts/Custom/Bridge/BridgeAssetValidator.cs b/overlay/Scripts/Custom/Bridge/BridgeAssetValidator.cs new file mode 100644 index 0000000..e4ed921 --- /dev/null +++ b/overlay/Scripts/Custom/Bridge/BridgeAssetValidator.cs @@ -0,0 +1,751 @@ +using System; +using System.IO; + +using Ultima; + +namespace Server.Custom.Bridge +{ + /// + /// **Validate before calling** (docs/link/v8.md §4.5) — the boundary between this protocol + /// and ServUO's vendored Ultima decoders. Phase 0 prototyped it in + /// tools/scaffolding/BridgeAssetProbe.cs and measured it both ways; phase 1 promoted + /// it here, into the overlay, and extended it to animations. + /// + /// The principle: `Ultima`'s decoders take their bounds from the file they are reading, so + /// the extractor must decide whether a record is worth handing over *before* handing it + /// over. Every check below is against the index entry and the record header — cheap, and + /// enough to turn an uncatchable corrupted-state exception into a skipped asset. + /// + /// **The failure this exists for is a wrong picture, not a crash.** `LoadStatic`, + /// `LoadLand` and `GetAnimation` all decode out of a shared m_StreamBuffer that is + /// reused, only ever grown, and filled by a stream.Read whose return value is + /// discarded. A record that is short, absent or out of bounds therefore renders **whatever + /// the previously-decoded asset left behind**, reports success, and is undetectable by + /// anything downstream. On the stock client on the machine phase 0 ran on that is 22,102 + /// ids whose index entry reads lookup 0, length 0. + /// + /// It cannot be complete and does not claim to be. It closes the shapes that reading the + /// source showed are reachable. What says the boundary is in the right place is the second + /// measurement rather than the first: against a client patched 21 ways it refused all eight + /// record-level defects, and against the **stock** client it refused **nothing** across + /// 49,151 statics and 16,384 land tiles. A checker that refuses real art would be worse + /// than no checker. + /// + public static class BridgeAssetValidator + { + public enum Verdict + { + /// Nothing at this id, and the index says so honestly. + Absent, + + /// The entry is self-consistent and inside its file. + Ok, + + /// The entry claims something the file cannot support. Do not decode it. + Refused + } + + /// Land tiles decode a fixed 44×44 diamond: 2 × (2+4+…+44) ushorts. + public const int LandRecordBytes = 2024; + + /// + /// A ceiling on decoded art dimensions. `LoadStatic` allocates + /// new Bitmap(width, height) straight from two ushorts in the record, so a + /// corrupt header asks for up to 65535×65535 — an 8 GB allocation, from a file. Real + /// art is a couple of hundred pixels at most. + /// + public const int MaxArtDimension = 1024; + + /// + /// Builds our own index over the same files, with the same constructor arguments + /// Art uses — including hasExtra: false, which is the whole reason the + /// art path is safe where the gump path is not (§4.1). + /// + public static FileIndex OpenArtIndex() + { + if (ArtDataPath() == null) + return null; + + return new FileIndex("Artidx.mul", "Art.mul", "artLegacyMUL.uop", 0x10000, 4, ".tga", 0x13FDC, false); + } + + /// + /// The file an art index entry's lookup is an offset **into** — which is not + /// art.mul on any current client. + /// + /// This cost a whole probe run to learn and it is the single most important thing + /// phase 1 must not get wrong. FileIndex's UOP constructor ends with a bare + /// MulPath = uopPath: **when artLegacyMUL.uop exists it wins outright**, + /// and art.mul / artidx.mul are never opened at all. A validator that + /// bounds offsets against art.mul while the index holds UOP offsets is not + /// merely approximate, it is nonsense — the first run of this probe refused 34,299 + /// perfectly good statics for "declaring 10533x2085" because it was reading UOP + /// offsets into the wrong file. + /// + /// So the resolution order here mirrors FileIndex's exactly, and anything that + /// needs the bytes behind an entry must ask this rather than assume. + /// + public static string ArtDataPath() + { + var uop = Files.GetFilePath("artlegacymul.uop"); + + if (uop != null) + return uop; + + return Files.GetFilePath("art.mul"); + } + + public static long MulLength(string path) + { + if (path == null) + return 0; + + try + { + return new FileInfo(path).Length; + } + catch + { + return 0; + } + } + + /// + /// Judges one index entry. + /// + /// The check FileIndex.Seek is missing is the last one: it tests + /// Stream.Length < e.lookup — that the record *starts* inside the file — and + /// never that it *ends* inside it. A record that begins two bytes before EOF and + /// declares a length of 4,000 passes, and stream.Read then returns a short count + /// that the decoders discard, leaving the previous asset's bytes in the shared buffer. + /// + public static Verdict CheckEntry(FileIndex index, int at, long mulLength, long verdataLength, out string reason) + { + reason = null; + + if (index == null || index.Index == null || at < 0 || at >= index.Index.Length) + { + reason = "index " + at + " out of range"; + return Verdict.Absent; + } + + Entry3D e = index.Index[at]; + + if (e.lookup < 0) + { + reason = "lookup " + e.lookup; + return Verdict.Absent; + } + + bool patched = (e.length & (1 << 31)) != 0; + int length = e.length & 0x7FFFFFFF; + + if (!patched && e.length < 0) + { + reason = "length " + e.length; + return Verdict.Absent; + } + + if (length == 0) + { + reason = "lookup " + e.lookup + ", length 0"; + return Verdict.Absent; + } + + long ceiling = patched ? verdataLength : mulLength; + + if (ceiling <= 0) + { + reason = (patched ? "verdata.mul" : "the art data file") + " has no length"; + return Verdict.Refused; + } + + if (e.lookup >= ceiling) + { + reason = "lookup " + e.lookup + " past the end of " + + (patched ? "verdata.mul" : "the mul") + " (" + ceiling + ")"; + return Verdict.Refused; + } + + // The missing check. A short read is silent, and its consequence is the PREVIOUS + // asset's picture served under this id. + if (e.lookup + (long)length > ceiling) + { + reason = "record runs " + (e.lookup + (long)length - ceiling) + " bytes past the end of " + + (patched ? "verdata.mul" : "the mul"); + return Verdict.Refused; + } + + return Verdict.Ok; + } + + /// + /// `LoadLand` reads 2,024 bytes regardless of the declared length, so a shorter record + /// reads past the end of a buffer sized from that length. + /// + public static bool LandLengthSane(FileIndex index, int at, out string reason) + { + reason = null; + + if (index == null || index.Index == null || at < 0 || at >= index.Index.Length) + return true; + + int length = index.Index[at].length & 0x7FFFFFFF; + + if (length > 0 && length < LandRecordBytes) + { + reason = "land record is " + length + " bytes; LoadLand always reads " + LandRecordBytes; + return false; + } + + return true; + } + + /// + /// Walks a static record's own row table the way LoadStatic will, and refuses + /// it if that walk would read outside the record. + /// + /// This is the check with teeth. LoadStatic's inner loop guards the write into + /// the bitmap (xOffset > delta, xOffset + xRun > delta) and does + /// nothing at all about the read cursor, which advances until it happens to find a + /// zero pair — potentially far outside a pinned array. Simulating the same walk with + /// a bound is the cheapest way to know whether handing the id over is safe. + /// + public static bool StaticRecordSane(byte[] record, int length, out string reason) + { + reason = null; + + if (length < 8) + { + reason = "record is " + length + " bytes; a static header needs 8"; + return false; + } + + int words = length / 2; + int width = ReadUInt16(record, 4); + int height = ReadUInt16(record, 6); + + // LoadStatic returns null for these rather than misbehaving, so it is not a refusal. + if (width <= 0 || height <= 0) + return true; + + if (width > MaxArtDimension || height > MaxArtDimension) + { + reason = "declares " + width + "x" + height + ", past the " + MaxArtDimension + "px ceiling"; + return false; + } + + // The row-lookup table: height ushorts starting at word 4. + if (4 + height > words) + { + reason = "row table (" + height + " entries) does not fit in a " + length + "-byte record"; + return false; + } + + int start = height + 4; + + for (int y = 0; y < height; y++) + { + int cursor = start + ReadUInt16(record, (4 + y) * 2); + + while (true) + { + // Two ushorts for the run header, and they must both be inside the record. + if (cursor < 0 || cursor + 1 >= words) + { + reason = "row " + y + " reads at word " + cursor + ", past the record's " + words; + return false; + } + + int xOffset = ReadUInt16(record, cursor * 2); + int xRun = ReadUInt16(record, (cursor + 1) * 2); + cursor += 2; + + if (xOffset + xRun == 0) + break; + + // LoadStatic stops the row here, so the read cursor stops with it. + if (xOffset > width || xOffset + xRun > width) + break; + + if (cursor + xRun > words) + { + reason = "row " + y + " declares a " + xRun + "-pixel run running past the record"; + return false; + } + + cursor += xRun; + } + } + + return true; + } + + // ── animations (phase 1) ───────────────────────────────────────────────────────────── + // + // Phase 0 measured the art path and left this half unbuilt, and then proved it was + // needed: the patched client's verdata entry for body 34 points past verdata.mul's own + // end, and the wolf still "decoded" — counted among the 1,144 successes while rendering + // something else entirely. `GetAnimation` has every weakness `LoadStatic` has and one + // more, because the buffer it decodes from is longer than the record it read. + + /// The palette every animation record opens with: 0x100 ushorts. + public const int AnimPaletteBytes = 0x100 * 2; + + /// + /// A ceiling on an animation's declared frame count. GetAnimation does + /// new int[frameCount] straight from four bytes in the file, before it has + /// looked at anything else. Real actions are tens of frames. + /// + public const int MaxAnimFrames = 1024; + + /// The xor Frame applies to every run header before decoding it. + private const int DoubleXor = (0x200 << 22) | (0x200 << 12); + + /// + /// The anim*.mul an animation index entry's lookup is an offset into. + /// + /// Unlike art (§4.6) there is no UOP precedence to get wrong here, and that is not + /// luck: Animations constructs its five FileIndexes with the four-argument + /// constructor, which passes uopFile: null. It never reads + /// AnimationFrame*.uop at all — which is the same fact that leaves six of the + /// twelve player-character bodies undecodable until §4.3's reader lands in phase 4. + /// + public static string AnimDataPath(int fileType) + { + switch (fileType) + { + case 1: return Files.GetFilePath("anim.mul"); + case 2: return Files.GetFilePath("anim2.mul"); + case 3: return Files.GetFilePath("anim3.mul"); + case 4: return Files.GetFilePath("anim4.mul"); + case 5: return Files.GetFilePath("anim5.mul"); + default: return null; + } + } + + /// + /// Builds our own index over one anim file, with the same constructor arguments + /// Animations uses — the entry lengths especially, since they decide how far + /// into the file an index runs. + /// + public static FileIndex OpenAnimIndex(int fileType) + { + if (AnimDataPath(fileType) == null) + return null; + + switch (fileType) + { + case 1: return new FileIndex("Anim.idx", "Anim.mul", 0x40000, 6); + case 2: return new FileIndex("Anim2.idx", "Anim2.mul", 0x10000, -1); + case 3: return new FileIndex("Anim3.idx", "Anim3.mul", 0x20000, -1); + case 4: return new FileIndex("Anim4.idx", "Anim4.mul", 0x20000, -1); + case 5: return new FileIndex("Anim5.idx", "Anim5.mul", 0x20000, -1); + default: return null; + } + } + + /// + /// Where a body's animation actually lives: which anim file, and which index in it. + /// + /// **This is the never-sweep-file-types rule, written as code** (§4.3). It asks + /// BodyConverter.Convert once, takes its answer, and if that answer leads + /// nowhere it reports nowhere. There is deliberately no loop here and no fallback, + /// because asking the *other* anim files for an index they do not own does not fail — + /// it returns 175 decodable action/direction combinations of **a giant spider** for + /// gargoyle 666, and misaligned colour fragments for the other two. Every one of those + /// reads reports success, and nothing downstream can tell them from art. + /// + /// A false return with set is the ordinary, expected answer + /// for a body this client has no art for — the caller reports absent, not an error. + /// + public static bool ResolveAnimation( + int body, int action, int direction, out int fileType, out int index, out string reason) + { + reason = null; + fileType = 0; + index = -1; + + if (body <= 0 || action < 0) + { + reason = "body " + body + " action " + action + " is not addressable"; + return false; + } + + // Directions 5-7 are the client mirroring 1-3, and `Frame` decodes them through its + // flip branch — different pointer arithmetic, which nothing below has checked. + // §5.1 fixed this protocol at direction 0 or 1, so refusing the rest costs nothing + // and keeps the validator honest about what it has actually verified. + if (direction < 0 || direction > 4) + { + reason = "direction " + direction + " is mirrored; this protocol reads 0-4 only"; + return false; + } + + int translated = body; + int hue = 0; + + try + { + // Exactly what GetAnimation(..., preserveHue: false, ...) does first. + Animations.Translate(ref translated, ref hue); + fileType = BodyConverter.Convert(ref translated); + } + catch (Exception e) + { + reason = "body.def/bodyconv.def lookup failed: " + e.GetType().Name; + return false; + } + + if (AnimDataPath(fileType) == null) + { + // Gargoyle 666 lands here: Bodyconv.def maps it to anim5, and this client has no + // anim5. Absent is the correct answer and the ONLY safe one. + reason = "bodyconv sends body " + body + " to file type " + fileType + + ", which this client does not have"; + return false; + } + + index = AnimIndexOf(translated, fileType) + (action * 5) + direction; + + return true; + } + + /// + /// Animations.GetFileIndex's own arithmetic, which is private. The banding is + /// per file type and the boundaries differ between them, so this is transcribed rather + /// than generalised — an index that disagrees with the library's by one is a picture + /// of the wrong creature, validated. + /// + private static int AnimIndexOf(int body, int fileType) + { + switch (fileType) + { + case 2: + return body < 200 ? body * 110 : 22000 + ((body - 200) * 65); + + case 3: + if (body < 300) + return body * 65; + + return body < 400 ? 33000 + ((body - 300) * 110) : 35000 + ((body - 400) * 175); + + case 5: + // "looks strange, though it works" — the library's own comment. Body 34 is + // excluded from the first band here and nowhere else. + if (body < 200 && body != 34) + return body * 110; + + return body < 400 ? 22000 + ((body - 200) * 65) : 35000 + ((body - 400) * 175); + + default: // 1 and 4 share their banding + if (body < 200) + return body * 110; + + return body < 400 ? 22000 + ((body - 200) * 65) : 35000 + ((body - 400) * 175); + } + } + + /// + /// Walks an animation record the way GetAnimation and Frame will, and + /// refuses it if that walk would read outside the record or write outside the bitmap. + /// + /// Two things make this stricter than the static walk, and both come from the library: + /// + /// GetAnimation decodes through new MemoryStream(m_StreamBuffer, false) — + /// the whole shared buffer, not the length bytes it just read into it. So a + /// truncated record does not hit end-of-stream and throw; the reader sails on into the + /// **previous** animation's bytes and returns a plausible frame. Bounding against + /// rather than against the buffer is the entire point. + /// + /// And Frame's run loop is a *write* through a LockBits pointer whose + /// origin comes from two signed shorts in the file (xCenter, yCenter), + /// with no bound of any kind. LoadStatic at least guards its writes; this does + /// not, so the destination of every run is checked against the bitmap it locked. + /// + /// is how many frames the caller will actually decode — + /// 1 for the catalogue's thumbnail (FirstFrame: true), 0 for all of them. + /// Checking frames nobody decodes would invent refusals, which §4.5 costs more than + /// it saves. + /// + public static bool AnimationRecordSane(byte[] record, int length, int maxFrames, out string reason) + { + reason = null; + + if (length < AnimPaletteBytes + 4) + { + reason = "record is " + length + " bytes; an animation needs " + + (AnimPaletteBytes + 4) + " for its palette and frame count"; + return false; + } + + int start = AnimPaletteBytes; + int frameCount = ReadInt32(record, start); + + if (frameCount <= 0) + { + reason = "declares " + frameCount + " frames"; + return false; + } + + if (frameCount > MaxAnimFrames) + { + reason = "declares " + frameCount + " frames, past the " + MaxAnimFrames + " ceiling"; + return false; + } + + // The lookup table is read in full whatever FirstFrame says, so it is bounded in full. + long tableEnd = (long)start + 4 + ((long)frameCount * 4); + + if (tableEnd > length) + { + reason = "frame table (" + frameCount + " entries) does not fit in a " + + length + "-byte record"; + return false; + } + + int check = maxFrames > 0 && maxFrames < frameCount ? maxFrames : frameCount; + + for (int i = 0; i < check; i++) + { + int at = start + ReadInt32(record, start + 4 + (i * 4)); + + if (!FrameSane(record, length, at, i, out reason)) + return false; + } + + return true; + } + + private static bool FrameSane(byte[] record, int length, int at, int frame, out string reason) + { + reason = null; + + if (at < 0 || at + 8 > length) + { + reason = "frame " + frame + " starts at " + at + ", outside the " + + length + "-byte record"; + return false; + } + + int xCenter = ReadInt16(record, at); + int yCenter = ReadInt16(record, at + 2); + int width = ReadUInt16(record, at + 4); + int height = ReadUInt16(record, at + 6); + + // Frame's constructor returns before locking anything for these, so they are empty + // rather than dangerous — and an empty frame is a real thing in this format. + if (width == 0 || height == 0) + return true; + + if (width > MaxArtDimension || height > MaxArtDimension) + { + reason = "frame " + frame + " declares " + width + "x" + height + ", past the " + + MaxArtDimension + "px ceiling"; + return false; + } + + // Settings.PixelFormat is 16bpp and GDI+ pads each scanline to four bytes, so a row + // is `delta` ushorts wide and the locked region is height*delta of them. This is the + // same `bd.Stride >> 1` Frame computes. + int delta = (((width * 2) + 3) & ~3) >> 1; + long pixels = (long)height * delta; + + long origin = (xCenter - 0x200) + ((long)((yCenter + height) - 0x200) * delta); + int cursor = at + 8; + + while (true) + { + if (cursor + 4 > length) + { + reason = "frame " + frame + + " runs off the end of the record looking for its terminator"; + return false; + } + + int header = ReadInt32(record, cursor); + cursor += 4; + + if (header == 0x7FFF7FFF) + break; + + header ^= DoubleXor; + + long dy = (header >> 12) & 0x3FF; + long dx = (header >> 22) & 0x3FF; + int run = header & 0xFFF; + + long first = origin + (dy * delta) + dx; + + if (first < 0 || first + run > pixels) + { + reason = "frame " + frame + " writes pixels " + first + ".." + (first + run) + + " outside its own " + pixels + "-pixel bitmap"; + return false; + } + + // One palette byte per pixel, read straight out of the record. + if (cursor + run > length) + { + reason = "frame " + frame + " declares a " + run + + "-pixel run running past the record"; + return false; + } + + cursor += run; + } + + return true; + } + + private static int ReadUInt16(byte[] b, int at) + { + return b[at] | (b[at + 1] << 8); + } + + private static int ReadInt16(byte[] b, int at) + { + return (short)(b[at] | (b[at + 1] << 8)); + } + + private static int ReadInt32(byte[] b, int at) + { + return b[at] | (b[at + 1] << 8) | (b[at + 2] << 16) | (b[at + 3] << 24); + } + + /// + /// Reads a record's actual bytes so or + /// can walk it. + /// + /// Holds its own handles rather than borrowing the library's, because FileIndex + /// hands out the stream it decodes from and moving that stream's position underneath + /// the decoder would be its own bug. Opened FileShare.ReadWrite to match how + /// FileIndex opens the same files. + /// + /// One reader serves one data file, so an animation sweep wants one per file type, + /// built from . + /// + public sealed class RecordReader : IDisposable + { + private readonly FileStream _mul; + private readonly FileStream _verdata; + private byte[] _scratch = new byte[64 * 1024]; + + public RecordReader(string mulPath, string verdataPath) + { + _mul = Open(mulPath); + _verdata = Open(verdataPath); + } + + private static FileStream Open(string path) + { + if (path == null || !File.Exists(path)) + return null; + + try + { + return new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite); + } + catch + { + return null; + } + } + + /// + /// True when the record at is safe to hand to + /// Art.GetStatic. A record that cannot be read at all is reported sane — + /// has already judged the entry, and this must not + /// invent a second reason to refuse. + /// + public bool StaticSane(FileIndex index, int at, out string reason) + { + int length = ReadRecord(index, at, out reason); + + if (length < 0) + return true; + + if (length == 0) + return false; + + return StaticRecordSane(_scratch, length, out reason); + } + + /// + /// True when the record at is safe to hand to + /// Animations.GetAnimation. is how many frames + /// the caller will decode — 1 for a FirstFrame call, 0 for all of them. + /// + public bool AnimationSane(FileIndex index, int at, int maxFrames, out string reason) + { + int length = ReadRecord(index, at, out reason); + + if (length < 0) + return true; + + if (length == 0) + return false; + + return AnimationRecordSane(_scratch, length, maxFrames, out reason); + } + + /// + /// Reads one record into . Returns its length, 0 for a + /// failure (with set), or -1 when there is nothing to + /// read at all — has already judged the entry, and this + /// must not invent a second reason to refuse. + /// + private int ReadRecord(FileIndex index, int at, out string reason) + { + reason = null; + + if (index == null || index.Index == null || at < 0 || at >= index.Index.Length) + return -1; + + Entry3D e = index.Index[at]; + bool patched = (e.length & (1 << 31)) != 0; + int length = e.length & 0x7FFFFFFF; + + var stream = patched ? _verdata : _mul; + + if (stream == null || length <= 0 || e.lookup < 0) + return -1; + + if (_scratch.Length < length) + _scratch = new byte[length]; + + int read; + + try + { + stream.Seek(e.lookup, SeekOrigin.Begin); + read = stream.Read(_scratch, 0, length); + } + catch (Exception ex) + { + reason = "cannot read the record: " + ex.GetType().Name; + return 0; + } + + // The short read the decoders discard. Refusing here is the whole point: the + // library would decode whatever the shared buffer happened to hold. + if (read < length) + { + reason = "short read — " + read + " of " + length + " bytes available"; + return 0; + } + + return length; + } + + public void Dispose() + { + if (_mul != null) + _mul.Dispose(); + + if (_verdata != null) + _verdata.Dispose(); + } + } + } +} diff --git a/overlay/Scripts/Custom/Bridge/BridgeAssets.cs b/overlay/Scripts/Custom/Bridge/BridgeAssets.cs new file mode 100644 index 0000000..a3e4714 --- /dev/null +++ b/overlay/Scripts/Custom/Bridge/BridgeAssets.cs @@ -0,0 +1,747 @@ +using System; +using System.Collections.Generic; +using System.Globalization; +using System.IO; +using System.Security.Cryptography; +using System.Text; +using System.Threading; + +using Ultima; + +namespace Server.Custom.Bridge +{ + /// + /// **The Asset Bridge's transport** (docs/link/v8.md §3, §6, §7 — protocol 8, phase 1). + /// + /// Everything else on this link answers on the Core thread, reads live world state, and + /// replies in microseconds. The asset plane cannot: it reads hundreds of megabytes of the + /// operator's client files and decodes pictures out of them, and doing either on the Core + /// thread would stop the world for every player on the shard. So this class is the one + /// genuinely new shape in protocol 8 — a handler that accepts on the Core thread, hands + /// the work to a **dedicated asset worker**, and returns immediately. + /// + /// Three rules hold it together, and each of them is answering a specific way this could + /// go wrong. + /// + /// **1. Replies, never events.** Every asset frame carries the caller's `reqId`, so + /// `rpc.rs`'s `try_route` consumes it before `app.rs` can persist it to SQLite and + /// broadcast it to every WebSocket subscriber. An asset stream on the event path would + /// grow the sidecar's store without bound and fan megabytes at every connected client, + /// forever. Nothing here emits an unsolicited frame — if a request has no `reqId` it is + /// refused rather than answered. + /// + /// **2. One request outstanding, always.** 's queue is bounded + /// drop-oldest in **lines, not bytes** — a design that is right for live events and + /// dangerous for bulk transfer, because 10,000 queued 200 KB replies is 2 GB of shard + /// memory. The bound that actually holds is flow control, not a bigger queue: this plane + /// has **one slot**, and a second asset request arriving while one is in flight is + /// answered `bridge.busy` (which the sidecar already maps to 425) rather than queued. + /// Queue depth therefore stays at approximately one by construction. A dropped or lost + /// reply just times out and is re-requested, which is safe because reading a client file + /// is idempotent and touches no world state. + /// + /// Note what that costs, deliberately: a status poll shares the slot with a batch, so + /// polling during a long import gets 425 until the batch lands. That is honest — this + /// plane really is doing one thing at a time — and the admin surface (phase 8) is where a + /// separate status lane would have to argue for itself. + /// + /// **3. Byte budgets, not counts.** Batches are cut by encoded size + /// (, 512 KB), not by item count, because the + /// ceilings this has to live inside are byte ceilings: the sidecar refuses an inbound line + /// over 1 MiB, and base64 costs 33% on top of whatever the payload measures. + /// is that budget, and every asset family shares it so the + /// envelope cannot drift apart between them. + /// + /// **Emitting from off the Core thread is safe here, and it is worth saying why.** + /// BridgeLink.Emit enqueues onto a ConcurrentQueue and never touches the + /// socket, so the enqueue itself is fine. The subtle part is + /// BridgeIdempotency.Observe, which Emit calls while a keyed command is in + /// flight: it captures a line only when that line's correlation field **exactly equals** + /// the open command's correlation value, and correlation values come from one monotonic + /// counter in the sidecar. An asset reply therefore cannot be mistaken for a keyed + /// command's reply, whatever the interleaving. + /// + public static class BridgeAssets + { + /// + /// What version of *our derivation* produced these bytes (§7). + /// + /// The source gate hashes the operator's client files, which answers "did the inputs + /// change". It cannot answer "did the way we read them change" — and that is the case + /// that bites, because a corrected frame offset or a fixed hue application changes + /// every derived byte while every source file stays identical. So this is folded into + /// stage 1 alongside the hashes, and bumping it makes the whole working set drift, + /// which is the intended and correct blast radius. + /// + /// Bump it whenever extraction changes what it produces from unchanged input. It is + /// the same rule spawnAtlasSource.js's `PARSER_VERSION` follows, and it applies + /// here more rather than less: this pipeline derives far more from far less. + /// + public const int EXTRACTOR_VERSION = 1; + + // ── the one slot (§3.2) ────────────────────────────────────────────────────────────── + + private static readonly object _sync = new object(); + private static Thread _worker; + private static readonly AutoResetEvent _wake = new AutoResetEvent(false); + private static Action _job; + private static string _inFlight; + private static DateTime _inFlightSince; + private static bool _running; + + private static long _served, _busied, _failed; + + // ── the hash cache (§6) ────────────────────────────────────────────────────────────── + + private static readonly Dictionary _hashes = + new Dictionary(StringComparer.OrdinalIgnoreCase); + + private static Thread _hasher; + private static volatile bool _hashing; + + private sealed class CachedHash + { + public long Size; + public long MTime; + public string Sha256; + } + + // ── imaging (§4.4) ─────────────────────────────────────────────────────────────────── + + private static bool _imagingChecked; + private static bool _imagingOk; + private static string _imagingReason; + + public static void Initialize() + { + if (!BridgeConfig.Enabled) + return; + + BridgeBoot.RegisterHandler("assets.sources", OnSources); + } + + public static string Status() + { + int cached; + + lock (_hashes) + { + cached = _hashes.Count; + } + + lock (_sync) + { + return String.Format( + "assets(served={0} busied={1} failed={2} inFlight={3} hashing={4} cached={5})", + _served, _busied, _failed, _inFlight ?? "-", _hashing, cached); + } + } + + // ── the request plane ──────────────────────────────────────────────────────────────── + + /// + /// Stage 1 of §6: what the shard's client files currently are. No pixels, no assets — + /// just the gate that lets the website decide whether anything needs importing at all, + /// because the normal case is a restart that changed nothing and it must cost nothing. + /// + private static void OnSources(Dictionary o) + { + var reqId = BridgeJson.GetString(o, "reqId"); + + if (reqId == null) + { + // Rule 1. Without a correlation id this would land on the event path, be persisted + // to the sidecar's store and broadcast to every subscriber. Refuse instead. + Fail(null, "assets.sources requires a reqId"); + return; + } + + if (!BridgeConfig.AssetsEnabled) + { + Fail(reqId, "asset extraction is disabled on this shard"); + return; + } + + Accept(reqId, "assets.sources", () => ReplySources(reqId)); + } + + /// + /// Claims the single slot and hands the work to the worker, or answers `bridge.busy`. + /// Runs on the Core thread and does nothing expensive; runs on + /// the worker and must touch no world state. + /// + private static void Accept(string reqId, string kind, Action job) + { + lock (_sync) + { + if (_inFlight != null) + { + _busied++; + Busy(reqId, kind); + return; + } + + _inFlight = kind; + _inFlightSince = DateTime.UtcNow; + _job = job; + + try + { + EnsureWorker(); + } + catch (Exception e) + { + // The slot is claimed and there is now nothing that will ever free it. Give it + // back here or this plane answers `bridge.busy` for the life of the process. + _inFlight = null; + _job = null; + + Console.WriteLine("[Bridge] cannot start the asset worker: {0}", e.Message); + Fail(reqId, "the shard could not start its asset worker"); + return; + } + } + + _wake.Set(); + } + + private static void Busy(string reqId, string kind) + { + var held = (DateTime.UtcNow - _inFlightSince).TotalSeconds; + + var sb = BridgeJson.Begin("bridge.busy"); + + sb.Str("reqId", reqId) + // `busyKind`, never a second `kind` — `Begin` has already written this frame's own, + // and a JSON object carrying two makes every parser take the last. Protocol 6 shipped + // that bug once and it made the sidecar answer 200 for a refusal. + .Str("busyKind", kind) + .Num("heldForSec", (long)held) + .Str("reason", "the asset plane serves one request at a time"); + + BridgeLink.Emit(sb.End()); + } + + private static void Fail(string reqId, string reason) + { + var sb = BridgeJson.Begin("assets.error"); + + if (reqId != null) + sb.Str("reqId", reqId); + + sb.Str("reason", reason); + BridgeLink.Emit(sb.End()); + } + + // ── the worker ─────────────────────────────────────────────────────────────────────── + + /// + /// Started on first use rather than at boot, so a shard that never imports an asset + /// never carries the thread. Caller must hold . + /// + private static void EnsureWorker() + { + if (_worker != null) + return; + + _running = true; + + _worker = new Thread(WorkLoop) + { + Name = "BridgeAssets", + IsBackground = true + }; + + _worker.Start(); + } + + private static void WorkLoop() + { + while (_running) + { + _wake.WaitOne(1000); + + Action job; + + lock (_sync) + { + job = _job; + _job = null; + } + + if (job == null) + continue; + + try + { + job(); + Interlocked.Increment(ref _served); + } + catch (Exception e) + { + // A handler that throws must still free the slot, or this plane is wedged for + // the life of the process and every later request answers `bridge.busy`. + Interlocked.Increment(ref _failed); + Console.WriteLine("[Bridge] asset worker: {0}: {1}", e.GetType().Name, e.Message); + } + finally + { + lock (_sync) + { + _inFlight = null; + } + } + } + } + + // ── assets.sources ─────────────────────────────────────────────────────────────────── + + /// + /// The client files whose bytes decide everything downstream. + /// + /// Resolved through Ultima.Files rather than by joining a configured directory, + /// because that is what the decoders themselves do — a file this reports is a file + /// they would actually open. + /// + private static readonly string[] SourceFiles = + { + "cliloc.enu", + "artlegacymul.uop", "art.mul", "artidx.mul", + "anim.idx", "anim.mul", + "anim2.idx", "anim2.mul", + "anim3.idx", "anim3.mul", + "anim4.idx", "anim4.mul", + "anim5.idx", "anim5.mul", + "body.def", "bodyconv.def", + "hues.mul", + "verdata.mul" + }; + + private static void ReplySources(string reqId) + { + var sb = BridgeJson.Begin("assets.sources.ok"); + + sb.Str("reqId", reqId) + .Num("extractorVersion", EXTRACTOR_VERSION); + + WriteImaging(sb); + + // §4.6: whichever of art.mul / artLegacyMUL.uop `FileIndex` would actually open. An + // operator who added custom graphics to art.mul while the UOP is present is getting + // nothing, silently, and this is the only place that can tell them so. + string artData = BridgeAssetValidator.ArtDataPath(); + + sb.Str("artDataFile", artData == null ? null : Path.GetFileName(artData)); + + var page = new PageBuilder(sb, "files", BridgeConfig.AssetBatchBytes); + bool anyMissingHash = false; + + for (int i = 0; i < SourceFiles.Length; i++) + { + string name = SourceFiles[i]; + string path = ResolvePath(name); + + if (path == null) + continue; + + var item = new StringBuilder(256); + + item.Append("{\"name\":"); + BridgeJson.Text(item, name); + item.Append(",\"path\":"); + BridgeJson.Text(item, path); + + long size = 0, mtime = 0; + + try + { + var info = new FileInfo(path); + size = info.Length; + mtime = ToUnixMs(info.LastWriteTimeUtc); + } + catch (Exception e) + { + item.Append(",\"unreadable\":"); + BridgeJson.Text(item, e.GetType().Name); + } + + item.Append(",\"size\":").Append(size.ToString(CultureInfo.InvariantCulture)); + item.Append(",\"mtime\":").Append(mtime.ToString(CultureInfo.InvariantCulture)); + + string hash = CachedHashFor(path, size, mtime); + + if (hash == null) + anyMissingHash = true; + + item.Append(",\"sha256\":"); + BridgeJson.Text(item, hash); + + // The one diagnostic §4.6 asks for: art.mul is present, and unread. + if (artData != null + && (name == "art.mul" || name == "artidx.mul") + && !artData.EndsWith(".mul", StringComparison.OrdinalIgnoreCase)) + { + item.Append(",\"shadowedBy\":"); + BridgeJson.Text(item, Path.GetFileName(artData)); + } + + item.Append('}'); + + if (!page.TryAdd(item.ToString(), name)) + break; + } + + page.Close(); + + // §6's gate is (size, mtime) first and a content hash only when those differ, because + // anim.mul and art.mul are 195 MB and 148 MB and a full hash on every status poll + // would make the admin panel feel broken. It would also blow the sidecar's 10 s reply + // timeout outright on the first call. So a hash that is not cached is reported `null` + // and computed in the background: this reply is always fast, and the next poll — after + // `hashing` goes false — carries the answer. + if (anyMissingHash) + StartHashing(); + + sb.Bool("hashing", _hashing); + sb.Bool("complete", !anyMissingHash); + + BridgeLink.Emit(sb.End()); + } + + private static string ResolvePath(string name) + { + try + { + return Files.GetFilePath(name); + } + catch + { + return null; + } + } + + private static long ToUnixMs(DateTime utc) + { + return (long)(utc - new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc)).TotalMilliseconds; + } + + // ── imaging (§4.4) ─────────────────────────────────────────────────────────────────── + + /// + /// Whether this host can turn a record into a picture at all. + /// + /// ServUO targets net48, so a Linux shard runs it under Mono, where + /// System.Drawing is a thin layer over **libgdiplus** — and §4.2 put + /// System.Drawing in the *decode* path, not merely the encode: Frame + /// writes ARGB1555 through a LockBits pointer. Without that library a Linux + /// shard cannot read a sprite at all, while clilocs and the atlas are unaffected + /// because neither touches pixels. + /// + /// It must never present as a stack trace or a 500. It is a named, actionable outcome + /// in the same family as the cliloc reader's `COMPRESSED`, and it is reported here — + /// on the *source gate*, the first call any import makes — so an operator learns it + /// while setting the shard up rather than from an empty bestiary weeks later. + /// + private static void WriteImaging(StringBuilder sb) + { + CheckImaging(); + + sb.Append(",\"imaging\":{\"ok\":").Append(_imagingOk ? "true" : "false"); + + if (!_imagingOk) + { + sb.Append(",\"code\":\"NO_IMAGING\",\"reason\":"); + BridgeJson.Text(sb, + "This shard host cannot render images — Mono's System.Drawing needs libgdiplus. " + + "Install it (apt-get install libgdiplus) and re-run the import. Cliloc and atlas " + + "import are unaffected. (" + _imagingReason + ")"); + } + + sb.Append('}'); + } + + private static void CheckImaging() + { + if (_imagingChecked) + return; + + _imagingChecked = true; + + try + { + TouchImaging(); + _imagingOk = true; + } + catch (Exception e) + { + // On a host with no libgdiplus this is a TypeInitializationException wrapping a + // DllNotFoundException, and it can surface as the method failing to JIT rather + // than as a throw from inside it — which is why the construction lives in its own + // method, so the failure is contained here instead of taking this class's + // static initialisation with it. + _imagingOk = false; + _imagingReason = e.GetType().Name + ": " + e.Message; + } + } + + private static void TouchImaging() + { + using (var bmp = new System.Drawing.Bitmap(1, 1)) + { + bmp.SetPixel(0, 0, System.Drawing.Color.Black); + } + } + + // ── the hash cache (§6) ────────────────────────────────────────────────────────────── + + private static string CachedHashFor(string path, long size, long mtime) + { + lock (_hashes) + { + CachedHash cached; + + if (_hashes.TryGetValue(path, out cached) + && cached.Size == size + && cached.MTime == mtime) + { + return cached.Sha256; + } + } + + return null; + } + + /// + /// Rehashes whatever the cache is missing, on its own thread. + /// + /// Deliberately **not** a job on the asset worker: hashing 343 MB takes seconds to + /// tens of seconds, and holding the single slot for that long would answer every + /// status poll `bridge.busy` for the whole pass — which is exactly the moment an + /// operator is watching the panel. It emits nothing and correlates with nothing; it + /// only fills the cache that the next `assets.sources` reads. + /// + private static void StartHashing() + { + lock (_sync) + { + if (_hashing) + return; + + _hashing = true; + + _hasher = new Thread(HashLoop) + { + Name = "BridgeAssetHash", + IsBackground = true + }; + + _hasher.Start(); + } + } + + private static void HashLoop() + { + try + { + for (int i = 0; i < SourceFiles.Length; i++) + { + string path = ResolvePath(SourceFiles[i]); + + if (path == null) + continue; + + long size, mtime; + + try + { + var info = new FileInfo(path); + size = info.Length; + mtime = ToUnixMs(info.LastWriteTimeUtc); + } + catch + { + continue; + } + + if (CachedHashFor(path, size, mtime) != null) + continue; + + string hash = HashFile(path); + + if (hash == null) + continue; + + lock (_hashes) + { + _hashes[path] = new CachedHash { Size = size, MTime = mtime, Sha256 = hash }; + } + } + } + catch (Exception e) + { + Console.WriteLine("[Bridge] asset hash pass: {0}: {1}", e.GetType().Name, e.Message); + } + finally + { + // Under _sync, matching StartHashing: cleared outside it, two passes could both + // pass the guard and hash the same 343 MB twice. + lock (_sync) + { + _hashing = false; + } + } + } + + private static string HashFile(string path) + { + try + { + using (var sha = SHA256.Create()) + using (var stream = new FileStream( + path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite, 1 << 20)) + { + var buffer = new byte[1 << 20]; + int read; + + while ((read = stream.Read(buffer, 0, buffer.Length)) > 0) + sha.TransformBlock(buffer, 0, read, null, 0); + + sha.TransformFinalBlock(buffer, 0, 0); + + return Hex(sha.Hash); + } + } + catch (Exception e) + { + Console.WriteLine("[Bridge] cannot hash {0}: {1}", path, e.Message); + return null; + } + } + + private static string Hex(byte[] bytes) + { + var sb = new StringBuilder(bytes.Length * 2); + + for (int i = 0; i < bytes.Length; i++) + sb.Append(bytes[i].ToString("x2", CultureInfo.InvariantCulture)); + + return sb.ToString(); + } + + // ── the paging envelope (§3.3) ─────────────────────────────────────────────────────── + + /// + /// **One envelope for every asset family**, defined here in phase 1 so that clilocs + /// (phase 2), the body catalogue (3), statics and land (5), deep animation keys (6) + /// and the ServUO tree files (7) all page the same way. They are otherwise five + /// chances to invent five slightly different shapes, and the website would have to + /// learn each one. + /// + /// The envelope a reply closes with: + /// + /// + /// "items": [ … ], + /// "more": true, // ask again with this cursor + /// "cursor": "s:4104", // opaque to everyone but the shard; absent when more:false + /// "cut": "budget" // budget | end | limit — WHY this page stopped + /// + /// + /// **The budget is bytes, and it is UTF-8 bytes.** Not item count, because the ceiling + /// this lives inside is the sidecar's inbound line cap; and not chars, because a + /// cliloc row is real text and a `StringBuilder`'s Length would undercount every + /// non-ASCII character in it. + /// + /// `cut` exists because "the page is short" has three different meanings and the + /// website must not have to guess which: the source ran out (`end`), the byte budget + /// was spent (`budget`), or the family stopped at its own limit (`limit`). Only the + /// first means the import is finished. + /// + /// **The first item is always admitted**, even if it alone exceeds the budget. + /// Otherwise an oversized item would make its family unable to make any progress at + /// all — it would be skipped for the budget on every page, forever. That is safe + /// precisely because the budget is set to half the sidecar's line cap + /// (), so one such item still fits the wire. + /// + public sealed class PageBuilder + { + private readonly StringBuilder _sb; + private readonly int _budget; + private int _bytes; + private int _count; + private string _cursor; + private string _cut = "end"; + + /// + /// Room kept back for the fields the envelope must still be able to write after + /// the last item — `more`, `cursor`, `cut` and the closing brace. Without it a + /// page could fill the budget exactly and then overrun it closing itself. + /// + private const int Reserve = 256; + + public PageBuilder(StringBuilder sb, string arrayName, int budget) + { + _sb = sb; + _budget = budget; + + sb.Append(",\"").Append(arrayName).Append("\":["); + + // The prefix is already written, and it counts: the cap the sidecar enforces is + // on the whole line, not on the array. + _bytes = Encoding.UTF8.GetByteCount(sb.ToString()); + } + + public int Count { get { return _count; } } + + /// + /// Adds one already-serialised item. is where the + /// family should resume if this turns out to be the last item on the page. + /// Returns false when the budget is spent — the caller stops, and `more` is true. + /// + public bool TryAdd(string item, string cursorAfter) + { + if (item == null) + return true; + + int cost = Encoding.UTF8.GetByteCount(item) + (_count > 0 ? 1 : 0); + + if (_count > 0 && _bytes + cost + Reserve > _budget) + { + _cut = "budget"; + return false; + } + + if (_count > 0) + _sb.Append(','); + + _sb.Append(item); + + _bytes += cost; + _count++; + _cursor = cursorAfter; + + return true; + } + + /// + /// Stops the page for a reason of the family's own — a per-request limit, say — + /// rather than because the budget ran out. + /// + public void Cut(string why) + { + _cut = why; + } + + public void Close() + { + bool more = _cut != "end"; + + _sb.Append(']'); + _sb.Bool("more", more); + _sb.Str("cut", _cut); + + if (more && _cursor != null) + _sb.Str("cursor", _cursor); + } + } + } +} diff --git a/overlay/Scripts/Custom/Bridge/BridgeBoot.cs b/overlay/Scripts/Custom/Bridge/BridgeBoot.cs index 3d6c131..d09dc43 100644 --- a/overlay/Scripts/Custom/Bridge/BridgeBoot.cs +++ b/overlay/Scripts/Custom/Bridge/BridgeBoot.cs @@ -260,6 +260,7 @@ namespace Server.Custom.Bridge e.Mobile.SendMessage("Bridge: {0}", BridgeParticipation.Status()); e.Mobile.SendMessage("Bridge: {0}", BridgeWorld.Status()); e.Mobile.SendMessage("Bridge: {0}", BridgeOneShots.Status()); + e.Mobile.SendMessage("Bridge: {0}", BridgeAssets.Status()); break; } } diff --git a/overlay/Scripts/Custom/Bridge/BridgeConfig.cs b/overlay/Scripts/Custom/Bridge/BridgeConfig.cs index dedf4bc..0f140f4 100644 --- a/overlay/Scripts/Custom/Bridge/BridgeConfig.cs +++ b/overlay/Scripts/Custom/Bridge/BridgeConfig.cs @@ -87,6 +87,16 @@ namespace Server.Custom.Bridge // morning. Those are different consents, and one switch cannot express both. public static bool EventsEnabled { get; private set; } + // ---- the asset plane (docs/link/v8.md §3, protocol 8) ---- + // + // Its own gate again, and for the same reason the event plane got one: enabling this is + // an operator consenting to the WEBSITE READING THEIR CLIENT FILES -- art, animations and + // the string table, off the host's disk, over the link. That is a different consent from + // publishing world state, and one switch cannot express both. Reads only: nothing on this + // plane writes anything, anywhere. + public static bool AssetsEnabled { get; private set; } + public static int AssetBatchBytes { get; private set; } + public static int LeaseMaxDurationSec { get; private set; } public static int LeaseGraceSec { get; private set; } @@ -144,6 +154,20 @@ namespace Server.Custom.Bridge Port = Config.Get("Bridge.Port", 7788); QueueCap = Config.Get("Bridge.QueueCap", 10000); + AssetsEnabled = Config.Get("Bridge.AssetsEnabled", true); + + // The largest reply this plane will build, in ENCODED bytes -- not items, because the + // ceiling it has to live inside is a byte ceiling. Clamped to half the sidecar's 1 MiB + // inbound line cap, and the halving is load-bearing rather than cautious: a page + // always admits its first item even when that item alone exceeds the budget (the + // alternative is an oversized item being skipped forever and its family never making + // progress), so the wire must still have room for one such overshoot. + AssetBatchBytes = Config.Get("Bridge.AssetBatchBytes", 512 * 1024); + if (AssetBatchBytes < 64 * 1024) + AssetBatchBytes = 64 * 1024; + if (AssetBatchBytes > 512 * 1024) + AssetBatchBytes = 512 * 1024; + StatSweepSeconds = Config.Get("Bridge.StatSweepSeconds", 30); DecaySweepSeconds = Config.Get("Bridge.DecaySweepSeconds", 60); EconomySweepSeconds = Config.Get("Bridge.EconomySweepSeconds", 300); diff --git a/tools/scaffolding/BridgeAssetProbe.cs b/tools/scaffolding/BridgeAssetProbe.cs index 6f0bfb9..7d42604 100644 --- a/tools/scaffolding/BridgeAssetProbe.cs +++ b/tools/scaffolding/BridgeAssetProbe.cs @@ -7,6 +7,7 @@ using System.Text; using System.Threading; using Server.Commands; +using Server.Custom.Bridge; using Ultima; @@ -475,76 +476,109 @@ namespace Server.Custom /// nothing downstream can tell. A "0 rows" outcome is the correct answer. /// /// So the sweep records the file type each body resolved to and whether that file - /// answered — and never a second opinion. + /// answered — and never a second opinion. ResolveAnimation is that rule as + /// code, and this sweep is now one of its callers rather than its own transcription. + /// + /// **Phase 1 added the validator to this sweep**, which phase 0 ran without one: + /// it reported "1,144 decoded, 0 faults" while the patched client's verdata entry for + /// body 34 pointed past verdata.mul's own end and the wolf rendered something else, + /// counted among those 1,144. REFUSED-BUT-DECODED is the cell that says so, and it is + /// the same cell the art sweeps have had since phase 0. /// private static void SectionBodies() { Head("bodies — Animations.GetAnimation, one direction, first frame"); int direction = Config.Get("Bridge.AssetProbeCreatureDirection", 1); - int decoded = 0, empty = 0, faulted = 0; + var tally = new Tally(); var byFileType = new int[8]; - var faults = new List(); + int unresolved = 0; - for (int body = 0; body < 2048; body++) + var indexes = new Dictionary(); + var readers = new Dictionary(); + var lengths = new Dictionary(); + + string verdataPath = Files.GetFilePath("verdata.mul"); + long verdataLength = BridgeAssetValidator.MulLength(verdataPath); + + try { - Checkpoint("bodies", body); - - int translated = body; - int fileType; - - try + for (int body = 0; body < 2048; body++) { - fileType = BodyConverter.Convert(ref translated); - } - catch (Exception e) - { - faulted++; - faults.Add("body " + body + " BodyConverter.Convert: " + e.GetType().Name + ": " + e.Message); - continue; - } + Checkpoint("bodies", body); - if (fileType >= 0 && fileType < byFileType.Length) - byFileType[fileType]++; + int fileType, at; + string reason; - try - { - int hue = 0; - var frames = Animations.GetAnimation(body, 0, direction, ref hue, false, true); - - if (frames != null && frames.Length > 0 && frames[0] != null && frames[0].Bitmap != null) + if (!BridgeAssetValidator.ResolveAnimation(body, 0, direction, out fileType, out at, out reason)) { - decoded++; - frames[0].Bitmap.Dispose(); + // The never-sweep-file-types rule's own outcome (§4.3): bodyconv sent this + // body to a file this client does not have, so we report nothing and ask + // no other file. Asking anim2 for gargoyle 666 returns a giant spider. + unresolved++; + continue; } - else - { - empty++; - } - } - catch (Exception e) - { - faulted++; - if (faults.Count < 40) - faults.Add("body " + body + " (fileType " + fileType + "): " + e.GetType().Name + ": " + e.Message); + if (fileType >= 0 && fileType < byFileType.Length) + byFileType[fileType]++; + + if (!indexes.ContainsKey(fileType)) + { + string dataPath = BridgeAssetValidator.AnimDataPath(fileType); + + indexes[fileType] = BridgeAssetValidator.OpenAnimIndex(fileType); + lengths[fileType] = BridgeAssetValidator.MulLength(dataPath); + readers[fileType] = new BridgeAssetValidator.RecordReader(dataPath, verdataPath); + } + + var index = indexes[fileType]; + var verdict = BridgeAssetValidator.CheckEntry( + index, at, lengths[fileType], verdataLength, out reason); + + // Only the entry has been judged so far. The record behind it is where the + // frame table and the unbounded run headers live. + if (verdict == BridgeAssetValidator.Verdict.Ok + && !readers[fileType].AnimationSane(index, at, 1, out reason)) + { + verdict = BridgeAssetValidator.Verdict.Refused; + } + + bool decoded = false; + string thrown = null; + + try + { + int hue = 0; + var frames = Animations.GetAnimation(body, 0, direction, ref hue, false, true); + + if (frames != null && frames.Length > 0 && frames[0] != null && frames[0].Bitmap != null) + { + decoded = true; + frames[0].Bitmap.Dispose(); + } + } + catch (Exception e) + { + thrown = e.GetType().Name + ": " + e.Message; + } + + Record(tally, verdict, reason, decoded, thrown, "body/" + body + "/a0"); + } + } + finally + { + foreach (var reader in readers.Values) + { + if (reader != null) + reader.Dispose(); } } Say("direction " + direction + " (creature default — §5.1)"); - Say(string.Format(" decoded {0} empty {1} FAULTED {2}", decoded, empty, faulted)); + Say(string.Format(" bodyconv resolves nowhere (correct — never swept): {0:N0}", unresolved)); Say(" by file type: " + string.Join(", ", FileTypeCounts(byFileType))); - - if (faults.Count > 0) - { - Say(""); - Say(" faults:"); - - foreach (var f in faults) - Say(" " + f); - } - Say(""); + tally.Report("bodies 0..2047, action 0, first frame"); } private static string[] FileTypeCounts(int[] byFileType) @@ -566,10 +600,16 @@ namespace Server.Custom /// The twelve (on stock 57.4) player-character bodies, each at direction 0 — head-on, /// because a character is a portrait and should look at you (§5.1). /// - /// Six of them are expected to report nothing on the legacy path: both human ghosts and - /// every gargoyle body are UOP-only. **That is the measurement, not a failure** — it is - /// what phase 4's UOP reader exists for, and a probe that flagged it red would teach an + /// Most of them are expected to have no art on the legacy path — the ghosts and every + /// gargoyle body are UOP-only. **That is the measurement, not a failure**: it is what + /// phase 4's UOP reader exists for, and a probe that flagged it red would teach an /// operator to ignore the panel. + /// + /// **What is a failure is the middle column.** Phase 0 ran this without the validator + /// and read the library's answer as the truth, which made it report six of twelve + /// decoding. Two of those six — the elf ghosts — have an index entry of `length 0` and + /// were returning whatever body was decoded immediately before them, at that body's + /// exact dimensions. Four of twelve have art on a stock client, not six. /// private static void SectionPlayers() { @@ -582,14 +622,42 @@ namespace Server.Custom } int direction = Config.Get("Bridge.AssetProbePlayerDirection", 0); - int decoded = 0, absent = 0; + int real = 0, absent = 0, wrong = 0; + + string verdataPath = Files.GetFilePath("verdata.mul"); + long verdataLength = BridgeAssetValidator.MulLength(verdataPath); foreach (var pb in _playerBodies) { Checkpoint("players", pb.Body); - int translated = pb.Body; - int fileType = BodyConverter.Convert(ref translated); + int fileType, at; + string reason; + bool resolved = BridgeAssetValidator.ResolveAnimation( + pb.Body, 0, direction, out fileType, out at, out reason); + + // What the validator says BEFORE the library is asked. This is the whole point of + // the section: phase 0 reported "6 of 12 decode" from the library's answer alone, + // and two of those six were the previous body's picture. + var verdict = BridgeAssetValidator.Verdict.Absent; + + if (resolved) + { + string dataPath = BridgeAssetValidator.AnimDataPath(fileType); + var index = BridgeAssetValidator.OpenAnimIndex(fileType); + long length = BridgeAssetValidator.MulLength(dataPath); + + using (var reader = new BridgeAssetValidator.RecordReader(dataPath, verdataPath)) + { + verdict = BridgeAssetValidator.CheckEntry(index, at, length, verdataLength, out reason); + + if (verdict == BridgeAssetValidator.Verdict.Ok + && !reader.AnimationSane(index, at, 1, out reason)) + { + verdict = BridgeAssetValidator.Verdict.Refused; + } + } + } string outcome; @@ -597,17 +665,32 @@ namespace Server.Custom { int hue = 0; var frames = Animations.GetAnimation(pb.Body, 0, direction, ref hue, false, true); + bool gotBitmap = frames != null && frames.Length > 0 + && frames[0] != null && frames[0].Bitmap != null; + string size = null; - if (frames != null && frames.Length > 0 && frames[0] != null && frames[0].Bitmap != null) + if (gotBitmap) { var bmp = frames[0].Bitmap; - outcome = "decoded " + bmp.Width + "x" + bmp.Height; + size = bmp.Width + "x" + bmp.Height; bmp.Dispose(); - decoded++; + } + + if (verdict == BridgeAssetValidator.Verdict.Ok && gotBitmap) + { + outcome = "art, " + size; + real++; + } + else if (gotBitmap) + { + // The elf ghosts land here on a stock client: index entry `length 0`, and + // a bitmap the exact size of whatever was decoded last. + outcome = "WRONG PICTURE " + size + " — " + reason; + wrong++; } else { - outcome = "no art on the legacy path (UOP-only — phase 4)"; + outcome = "no art on the legacy path (UOP-only — phase 4): " + reason; absent++; } } @@ -617,11 +700,12 @@ namespace Server.Custom } Say(string.Format(" {0,-10} {1,-14} body {2,-5} fileType {3,-3} {4}", - pb.Race, pb.Slot, pb.Body, fileType, outcome)); + pb.Race, pb.Slot, pb.Body, resolved ? fileType.ToString() : "-", outcome)); } Say(""); - Say(string.Format(" {0} decoded, {1} absent, of {2}", decoded, absent, _playerBodies.Count)); + Say(string.Format(" {0} with art, {1} absent, {2} WRONG PICTURES, of {3}", + real, absent, wrong, _playerBodies.Count)); Say(""); } @@ -978,369 +1062,4 @@ namespace Server.Custom to.SendMessage(text); } } - - /// - /// **Validate before calling** — the response the org lead chose for §4.2's residual risk, - /// prototyped here so phase 1 adopts it with measurements rather than on faith. - /// - /// The principle: `Ultima`'s decoders take their bounds from the file they are reading, so - /// the extractor must decide whether a record is worth handing over *before* handing it - /// over. Every check below is against the index entry and the record header — cheap, and - /// enough to turn an uncatchable corrupted-state exception into a skipped asset. - /// - /// It cannot be complete and does not claim to be. It closes the shapes that reading the - /// source showed are reachable; the probe's REFUSED-BUT-DECODED count is what says whether - /// the boundary is drawn in the right place. - /// - /// Promoted into the overlay in phase 1. - /// - public static class BridgeAssetValidator - { - public enum Verdict - { - /// Nothing at this id, and the index says so honestly. - Absent, - - /// The entry is self-consistent and inside its file. - Ok, - - /// The entry claims something the file cannot support. Do not decode it. - Refused - } - - /// Land tiles decode a fixed 44×44 diamond: 2 × (2+4+…+44) ushorts. - public const int LandRecordBytes = 2024; - - /// - /// A ceiling on decoded art dimensions. `LoadStatic` allocates - /// new Bitmap(width, height) straight from two ushorts in the record, so a - /// corrupt header asks for up to 65535×65535 — an 8 GB allocation, from a file. Real - /// art is a couple of hundred pixels at most. - /// - public const int MaxArtDimension = 1024; - - /// - /// Builds our own index over the same files, with the same constructor arguments - /// Art uses — including hasExtra: false, which is the whole reason the - /// art path is safe where the gump path is not (§4.1). - /// - public static FileIndex OpenArtIndex() - { - if (ArtDataPath() == null) - return null; - - return new FileIndex("Artidx.mul", "Art.mul", "artLegacyMUL.uop", 0x10000, 4, ".tga", 0x13FDC, false); - } - - /// - /// The file an art index entry's lookup is an offset **into** — which is not - /// art.mul on any current client. - /// - /// This cost a whole probe run to learn and it is the single most important thing - /// phase 1 must not get wrong. FileIndex's UOP constructor ends with a bare - /// MulPath = uopPath: **when artLegacyMUL.uop exists it wins outright**, - /// and art.mul / artidx.mul are never opened at all. A validator that - /// bounds offsets against art.mul while the index holds UOP offsets is not - /// merely approximate, it is nonsense — the first run of this probe refused 34,299 - /// perfectly good statics for "declaring 10533x2085" because it was reading UOP - /// offsets into the wrong file. - /// - /// So the resolution order here mirrors FileIndex's exactly, and anything that - /// needs the bytes behind an entry must ask this rather than assume. - /// - public static string ArtDataPath() - { - var uop = Files.GetFilePath("artlegacymul.uop"); - - if (uop != null) - return uop; - - return Files.GetFilePath("art.mul"); - } - - public static long MulLength(string path) - { - if (path == null) - return 0; - - try - { - return new FileInfo(path).Length; - } - catch - { - return 0; - } - } - - /// - /// Judges one index entry. - /// - /// The check FileIndex.Seek is missing is the last one: it tests - /// Stream.Length < e.lookup — that the record *starts* inside the file — and - /// never that it *ends* inside it. A record that begins two bytes before EOF and - /// declares a length of 4,000 passes, and stream.Read then returns a short count - /// that the decoders discard, leaving the previous asset's bytes in the shared buffer. - /// - public static Verdict CheckEntry(FileIndex index, int at, long mulLength, long verdataLength, out string reason) - { - reason = null; - - if (index == null || index.Index == null || at < 0 || at >= index.Index.Length) - { - reason = "index " + at + " out of range"; - return Verdict.Absent; - } - - Entry3D e = index.Index[at]; - - if (e.lookup < 0) - { - reason = "lookup " + e.lookup; - return Verdict.Absent; - } - - bool patched = (e.length & (1 << 31)) != 0; - int length = e.length & 0x7FFFFFFF; - - if (!patched && e.length < 0) - { - reason = "length " + e.length; - return Verdict.Absent; - } - - if (length == 0) - { - reason = "lookup " + e.lookup + ", length 0"; - return Verdict.Absent; - } - - long ceiling = patched ? verdataLength : mulLength; - - if (ceiling <= 0) - { - reason = (patched ? "verdata.mul" : "the art data file") + " has no length"; - return Verdict.Refused; - } - - if (e.lookup >= ceiling) - { - reason = "lookup " + e.lookup + " past the end of " - + (patched ? "verdata.mul" : "the mul") + " (" + ceiling + ")"; - return Verdict.Refused; - } - - // The missing check. A short read is silent, and its consequence is the PREVIOUS - // asset's picture served under this id. - if (e.lookup + (long)length > ceiling) - { - reason = "record runs " + (e.lookup + (long)length - ceiling) + " bytes past the end of " - + (patched ? "verdata.mul" : "the mul"); - return Verdict.Refused; - } - - return Verdict.Ok; - } - - /// - /// `LoadLand` reads 2,024 bytes regardless of the declared length, so a shorter record - /// reads past the end of a buffer sized from that length. - /// - public static bool LandLengthSane(FileIndex index, int at, out string reason) - { - reason = null; - - if (index == null || index.Index == null || at < 0 || at >= index.Index.Length) - return true; - - int length = index.Index[at].length & 0x7FFFFFFF; - - if (length > 0 && length < LandRecordBytes) - { - reason = "land record is " + length + " bytes; LoadLand always reads " + LandRecordBytes; - return false; - } - - return true; - } - - /// - /// Walks a static record's own row table the way LoadStatic will, and refuses - /// it if that walk would read outside the record. - /// - /// This is the check with teeth. LoadStatic's inner loop guards the write into - /// the bitmap (xOffset > delta, xOffset + xRun > delta) and does - /// nothing at all about the read cursor, which advances until it happens to find a - /// zero pair — potentially far outside a pinned array. Simulating the same walk with - /// a bound is the cheapest way to know whether handing the id over is safe. - /// - public static bool StaticRecordSane(byte[] record, int length, out string reason) - { - reason = null; - - if (length < 8) - { - reason = "record is " + length + " bytes; a static header needs 8"; - return false; - } - - int words = length / 2; - int width = ReadUInt16(record, 4); - int height = ReadUInt16(record, 6); - - // LoadStatic returns null for these rather than misbehaving, so it is not a refusal. - if (width <= 0 || height <= 0) - return true; - - if (width > MaxArtDimension || height > MaxArtDimension) - { - reason = "declares " + width + "x" + height + ", past the " + MaxArtDimension + "px ceiling"; - return false; - } - - // The row-lookup table: height ushorts starting at word 4. - if (4 + height > words) - { - reason = "row table (" + height + " entries) does not fit in a " + length + "-byte record"; - return false; - } - - int start = height + 4; - - for (int y = 0; y < height; y++) - { - int cursor = start + ReadUInt16(record, (4 + y) * 2); - - while (true) - { - // Two ushorts for the run header, and they must both be inside the record. - if (cursor < 0 || cursor + 1 >= words) - { - reason = "row " + y + " reads at word " + cursor + ", past the record's " + words; - return false; - } - - int xOffset = ReadUInt16(record, cursor * 2); - int xRun = ReadUInt16(record, (cursor + 1) * 2); - cursor += 2; - - if (xOffset + xRun == 0) - break; - - // LoadStatic stops the row here, so the read cursor stops with it. - if (xOffset > width || xOffset + xRun > width) - break; - - if (cursor + xRun > words) - { - reason = "row " + y + " declares a " + xRun + "-pixel run running past the record"; - return false; - } - - cursor += xRun; - } - } - - return true; - } - - private static int ReadUInt16(byte[] b, int at) - { - return b[at] | (b[at + 1] << 8); - } - - /// - /// Reads a record's actual bytes so can walk it. - /// - /// Holds its own handles rather than borrowing the library's, because FileIndex - /// hands out the stream it decodes from and moving that stream's position underneath - /// the decoder would be its own bug. Opened FileShare.ReadWrite to match how - /// FileIndex opens the same files. - /// - public sealed class RecordReader : IDisposable - { - private readonly FileStream _mul; - private readonly FileStream _verdata; - private byte[] _scratch = new byte[64 * 1024]; - - public RecordReader(string mulPath, string verdataPath) - { - _mul = Open(mulPath); - _verdata = Open(verdataPath); - } - - private static FileStream Open(string path) - { - if (path == null || !File.Exists(path)) - return null; - - try - { - return new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite); - } - catch - { - return null; - } - } - - /// - /// True when the record at is safe to hand to - /// Art.GetStatic. A record that cannot be read at all is reported sane — - /// has already judged the entry, and this must not - /// invent a second reason to refuse. - /// - public bool StaticSane(FileIndex index, int at, out string reason) - { - reason = null; - - if (index == null || index.Index == null || at < 0 || at >= index.Index.Length) - return true; - - Entry3D e = index.Index[at]; - bool patched = (e.length & (1 << 31)) != 0; - int length = e.length & 0x7FFFFFFF; - - var stream = patched ? _verdata : _mul; - - if (stream == null || length <= 0 || e.lookup < 0) - return true; - - if (_scratch.Length < length) - _scratch = new byte[length]; - - int read; - - try - { - stream.Seek(e.lookup, SeekOrigin.Begin); - read = stream.Read(_scratch, 0, length); - } - catch (Exception ex) - { - reason = "cannot read the record: " + ex.GetType().Name; - return false; - } - - // The short read the decoders discard. Refusing here is the whole point: the - // library would decode whatever the shared buffer happened to hold. - if (read < length) - { - reason = "short read — " + read + " of " + length + " bytes available"; - return false; - } - - return StaticRecordSane(_scratch, length, out reason); - } - - public void Dispose() - { - if (_mul != null) - _mul.Dispose(); - - if (_verdata != null) - _verdata.Dispose(); - } - } - } } -- 2.49.1 From 73b07eed227ca95c832031d741d9f1e959075da8 Mon Sep 17 00:00:00 2001 From: wtclaude Date: Thu, 10 Sep 2026 11:12:43 -0500 Subject: [PATCH 03/11] feat(asset-bridge): the cliloc table, decompressed on the shard (Phase 2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The shard reads its own client's `Cliloc.enu` and serves it over the bridge, so the operator stops installing UOFiddler, building a converter against its `Ultima.dll`, and copying a 5 MB file to the web host every time they patch. `BridgeCliloc.cs` is the one decoder protocol 8 writes rather than calls (docs/link/v8.md §4, §9): a port of UOFiddler's `MythicDecompress` + `MoveToFront` — Beerware, so clean to bring into a GPL-3.0-or-later tree — rewritten against plain arrays, because the upstream is `Span` / `ArrayPool` / `BinaryPrimitives` code and ServUO targets `net48`. The algorithm is deliberately unchanged, including the parts that read oddly. The three-region count/cursor/end table and the symbol-table shifts are upstream's, because a tidier rewrite of somebody else's format decoder is a chance to be subtly wrong in a way that produces plausible text. Two bounds checks were added and they are the only behavioural difference: the upstream indexes its payload unchecked, which is safe for a file the client wrote and is not safe for a file this shard was handed. Measured on a stock client: 4,989,921 bytes read, decompressed and parsed in **290 ms**, yielding **67,496** non-blank rows in id order. That number is the acceptance test — it is what UOFiddler's own DLL produced from this same client through the converter this phase deletes, so an independent implementation agrees to the row. Zero U+FFFD; the 696 non-ASCII rows carry correct curly quotes; the longest row is a 12,149-character EULA, which is why the record length is read unsigned. Blanks never reach the wire — ~56,000 of the 123,490 entries are empty strings the client reserves, and the website discards them at import anyway. Also on this plane: * `assets.error` gains a `code`. Phase 1 chose between 403 and 400 by looking for the word "disabled" in an operator-facing sentence, which makes prose load-bearing; `DISABLED` / `NOT_FOUND` / `UNREADABLE` / `UNAVAILABLE` / `BAD_REQUEST` say it directly. * `Accept` and `Fail` are internal rather than private, because the asset plane's single slot and its refusal frame are shared by every family on it. The cursor is a cliloc NUMBER, not an offset: the decoded table is cached for five idle minutes and released after the last page, so it can be dropped and rebuilt between two pages of one import, and an index would then silently mean something else. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4 --- overlay/Scripts/Custom/Bridge/BridgeAssets.cs | 25 +- overlay/Scripts/Custom/Bridge/BridgeCliloc.cs | 767 ++++++++++++++++++ 2 files changed, 786 insertions(+), 6 deletions(-) create mode 100644 overlay/Scripts/Custom/Bridge/BridgeCliloc.cs diff --git a/overlay/Scripts/Custom/Bridge/BridgeAssets.cs b/overlay/Scripts/Custom/Bridge/BridgeAssets.cs index a3e4714..3cca03e 100644 --- a/overlay/Scripts/Custom/Bridge/BridgeAssets.cs +++ b/overlay/Scripts/Custom/Bridge/BridgeAssets.cs @@ -152,13 +152,13 @@ namespace Server.Custom.Bridge { // Rule 1. Without a correlation id this would land on the event path, be persisted // to the sidecar's store and broadcast to every subscriber. Refuse instead. - Fail(null, "assets.sources requires a reqId"); + Fail(null, "BAD_REQUEST", "assets.sources requires a reqId"); return; } if (!BridgeConfig.AssetsEnabled) { - Fail(reqId, "asset extraction is disabled on this shard"); + Fail(reqId, "DISABLED", "asset extraction is disabled on this shard"); return; } @@ -170,7 +170,7 @@ namespace Server.Custom.Bridge /// Runs on the Core thread and does nothing expensive; runs on /// the worker and must touch no world state. /// - private static void Accept(string reqId, string kind, Action job) + internal static void Accept(string reqId, string kind, Action job) { lock (_sync) { @@ -197,7 +197,7 @@ namespace Server.Custom.Bridge _job = null; Console.WriteLine("[Bridge] cannot start the asset worker: {0}", e.Message); - Fail(reqId, "the shard could not start its asset worker"); + Fail(reqId, "UNAVAILABLE", "the shard could not start its asset worker"); return; } } @@ -222,14 +222,27 @@ namespace Server.Custom.Bridge BridgeLink.Emit(sb.End()); } - private static void Fail(string reqId, string reason) + /// + /// The asset plane's one refusal frame, shared by every family on it. + /// + /// is what the sidecar maps to a status, and it exists because + /// the alternative it replaced — matching on the words in — + /// makes an operator-facing sentence load-bearing. Rewording "disabled" would silently + /// turn a 403 into a 400. The codes are `DISABLED` (the operator switched this plane + /// off), `NOT_FOUND` (the shard has no such file), `UNREADABLE` (it has it and cannot + /// decode it), `UNAVAILABLE` (the shard cannot do this right now) and `BAD_REQUEST` + /// (the default, and the caller's fault). + /// + internal static void Fail(string reqId, string code, string reason) { var sb = BridgeJson.Begin("assets.error"); if (reqId != null) sb.Str("reqId", reqId); - sb.Str("reason", reason); + sb.Str("code", code) + .Str("reason", reason); + BridgeLink.Emit(sb.End()); } diff --git a/overlay/Scripts/Custom/Bridge/BridgeCliloc.cs b/overlay/Scripts/Custom/Bridge/BridgeCliloc.cs new file mode 100644 index 0000000..1118769 --- /dev/null +++ b/overlay/Scripts/Custom/Bridge/BridgeCliloc.cs @@ -0,0 +1,767 @@ +using System; +using System.Collections.Generic; +using System.Globalization; +using System.IO; +using System.Text; + +using Ultima; + +namespace Server.Custom.Bridge +{ + /// + /// **The cliloc table, over the bridge** (docs/link/v8.md §9 — protocol 8, phase 2). + /// + /// A "cliloc" is UO's localization table: an integer id mapped to a display string. Items + /// on the wire carry a `LabelNumber`, never a name, so without this table the website can + /// only render `id 1023721` where the game renders "quarter staff". The number was never + /// the missing piece; the table was. + /// + /// Until this phase the operator supplied it by hand: install UOFiddler, build a converter + /// against its `Ultima.dll`, run it over their own `Cliloc.enu`, copy a 5 MB file to the + /// web host and point a setting at it. That whole pipeline existed for one reason — the + /// file is compressed and **nothing in this stack could read it**. ServUO's own bundled + /// `Ultima.StringList` implements the plain layout only and throws on a modern client's + /// file, which is also why the shard's `VendorSearch.GetItemName` has always been inert. + /// + /// So this class is the one decoder protocol 8 **writes** rather than calls (§4): a port + /// of UOFiddler's Mythic decompressor into the overlay, after which the shard can read its + /// own client's table and hand it to the website over the same request/reply path as + /// everything else. The operator installs nothing. + /// + /// **Attribution.** The decompression below is a port of `Ultima/Helpers/MythicDecompress` + /// and `MoveToFront` from UOFiddler (https://github.com/polserver/UOFiddler), which is + /// released under the **Beerware** licence — compatible with this tree's GPL-3.0-or-later. + /// It is rewritten for .NET Framework 4.8: the original is written against `Span<T>`, + /// `ArrayPool<T>` and `BinaryPrimitives`, none of which ServUO's `net48` target has. + /// + /// **What is NOT here, deliberately.** Shard-added items carry cliloc ids no client table + /// contains, and ServUO has no server-side notion of a custom cliloc — there is nothing in + /// the tree to read. That gap is in the *game*, not in this pipeline, so the website keeps + /// its `custom/` overlay directory and merges it over whatever arrives here. This class + /// answers exactly one question: what does the client's own table say. + /// + public static class BridgeCliloc + { + /// + /// Languages this can serve. + /// + /// Not an arbitrary code: Ultima.Files resolves only the names in its own file + /// table, and cliloc files are represented there by these four. Asking for anything + /// else cannot resolve to a path however the client is laid out, so it is refused by + /// name rather than answered with an empty table. + /// + /// `custom1` / `custom2` are the *client-side* custom cliloc files a shard ships to + /// its players. Nothing on the website imports them today — its `custom/` overlay + /// directory is the supported answer — but they are the shard's files and they are + /// readable, so they are not artificially excluded. + /// + private static readonly string[] Languages = { "enu", "deu", "custom1", "custom2" }; + + private const string DefaultLanguage = "enu"; + + /// + /// How long a decoded table is kept in memory after its last page. + /// + /// A stock `Cliloc.enu` decodes to ~67,000 live strings; holding that forever on a + /// shard that imports once a month is rude, and decoding it again costs about a + /// second. So it is cached only for as long as an import is plausibly still running: + /// freed when the last page is served, and expired on the next request if one never + /// comes (an import abandoned halfway leaves nothing behind). + /// + private static readonly TimeSpan CacheIdle = TimeSpan.FromMinutes(5); + + private static readonly object _sync = new object(); + private static Table _cached; + + public static void Initialize() + { + if (!BridgeConfig.Enabled) + return; + + BridgeBoot.RegisterHandler("cliloc.table", OnTable); + } + + // ── the request plane ──────────────────────────────────────────────────────────────── + + /// + /// Core thread. Validates, then hands the decode to the asset worker — reading and + /// decompressing five megabytes is emphatically not something to do while the world + /// is waiting, and 's single slot is what keeps the shard's + /// outbound queue at a depth of about one while it happens. + /// + private static void OnTable(Dictionary o) + { + var reqId = BridgeJson.GetString(o, "reqId"); + + if (reqId == null) + { + // Without a correlation id this reply would land on the event path, be persisted + // to the sidecar's store and broadcast to every subscriber — a megabyte of + // strings to every connected client, forever. Refuse instead (§3.1). + BridgeAssets.Fail(null, "BAD_REQUEST", "cliloc.table requires a reqId"); + return; + } + + if (!BridgeConfig.AssetsEnabled) + { + BridgeAssets.Fail(reqId, "DISABLED", "asset extraction is disabled on this shard"); + return; + } + + var lang = BridgeJson.GetString(o, "lang"); + + if (String.IsNullOrEmpty(lang)) + lang = DefaultLanguage; + + lang = lang.ToLowerInvariant(); + + if (Array.IndexOf(Languages, lang) < 0) + { + BridgeAssets.Fail(reqId, "NOT_FOUND", + "no cliloc file for language '" + lang + "' (this shard can serve: " + + String.Join(", ", Languages) + ")"); + return; + } + + // The cursor is this family's own resume point and it is a cliloc NUMBER, not an + // offset into anything. That matters: the cache behind it can be dropped and rebuilt + // between two pages of the same import (idle expiry, a second import, a restart), and + // an index into a list would silently mean something different afterwards. "Resume + // after id N" survives all of it, because the table is served in id order. + int after = -1; + var cursor = BridgeJson.GetString(o, "cursor"); + + if (!String.IsNullOrEmpty(cursor)) + { + if (!TryParseCursor(cursor, out after)) + { + BridgeAssets.Fail(reqId, "BAD_REQUEST", "malformed cursor: " + cursor); + return; + } + } + + string language = lang; + int resumeAfter = after; + + BridgeAssets.Accept(reqId, "cliloc.table", () => ReplyTable(reqId, language, resumeAfter)); + } + + private static bool TryParseCursor(string cursor, out int after) + { + after = -1; + + if (!cursor.StartsWith("n:", StringComparison.Ordinal)) + return false; + + return Int32.TryParse( + cursor.Substring(2), NumberStyles.Integer, CultureInfo.InvariantCulture, out after); + } + + /// + /// Asset worker. Decodes (or reuses) the table and writes one page of it. + /// + private static void ReplyTable(string reqId, string lang, int after) + { + string path = ResolvePath(lang); + + if (path == null) + { + BridgeAssets.Fail(reqId, "NOT_FOUND", + "this shard's client has no cliloc." + lang + " (looked where ServUO's own " + + "data path points)"); + return; + } + + Table table; + string code, reason; + + if (!TryLoad(lang, path, out table, out code, out reason)) + { + BridgeAssets.Fail(reqId, code, reason); + return; + } + + var sb = BridgeJson.Begin("cliloc.table.ok"); + + sb.Str("reqId", reqId) + .Str("lang", lang) + .Num("extractorVersion", BridgeAssets.EXTRACTOR_VERSION) + .Str("file", Path.GetFileName(path)) + // The website pages this table over several round trips and must be able to tell + // that the file changed underneath it — an operator patching their client mid-import + // would otherwise produce one table stitched from two, with no error anywhere. It + // compares these two fields across pages and starts over if they move. + .Num("size", table.Size) + .Num("mtime", table.MTime) + .Num("total", table.Count) + .Bool("compressed", table.Compressed); + + var page = new BridgeAssets.PageBuilder(sb, "rows", BridgeConfig.AssetBatchBytes); + + int start = table.IndexAfter(after); + int i = start; + + for (; i < table.Count; i++) + { + var item = new StringBuilder(96); + + item.Append("{\"n\":").Append(table.Numbers[i].ToString(CultureInfo.InvariantCulture)); + item.Append(",\"f\":").Append(table.Flags[i].ToString(CultureInfo.InvariantCulture)); + item.Append(",\"t\":"); + BridgeJson.Text(item, table.Texts[i]); + item.Append('}'); + + if (!page.TryAdd(item.ToString(), "n:" + table.Numbers[i].ToString(CultureInfo.InvariantCulture))) + break; + } + + page.Close(); + + bool finished = i >= table.Count; + + sb.Num("from", start); + BridgeLink.Emit(sb.End()); + + // The last page is also the end of the import, so let the strings go. A retry of that + // page re-decodes, which costs a second and happens approximately never; holding ~67k + // strings against that is the wrong trade. + if (finished) + Release(lang); + } + + private static string ResolvePath(string lang) + { + try + { + // ServUO's own `Scripts/Misc/DataPath.cs` calls `Files.SetMulPath` for every + // configured data directory at Configure time, so this resolves against the + // client the SHARD is running on — including on Linux, where `Ultima.Files`'s + // registry lookup finds nothing on its own. + return Files.GetFilePath("cliloc." + lang); + } + catch + { + return null; + } + } + + // ── the decoded table ──────────────────────────────────────────────────────────────── + + private sealed class Table + { + public string Lang; + public long Size; + public long MTime; + public bool Compressed; + public int[] Numbers; + public byte[] Flags; + public string[] Texts; + public DateTime LastUsed; + + public int Count { get { return Numbers.Length; } } + + /// + /// Index of the first row with a number greater than . + /// Binary search, because the rows are in id order by construction and a page + /// deep into the table would otherwise walk everything before it. + /// + public int IndexAfter(int after) + { + if (after < 0) + return 0; + + int lo = 0, hi = Numbers.Length; + + while (lo < hi) + { + int mid = lo + ((hi - lo) >> 1); + + if (Numbers[mid] <= after) + lo = mid + 1; + else + hi = mid; + } + + return lo; + } + } + + private static bool TryLoad(string lang, string path, out Table table, out string code, out string reason) + { + code = null; + reason = null; + + long size, mtime; + + try + { + var info = new FileInfo(path); + size = info.Length; + mtime = (long)(info.LastWriteTimeUtc - new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc)) + .TotalMilliseconds; + } + catch (Exception e) + { + table = null; + code = "UNREADABLE"; + reason = "cannot stat " + Path.GetFileName(path) + ": " + e.Message; + return false; + } + + lock (_sync) + { + if (_cached != null) + { + bool stale = _cached.Lang != lang + || _cached.Size != size + || _cached.MTime != mtime + || DateTime.UtcNow - _cached.LastUsed > CacheIdle; + + if (stale) + _cached = null; + } + + if (_cached != null) + { + _cached.LastUsed = DateTime.UtcNow; + table = _cached; + return true; + } + } + + byte[] raw; + + try + { + raw = File.ReadAllBytes(path); + } + catch (Exception e) + { + table = null; + code = "UNREADABLE"; + reason = "cannot read " + Path.GetFileName(path) + ": " + e.Message; + return false; + } + + bool compressed = IsCompressed(raw); + byte[] plain; + + if (compressed) + { + try + { + plain = Mythic.Decompress(raw); + } + catch (Exception e) + { + table = null; + code = "UNREADABLE"; + reason = "cannot decompress " + Path.GetFileName(path) + ": " + e.Message; + return false; + } + } + else + { + plain = raw; + } + + var built = new Table + { + Lang = lang, + Size = size, + MTime = mtime, + Compressed = compressed, + LastUsed = DateTime.UtcNow + }; + + if (!TryParseRecords(plain, built, out reason)) + { + table = null; + code = "UNREADABLE"; + return false; + } + + lock (_sync) + { + _cached = built; + } + + table = built; + return true; + } + + private static void Release(string lang) + { + lock (_sync) + { + if (_cached != null && _cached.Lang == lang) + _cached = null; + } + } + + /// + /// Every compressed cliloc begins with a DWORD whose high byte is 0x8E — the + /// top byte of UOFiddler's `HeaderXorKey`, showing through because the value it hides + /// (a length) is far smaller than the key. That single byte is what tells a modern + /// client's file from the pre-2010 plain layout, and both are accepted here: a shard + /// running an old or hand-built client is not a broken shard. + /// + private static bool IsCompressed(byte[] buffer) + { + return buffer.Length >= 4 && buffer[3] == 0x8E; + } + + // ── the plain layout ───────────────────────────────────────────────────────────────── + + private const int HeaderBytes = 6; // int32 version + int16 language marker + private const int RecordHeaderBytes = 7; // int32 number + byte flag + uint16 length + + /// + /// Parses the plain layout into the sorted, blank-free arrays the wire wants. + /// + /// **Strict about truncation**, and that strictness is the point: a half-decoded table + /// is indistinguishable from a complete one downstream — you would simply see some + /// items named and some not, which is exactly what "no table at all" looks like. So a + /// record running past the end of the buffer is an error naming its offset, never a + /// short table. + /// + /// **Blanks are dropped here rather than on the website.** Roughly 56,000 of a stock + /// table's 123,490 entries are empty strings the client reserves and never uses, the + /// website discards them at import already, and a row that resolves to no name is + /// indistinguishable from no row at all to every caller. Dropping them halves what + /// crosses the wire for data that would be thrown away on arrival. + /// + /// **A repeated id is resolved last-wins**, matching the client's own loader (its + /// dictionary assignment overwrites). The plain format permits it, so a file the game + /// itself would load happily must not fail here. + /// + private static bool TryParseRecords(byte[] data, Table into, out string reason) + { + reason = null; + + if (data.Length < HeaderBytes) + { + reason = "cliloc file is shorter than its 6-byte header"; + return false; + } + + var byNumber = new Dictionary(140000); + int offset = HeaderBytes; + int read = 0; + + while (offset < data.Length) + { + if (offset + RecordHeaderBytes > data.Length) + { + reason = "truncated record header at byte " + offset + " (" + read + " entries read)"; + return false; + } + + int number = ReadInt32(data, offset); + byte flag = data[offset + 4]; + // Unsigned: reading this signed (as ServUO's own SDK does) turns any string over + // 32 KB into a negative length. Real tables top out around 12 KB, so it changes + // nothing today and costs nothing to get right. + int length = data[offset + 5] | (data[offset + 6] << 8); + + offset += RecordHeaderBytes; + + if (offset + length > data.Length) + { + reason = "truncated record body at byte " + offset + " (" + read + " entries read)"; + return false; + } + + string text; + + try + { + text = Encoding.UTF8.GetString(data, offset, length); + } + catch (Exception e) + { + reason = "entry " + number + " at byte " + offset + " is not valid UTF-8: " + e.Message; + return false; + } + + offset += length; + read++; + + byNumber[number] = new Entry { Flag = flag, Text = text }; + } + + var numbers = new List(byNumber.Count); + + foreach (var pair in byNumber) + { + if (IsBlank(pair.Value.Text)) + continue; + + numbers.Add(pair.Key); + } + + numbers.Sort(); + + into.Numbers = numbers.ToArray(); + into.Flags = new byte[numbers.Count]; + into.Texts = new string[numbers.Count]; + + for (int i = 0; i < numbers.Count; i++) + { + var entry = byNumber[numbers[i]]; + + into.Flags[i] = entry.Flag; + into.Texts[i] = entry.Text; + } + + return true; + } + + private struct Entry + { + public byte Flag; + public string Text; + } + + private static bool IsBlank(string text) + { + if (String.IsNullOrEmpty(text)) + return true; + + for (int i = 0; i < text.Length; i++) + { + if (!Char.IsWhiteSpace(text[i])) + return false; + } + + return true; + } + + private static int ReadInt32(byte[] data, int at) + { + return data[at] | (data[at + 1] << 8) | (data[at + 2] << 16) | (data[at + 3] << 24); + } + + // ── the Mythic container ───────────────────────────────────────────────────────────── + + /// + /// The decompressor, ported from UOFiddler (Beerware; see this class's summary). + /// + /// The container is two stages over the plain cliloc bytes, undone in reverse: + /// + /// 1. A 4-byte header holding the decompressed length, XORed with `0x8E2C9A3D` — + /// which is where the `0x8E` sniff byte comes from. + /// 2. A **move-to-front** coding of… + /// 3. …a Burrows-Wheeler-style transform whose 1 KB frequency header (256 little-endian + /// counts, one per byte value) is both the table sizes and the total output length. + /// + /// Rewritten against plain arrays: the upstream is `Span<T>`/`ArrayPool<T>` + /// code and ServUO targets `net48`, which has neither without a package this tree does + /// not vendor. The algorithm is unchanged, including the parts that read oddly — the + /// three-region `partial` table (counts, cursors, ends) and the symbol-table shifts are + /// the original's, deliberately, because this is a format decoder and a tidier + /// rewrite is a chance to be subtly wrong about someone else's bytes. + /// + private static class Mythic + { + private const uint HeaderXorKey = 0x8E2C9A3D; + private const int FrequencyHeaderSize = 1024; // 256 little-endian ints + + public static byte[] Decompress(byte[] source) + { + if (source.Length < 4) + throw new InvalidDataException("compressed cliloc is shorter than its header"); + + uint declared = (uint)ReadInt32(source, 0) ^ HeaderXorKey; + + if (declared == 0 || declared > Int32.MaxValue) + throw new InvalidDataException("compressed cliloc declares an impossible length"); + + var mtf = new byte[source.Length - 4]; + MoveToFrontDecode(source, 4, mtf); + + var output = new byte[(int)declared]; + int written = InverseTransform(mtf, output); + + if (written != (int)declared) + { + throw new InvalidDataException( + "decompressed length " + written + " does not match the declared " + declared); + } + + return output; + } + + private static void MoveToFrontDecode(byte[] input, int from, byte[] output) + { + var symbols = new byte[256]; + + for (int i = 0; i < 256; i++) + symbols[i] = (byte)i; + + for (int i = 0; i < output.Length; i++) + { + int index = input[from + i]; + byte symbol = symbols[index]; + + output[i] = symbol; + + for (int j = index; j > 0; j--) + symbols[j] = symbols[j - 1]; + + symbols[0] = symbol; + } + } + + private static int InverseTransform(byte[] input, byte[] destination) + { + if (input.Length < FrequencyHeaderSize) + throw new InvalidDataException("compressed cliloc is smaller than its frequency header"); + + // Three regions of 256: [0..255] the counts read from the header, [256..511] a + // moving cursor per symbol, [512..767] where that symbol's run ends. + var partial = new int[256 * 3]; + + for (int i = 0; i < 256; i++) + partial[i] = ReadInt32(input, i * 4); + + int sum = 0; + + for (int i = 0; i < 256; i++) + { + if (partial[i] < 0) + throw new InvalidDataException("compressed cliloc has a negative symbol count"); + + sum += partial[i]; + } + + if (sum == 0) + return 0; + + if (destination.Length < sum) + throw new InvalidDataException("compressed cliloc's frequency header outruns its declared length"); + + int nonZero = 0; + + for (int i = 0; i < 256; i++) + { + if (partial[i] != 0) + nonZero++; + } + + var frequency = new byte[256]; + Frequency(partial, frequency); + + var symbols = new byte[256]; + + for (int i = 0; i < 256; i++) + symbols[i] = (byte)i; + + for (int i = 0, m = 0; i < nonZero; ++i) + { + byte freq = frequency[i]; + + Need(input, m + FrequencyHeaderSize); + + symbols[input[m + FrequencyHeaderSize]] = freq; + partial[freq + 256] = m + 1; + m += partial[freq]; + partial[freq + 512] = m; + } + + byte val = symbols[0]; + int count = 0; + + do + { + destination[count] = val; + + if (partial[val + 256] < partial[val + 512]) + { + Need(input, partial[val + 256] + FrequencyHeaderSize); + + byte idx = input[partial[val + 256] + FrequencyHeaderSize]; + partial[val + 256]++; + + if (idx != 0) + { + ShiftLeft(symbols, idx); + + symbols[idx] = val; + val = symbols[0]; + } + } + else if (nonZero-- > 0) + { + ShiftLeft(symbols, nonZero); + + val = symbols[0]; + } + + count++; + } + while (count < sum); + + return sum; + } + + /// + /// The upstream indexes the payload without bounds-checking it, which is safe for + /// a file the client wrote and is not safe for a file this shard was handed. A + /// truncated or hand-edited container would otherwise read whatever follows the + /// buffer in memory — or, on .NET, throw an `IndexOutOfRangeException` from inside + /// a decoder, which says nothing useful to an operator. This turns both into one + /// named, reportable failure. + /// + private static void Need(byte[] input, int at) + { + if (at < 0 || at >= input.Length) + throw new InvalidDataException("compressed cliloc ends mid-stream (wanted byte " + at + ")"); + } + + /// + /// Symbol values ordered by descending count — the order the coder assigned its + /// runs in. Repeated max-finding rather than a sort, as upstream: 256 passes over + /// 256 entries is nothing, and it reproduces the original's tie-breaking (the + /// lowest index wins), which a comparison sort would not. + /// + private static void Frequency(int[] counts, byte[] output) + { + var tmp = new int[256]; + Array.Copy(counts, tmp, 256); + + for (int i = 0; i < 256; i++) + { + int value = 0; + byte index = 0; + + for (int j = 0; j < 256; j++) + { + if (tmp[j] > value) + { + index = (byte)j; + value = tmp[j]; + } + } + + if (value == 0) + break; + + output[i] = index; + tmp[index] = 0; + } + } + + private static void ShiftLeft(byte[] symbols, int upTo) + { + for (int i = 0; i < upTo; ++i) + symbols[i] = symbols[i + 1]; + } + + private static int ReadInt32(byte[] data, int at) + { + return data[at] | (data[at + 1] << 8) | (data[at + 2] << 16) | (data[at + 3] << 24); + } + } + } +} -- 2.49.1 From cbdbc9fe5c32abed41ffc58c455d21dbb2708092 Mon Sep 17 00:00:00 2001 From: wtclaude Date: Thu, 10 Sep 2026 11:18:02 -0500 Subject: [PATCH 04/11] refactor(asset-bridge): write `from` before the page opens, not after it closes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `PageBuilder` keeps 256 bytes back for the envelope it still has to write, so a field appended after `Close()` is spent outside that reserve. It fits today by a wide margin — the largest measured page was 524,086 of a 524,288 budget, and the budget is half the line cap on purpose — but "nothing is written after Close()" is the invariant worth having, because the next family to page will copy this. No behaviour change; the field moves earlier in the same object. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4 --- overlay/Scripts/Custom/Bridge/BridgeCliloc.cs | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/overlay/Scripts/Custom/Bridge/BridgeCliloc.cs b/overlay/Scripts/Custom/Bridge/BridgeCliloc.cs index 1118769..8d53b9b 100644 --- a/overlay/Scripts/Custom/Bridge/BridgeCliloc.cs +++ b/overlay/Scripts/Custom/Bridge/BridgeCliloc.cs @@ -196,9 +196,15 @@ namespace Server.Custom.Bridge .Num("total", table.Count) .Bool("compressed", table.Compressed); + // Before the page opens, not after it closes: PageBuilder reserves room for the + // envelope it still has to write, and a field appended past Close() is spent outside + // that reserve. It fits today by a wide margin, and it is the kind of thing the next + // family copies. + int start = table.IndexAfter(after); + sb.Num("from", start); + var page = new BridgeAssets.PageBuilder(sb, "rows", BridgeConfig.AssetBatchBytes); - int start = table.IndexAfter(after); int i = start; for (; i < table.Count; i++) @@ -219,7 +225,6 @@ namespace Server.Custom.Bridge bool finished = i >= table.Count; - sb.Num("from", start); BridgeLink.Emit(sb.End()); // The last page is also the end of the import, so let the strings go. A retry of that -- 2.49.1 From 64c0ec00b1a3386abb33d7ac56fa57171bc79cbf Mon Sep 17 00:00:00 2001 From: wtclaude Date: Thu, 10 Sep 2026 18:40:12 -0500 Subject: [PATCH 05/11] feat(asset-bridge): the body catalogue and slug to body id (Phase 3) Two request families, and they run on opposite threads on purpose. `assets.bodies` (BridgeBodies) answers the question only code inside ServUO can: the atlas knows a creature by the class name in Spawns/*.xml, the client knows it by a body id, and nothing in the tree declares the mapping. Construct the type, read Body.BodyID, Delete(). That is world mutation, so it answers on the CORE thread and is the one family here that does not take the asset worker's slot -- and the batch is capped at 100 names, REFUSED rather than truncated, because a truncated answer is indistinguishable from a complete one from the website's side. `assets.manifest` / `assets.fetch` (BridgeCatalog) are the catalogue, on the worker. The manifest carries { key, sha256, bytes, width, height } and no pixels, so an Update fetches only what moved; the fetch carries base64 PNG. The scan keeps the bytes it hashed rather than decoding all 787 sprites twice. Three things worth stating about the shapes: - It pages on the WALL CLOCK as well as on bytes. The rows are ~90 bytes and the whole catalogue is one page by the byte budget, but building it means decoding hundreds of sprites against a 10 s reply timeout. - `catalog` is derived from the client files (sizes, mtimes, both direction settings, EXTRACTOR_VERSION), not minted per build -- the cache is released when idle, and a fresh id per build would force a restart mid-import although nothing about the client moved. - ARGB1555 is expanded to 32bpp here rather than handed to GDI+, because what it does with a one-bit alpha channel varies by platform and a black rectangle behind every sprite would pass any test that only checked the bytes decoded. Nothing trusts the library's success. Every body goes through CheckEntry and AnimationRecordSane before it is decoded, which is what keeps the 357 bodies whose index entry reads `length 0` -- and which the decoder hands back the PREVIOUS creature's bitmap for -- out of the catalogue. Walked on a live shard: 787 rows in one 734 ms page; bodies 320, 607, 666 all absent rather than wrong; 783 at direction 1 and 4 at direction 0; all 455 stock creature classes resolved at ~190 ms per 100 with zero mobiles leaked. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4 --- overlay/Config/Bridge.cfg | 33 + overlay/Scripts/Custom/Bridge/BridgeAssets.cs | 15 + overlay/Scripts/Custom/Bridge/BridgeBodies.cs | 254 +++++ overlay/Scripts/Custom/Bridge/BridgeBoot.cs | 1 + .../Scripts/Custom/Bridge/BridgeCatalog.cs | 1010 +++++++++++++++++ overlay/Scripts/Custom/Bridge/BridgeConfig.cs | 62 + 6 files changed, 1375 insertions(+) create mode 100644 overlay/Scripts/Custom/Bridge/BridgeBodies.cs create mode 100644 overlay/Scripts/Custom/Bridge/BridgeCatalog.cs diff --git a/overlay/Config/Bridge.cfg b/overlay/Config/Bridge.cfg index 5be83c3..9fc1141 100644 --- a/overlay/Config/Bridge.cfg +++ b/overlay/Config/Bridge.cfg @@ -307,6 +307,39 @@ AssetsEnabled=true # still fits. AssetBatchBytes=524288 +# How many ServUO class names one `assets.bodies` request may carry (phase 3). The only +# bound on this plane counted in items rather than bytes, because what it bounds is not +# reply size -- it is constructing and deleting that many real mobiles ON THE CORE +# THREAD, between two ticks of the world. A larger request is refused, never truncated. +# Clamped to [1, 500]. +AssetBodyBatch=100 + +# How many keys one `assets.fetch` request may name. The byte budget above still decides +# where a page is cut; this only bounds how large a request the shard will parse at all. +# Clamped to [1, 10000]. +AssetFetchKeys=2000 + +# The wall-clock budget for one catalogue page, in milliseconds. The catalogue's manifest +# rows are ~90 bytes so the byte budget never stops it -- but building them means +# decoding hundreds of animations, and the sidecar waits 10 s for a reply. Kept well +# under that, because the page still has to be serialised and written afterwards. +# Clamped to [250, 5000]. +AssetScanMs=3000 + +# Which direction the catalogue renders. NOT part of the asset key: five directions +# would five-fold every count in the working set to express a choice nobody varies. +# +# The split was found by RENDERING all five, not from a table. 0 is head-on, facing the +# viewer -- what a character portrait wants, and the least legible view there is of a +# four-legged creature (a wolf seen from the front is a dark blob). 1 is the front +# three-quarter, where the same wolf is unmistakably a wolf. +# +# Which bodies count as player bodies is asked of the shard (every registered race's +# male/female/ghost ids), never hardcoded. Clamped to [0, 4]: 5-7 are the client +# mirroring 1-3 through a decode branch this overlay has not verified. +AssetPlayerDirection=0 +AssetCreatureDirection=1 + # The test scaffolding in tools/scaffolding/ reads its own flags from this file # (SeedOnStart, CensusOnStart, ProbeOnStart). They are absent here on purpose: # Config.Get returns the default of false when a key is missing, so a deployed diff --git a/overlay/Scripts/Custom/Bridge/BridgeAssets.cs b/overlay/Scripts/Custom/Bridge/BridgeAssets.cs index 3cca03e..d809aa6 100644 --- a/overlay/Scripts/Custom/Bridge/BridgeAssets.cs +++ b/overlay/Scripts/Custom/Bridge/BridgeAssets.cs @@ -473,6 +473,21 @@ namespace Server.Custom.Bridge sb.Append('}'); } + /// + /// Whether this host can produce a picture, for the families that produce pictures. + /// + /// reports this on the source gate so an operator learns it + /// while setting the shard up. The catalogue needs the same answer as a *decision* — + /// it must refuse rather than throw a DllNotFoundException out of the middle of + /// a decode loop — so the check itself is shared and this is its one accessor. + /// + internal static bool ImagingOk(out string reason) + { + CheckImaging(); + reason = _imagingReason; + return _imagingOk; + } + private static void CheckImaging() { if (_imagingChecked) diff --git a/overlay/Scripts/Custom/Bridge/BridgeBodies.cs b/overlay/Scripts/Custom/Bridge/BridgeBodies.cs new file mode 100644 index 0000000..00b599f --- /dev/null +++ b/overlay/Scripts/Custom/Bridge/BridgeBodies.cs @@ -0,0 +1,254 @@ +using System; +using System.Collections.Generic; +using System.Globalization; +using System.Text; + +using Server.Mobiles; + +namespace Server.Custom.Bridge +{ + /// + /// **Slug to body id — the part only the shard can do** (docs/link/v8.md §8, protocol 8, + /// phase 3). + /// + /// The spawn atlas knows a creature by a **slug** derived from the class name it found in + /// `Spawns/*.xml` ("giant-spider"). The client knows the same creature by a **body id** + /// (28). Nothing in the ServUO tree declares that mapping as data. Today an operator + /// bridges it by hand, grepping `Scripts/Mobiles/Normal/<Name>.cs` for `Body =`, + /// which appears as a decimal, as hex (`0xD1`), as `Utility.RandomList(35, 36)` and as an + /// `m_IDs[]` table — a parse that is wrong on the shard's own custom creatures, which is + /// precisely the set an operator most wants pictures for. + /// + /// Inside ServUO the problem does not exist: construct the type, read `Body.BodyID`, + /// delete it. BridgeWorld.cs already does exactly that for a different feature. + /// + /// **This is the one asset-plane family that does NOT run on the asset worker**, and the + /// reason is the whole point of §8. Constructing and deleting a mobile is world mutation, + /// so it must happen on the Core thread — while the decode in + /// must happen off it, because it reads hundreds of megabytes and would stop the world for + /// every player on the shard. That split is why body resolution is its own request kind + /// rather than a step inside asset extraction. + /// + /// Two consequences follow from answering on the Core thread, and both are bounds: + /// + /// **The batch is small and the shard enforces the cap itself.** Every type constructed + /// here runs a real constructor — packing items, rolling skills, starting AI timers — and + /// all of that happens between two ticks of the world. The website chunks its own list; + /// a request over names is **refused** rather + /// than truncated, so the two sides cannot quietly disagree about what was answered. + /// + /// **It does not take the asset plane's single slot.** The slot exists to stop several + /// large replies queueing at once (§3.2); this reply is a few kilobytes and the work is + /// not on the worker, so claiming the slot would only make a body pass and a catalogue + /// page refuse each other for no benefit. + /// + /// **A creature whose constructor randomises its body reports one of its variants**, not + /// an error and not a set. Constructing twice to detect that would double every side + /// effect above to learn something the bestiary does not render differently — both ids are + /// the same creature. The answer is stable enough to cache and cheap enough to redo. + /// + public static class BridgeBodies + { + public static void Initialize() + { + if (!BridgeConfig.Enabled) + return; + + BridgeBoot.RegisterHandler("assets.bodies", OnBodies); + } + + // ── the request ────────────────────────────────────────────────────────────────────── + + private static void OnBodies(Dictionary o) + { + var reqId = BridgeJson.GetString(o, "reqId"); + + if (reqId == null) + { + // Rule 1 of the asset plane: without a correlation id this reply lands on the + // event path, is persisted to the sidecar's store and broadcast to every + // subscriber. Refuse rather than answer. + BridgeAssets.Fail(null, "BAD_REQUEST", "assets.bodies requires a reqId"); + return; + } + + if (!BridgeConfig.AssetsEnabled) + { + BridgeAssets.Fail(reqId, "DISABLED", "asset extraction is disabled on this shard"); + return; + } + + var types = BridgeJson.GetStringList(o, "types"); + + if (types.Count == 0) + { + BridgeAssets.Fail(reqId, "BAD_REQUEST", + "assets.bodies requires a non-empty `types` array of ServUO class names"); + return; + } + + if (types.Count > BridgeConfig.AssetBodyBatch) + { + // Refuse, never truncate. A silently shortened answer looks identical to a + // complete one from the website's side, and the types that fell off the end would + // be recorded as "asked and unanswerable" rather than "never asked". + BridgeAssets.Fail(reqId, "BAD_REQUEST", + "assets.bodies takes at most " + BridgeConfig.AssetBodyBatch + + " types per request (asked for " + types.Count + "); send them in chunks"); + return; + } + + Reply(reqId, types); + } + + /// + /// Core thread. Constructs each type once, reads its body, deletes it. + /// + /// Every outcome is a **row**, never a failed request: a shard is expected to be asked + /// about types it does not have (an atlas built from a tree that has since changed, a + /// spawn file naming a creature from a script package the operator removed), and a + /// status screen that fails the whole pass over one of those teaches an operator to + /// stop pressing the button. + /// + private static void Reply(string reqId, List types) + { + var sb = BridgeJson.Begin("assets.bodies.ok"); + + sb.Str("reqId", reqId) + .Num("extractorVersion", BridgeAssets.EXTRACTOR_VERSION) + .Num("asked", types.Count); + + // The envelope is shared with every other family (§3.4) even though this one never + // pages: the website drives the chunking, so `more` is always false and `cut` always + // "end". Writing it anyway means one reader shape on the other side rather than two. + var page = new BridgeAssets.PageBuilder(sb, "rows", BridgeConfig.AssetBatchBytes); + + int resolved = 0; + + foreach (var name in types) + { + string status; + int body; + + Resolve(name, out body, out status); + + if (status == "ok") + resolved++; + + var item = new StringBuilder(96); + + item.Append("{\"type\":"); + BridgeJson.Text(item, name); + item.Append(",\"status\":\"").Append(status).Append('"'); + + if (status == "ok") + item.Append(",\"body\":").Append(body.ToString(CultureInfo.InvariantCulture)); + + item.Append('}'); + + // A chunk this small cannot spend the budget — the cap above is a hundred names + // and the budget is half a megabyte — but the check costs nothing and the day + // someone raises `AssetBodyBatch` it is the difference between a short page and a + // line the sidecar drops. + if (!page.TryAdd(item.ToString(), null)) + break; + } + + page.Close(); + + sb.Num("resolved", resolved); + + BridgeLink.Emit(sb.End()); + } + + /// + /// One type name to one body id. + /// + /// `status` is the field the website records, and the four values are four different + /// things an operator can act on: + /// + /// ok — constructed, body read. + /// unknown — no such type on this shard. The spawn file names something the + /// scripts do not define, which is a real drift an operator wants to see. + /// notCreature — the type exists but is not a `BaseCreature`. Spawn files + /// legitimately name items and static decorations; those have no body and never will, + /// so this is a permanent answer rather than a retryable failure. + /// failed — the constructor threw, or the type has none that takes no + /// arguments. Caught per type, because one creature whose constructor depends on a + /// script package the operator removed must not cost the other ninety-nine. + /// + private static void Resolve(string name, out int body, out string status) + { + body = 0; + status = "failed"; + + Type type; + + try + { + // `true` is ignoreCase — spawn files are hand-edited and their casing drifts from + // the class it names far more often than the name itself does. + type = ScriptCompiler.FindTypeByName(name, true); + } + catch + { + status = "failed"; + return; + } + + if (type == null) + { + status = "unknown"; + return; + } + + if (!typeof(BaseCreature).IsAssignableFrom(type) || type.IsAbstract) + { + status = "notCreature"; + return; + } + + BaseCreature creature = null; + + try + { + creature = Activator.CreateInstance(type) as BaseCreature; + + if (creature == null) + { + status = "failed"; + return; + } + + body = creature.Body.BodyID; + status = body > 0 ? "ok" : "failed"; + } + catch (Exception e) + { + Console.WriteLine("[Bridge] assets.bodies: {0}: {1}: {2}", + name, e.GetType().Name, e.Message); + + status = "failed"; + } + finally + { + if (creature != null) + { + try + { + // Deleting the mobile deletes the items it packed — `Mobile.Delete` walks + // `Items`, and `Item.Delete` walks what each contains — and stops its AI + // timer. A creature left alive here is a creature standing at (0,0,0) on + // the internal map forever, saved with the world, once per import. + creature.Delete(); + } + catch + { + // Nothing useful is left to do, and throwing out of `finally` would lose + // whatever the try block was already reporting. + } + } + } + } + } +} diff --git a/overlay/Scripts/Custom/Bridge/BridgeBoot.cs b/overlay/Scripts/Custom/Bridge/BridgeBoot.cs index d09dc43..1511e8f 100644 --- a/overlay/Scripts/Custom/Bridge/BridgeBoot.cs +++ b/overlay/Scripts/Custom/Bridge/BridgeBoot.cs @@ -261,6 +261,7 @@ namespace Server.Custom.Bridge e.Mobile.SendMessage("Bridge: {0}", BridgeWorld.Status()); e.Mobile.SendMessage("Bridge: {0}", BridgeOneShots.Status()); e.Mobile.SendMessage("Bridge: {0}", BridgeAssets.Status()); + e.Mobile.SendMessage("Bridge: {0}", BridgeCatalog.Status()); break; } } diff --git a/overlay/Scripts/Custom/Bridge/BridgeCatalog.cs b/overlay/Scripts/Custom/Bridge/BridgeCatalog.cs new file mode 100644 index 0000000..00b3ad0 --- /dev/null +++ b/overlay/Scripts/Custom/Bridge/BridgeCatalog.cs @@ -0,0 +1,1010 @@ +using System; +using System.Collections.Generic; +using System.Drawing; +using System.Drawing.Imaging; +using System.Globalization; +using System.IO; +using System.Runtime.InteropServices; +using System.Security.Cryptography; +using System.Text; + +using Ultima; + +namespace Server.Custom.Bridge +{ + /// + /// **The body catalogue** (docs/link/v8.md §4.8, §5, §6, §11 — protocol 8, phase 3). + /// + /// One thumbnail per creature body: the working set that makes a bestiary, a marketplace + /// listing and a character sheet render. Everything deeper — every action, every frame — + /// is the same addressing scheme at a deeper key, fetched on demand in a later phase; this + /// is the set that is worth importing before anything asks for it, because on this + /// machine's client it is **787 sprites at about a kilobyte each**. + /// + /// Two request kinds, which are §6's two stages for assets rather than for sources: + /// + /// assets.manifest — `[{ key, sha256, bytes, width, height }]`, no pixels. The + /// website diffs it against what it already holds and asks only for what changed. That is + /// the whole difference between an Update and a re-download. + /// + /// assets.fetch — the pixels, for an explicit list of keys. + /// + /// ── **Why the manifest builds the pictures it refuses to send** ── + /// + /// A manifest row carries a hash of the bytes, and the only way to hash bytes is to have + /// them. So the scan decodes, encodes to PNG and hashes, then sends the row and **keeps + /// the bytes** — a megabyte for the whole catalogue, against re-decoding all 787 sprites a + /// second time when the fetch arrives moments later. + /// + /// ── **Why the manifest pages on TIME rather than on bytes** ── + /// + /// Every other family on this plane pages because its rows are large. This one's rows are + /// ninety bytes and the whole catalogue is one page by the byte budget — but producing + /// that page means decoding 787 animations, and the sidecar gives a reply ten seconds + /// (§3.3). So the scan carries a **wall-clock budget** as well + /// () and cuts the page `limit` when it is spent, + /// resuming from its cursor on the next call. The byte budget is still enforced, because + /// the day a family's rows grow is not the day to discover only one of the two bounds was + /// real. + /// + /// ── **Why `catalog` is derived from the sources and not minted per build** ── + /// + /// A manifest walk and the fetch that follows it must be talking about the same client + /// files, or the website stitches one catalogue out of two. The obvious answer is a fresh + /// id per build, and it is wrong: this cache is released when it goes idle, so a rebuild + /// halfway through a slow import would change the id and force a restart although nothing + /// about the client moved. So the id is a hash of what actually decides the bytes — every + /// anim file's size and mtime, both direction settings and + /// . It is stable across a rebuild and it + /// changes exactly when an operator patches their client. + /// + /// ── **The never-sweep rule, and the 357** ── + /// + /// Nothing here asks a file type for an index it does not own, and nothing here trusts the + /// library's own success. takes + /// `BodyConverter.Convert`'s answer and reports nothing if it leads nowhere (sweeping + /// instead puts a giant spider on the gargoyle page, decoding cleanly); and every body is + /// put through and + /// RecordReader.AnimationSane **before** it is decoded, because a body whose index + /// entry reads `length 0` gets a bitmap back anyway — the previously-decoded creature's, + /// from the library's reused stream buffer. That is 357 of the 1,144 bodies the library + /// claims on a stock client, and importing them would have written 357 duplicate + /// portraits whose subject depended on the order this walk happened to run in. + /// + public static class BridgeCatalog + { + /// The only family this phase serves. §5's key scheme covers the rest. + private const string Family = "body"; + + /// Bodies are addressable to 2047; the sweep behind §4.8 covered exactly this. + private const int MaxBody = 2047; + + /// The catalogue is first frames only. Deep keys are phase 6. + private const int CatalogAction = 0; + + public static void Initialize() + { + if (!BridgeConfig.Enabled) + return; + + BridgeBoot.RegisterHandler("assets.manifest", OnManifest); + BridgeBoot.RegisterHandler("assets.fetch", OnFetch); + } + + // ── the cache ──────────────────────────────────────────────────────────────────────── + + private sealed class Sprite + { + public string Key; + public int Body; + public int Direction; + public int FileType; + public string Sha256; + public byte[] Png; + public int Width; + public int Height; + } + + private sealed class Catalog + { + public string Id; + public readonly Dictionary ByKey = + new Dictionary(StringComparer.Ordinal); + public readonly List Order = new List(); + + /// The next body the scan has yet to look at. + public int Next = 1; + + public bool Complete; + public DateTime LastUsed; + } + + private static readonly object _sync = new object(); + private static Catalog _catalog; + + private static readonly TimeSpan IdleFor = TimeSpan.FromMinutes(5); + + // ── assets.manifest ────────────────────────────────────────────────────────────────── + + private static void OnManifest(Dictionary o) + { + string reqId; + + if (!Admit(o, "assets.manifest", out reqId)) + return; + + var family = BridgeJson.GetString(o, "family") ?? Family; + + if (!String.Equals(family, Family, StringComparison.Ordinal)) + { + // Named rather than ignored: `family` exists so §5's statics and land can join + // this envelope in phase 5 without a second request kind, and a website that + // asked for one of those against a phase-3 overlay must be told it asked too + // early rather than handed a body catalogue it did not request. + BridgeAssets.Fail(reqId, "BAD_REQUEST", + "this shard serves the '" + Family + "' asset family only (asked for '" + + family + "')"); + return; + } + + var cursor = BridgeJson.GetString(o, "cursor"); + + BridgeAssets.Accept(reqId, "assets.manifest", () => ReplyManifest(reqId, cursor)); + } + + /// + /// Worker thread. Scans forward from the cursor until the byte budget or the time + /// budget is spent, hashing what it decodes and keeping the bytes for the fetch. + /// + private static void ReplyManifest(string reqId, string cursor) + { + string imagingReason; + + if (!BridgeAssets.ImagingOk(out imagingReason)) + { + // Never a stack trace and never a 500: on a Linux host without libgdiplus this is + // the expected outcome, and it is actionable in one line (§4.4). + BridgeAssets.Fail(reqId, "UNAVAILABLE", + "this shard host cannot render images - Mono's System.Drawing needs " + + "libgdiplus. Install it (apt-get install libgdiplus) and re-run the import. " + + "Cliloc and atlas import are unaffected. (" + imagingReason + ")"); + return; + } + + string id = SourceId(); + + Catalog catalog; + + lock (_sync) + { + if (_catalog == null || _catalog.Id != id) + _catalog = new Catalog { Id = id }; + + catalog = _catalog; + catalog.LastUsed = DateTime.UtcNow; + } + + int from = ParseBodyCursor(cursor); + + var sb = BridgeJson.Begin("assets.manifest.ok"); + + sb.Str("reqId", reqId) + .Str("family", Family) + // What the website compares across pages, and across the fetch that follows. A + // change means the operator patched their client mid-import and the half already + // read describes files that no longer exist. + .Str("catalog", catalog.Id) + .Num("extractorVersion", BridgeAssets.EXTRACTOR_VERSION) + .Num("maxBody", MaxBody) + .Num("from", from); + + WritePlayerBodies(sb); + + var page = new BridgeAssets.PageBuilder(sb, "rows", BridgeConfig.AssetBatchBytes); + + int scanned = 0; + int last = from - 1; + bool timedOut = false; + bool budgetCut = false; + + var deadline = DateTime.UtcNow.AddMilliseconds(BridgeConfig.AssetScanMs); + + using (var readers = new Readers()) + { + int body = from; + + for (; body <= MaxBody; body++) + { + // Checked before the body rather than after it, so the budget bounds the reply + // rather than the reply plus one more decode. One sprite is milliseconds; the + // ceiling this lives under is ten seconds and the cost of overshooting it is + // the whole page, retried. + if (body > from && DateTime.UtcNow >= deadline) + { + timedOut = true; + break; + } + + scanned++; + last = body; + + Sprite sprite = Resolve(catalog, readers, body); + + if (sprite == null) + continue; + + var item = new StringBuilder(128); + + item.Append("{\"key\":"); + BridgeJson.Text(item, sprite.Key); + item.Append(",\"sha256\":\"").Append(sprite.Sha256).Append('"'); + item.Append(",\"bytes\":").Append(sprite.Png.Length.ToString(CultureInfo.InvariantCulture)); + item.Append(",\"width\":").Append(sprite.Width.ToString(CultureInfo.InvariantCulture)); + item.Append(",\"height\":").Append(sprite.Height.ToString(CultureInfo.InvariantCulture)); + item.Append(",\"body\":").Append(sprite.Body.ToString(CultureInfo.InvariantCulture)); + item.Append(",\"direction\":").Append(sprite.Direction.ToString(CultureInfo.InvariantCulture)); + item.Append('}'); + + if (!page.TryAdd(item.ToString(), "b:" + body.ToString(CultureInfo.InvariantCulture))) + { + // The budget stopped this page BEFORE this body's row went on it, so the + // next page must resume AT this body rather than after it. Getting this + // one line wrong drops exactly one creature from the catalogue per page, + // which nothing downstream could ever notice. + budgetCut = true; + last = body - 1; + scanned--; + break; + } + } + } + + if (timedOut) + page.Cut("limit"); + + // The walk reached the end of the addressable range without either budget stopping it. + // Derived from the two flags rather than from the row count, because a page that ends + // exactly on a boundary is indistinguishable from a finished one by count alone — + // §3.4's whole argument for `cut` existing. + bool finished = !timedOut && !budgetCut && last >= MaxBody; + + int held; + + lock (_sync) + { + if (_catalog == catalog) + { + catalog.Next = Math.Max(catalog.Next, last + 1); + catalog.LastUsed = DateTime.UtcNow; + + if (finished) + catalog.Complete = true; + } + + held = catalog.Order.Count; + } + + page.Close(); + + // Past Close(), which is normally the mistake BridgeCliloc's `from` comment warns + // about — but these three are not knowable until the scan has run, and they cost + // about fifty bytes against PageBuilder's 256-byte reserve, of which Close() itself + // spends around forty. Anything larger than this belongs before the page opens. + sb.Num("scanned", scanned) + .Num("held", held) + .Bool("complete", finished); + + BridgeLink.Emit(sb.End()); + + Sweep(); + } + + // ── assets.fetch ───────────────────────────────────────────────────────────────────── + + private static void OnFetch(Dictionary o) + { + string reqId; + + if (!Admit(o, "assets.fetch", out reqId)) + return; + + var keys = BridgeJson.GetStringList(o, "keys"); + + if (keys.Count == 0) + { + BridgeAssets.Fail(reqId, "BAD_REQUEST", + "assets.fetch requires a non-empty `keys` array"); + return; + } + + if (keys.Count > BridgeConfig.AssetFetchKeys) + { + BridgeAssets.Fail(reqId, "BAD_REQUEST", + "assets.fetch takes at most " + BridgeConfig.AssetFetchKeys + + " keys per request (asked for " + keys.Count + ")"); + return; + } + + var catalog = BridgeJson.GetString(o, "catalog"); + var cursor = BridgeJson.GetString(o, "cursor"); + + BridgeAssets.Accept(reqId, "assets.fetch", () => ReplyFetch(reqId, keys, catalog, cursor)); + } + + private static void ReplyFetch(string reqId, List keys, string expected, string cursor) + { + string imagingReason; + + if (!BridgeAssets.ImagingOk(out imagingReason)) + { + BridgeAssets.Fail(reqId, "UNAVAILABLE", + "this shard host cannot render images - Mono's System.Drawing needs " + + "libgdiplus. (" + imagingReason + ")"); + return; + } + + string id = SourceId(); + + if (expected != null && expected != id) + { + // The client files moved between the manifest and this fetch. Refusing is the only + // honest answer: the keys were chosen against a catalogue that no longer describes + // what is on disk, and serving them would mix two clients in one import with no + // error anywhere. + BridgeAssets.Fail(reqId, "UNREADABLE", + "the shard's client files changed since that manifest was read (catalog " + + expected + " is now " + id + "); start the import again"); + return; + } + + Catalog catalog; + + lock (_sync) + { + if (_catalog == null || _catalog.Id != id) + _catalog = new Catalog { Id = id }; + + catalog = _catalog; + catalog.LastUsed = DateTime.UtcNow; + } + + int from = ParseKeyCursor(cursor); + + var sb = BridgeJson.Begin("assets.fetch.ok"); + + sb.Str("reqId", reqId) + .Str("family", Family) + .Str("catalog", catalog.Id) + .Num("extractorVersion", BridgeAssets.EXTRACTOR_VERSION) + .Num("asked", keys.Count) + .Num("from", from); + + var page = new BridgeAssets.PageBuilder(sb, "rows", BridgeConfig.AssetBatchBytes); + + int i = from; + + using (var readers = new Readers()) + { + for (; i < keys.Count; i++) + { + var item = Render(catalog, readers, keys[i]); + + if (!page.TryAdd(item, "k:" + (i + 1).ToString(CultureInfo.InvariantCulture))) + break; + } + } + + page.Close(); + + sb.Num("sent", page.Count); + + BridgeLink.Emit(sb.End()); + + Sweep(); + } + + /// + /// One key to one row, with the bytes. + /// + /// A key this shard cannot serve is a **row**, not a failed request: the website asked + /// for a list, and one key naming a body whose art this client does not carry must not + /// cost the other four hundred. `status` distinguishes the two ways that happens — + /// `absent` (this client has no art at that key, the expected answer for two thirds of + /// the player bodies) and `unsupported` (a key shape this phase does not serve, which + /// is a website bug rather than a client gap). + /// + private static string Render(Catalog catalog, Readers readers, string key) + { + int body; + + if (!TryParseKey(key, out body)) + { + var bad = new StringBuilder(96); + bad.Append("{\"key\":"); + BridgeJson.Text(bad, key); + bad.Append(",\"status\":\"unsupported\"}"); + return bad.ToString(); + } + + Sprite sprite = Resolve(catalog, readers, body); + + var item = new StringBuilder(2048); + + item.Append("{\"key\":"); + BridgeJson.Text(item, key); + + if (sprite == null) + { + item.Append(",\"status\":\"absent\"}"); + return item.ToString(); + } + + item.Append(",\"status\":\"ok\""); + item.Append(",\"sha256\":\"").Append(sprite.Sha256).Append('"'); + item.Append(",\"bytes\":").Append(sprite.Png.Length.ToString(CultureInfo.InvariantCulture)); + item.Append(",\"width\":").Append(sprite.Width.ToString(CultureInfo.InvariantCulture)); + item.Append(",\"height\":").Append(sprite.Height.ToString(CultureInfo.InvariantCulture)); + item.Append(",\"body\":").Append(sprite.Body.ToString(CultureInfo.InvariantCulture)); + item.Append(",\"direction\":").Append(sprite.Direction.ToString(CultureInfo.InvariantCulture)); + item.Append(",\"png\":\"").Append(Convert.ToBase64String(sprite.Png)).Append("\"}"); + + return item.ToString(); + } + + // ── decode ─────────────────────────────────────────────────────────────────────────── + + /// + /// The catalogue entry for one body, decoded and hashed on first sight and cached + /// after. Returns null when this client has no art for it — which is an ordinary + /// answer for well over half of the addressable range, not a failure. + /// + private static Sprite Resolve(Catalog catalog, Readers readers, int body) + { + string key = Key(body); + + lock (_sync) + { + Sprite cached; + + if (catalog.ByKey.TryGetValue(key, out cached)) + return cached; + } + + int direction = IsPlayerBody(body) + ? BridgeConfig.AssetPlayerDirection + : BridgeConfig.AssetCreatureDirection; + + int fileType, at; + string reason; + + if (!BridgeAssetValidator.ResolveAnimation(body, CatalogAction, direction, + out fileType, out at, out reason)) + return null; + + FileIndex index = readers.Index(fileType); + + if (index == null) + return null; + + if (BridgeAssetValidator.CheckEntry(index, at, readers.MulLength(fileType), + readers.VerdataLength, out reason) != BridgeAssetValidator.Verdict.Ok) + { + // The `length 0` case lands here, and it is the 357. The library would hand back + // the previously-decoded body's bitmap for every one of them. + return null; + } + + var reader = readers.Reader(fileType); + + if (reader == null) + return null; + + // `maxFrames: 1` because that is what `firstFrame: true` decodes. Checking frames + // nobody reads would invent refusals, and a checker that refuses real art is worse + // than no checker at all. + if (!reader.AnimationSane(index, at, 1, out reason)) + return null; + + Sprite sprite; + + try + { + sprite = Decode(key, body, direction, fileType); + } + catch (Exception e) + { + Console.WriteLine("[Bridge] catalogue: body {0}: {1}: {2}", + body, e.GetType().Name, e.Message); + return null; + } + + if (sprite == null) + return null; + + lock (_sync) + { + if (!catalog.ByKey.ContainsKey(key)) + { + catalog.ByKey[key] = sprite; + catalog.Order.Add(sprite); + } + + return catalog.ByKey[key]; + } + } + + private static Sprite Decode(string key, int body, int direction, int fileType) + { + int hue = 0; + + // `preserveHue: false` — the catalogue is the creature's own art, and a body-level hue + // from Body.def belongs to a specific mob rather than to the species. §5's key scheme + // is where a hued variant is expressed (`static/3922/h33`), not here. + Frame[] frames = Animations.GetAnimation(body, CatalogAction, direction, ref hue, false, true); + + if (frames == null || frames.Length == 0 || frames[0] == null) + return null; + + Bitmap bitmap = frames[0].Bitmap; + + if (bitmap == null || bitmap.Width <= 0 || bitmap.Height <= 0) + return null; + + byte[] png = ToPng(bitmap); + + if (png == null) + return null; + + return new Sprite + { + Key = key, + Body = body, + Direction = direction, + FileType = fileType, + Png = png, + Width = bitmap.Width, + Height = bitmap.Height, + Sha256 = Hash(png) + }; + } + + /// + /// ARGB1555 to a PNG with a transparent background. + /// + /// Frame writes 16-bit ARGB1555: a pixel the sprite does not cover is left as + /// zero and a pixel it does cover carries the top bit set. Saving that format straight + /// to PNG asks GDI+ to make the conversion, and what it does with a one-bit alpha + /// channel varies by platform — on Mono it is a different implementation entirely. A + /// sprite that came back with a black rectangle behind it would look fine in a test + /// that only checked the bytes decoded, and wrong on every page that showed it. + /// + /// So the expansion is done here, explicitly: alpha bit clear becomes fully + /// transparent, and each 5-bit channel is widened to 8 bits by repeating its high bits + /// ((c << 3) | (c >> 2)) rather than by shifting alone, which would + /// cap white at 248 and tint the whole catalogue. + /// + private static byte[] ToPng(Bitmap source) + { + var rect = new Rectangle(0, 0, source.Width, source.Height); + + if (source.PixelFormat != PixelFormat.Format16bppArgb1555) + { + // Not what this library has ever produced. Save it rather than reinterpret it: + // guessing at an unknown layout is how a catalogue fills with confident nonsense. + using (var ms = new MemoryStream()) + { + source.Save(ms, ImageFormat.Png); + return ms.ToArray(); + } + } + + using (var target = new Bitmap(source.Width, source.Height, PixelFormat.Format32bppArgb)) + { + BitmapData src = source.LockBits(rect, ImageLockMode.ReadOnly, PixelFormat.Format16bppArgb1555); + BitmapData dst = null; + + try + { + dst = target.LockBits(rect, ImageLockMode.WriteOnly, PixelFormat.Format32bppArgb); + + var line = new short[source.Width]; + var outLine = new int[source.Width]; + + for (int y = 0; y < source.Height; y++) + { + Marshal.Copy(new IntPtr(src.Scan0.ToInt64() + ((long)y * src.Stride)), + line, 0, source.Width); + + for (int x = 0; x < source.Width; x++) + { + int p = line[x] & 0xFFFF; + + if ((p & 0x8000) == 0) + { + outLine[x] = 0; + continue; + } + + int r = (p >> 10) & 0x1F; + int g = (p >> 5) & 0x1F; + int b = p & 0x1F; + + outLine[x] = unchecked((int)0xFF000000) + | (((r << 3) | (r >> 2)) << 16) + | (((g << 3) | (g >> 2)) << 8) + | ((b << 3) | (b >> 2)); + } + + Marshal.Copy(outLine, 0, new IntPtr(dst.Scan0.ToInt64() + ((long)y * dst.Stride)), + source.Width); + } + } + finally + { + if (dst != null) + target.UnlockBits(dst); + + source.UnlockBits(src); + } + + using (var ms = new MemoryStream()) + { + target.Save(ms, ImageFormat.Png); + return ms.ToArray(); + } + } + } + + private static string Hash(byte[] bytes) + { + using (var sha = SHA256.Create()) + { + byte[] digest = sha.ComputeHash(bytes); + var sb = new StringBuilder(digest.Length * 2); + + foreach (byte b in digest) + sb.Append(b.ToString("x2", CultureInfo.InvariantCulture)); + + return sb.ToString(); + } + } + + // ── player bodies (§5.2) ───────────────────────────────────────────────────────────── + + /// + /// Asked of the shard, never hardcoded. + /// + /// Every registered race carries four body ids, and a shard that calls `RegisterRace` + /// adds ids no table of ours could contain. Even on stock ServUO a hardcoded list + /// would already be wrong in a way that is easy to miss: `RaceDefinitions.cs` passes + /// the gargoyle's ghost bodies in the OPPOSITE order to the other two races. + /// + /// This set is the whole of what §5.1 gives direction 0 — head-on, facing the viewer, + /// because a character is a portrait and should look at you. Everything else takes + /// direction 1, the front three-quarter, because head-on is the least legible view of + /// a four-legged creature: a wolf seen from the front is a dark blob. + /// + /// + /// Cached for the life of the process: `RegisterRace` runs at Configure time, before + /// anything on this plane can be asked a question, so the set cannot change under a + /// running shard. Rebuilding it per body would enumerate every race 2,047 times per + /// scan to answer a question whose answer is twelve integers. + /// + private static HashSet _playerBodies; + + private static HashSet PlayerBodies() + { + var cached = _playerBodies; + + if (cached != null) + return cached; + + var set = new HashSet(); + + try + { + foreach (var race in Race.AllRaces) + { + if (race == null) + continue; + + set.Add(race.MaleBody); + set.Add(race.FemaleBody); + set.Add(race.MaleGhostBody); + set.Add(race.FemaleGhostBody); + } + } + catch (Exception e) + { + Console.WriteLine("[Bridge] catalogue: cannot enumerate races: {0}", e.Message); + } + + set.Remove(0); + + _playerBodies = set; + + return set; + } + + private static bool IsPlayerBody(int body) + { + return PlayerBodies().Contains(body); + } + + private static void WritePlayerBodies(StringBuilder sb) + { + var bodies = new List(PlayerBodies()); + bodies.Sort(); + + sb.Append(",\"playerBodies\":["); + + for (int i = 0; i < bodies.Count; i++) + { + if (i > 0) + sb.Append(','); + + sb.Append(bodies[i].ToString(CultureInfo.InvariantCulture)); + } + + sb.Append(']'); + } + + // ── keys, cursors and the source id ────────────────────────────────────────────────── + + private static string Key(int body) + { + return "body/" + body.ToString(CultureInfo.InvariantCulture) + + "/a" + CatalogAction.ToString(CultureInfo.InvariantCulture); + } + + /// + /// `body/<id>/a0`, and nothing else in this phase. A deeper key + /// (`body/400/a2/f3`) is well-formed under §5 and simply not served yet, so it comes + /// back `unsupported` rather than being silently read as its own first frame. + /// + private static bool TryParseKey(string key, out int body) + { + body = 0; + + if (key == null) + return false; + + string[] parts = key.Split('/'); + + if (parts.Length != 3 || parts[0] != "body") + return false; + + if (!Int32.TryParse(parts[1], NumberStyles.None, CultureInfo.InvariantCulture, out body)) + return false; + + if (body < 1 || body > MaxBody) + return false; + + return parts[2] == "a" + CatalogAction.ToString(CultureInfo.InvariantCulture); + } + + private static int ParseBodyCursor(string cursor) + { + if (cursor == null) + return 1; + + int value; + + if (cursor.StartsWith("b:", StringComparison.Ordinal) + && Int32.TryParse(cursor.Substring(2), NumberStyles.None, CultureInfo.InvariantCulture, out value)) + return Math.Max(1, value + 1); + + return 1; + } + + private static int ParseKeyCursor(string cursor) + { + if (cursor == null) + return 0; + + int value; + + if (cursor.StartsWith("k:", StringComparison.Ordinal) + && Int32.TryParse(cursor.Substring(2), NumberStyles.None, CultureInfo.InvariantCulture, out value)) + return Math.Max(0, value); + + return 0; + } + + /// + /// Everything that decides the bytes, hashed into one short id. + /// + /// Deliberately (size, mtime) rather than content: §6 makes exactly the same choice + /// for the source gate, and for the same reason — the anim files are 195 MB and + /// hashing them on every page of a walk would turn a manifest into a minute. + /// `assets.sources` is where an operator gets content hashes, computed off the request + /// path; this is a "did it move while I was reading" check, which (size, mtime) + /// answers. + /// + private static string SourceId() + { + var sb = new StringBuilder(256); + + sb.Append(BridgeAssets.EXTRACTOR_VERSION) + .Append(':').Append(BridgeConfig.AssetPlayerDirection) + .Append(':').Append(BridgeConfig.AssetCreatureDirection); + + for (int fileType = 1; fileType <= 5; fileType++) + { + sb.Append('|'); + + string path = BridgeAssetValidator.AnimDataPath(fileType); + + if (path == null) + continue; + + try + { + var info = new FileInfo(path); + + if (!info.Exists) + continue; + + sb.Append(info.Length).Append(',').Append(info.LastWriteTimeUtc.Ticks); + } + catch + { + // An unreadable file is itself a state, and one that must not change from page + // to page without being noticed. Leaving the slot empty does that. + } + } + + return Hash(Encoding.UTF8.GetBytes(sb.ToString())).Substring(0, 16); + } + + // ── shared plumbing ────────────────────────────────────────────────────────────────── + + /// + /// The two gates every request on this plane passes: a correlation id, and the + /// operator's consent. Both refuse rather than answer. + /// + private static bool Admit(Dictionary o, string kind, out string reqId) + { + reqId = BridgeJson.GetString(o, "reqId"); + + if (reqId == null) + { + BridgeAssets.Fail(null, "BAD_REQUEST", kind + " requires a reqId"); + return false; + } + + if (!BridgeConfig.AssetsEnabled) + { + BridgeAssets.Fail(reqId, "DISABLED", "asset extraction is disabled on this shard"); + return false; + } + + return true; + } + + /// + /// The five anim files' index and record readers, opened for one reply and closed with + /// it. Holding them across replies would keep handles on the operator's client files + /// for as long as the cache lives, for no gain: opening five is microseconds and a + /// page decodes hundreds of sprites through them. + /// + private sealed class Readers : IDisposable + { + private readonly FileIndex[] _index = new FileIndex[6]; + private readonly BridgeAssetValidator.RecordReader[] _reader = + new BridgeAssetValidator.RecordReader[6]; + private readonly long[] _length = new long[6]; + private readonly bool[] _open = new bool[6]; + + public readonly long VerdataLength; + + public Readers() + { + VerdataLength = BridgeAssetValidator.MulLength(VerdataPath()); + } + + private static string VerdataPath() + { + try + { + return Files.GetFilePath("verdata.mul"); + } + catch + { + return null; + } + } + + private void Ensure(int fileType) + { + if (fileType < 1 || fileType > 5 || _open[fileType]) + return; + + _open[fileType] = true; + + string path = BridgeAssetValidator.AnimDataPath(fileType); + + if (path == null) + return; + + try + { + _index[fileType] = BridgeAssetValidator.OpenAnimIndex(fileType); + _length[fileType] = BridgeAssetValidator.MulLength(path); + _reader[fileType] = new BridgeAssetValidator.RecordReader(path, VerdataPath()); + } + catch (Exception e) + { + Console.WriteLine("[Bridge] catalogue: anim file type {0}: {1}", + fileType, e.Message); + } + } + + public FileIndex Index(int fileType) + { + Ensure(fileType); + return fileType >= 1 && fileType <= 5 ? _index[fileType] : null; + } + + public long MulLength(int fileType) + { + Ensure(fileType); + return fileType >= 1 && fileType <= 5 ? _length[fileType] : 0; + } + + public BridgeAssetValidator.RecordReader Reader(int fileType) + { + Ensure(fileType); + return fileType >= 1 && fileType <= 5 ? _reader[fileType] : null; + } + + public void Dispose() + { + for (int i = 1; i <= 5; i++) + { + if (_reader[i] == null) + continue; + + try + { + _reader[i].Dispose(); + } + catch + { + // Closing a read-only handle. Nothing useful is left to do. + } + } + } + } + + /// + /// Lets a megabyte of the operator's client art go once nothing has asked for it in + /// five minutes. A rebuild costs one scan and, because is + /// derived from the files rather than minted per build, it produces the same catalogue + /// id — so an import that spans the drop does not have to start over. + /// + private static void Sweep() + { + lock (_sync) + { + if (_catalog == null) + return; + + if (DateTime.UtcNow - _catalog.LastUsed > IdleFor) + _catalog = null; + } + } + + public static string Status() + { + lock (_sync) + { + if (_catalog == null) + return "catalog(empty)"; + + return String.Format("catalog(id={0} held={1} next={2} complete={3})", + _catalog.Id, _catalog.Order.Count, _catalog.Next, _catalog.Complete); + } + } + } +} diff --git a/overlay/Scripts/Custom/Bridge/BridgeConfig.cs b/overlay/Scripts/Custom/Bridge/BridgeConfig.cs index 0f140f4..dede1bd 100644 --- a/overlay/Scripts/Custom/Bridge/BridgeConfig.cs +++ b/overlay/Scripts/Custom/Bridge/BridgeConfig.cs @@ -97,6 +97,33 @@ namespace Server.Custom.Bridge public static bool AssetsEnabled { get; private set; } public static int AssetBatchBytes { get; private set; } + // How many types one `assets.bodies` request may name (§8, phase 3). This is the ONLY + // asset-plane bound counted in items rather than bytes, and deliberately so: the cost it + // bounds is not the size of the reply, it is constructing and deleting that many real + // mobiles ON THE CORE THREAD, between two ticks of the world. + public static int AssetBodyBatch { get; private set; } + + // How many keys one `assets.fetch` request may name. Bytes still cut the page; this only + // bounds how large a request the shard will parse and walk at all. + public static int AssetFetchKeys { get; private set; } + + // The wall-clock budget for one catalogue page (§4.8, phase 3). The catalogue's rows are + // ninety bytes, so the byte budget never stops it -- but building them means decoding + // hundreds of animations, and the sidecar gives a reply ten seconds. Kept well under that, + // because the reply still has to be built, serialised and cross the wire afterwards. + public static int AssetScanMs { get; private set; } + + // Which direction the catalogue renders (§5.1). Both are settings and neither is in the + // asset key, because five directions would five-fold every count in §11 to express a + // choice nobody is going to vary. + // + // The split is not arbitrary and was found by RENDERING all five rather than from a table: + // index 0 is head-on, which is what a character portrait wants and the least legible view + // there is of a four-legged creature. A wolf seen from the front is a dark blob; at index + // 1, the front three-quarter, it is unmistakably a wolf. + public static int AssetPlayerDirection { get; private set; } + public static int AssetCreatureDirection { get; private set; } + public static int LeaseMaxDurationSec { get; private set; } public static int LeaseGraceSec { get; private set; } @@ -168,6 +195,33 @@ namespace Server.Custom.Bridge if (AssetBatchBytes > 512 * 1024) AssetBatchBytes = 512 * 1024; + AssetBodyBatch = Config.Get("Bridge.AssetBodyBatch", 100); + if (AssetBodyBatch < 1) + AssetBodyBatch = 1; + if (AssetBodyBatch > 500) + AssetBodyBatch = 500; + + AssetFetchKeys = Config.Get("Bridge.AssetFetchKeys", 2000); + if (AssetFetchKeys < 1) + AssetFetchKeys = 1; + if (AssetFetchKeys > 10000) + AssetFetchKeys = 10000; + + AssetScanMs = Config.Get("Bridge.AssetScanMs", 3000); + if (AssetScanMs < 250) + AssetScanMs = 250; + // Half the sidecar's 10 s reply timeout, so the page still has time to be serialised + // and written after the scan stops. A budget set at the timeout would produce replies + // that are always thrown away. + if (AssetScanMs > 5000) + AssetScanMs = 5000; + + // Clamped to 0-4: 5-7 are the client MIRRORING 1-3, which `Frame` decodes through a + // different pointer-arithmetic branch that nothing in BridgeAssetValidator has + // checked. Accepting one would hand an unverified write path a bitmap to fill. + AssetPlayerDirection = Clamp(Config.Get("Bridge.AssetPlayerDirection", 0), 0, 4); + AssetCreatureDirection = Clamp(Config.Get("Bridge.AssetCreatureDirection", 1), 0, 4); + StatSweepSeconds = Config.Get("Bridge.StatSweepSeconds", 30); DecaySweepSeconds = Config.Get("Bridge.DecaySweepSeconds", 60); EconomySweepSeconds = Config.Get("Bridge.EconomySweepSeconds", 300); @@ -518,6 +572,14 @@ namespace Server.Custom.Bridge return fallback; } + private static int Clamp(int value, int min, int max) + { + if (value < min) + return min; + + return value > max ? max : value; + } + public static string Describe() { return String.Format( -- 2.49.1 From efbd45685ce6526c2b4a51ffc1e43e4af78d208b Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 11 Sep 2026 04:55:11 -0500 Subject: [PATCH 06/11] feat(asset-bridge): the UOP animation reader, and 235 bodies the legacy path cannot see (Phase 4) ServUO's vendored `Ultima.Animations` reads legacy `anim*.mul` only -- it builds its five FileIndexes with the constructor that passes `uopFile: null` -- so everything a modern client moved into `AnimationFrame*.uop` is invisible to it. This adds the one reader docs/link/v8.md 4.3 reserved for phase 4, and wires it in as a fallback beneath the legacy path. What it actually recovers is not what the plan expected, and the difference was measured before any of this was written: - Of the EIGHT player-character bodies 4.8 assigned to this phase, only TWO are in the client at all: gargoyles 666 and 667, in AnimationFrame3.uop. The six ghost bodies (human 402/403, elf 607/608, gargoyle 694/695) are in no package. The five packages hold 10,724 entries between them and the `build/animationlegacyframe/%06d/%02d.bin` name scheme claims every one, so there is no other naming they could be hiding under. - The same fallback reaches 233 further bodies the catalogue had nothing for, so the working set goes from 787 to 1,022 (57 Monster, 26 Animal, 97 Equipment, 50 unlisted, 3 Human, 2 Sea). The catalogue was already 366 Equipment bodies before this, so its character does not change. Decided with the org lead before building: the fallback applies to every body rather than to player bodies alone; ghost ids leave the player-body set entirely (no client has art for any of them, and listing them only advertised keys that cannot exist); the UOP path gets its own PNG encoder rather than Bitmap.Save; and a host without libgdiplus keeps the flat NO_IMAGING refusal rather than serving a partial catalogue. Three things about the reader: - It is not the never-sweep rule being broken. That rule exists because a legacy index is addressed by POSITION, so asking the wrong file returns a giant spider that decodes cleanly. A UOP entry is addressed by the hash of a name containing the body id, and the payload declares that id again for `Group.TryOpen` to check, so a hit is proof of identity. Measured: no hash appears in two packages. - Validate-as-we-go replaces validate-before-calling, because here we ARE the library: the block chain is bounded against the file, the record against the file, the inflated length against the declared one, the frame table against the payload, and every run header against both the record's remaining bytes and the bitmap it writes into. Measured the way 4.5 was -- across every UOP body on a stock client it refuses nothing that carries art. The one body it refuses (286) declares a 0x0 frame, which the vendored decoder treats as absent too. - No System.Drawing anywhere in it, which is what 4.4 promised: the decode fills a ushort[] of ARGB1555 and BridgePng encodes that directly (zlib around net48's raw-deflate-only DeflateStream, CRC32, one IDAT, filter 0). EXTRACTOR_VERSION 1 -> 2: every client file is byte-identical and the answer is different, which is exactly what that number exists to say. The UOP packages join `assets.sources` and the catalogue id, so patching one is drift; `Ultima.Files` cannot resolve them (its table predates UOP animations) so BridgeUop.FindClientFile does it, case-insensitively by enumeration for Linux hosts. Manifest and fetch rows carry a new `source` field (`legacy` / `uop`). Protocol stays 8 -- no message shape changed, only fields added. Measured on the live rig (real sidecar, real ServUO, this machine's client): 1,022 rows in ONE page, 1,409 ms cold; six player bodies, all six with art for the first time (400/401/605/606 legacy, 666/667 uop), all at direction 0; 1,016 at direction 1; the six ghost bodies absent; 45 duplicate-hash groups of which exactly one is new, bodies 1531/1532, two distinct records whose first frames match -- legitimate, and provable only because each payload declares its own body id. The gargoyles were rendered and looked at, because 4.3's whole point is that this failure mode produces confident, wrong pictures. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4 --- overlay/Scripts/Custom/Bridge/BridgeAssets.cs | 24 +- .../Scripts/Custom/Bridge/BridgeCatalog.cs | 231 ++++- overlay/Scripts/Custom/Bridge/BridgePng.cs | 231 +++++ overlay/Scripts/Custom/Bridge/BridgeUop.cs | 823 ++++++++++++++++++ 4 files changed, 1281 insertions(+), 28 deletions(-) create mode 100644 overlay/Scripts/Custom/Bridge/BridgePng.cs create mode 100644 overlay/Scripts/Custom/Bridge/BridgeUop.cs diff --git a/overlay/Scripts/Custom/Bridge/BridgeAssets.cs b/overlay/Scripts/Custom/Bridge/BridgeAssets.cs index d809aa6..9bde8cb 100644 --- a/overlay/Scripts/Custom/Bridge/BridgeAssets.cs +++ b/overlay/Scripts/Custom/Bridge/BridgeAssets.cs @@ -76,8 +76,14 @@ namespace Server.Custom.Bridge /// Bump it whenever extraction changes what it produces from unchanged input. It is /// the same rule spawnAtlasSource.js's `PARSER_VERSION` follows, and it applies /// here more rather than less: this pipeline derives far more from far less. + /// + /// **2** — phase 4 (§4.3, §4.9). The catalogue now falls back to + /// AnimationFrame*.uop for bodies the legacy path has nothing for, which on a + /// stock client is 235 new sprites and two of them player-character bodies; and the + /// player-body set no longer carries ghost ids. Every client file is byte-identical + /// and the answer is different, which is precisely what this number exists to say. /// - public const int EXTRACTOR_VERSION = 1; + public const int EXTRACTOR_VERSION = 2; // ── the one slot (§3.2) ────────────────────────────────────────────────────────────── @@ -325,6 +331,8 @@ namespace Server.Custom.Bridge "anim3.idx", "anim3.mul", "anim4.idx", "anim4.mul", "anim5.idx", "anim5.mul", + "animationframe1.uop", "animationframe2.uop", "animationframe3.uop", + "animationframe4.uop", "animationframe6.uop", "body.def", "bodyconv.def", "hues.mul", "verdata.mul" @@ -425,12 +433,22 @@ namespace Server.Custom.Bridge { try { - return Files.GetFilePath(name); + string path = Files.GetFilePath(name); + + if (path != null) + return path; } catch { - return null; + // Ultima's lookup reads the registry on Windows; a host where that throws still + // has the directories ServUO itself booted from, which is what the fallback uses. } + + // `Ultima.Files` has a fixed table of file names that predates UOP animations, so it + // answers null for every `AnimationFrame*.uop` however present they are (§4.3). Phase + // 4 added those to this list, so the fallback is what makes their size, mtime and hash + // reachable at all. + return BridgeUop.FindClientFile(name); } private static long ToUnixMs(DateTime utc) diff --git a/overlay/Scripts/Custom/Bridge/BridgeCatalog.cs b/overlay/Scripts/Custom/Bridge/BridgeCatalog.cs index 00b3ad0..20f19fd 100644 --- a/overlay/Scripts/Custom/Bridge/BridgeCatalog.cs +++ b/overlay/Scripts/Custom/Bridge/BridgeCatalog.cs @@ -19,7 +19,9 @@ namespace Server.Custom.Bridge /// listing and a character sheet render. Everything deeper — every action, every frame — /// is the same addressing scheme at a deeper key, fetched on demand in a later phase; this /// is the set that is worth importing before anything asks for it, because on this - /// machine's client it is **787 sprites at about a kilobyte each**. + /// machine's client it is **1,022 sprites at about a kilobyte each** — 787 out of the + /// legacy `anim*.mul` files and, since phase 4, 235 more out of `AnimationFrame*.uop`, + /// which ServUO's vendored decoder never opens (§4.3, §4.9). /// /// Two request kinds, which are §6's two stages for assets rather than for sources: /// @@ -70,6 +72,17 @@ namespace Server.Custom.Bridge /// from the library's reused stream buffer. That is 357 of the 1,144 bodies the library /// claims on a stock client, and importing them would have written 357 duplicate /// portraits whose subject depended on the order this walk happened to run in. + /// + /// ── **The UOP fallback, and why it cannot reintroduce that** ── + /// + /// Phase 4 added beneath the legacy reader: a body the vendored + /// path has nothing for is looked for in the UOP packages before it is reported absent. + /// That is where two of the six player-character bodies live — `Bodyconv.def` sends + /// gargoyles 666 and 667 to `anim5`, at an index past the end of `anim5.idx` — and 233 + /// other bodies besides. It cannot produce a wrong picture the way a legacy sweep would, + /// because a UOP entry is addressed by the hash of a name that contains the body id and + /// the payload then declares that id again, which + /// checks. /// public static class BridgeCatalog { @@ -103,6 +116,15 @@ namespace Server.Custom.Bridge public byte[] Png; public int Width; public int Height; + + /// + /// Which reader produced it: `legacy` for ServUO's vendored Animations over + /// anim*.mul, `uop` for phase 4's own reader over + /// AnimationFrame*.uop (§4.3, §4.9). On the wire so that an operator + /// looking at a wrong picture can tell which half of the extractor to doubt, and + /// so the acceptance walk can prove the fallback fired at all. + /// + public string Source; } private sealed class Catalog @@ -243,6 +265,7 @@ namespace Server.Custom.Bridge item.Append(",\"height\":").Append(sprite.Height.ToString(CultureInfo.InvariantCulture)); item.Append(",\"body\":").Append(sprite.Body.ToString(CultureInfo.InvariantCulture)); item.Append(",\"direction\":").Append(sprite.Direction.ToString(CultureInfo.InvariantCulture)); + item.Append(",\"source\":\"").Append(sprite.Source).Append('"'); item.Append('}'); if (!page.TryAdd(item.ToString(), "b:" + body.ToString(CultureInfo.InvariantCulture))) @@ -446,6 +469,7 @@ namespace Server.Custom.Bridge item.Append(",\"height\":").Append(sprite.Height.ToString(CultureInfo.InvariantCulture)); item.Append(",\"body\":").Append(sprite.Body.ToString(CultureInfo.InvariantCulture)); item.Append(",\"direction\":").Append(sprite.Direction.ToString(CultureInfo.InvariantCulture)); + item.Append(",\"source\":\"").Append(sprite.Source).Append('"'); item.Append(",\"png\":\"").Append(Convert.ToBase64String(sprite.Png)).Append("\"}"); return item.ToString(); @@ -474,6 +498,34 @@ namespace Server.Custom.Bridge ? BridgeConfig.AssetPlayerDirection : BridgeConfig.AssetCreatureDirection; + // Legacy first, always. The vendored decoder is what 787 of this client's bodies come + // out of, it is what phase 3 measured, and the UOP packages hold a different and + // mostly disjoint set (measured: of the 244 bodies they carry, 8 also have legacy + // art). So this is a fallback rather than a choice, and no body changes reader while + // a client sits still. + Sprite sprite = ResolveLegacy(key, readers, body, direction) + ?? ResolveUop(key, readers, body, direction); + + if (sprite == null) + return null; + + lock (_sync) + { + if (!catalog.ByKey.ContainsKey(key)) + { + catalog.ByKey[key] = sprite; + catalog.Order.Add(sprite); + } + + return catalog.ByKey[key]; + } + } + + /// + /// ServUO's vendored Animations over anim*.mul, behind §4.5's validator. + /// + private static Sprite ResolveLegacy(string key, Readers readers, int body, int direction) + { int fileType, at; string reason; @@ -505,11 +557,9 @@ namespace Server.Custom.Bridge if (!reader.AnimationSane(index, at, 1, out reason)) return null; - Sprite sprite; - try { - sprite = Decode(key, body, direction, fileType); + return Decode(key, body, direction, fileType); } catch (Exception e) { @@ -517,20 +567,93 @@ namespace Server.Custom.Bridge body, e.GetType().Name, e.Message); return null; } + } - if (sprite == null) - return null; + /// + /// Phase 4's own reader over AnimationFrame*.uop (§4.3, §4.9), for the bodies + /// the legacy path has nothing for. + /// + /// On this machine's client that is **235 bodies** the catalogue could not reach + /// before, including the two gargoyle player bodies — `Bodyconv.def` sends 666 and 667 + /// to `anim5`, at an index past the end of `anim5.idx`, and the art has been in + /// `AnimationFrame3.uop` all along. + /// + /// Nothing here can produce §4.8's failure. A legacy index is addressed by position, + /// so a wrong lookup is another creature's picture; a UOP entry is addressed by the + /// hash of a name carrying the body id, and the payload repeats that id in its own + /// header for to check. A miss is a miss. + /// + private static Sprite ResolveUop(string key, Readers readers, int body, int direction) + { + ulong hash = BridgeUop.HashOf(body, CatalogAction); - lock (_sync) + byte[] payload = null; + string reason = null; + + foreach (int n in BridgeUop.Packages) { - if (!catalog.ByKey.ContainsKey(key)) + BridgeUop.Package package = readers.Package(n); + + if (package == null || !package.Has(hash)) + continue; + + if (!package.TryRead(hash, out payload, out reason)) { - catalog.ByKey[key] = sprite; - catalog.Order.Add(sprite); + Console.WriteLine("[Bridge] catalogue: body {0} in {1}: {2}", + body, BridgeUop.PackageName(n), reason); + return null; } - return catalog.ByKey[key]; + break; } + + if (payload == null) + return null; + + BridgeUop.Group group; + + if (!BridgeUop.Group.TryOpen(payload, body, out group, out reason)) + { + Console.WriteLine("[Bridge] catalogue: body {0} uop: {1}", body, reason); + return null; + } + + int frame = group.DirectionAt(direction); + + if (frame < 0) + return null; + + BridgeUop.Pixels pixels; + bool empty; + + if (!group.TryDecode(frame, out pixels, out empty, out reason)) + { + // A 0x0 frame is no art rather than damage — the vendored decoder returns early on + // exactly the same condition — so it is absent, silently. Anything else is a + // record this reader refused, and that is worth a line. + if (!empty) + Console.WriteLine("[Bridge] catalogue: body {0} uop: {1}", body, reason); + + return null; + } + + byte[] png = BridgePng.FromArgb1555(pixels.Argb1555, pixels.Width, pixels.Height); + + if (png == null) + return null; + + return new Sprite + { + Key = key, + Body = body, + Direction = direction, + FileType = 0, + Png = png, + Width = pixels.Width, + Height = pixels.Height, + Sha256 = Hash(png), + Source = "uop" + }; } private static Sprite Decode(string key, int body, int direction, int fileType) @@ -564,7 +687,8 @@ namespace Server.Custom.Bridge Png = png, Width = bitmap.Width, Height = bitmap.Height, - Sha256 = Hash(png) + Sha256 = Hash(png), + Source = "legacy" }; } @@ -674,21 +798,32 @@ namespace Server.Custom.Bridge /// /// Asked of the shard, never hardcoded. /// - /// Every registered race carries four body ids, and a shard that calls `RegisterRace` - /// adds ids no table of ours could contain. Even on stock ServUO a hardcoded list - /// would already be wrong in a way that is easy to miss: `RaceDefinitions.cs` passes - /// the gargoyle's ghost bodies in the OPPOSITE order to the other two races. + /// Every registered race carries its living male and female body ids, and a shard that + /// calls `RegisterRace` adds ids no table of ours could contain — which is the whole + /// argument against a hardcoded list, and it was never hypothetical: stock ServUO's + /// own `RaceDefinitions.cs` passes the gargoyle's ghost bodies in the OPPOSITE order + /// to the other two races. /// /// This set is the whole of what §5.1 gives direction 0 — head-on, facing the viewer, /// because a character is a portrait and should look at you. Everything else takes /// direction 1, the front three-quarter, because head-on is the least legible view of /// a four-legged creature: a wolf seen from the front is a dark blob. + /// + /// **Ghost bodies are deliberately not in it** (§5.2, decided 2026-09-10 in phase 4). + /// A race declares four ids and two of them are its ghosts, and no UO client has art + /// for any of them: 402/403 and 694/695 read `lookup -1` in `anim.idx`, 607/608 read + /// `length 0` — the §4.8 shape, where the library hands back the previously-decoded + /// body's picture — and none of the six is in any `AnimationFrame*.uop`, which phase 4 + /// established by claiming all 10,724 entries of the five packages with the one name + /// scheme. Listing them only advertised keys that cannot exist. A shard whose client + /// does ship ghost art still gets it: the body is catalogued like any other, at + /// direction 1 rather than 0. /// /// /// Cached for the life of the process: `RegisterRace` runs at Configure time, before /// anything on this plane can be asked a question, so the set cannot change under a /// running shard. Rebuilding it per body would enumerate every race 2,047 times per - /// scan to answer a question whose answer is twelve integers. + /// scan to answer a question whose answer is six integers. /// private static HashSet _playerBodies; @@ -710,8 +845,6 @@ namespace Server.Custom.Bridge set.Add(race.MaleBody); set.Add(race.FemaleBody); - set.Add(race.MaleGhostBody); - set.Add(race.FemaleGhostBody); } } catch (Exception e) @@ -829,12 +962,22 @@ namespace Server.Custom.Bridge .Append(':').Append(BridgeConfig.AssetPlayerDirection) .Append(':').Append(BridgeConfig.AssetCreatureDirection); + var paths = new List(); + for (int fileType = 1; fileType <= 5; fileType++) + paths.Add(BridgeAssetValidator.AnimDataPath(fileType)); + + // Since phase 4 the catalogue's bytes depend on the UOP packages too — 235 of its + // bodies come out of them — so patching one has to change the catalogue id, exactly as + // patching an anim*.mul does. Leaving them out would let an operator replace a + // gargoyle and have an Update find nothing to do. + foreach (int n in BridgeUop.Packages) + paths.Add(BridgeUop.PackagePath(n)); + + foreach (string path in paths) { sb.Append('|'); - string path = BridgeAssetValidator.AnimDataPath(fileType); - if (path == null) continue; @@ -883,10 +1026,11 @@ namespace Server.Custom.Bridge } /// - /// The five anim files' index and record readers, opened for one reply and closed with - /// it. Holding them across replies would keep handles on the operator's client files - /// for as long as the cache lives, for no gain: opening five is microseconds and a - /// page decodes hundreds of sprites through them. + /// The five anim files' index and record readers — and, since phase 4, the five UOP + /// packages beside them — opened for one reply and closed with it. Holding them across + /// replies would keep handles on the operator's client files for as long as the cache + /// lives, for no gain: opening them is microseconds and a page decodes hundreds of + /// sprites through them. /// private sealed class Readers : IDisposable { @@ -896,6 +1040,9 @@ namespace Server.Custom.Bridge private readonly long[] _length = new long[6]; private readonly bool[] _open = new bool[6]; + private readonly Dictionary _packages = + new Dictionary(); + public readonly long VerdataLength; public Readers() @@ -958,8 +1105,42 @@ namespace Server.Custom.Bridge return fileType >= 1 && fileType <= 5 ? _reader[fileType] : null; } + /// + /// One AnimationFrame*.uop, opened on first use. A package this client does + /// not ship is a null that is cached as one: the miss must not be re-resolved and + /// re-opened once per body across a 2,047-body walk. + /// + public BridgeUop.Package Package(int n) + { + BridgeUop.Package package; + + if (_packages.TryGetValue(n, out package)) + return package; + + package = BridgeUop.Package.Open(BridgeUop.PackagePath(n)); + + _packages[n] = package; + + return package; + } + public void Dispose() { + foreach (var package in _packages.Values) + { + if (package == null) + continue; + + try + { + package.Dispose(); + } + catch + { + // Closing a read-only handle. Nothing useful is left to do. + } + } + for (int i = 1; i <= 5; i++) { if (_reader[i] == null) diff --git a/overlay/Scripts/Custom/Bridge/BridgePng.cs b/overlay/Scripts/Custom/Bridge/BridgePng.cs new file mode 100644 index 0000000..47b89e1 --- /dev/null +++ b/overlay/Scripts/Custom/Bridge/BridgePng.cs @@ -0,0 +1,231 @@ +using System; +using System.IO; +using System.IO.Compression; + +namespace Server.Custom.Bridge +{ + /// + /// **A PNG encoder that does not go through GDI+** (docs/link/v8.md §4.4, §4.9 — phase 4). + /// + /// decodes into a ushort[] of ARGB1555 rather than into a + /// Bitmap, which is the whole point of §4.4's note that the UOP reader is written + /// without System.Drawing: libgdiplus was archived in March 2025, and every line of + /// extraction that does not depend on it is a line that survives its absence. That leaves + /// the encode, and Bitmap.Save(…, ImageFormat.Png) is GDI+ too — so this is the + /// other half. + /// + /// It is deliberately the smallest thing that produces a correct file: 8-bit RGBA, one + /// IDAT, filter type 0 on every row. No interlacing, no palette, no colour-type choice, no + /// filter heuristics. A sprite is a few hundred pixels across and the bytes go straight + /// into a base64 field; the compression difference between this and a tuned encoder is a + /// rounding error against the wire, and every knob not turned is a way this cannot be + /// subtly wrong. + /// + /// Phase 3's BridgeCatalog.ToPng is left exactly as it is. It is measured, shipped, + /// and its input really is a Bitmap from the vendored decoder — a path that needs + /// GDI+ to produce the pixels in the first place, so encoding them without it buys nothing. + /// + public static class BridgePng + { + private static readonly byte[] Signature = + { + 0x89, (byte)'P', (byte)'N', (byte)'G', 0x0D, 0x0A, 0x1A, 0x0A + }; + + private static readonly uint[] CrcTable = BuildCrcTable(); + + private static readonly byte[] Empty = new byte[0]; + + /// + /// ARGB1555 to an RGBA8 PNG with a transparent background. + /// + /// The expansion is the same one BridgeCatalog.ToPng documents and for the same + /// reason: alpha bit clear is fully transparent, and each 5-bit channel is widened by + /// repeating its high bits — (c << 3) | (c >> 2), not a plain shift, + /// which would cap white at 248 and tint every sprite. + /// + public static byte[] FromArgb1555(ushort[] pixels, int width, int height) + { + if (pixels == null || width <= 0 || height <= 0) + return null; + + if ((long)width * height > pixels.Length) + return null; + + // One filter byte per row, then RGBA per pixel. This is the PNG "raw" stream, the + // thing that gets deflated. Bounded by the caller's dimension ceiling + // (BridgeAssetValidator.MaxArtDimension), so the arithmetic cannot overflow an int — + // the check is here anyway, because that ceiling lives in another file. + long size = (((long)width * 4) + 1) * height; + + if (size > Int32.MaxValue / 2) + return null; + + var raw = new byte[size]; + + int at = 0; + + for (int y = 0; y < height; y++) + { + raw[at++] = 0; // filter: None + + int row = y * width; + + for (int x = 0; x < width; x++) + { + int p = pixels[row + x]; + + if ((p & 0x8000) == 0) + { + at += 4; // already zero: transparent black + continue; + } + + int r = (p >> 10) & 0x1F; + int g = (p >> 5) & 0x1F; + int b = p & 0x1F; + + raw[at++] = (byte)((r << 3) | (r >> 2)); + raw[at++] = (byte)((g << 3) | (g >> 2)); + raw[at++] = (byte)((b << 3) | (b >> 2)); + raw[at++] = 0xFF; + } + } + + using (var ms = new MemoryStream(raw.Length / 2)) + { + ms.Write(Signature, 0, Signature.Length); + + var header = new byte[13]; + + WriteBigEndian(header, 0, (uint)width); + WriteBigEndian(header, 4, (uint)height); + + header[8] = 8; // bit depth + header[9] = 6; // colour type: truecolour with alpha + header[10] = 0; // compression: deflate + header[11] = 0; // filter method 0 + header[12] = 0; // no interlace + + WriteChunk(ms, "IHDR", header, 0, header.Length); + + byte[] deflated = Zlib(raw); + + WriteChunk(ms, "IDAT", deflated, 0, deflated.Length); + WriteChunk(ms, "IEND", Empty, 0, 0); + + return ms.ToArray(); + } + } + + /// + /// A zlib stream around .NET Framework's raw-deflate-only DeflateStream: the + /// two-byte header PNG requires, the deflate data, and the adler32 trailer computed + /// here because nothing in the framework will do it. Written by hand for exactly the + /// same reason reads one by hand — net48 exposes deflate and + /// calls it zlib, and the two are not the same format. + /// + private static byte[] Zlib(byte[] data) + { + using (var ms = new MemoryStream(data.Length / 2)) + { + // CMF 0x78 (deflate, 32K window) and FLG 0x9C (default level, no dictionary): + // 0x789C is the pair whose value is divisible by 31, which is the check a decoder + // applies. + ms.WriteByte(0x78); + ms.WriteByte(0x9C); + + using (var deflate = new DeflateStream(ms, CompressionMode.Compress, true)) + deflate.Write(data, 0, data.Length); + + uint adler = Adler32(data); + + ms.WriteByte((byte)(adler >> 24)); + ms.WriteByte((byte)(adler >> 16)); + ms.WriteByte((byte)(adler >> 8)); + ms.WriteByte((byte)adler); + + return ms.ToArray(); + } + } + + private static void WriteChunk(Stream to, string type, byte[] data, int offset, int length) + { + var head = new byte[8]; + + WriteBigEndian(head, 0, (uint)length); + + head[4] = (byte)type[0]; + head[5] = (byte)type[1]; + head[6] = (byte)type[2]; + head[7] = (byte)type[3]; + + to.Write(head, 0, head.Length); + + if (length > 0) + to.Write(data, offset, length); + + // The CRC covers the type and the data, and not the length. + uint crc = Crc32(head, 4, 4, 0xFFFFFFFF); + + if (length > 0) + crc = Crc32(data, offset, length, crc); + + crc ^= 0xFFFFFFFF; + + var tail = new byte[4]; + + WriteBigEndian(tail, 0, crc); + + to.Write(tail, 0, tail.Length); + } + + private static void WriteBigEndian(byte[] into, int at, uint value) + { + into[at] = (byte)(value >> 24); + into[at + 1] = (byte)(value >> 16); + into[at + 2] = (byte)(value >> 8); + into[at + 3] = (byte)value; + } + + private static uint[] BuildCrcTable() + { + var table = new uint[256]; + + for (uint n = 0; n < 256; n++) + { + uint c = n; + + for (int k = 0; k < 8; k++) + c = (c & 1) != 0 ? 0xEDB88320 ^ (c >> 1) : c >> 1; + + table[n] = c; + } + + return table; + } + + private static uint Crc32(byte[] data, int offset, int length, uint crc) + { + for (int i = 0; i < length; i++) + crc = CrcTable[(crc ^ data[offset + i]) & 0xFF] ^ (crc >> 8); + + return crc; + } + + private static uint Adler32(byte[] data) + { + const uint Mod = 65521; + + uint a = 1, b = 0; + + for (int i = 0; i < data.Length; i++) + { + a = (a + data[i]) % Mod; + b = (b + a) % Mod; + } + + return (b << 16) | a; + } + } +} diff --git a/overlay/Scripts/Custom/Bridge/BridgeUop.cs b/overlay/Scripts/Custom/Bridge/BridgeUop.cs new file mode 100644 index 0000000..da7c41c --- /dev/null +++ b/overlay/Scripts/Custom/Bridge/BridgeUop.cs @@ -0,0 +1,823 @@ +using System; +using System.Collections.Generic; +using System.Globalization; +using System.IO; +using System.IO.Compression; +using System.Text; + +using Ultima; + +namespace Server.Custom.Bridge +{ + /// + /// **The UOP animation reader** (docs/link/v8.md §4.3, §4.9 — protocol 8, phase 4): the + /// second and last decoder this protocol writes rather than calls. + /// + /// ServUO's vendored Ultima.Animations reads legacy anim*.mul only — it + /// constructs its five FileIndexes with the four-argument constructor, which passes + /// uopFile: null, so AnimationFrame*.uop is never opened. Everything a + /// modern client added there is invisible to it. This class opens those packages directly. + /// + /// ── **Why this is not the never-sweep rule being broken** ── + /// + /// §4.3's rule is that a body's file type comes from BodyConverter.Convert and is + /// never guessed, because asking another anim*.mul for an index it does not own + /// returns a decodable picture of something else — a giant spider on the gargoyle page. + /// That rule exists because a legacy index is addressed **by position**: nothing in the + /// file says which body a record belongs to. + /// + /// A UOP package is addressed by the **hash of a name that contains the body id** + /// (build/animationlegacyframe/000666/00.bin). Looking in all five packages for one + /// hash is therefore not a sweep — a hit is proof of identity, not a coincidence of + /// position, and the payload repeats the body id in its own header for us to check against. + /// Measured on this machine's client: 10,724 entries across the five packages, every one + /// of them claimed by that name scheme, and **no hash appears in more than one package**. + /// + /// ── **Validate as we go, because here we are the library** ── + /// + /// §4.5's rule is "validate before calling", and it exists because Ultima's decoders + /// take their bounds from the file they are reading. Nothing about this code can be + /// validated from outside — it *is* the decode — so the same discipline appears as a bound + /// on every read: the block chain against the file length, an entry's record against the + /// file, the inflated length against the declared one, the frame table against the + /// payload, and every run header against **both** the record's remaining bytes and the + /// bitmap it is writing into. A record that fails any of them is reported absent and no + /// pixel of it is kept. + /// + /// Measured the same way §4.5 was, which is the only measurement that says the boundary is + /// in the right place: across every UOP body on a stock client the walk refused **nothing** + /// that carries art, and the one body it does refuse (286) declares a 0×0 frame, which the + /// legacy decoder treats as absent too. + /// + /// ── **No System.Drawing, deliberately** ── + /// + /// §4.4 states it: libgdiplus was archived in March 2025, and the long-term argument for + /// moving extraction off System.Drawing is that a Linux shard depends on an + /// unmaintained library to see a sprite. This decoder writes ARGB1555 into a + /// ushort[] of its own and encodes that directly, so the + /// door stays open. (Phase 4 does not walk through it: the catalogue still refuses the + /// whole family when imaging is unavailable, because most of it genuinely needs GDI+.) + /// + public static class BridgeUop + { + /// 'MYP\0' — the Mythic package magic, as FileIndex reads it. + private const int PackageMagic = 0x50594D; + + /// 'AMOU' — the animation payload's own magic, little-endian. + private const int PayloadMagic = 0x554F4D41; + + /// Each frame record opens with its own palette: 0x100 ARGB1555 entries. + private const int PaletteBytes = 0x100 * 2; + + /// The frame table's row width: group, frame id, two unknowns, pixel offset. + private const int FrameRowBytes = 16; + + /// One block-chain record: offset, three lengths, hash, adler32, flag. + private const int BlockEntryBytes = 34; + + /// The xor Frame applies to every run header, and so must this. + private const int DoubleXor = (0x200 << 22) | (0x200 << 12); + + /// + /// A ceiling on a declared decompressed payload. One group file is a whole action for + /// one body across every direction; the largest on this machine's client is body + /// 1248's at 4.3 MB, so this is two orders of magnitude of headroom over real data and + /// still small enough that a corrupt length cannot ask for the host's memory. + /// + public const int MaxPayloadBytes = 64 * 1024 * 1024; + + /// + /// A ceiling on the block chain. Five packages hold 10,724 entries between them; this + /// bounds a cyclic or corrupt chain into a refusal rather than a hang. + /// + private const int MaxEntries = 1 << 20; + + /// The five packages this client ships. There is no AnimationFrame5.uop. + private static readonly int[] PackageNumbers = { 1, 2, 3, 4, 6 }; + + public static IEnumerable Packages + { + get { return PackageNumbers; } + } + + public static string PackageName(int n) + { + return "AnimationFrame" + n.ToString(CultureInfo.InvariantCulture) + ".uop"; + } + + /// + /// Where a UOP animation package lives. + /// + /// Ultima.Files.GetFilePath cannot answer this: its table of known client files + /// predates UOP animations and contains no AnimationFrame*.uop entry, so it + /// returns null for every one of them. So the lookup is done here, against the same + /// directories ServUO itself resolved at boot — Files.RootDir first, then + /// Core.DataDirectories, which §1 is built on. + /// + /// The comparison is case-insensitive **by enumeration** rather than by trying one + /// spelling. On Windows either would work; on a Linux shard host the client directory + /// is case-sensitive and the file may be shipped as `AnimationFrame1.uop`, + /// `animationframe1.uop` or anything between, which is exactly the shape of bug that + /// presents as "the gargoyles import on my machine and not on the server". + /// + /// is the general form, and `assets.sources` uses it for + /// the same reason: a file Ultima's table predates has to be found some other way. + /// + public static string PackagePath(int n) + { + return FindClientFile(PackageName(n)); + } + + private static readonly object _pathSync = new object(); + + private static readonly Dictionary _paths = + new Dictionary(StringComparer.OrdinalIgnoreCase); + + /// + /// Finds a client file Ultima.Files has never heard of. + /// + /// Only successful answers are cached: a file an operator copies in while the shard is + /// up should be found by the next import, and nothing here is hot enough for a + /// negative cache to be worth that. + /// + public static string FindClientFile(string name) + { + if (String.IsNullOrEmpty(name)) + return null; + + lock (_pathSync) + { + string cached; + + if (_paths.TryGetValue(name, out cached)) + return cached; + } + + foreach (string dir in Directories()) + { + if (String.IsNullOrEmpty(dir)) + continue; + + try + { + if (!Directory.Exists(dir)) + continue; + + string direct = Path.Combine(dir, name); + string hit = File.Exists(direct) ? direct : null; + + if (hit == null) + { + foreach (string found in Directory.GetFiles(dir)) + { + if (String.Equals(Path.GetFileName(found), name, + StringComparison.OrdinalIgnoreCase)) + { + hit = found; + break; + } + } + } + + if (hit == null) + continue; + + lock (_pathSync) + _paths[name] = hit; + + return hit; + } + catch (Exception e) + { + Console.WriteLine("[Bridge] uop: cannot look in {0}: {1}", dir, e.Message); + } + } + + return null; + } + + private static IEnumerable Directories() + { + string root = null; + + try + { + root = Files.RootDir; + } + catch + { + // Ultima's static initialiser reads the registry on Windows. A host where that + // throws still has Core.DataDirectories, which is the path ServUO actually booted + // from. + } + + if (!String.IsNullOrEmpty(root)) + yield return root; + + List dirs = null; + + try + { + dirs = Core.DataDirectories; + } + catch + { + // Same reasoning; an empty list is a real answer and the caller reports absent. + } + + if (dirs == null) + yield break; + + foreach (string dir in dirs) + yield return dir; + } + + /// + /// The name a body's action file is stored under, hashed the way the container indexes + /// it. Ultima.FileIndex.HashFileName is pure arithmetic over a string — no file + /// is touched and no decoder is entered — so this is the one place phase 4 leans on + /// the vendored code, and it leans on it precisely so that our lookup cannot disagree + /// with the container's own. + /// + public static ulong HashOf(int body, int action) + { + string name = String.Format(CultureInfo.InvariantCulture, + "build/animationlegacyframe/{0:D6}/{1:D2}.bin", body, action); + + return FileIndex.HashFileName(name); + } + + // ── the container ──────────────────────────────────────────────────────────────────── + + private struct Entry + { + public long At; + public int CompressedLength; + public int DecompressedLength; + public short Flag; + } + + /// + /// One opened AnimationFrame*.uop: its entry table in memory, its bytes on + /// demand. Opening one is a single pass over the block chain — 10,724 entries across + /// all five on this client — and the handle is held for the life of a reply, exactly + /// like the legacy readers next to it. + /// + public sealed class Package : IDisposable + { + private readonly Dictionary _entries; + private readonly FileStream _stream; + + public readonly string Path; + + private Package(string path, FileStream stream, Dictionary entries) + { + Path = path; + _stream = stream; + _entries = entries; + } + + public int Count + { + get { return _entries.Count; } + } + + /// + /// Reads the block chain, refusing anything that does not fit inside the file. + /// Returns null — never throws — because a client that ships a truncated package + /// is an ordinary thing to survive, not an error to raise. + /// + public static Package Open(string path) + { + if (String.IsNullOrEmpty(path)) + return null; + + FileStream stream = null; + + try + { + stream = new FileStream(path, FileMode.Open, FileAccess.Read, + FileShare.ReadWrite); + + long length = stream.Length; + + var entries = new Dictionary(); + + using (var br = new BinaryReader(stream, Encoding.UTF8, true)) + { + if (length < 28 || br.ReadInt32() != PackageMagic) + { + Console.WriteLine("[Bridge] uop: {0} is not a Mythic package", path); + stream.Dispose(); + return null; + } + + br.ReadInt32(); // version + br.ReadUInt32(); // signature + + long nextBlock = br.ReadInt64(); + + br.ReadInt32(); // block capacity + br.ReadInt32(); // declared file count + + while (nextBlock > 0) + { + // A block header is 12 bytes. Anything that does not leave room for + // one is a corrupt or cyclic chain, and this is where it stops. + if (nextBlock + 12 > length) + break; + + stream.Seek(nextBlock, SeekOrigin.Begin); + + int filesCount = br.ReadInt32(); + long following = br.ReadInt64(); + + if (filesCount < 0 + || nextBlock + 12 + ((long)filesCount * BlockEntryBytes) > length) + { + break; + } + + for (int i = 0; i < filesCount; i++) + { + long offset = br.ReadInt64(); + int headerLength = br.ReadInt32(); + int compressedLength = br.ReadInt32(); + int decompressedLength = br.ReadInt32(); + ulong hash = br.ReadUInt64(); + + br.ReadUInt32(); // adler32 + + short flag = br.ReadInt16(); + + if (offset <= 0 || headerLength < 0 || compressedLength <= 0) + continue; + + if (decompressedLength <= 0 || decompressedLength > MaxPayloadBytes) + continue; + + long at = offset + headerLength; + + // The check FileIndex.Seek is missing, in the place it matters + // here too: that the record ENDS inside the file, not merely that + // it starts inside it (§4.5). + if (at < 0 || at + compressedLength > length) + continue; + + if (entries.Count >= MaxEntries) + break; + + // First writer wins. Nothing on this client produces a collision + // — measured: no hash appears in two packages, and none twice in + // one — and if a patched client ever did, taking the first is the + // answer that does not depend on chain order. + if (!entries.ContainsKey(hash)) + entries[hash] = new Entry + { + At = at, + CompressedLength = compressedLength, + DecompressedLength = decompressedLength, + Flag = flag + }; + } + + if (following <= nextBlock) + break; // a chain that does not move forward is a loop + + nextBlock = following; + } + } + + return new Package(path, stream, entries); + } + catch (Exception e) + { + Console.WriteLine("[Bridge] uop: cannot open {0}: {1}: {2}", + path, e.GetType().Name, e.Message); + + if (stream != null) + { + try + { + stream.Dispose(); + } + catch + { + // Closing a read-only handle. + } + } + + return null; + } + } + + public bool Has(ulong hash) + { + return _entries.ContainsKey(hash); + } + + /// + /// The bytes behind one entry, decompressed. False with a reason is the ordinary + /// answer for "this package does not hold it". + /// + public bool TryRead(ulong hash, out byte[] payload, out string reason) + { + payload = null; + reason = null; + + Entry entry; + + if (!_entries.TryGetValue(hash, out entry)) + { + reason = "not in " + System.IO.Path.GetFileName(Path); + return false; + } + + byte[] raw; + + try + { + _stream.Seek(entry.At, SeekOrigin.Begin); + + raw = new byte[entry.CompressedLength]; + + if (!Fill(_stream, raw, raw.Length)) + { + // The §4.5 failure, in our own code this time: a short read that nobody + // checked is how the library ends up decoding the previous asset. + reason = "record is shorter than the index claims"; + return false; + } + } + catch (Exception e) + { + reason = "read failed: " + e.GetType().Name; + return false; + } + + if (entry.Flag != 1) + { + if (raw.Length != entry.DecompressedLength) + { + reason = "stored record is " + raw.Length + " bytes, not the declared " + + entry.DecompressedLength; + return false; + } + + payload = raw; + return true; + } + + return TryInflate(raw, entry.DecompressedLength, out payload, out reason); + } + + public void Dispose() + { + try + { + _stream.Dispose(); + } + catch + { + // Closing a read-only handle. Nothing useful is left to do. + } + } + } + + private static bool Fill(Stream stream, byte[] into, int count) + { + int read = 0; + + while (read < count) + { + int n = stream.Read(into, read, count - read); + + if (n <= 0) + return false; + + read += n; + } + + return true; + } + + /// + /// zlib, which .NET Framework 4.8 does not expose — only raw deflate. The two-byte + /// zlib header is checked and skipped rather than assumed, because handing a + /// DeflateStream a stream that is not deflate produces garbage as readily as an + /// exception, and the trailing adler32 is left to the length check below: a stream + /// that inflates to exactly the declared number of bytes did not silently truncate. + /// + private static bool TryInflate(byte[] raw, int declared, out byte[] payload, out string reason) + { + payload = null; + reason = null; + + if (raw.Length < 3) + { + reason = "compressed record is too short to be zlib"; + return false; + } + + int cmf = raw[0]; + int flg = raw[1]; + + if ((cmf & 0x0F) != 8 || (((cmf << 8) + flg) % 31) != 0 || (flg & 0x20) != 0) + { + reason = "compressed record is not a zlib stream"; + return false; + } + + try + { + var output = new byte[declared]; + + using (var source = new MemoryStream(raw, 2, raw.Length - 2, false)) + using (var inflate = new DeflateStream(source, CompressionMode.Decompress)) + { + int read = 0; + + while (read < declared) + { + int n = inflate.Read(output, read, declared - read); + + if (n <= 0) + break; + + read += n; + } + + if (read != declared) + { + reason = "inflated " + read + " bytes, not the declared " + declared; + return false; + } + + // One more byte would mean the record is longer than its own header says, + // which is a different file from the one we were promised. + if (inflate.ReadByte() != -1) + { + reason = "inflated past the declared " + declared + " bytes"; + return false; + } + } + + payload = output; + return true; + } + catch (Exception e) + { + reason = "inflate failed: " + e.GetType().Name; + return false; + } + } + + // ── the payload ────────────────────────────────────────────────────────────────────── + + /// One decoded frame: ARGB1555 in our own array, no Bitmap anywhere. + public sealed class Pixels + { + public int Width; + public int Height; + public int CenterX; + public int CenterY; + public ushort[] Argb1555; + } + + /// + /// One action of one body — every direction of it, concatenated. + /// + /// The legacy files address a frame as index + action * 5 + direction; a UOP + /// group file holds the whole action in one record and the directions are equal-length + /// runs inside its frame table. So is where "direction 1" is + /// turned into a frame number, and it is integer division exactly as the reference + /// implementations do it — see the note there for the nine bodies where that matters. + /// + public sealed class Group + { + private readonly byte[] _buf; + private readonly int _dataStart; + + public readonly int FrameCount; + public readonly int Body; + + private Group(byte[] buf, int body, int frameCount, int dataStart) + { + _buf = buf; + Body = body; + FrameCount = frameCount; + _dataStart = dataStart; + } + + public static bool TryOpen(byte[] buf, int expectedBody, out Group group, out string reason) + { + group = null; + reason = null; + + if (buf == null || buf.Length < 40) + { + reason = "payload is too short to carry a header"; + return false; + } + + if (BitConverter.ToInt32(buf, 0) != PayloadMagic) + { + reason = "payload is not an AMOU animation record"; + return false; + } + + int body = BitConverter.ToInt32(buf, 12); + + // The container said which body this is, by the name it was stored under; the + // payload says it again. They agree on every record of this client, and the day + // they do not is the day something is being read that was not asked for. + if (body != expectedBody) + { + reason = "payload declares body " + body + ", not " + expectedBody; + return false; + } + + int frameCount = BitConverter.ToInt32(buf, 32); + int dataStart = BitConverter.ToInt32(buf, 36); + + if (frameCount <= 0 || frameCount > BridgeAssetValidator.MaxAnimFrames) + { + reason = "payload declares " + frameCount + " frames"; + return false; + } + + if (dataStart < 40 || dataStart > buf.Length) + { + reason = "frame table starts at " + dataStart + " of " + buf.Length; + return false; + } + + if ((long)dataStart + ((long)frameCount * FrameRowBytes) > buf.Length) + { + reason = "frame table of " + frameCount + " rows runs past the record"; + return false; + } + + group = new Group(buf, body, frameCount, dataStart); + return true; + } + + /// + /// Which frame of this action faces a given direction. + /// + /// Five directions share the action's frames equally, so direction *d* starts at + /// d * (FrameCount / 5). On nine of this client's 244 UOP bodies the frame + /// count is **not** a multiple of five (41, 42, 46…), and integer division then + /// lands a direction or so early in the run. That is what ClassicUO does, it is + /// the right trade, and the reason is §4.8's: the failure being guarded against is + /// a picture of the **wrong creature**, and this cannot produce one — the worst + /// case is the right creature at a slightly different angle, on nine bodies, where + /// refusing them instead would lose nine creatures outright. + /// + public int DirectionAt(int direction) + { + int perDirection = FrameCount / 5; + + if (perDirection <= 0) + return direction == 0 ? 0 : -1; + + if (direction < 0 || direction > 4) + return -1; + + int at = direction * perDirection; + + return at < FrameCount ? at : -1; + } + + /// + /// Decodes one frame, bounding every read against the record and every write + /// against the bitmap. + /// + /// The run loop is Ultima.Frame's, with the two bounds it does not have. + /// Frame writes through a LockBits pointer whose origin comes from + /// two signed shorts in the file and never checks where a run lands; here a run + /// that would leave the bitmap, or read past the record, refuses the frame. Across + /// every UOP body on a stock client that refuses nothing that carries art. + /// + /// A 0×0 frame returns false with set: the legacy decoder + /// treats that as no art rather than as damage, and so must this, or body 286 + /// would be logged as a defect on every scan. + /// + public bool TryDecode(int index, out Pixels pixels, out bool empty, out string reason) + { + pixels = null; + empty = false; + reason = null; + + if (index < 0 || index >= FrameCount) + { + reason = "frame " + index + " of " + FrameCount; + return false; + } + + int row = _dataStart + (index * FrameRowBytes); + + long at = (long)row + (uint)BitConverter.ToInt32(_buf, row + 12); + + if (at < 0 || at + PaletteBytes + 8 > _buf.Length) + { + reason = "frame " + index + " points outside the record"; + return false; + } + + int pixelAt = (int)at; + + int centerX = BitConverter.ToInt16(_buf, pixelAt + PaletteBytes); + int centerY = BitConverter.ToInt16(_buf, pixelAt + PaletteBytes + 2); + int width = BitConverter.ToUInt16(_buf, pixelAt + PaletteBytes + 4); + int height = BitConverter.ToUInt16(_buf, pixelAt + PaletteBytes + 6); + + if (width <= 0 || height <= 0) + { + empty = true; + reason = "frame " + index + " is " + width + "x" + height; + return false; + } + + if (width > BridgeAssetValidator.MaxArtDimension + || height > BridgeAssetValidator.MaxArtDimension) + { + reason = "frame " + index + " declares " + width + "x" + height; + return false; + } + + var palette = new ushort[0x100]; + + for (int i = 0; i < palette.Length; i++) + { + // The library's own xor: the stored entry has its alpha bit clear and every + // palette colour is opaque. A pixel no run covers stays zero, which is how a + // sprite keeps its transparent background. + palette[i] = (ushort)(BitConverter.ToUInt16(_buf, pixelAt + (i * 2)) ^ 0x8000); + } + + var canvas = new ushort[width * height]; + + int p = pixelAt + PaletteBytes + 8; + + int xBase = centerX - 0x200; + int yBase = (centerY + height) - 0x200; + + while (true) + { + if (p + 4 > _buf.Length) + { + reason = "frame " + index + " has no terminator inside the record"; + return false; + } + + int header = BitConverter.ToInt32(_buf, p); + p += 4; + + if (header == 0x7FFF7FFF) + break; + + header ^= DoubleXor; + + int x = ((header >> 22) & 0x3FF) + xBase; + int y = ((header >> 12) & 0x3FF) + yBase; + int run = header & 0xFFF; + + if (run == 0) + continue; + + if (p + run > _buf.Length) + { + reason = "frame " + index + " has a run past the end of the record"; + return false; + } + + if (y < 0 || y >= height || x < 0 || x + run > width) + { + reason = "frame " + index + " has a run at " + x + "," + y + " of " + + run + " outside " + width + "x" + height; + return false; + } + + int cursor = (y * width) + x; + + for (int i = 0; i < run; i++) + canvas[cursor + i] = palette[_buf[p + i]]; + + p += run; + } + + pixels = new Pixels + { + Width = width, + Height = height, + CenterX = centerX, + CenterY = centerY, + Argb1555 = canvas + }; + + return true; + } + } + } +} -- 2.49.1 From 1be1f24562b7da18915dd732d52f3b1acbe1dfab Mon Sep 17 00:00:00 2001 From: wtclaude Date: Fri, 11 Sep 2026 05:58:45 -0500 Subject: [PATCH 07/11] feat(asset-bridge): item and land art on demand, hued where the files are (Phase 5) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The body catalogue is a set; this is not. This client addresses 49,152 static ids and has art for 39,189 of them, plus 4,244 land tiles of 16,384 -- and hues multiply the statics by three thousand. So there is no manifest and no scan: `assets.fetch` grows two more families (`static`, `land`) and answers the keys the website's own data names. `assets.fetch` becomes shared plumbing. BridgeAssets now owns the command, does the reqId/consent/key-ceiling checks once, derives the family from the keys themselves (§5 made the key the address; a request that also named its family would have two places to be wrong and one of them silent) and dispatches to the reader that registered it. A batch must be of one family, because the reply carries one `catalog` id. `assets.sources` gains `families` -- additive, so the protocol stays 8, and EXTRACTOR_VERSION stays 2 because no existing key's bytes change. Two traps, both in §4.5's family -- a confident, plausible, wrong picture: - `Art.GetStatic` memoises into a static Bitmap[0xFFFF] and hands back the SAME instance, while `Hue.ApplyTo` repaints in place. Hue a static once and the plain key comes back hued from then on, and the next hue stacks on the last. Measured on this client before the fix. `Files.CacheData` is now off for the life of the process; `TryHue` re-checks it and refuses rather than risk it, and the same flag decides whether a bitmap is ours to dispose. - `PartialHue` decides whether a hue repaints every pixel or only the grey ones, per item id, out of `tiledata.mul` -- 13,259 of 65,536 ids on this client. Item 597 is a wooden screen with painted flowers: one mode reddens the flowers, the other the whole screen. Both decode. The first cut of this reader bound `TileData` to ServUO's OWN `Server.TileData` (the enclosing namespace beats `using Ultima;`, and it has a PartialHue flag too), which compiled and refused every hued key at runtime. Every such type is spelled `Ultima.` now. Land takes no hue segment: the mode that decides how is an item flag and land has no equivalent, so `land/3/h33` is refused rather than guessed. `h0` is not a key either -- hue 0 means "not hued", and the plain key already names it. Measured through the reader over the whole range: 39,189 statics and 4,244 land tiles served, and the only refusals are the 9,963 + 12,140 empty index slots §4.5 predicted. Nothing that carries art is refused. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4 --- overlay/Scripts/Custom/Bridge/BridgeArt.cs | 842 ++++++++++++++++++ overlay/Scripts/Custom/Bridge/BridgeAssets.cs | 314 +++++++ overlay/Scripts/Custom/Bridge/BridgeBoot.cs | 1 + .../Scripts/Custom/Bridge/BridgeCatalog.cs | 152 +--- overlay/Scripts/Custom/Bridge/BridgeConfig.cs | 17 + 5 files changed, 1187 insertions(+), 139 deletions(-) create mode 100644 overlay/Scripts/Custom/Bridge/BridgeArt.cs diff --git a/overlay/Scripts/Custom/Bridge/BridgeArt.cs b/overlay/Scripts/Custom/Bridge/BridgeArt.cs new file mode 100644 index 0000000..3cdbfe1 --- /dev/null +++ b/overlay/Scripts/Custom/Bridge/BridgeArt.cs @@ -0,0 +1,842 @@ +using System; +using System.Collections.Generic; +using System.Drawing; +using System.Globalization; +using System.IO; +using System.Text; + +using Ultima; + +namespace Server.Custom.Bridge +{ + /// + /// **Item and land art, on demand** (docs/link/v8.md §5, §11 — protocol 8, phase 5). + /// + /// The body catalogue is a *set*: 1,022 sprites, enumerated, hashed and imported in one + /// pass because a bestiary needs all of them. This is the opposite shape. This client + /// addresses **49,152 static ids** and has real art for **39,189** of them, plus 4,244 land + /// tiles of 16,384 — and then there are hues, which multiply the statics by three thousand. + /// Nothing enumerates that. So there is no manifest here and no scan: the website asks for + /// the handful of keys its own data actually names, and this answers them. + /// + /// (49,152 rather than the 81,884 entries `artidx.mul` declares: FileIndex sizes its + /// table from the **length argument it is constructed with**, `0x10000`, not from the idx + /// file — so the addressable range is `0x10000 - 0x4000`. Reading the ceiling off the file + /// instead would invent 16,348 ids, every one of them answered out of an array nobody + /// bounded.) + /// + /// ── **The keys** (§5) ── + /// + /// + /// static/3922 one item graphic, as the client files hold it + /// static/3922/h33 the same graphic with hue 33 applied + /// land/3 one land tile + /// + /// + /// ── **Why the hue is applied HERE and not on the website** ── + /// + /// Because it cannot be applied correctly anywhere else, and the incorrect version looks + /// fine. + /// + /// A hue is not a tint. It is a 32-entry colour ramp out of `hues.mul` indexed by a + /// pixel's own red channel — and whether it replaces *every* pixel or only the grey ones + /// is decided by the PartialHue flag in tiledata.mul, per item id. On this + /// client **13,259 of 65,536 item ids carry that flag**. Get it wrong on one of them and + /// you do not get an error: item 597 is a wooden screen with painted flowers, and hued red + /// the right way the flowers turn red, the wrong way the whole screen turns red. Both + /// decode. Both are the right size. One is wrong. + /// + /// The website has neither file and never will — shipping `Hues.mul` semantics and a + /// 65,536-row flag table into Node to answer a question the shard can answer for free is + /// the same trade §2.1 already refused. So hue is part of the key, and the key is resolved + /// where the files are. + /// + /// ── **The trap this phase existed to find** ── + /// + /// Art.GetStatic memoises into a static Bitmap[0xFFFF] and returns **the same + /// instance** every time; Hue.ApplyTo repaints a bitmap **in place**. Hue a static + /// once and the library's own copy is hued from then on — the plain key comes back hued, + /// and the next hue stacks on the last. It is §4.5's failure mode (a confident, plausible, + /// wrong picture that every success count agrees with) reached through a door §4.5 never + /// looked at, because phase 0 was auditing *records* and this is the library's *cache*. + /// + /// turns Files.CacheData off for the life of + /// the process, which makes every bitmap this file receives its own. That invariant is + /// load-bearing enough that **re-checks it** before applying a hue and + /// refuses rather than risk it: an invariant nothing verifies is a comment. + /// + /// ── **What is validated, and against what** ── + /// + /// Everything §4.5 built, reused as-is. An index entry is judged before the id is handed to + /// Ultima (), a static's record header + /// and row table are walked bounded (StaticSane), a land record is checked against + /// the 2,024 bytes LoadLand reads whatever the length says (LandLengthSane), + /// and the bound is taken against **whichever file FileIndex actually opened** — + /// artLegacyMUL.uop on every current client, never art.mul (§4.6). + /// + /// Two of §4.5's measurements are this family's, not the catalogue's, and they are the + /// reason all of it is here: on a **stock** client **9,963 static ids and 12,140 land ids** + /// have an index entry reading `lookup 0, length 0`, which FileIndex.Seek treats as + /// a hit and the decoder answers with whatever was decoded last. Measured through this + /// reader over the whole range, those are the ONLY refusals — every one of the 39,189 + /// statics and 4,244 land tiles that carries art is served, which is the half of the + /// measurement that says the boundary is in the right place (§4.5). + /// + public static class BridgeArt + { + /// Item graphics. static/<id>, optionally /h<hue>. + private const string StaticFamily = "static"; + + /// Land tiles. land/<id>, and no hue segment — see . + private const string LandFamily = "land"; + + /// The art index addresses land at its own id and statics at 0x4000 + id. + private const int StaticBase = 0x4000; + + /// Land is addressed with index & 0x3FFF by the library itself. + private const int LandCount = 0x4000; + + /// hues.mul holds 3,000 slots; the wire's hue 1 is slot 0. + private const int MaxHue = 3000; + + public static void Initialize() + { + if (!BridgeConfig.Enabled) + return; + + BridgeAssets.RegisterFamily(StaticFamily, ReplyFetch); + BridgeAssets.RegisterFamily(LandFamily, ReplyFetch); + } + + // ── the cache (§11) ────────────────────────────────────────────────────────────────── + + private sealed class Rendered + { + public string Key; + public string Status; + public string Reason; + public string Sha256; + public byte[] Png; + public int Width; + public int Height; + public int Hue; + public bool PartialHue; + public string Source; + + public int Weight + { + get { return Png == null ? 128 : Png.Length + 128; } + } + } + + private sealed class Cache + { + public string Id; + + public readonly Dictionary ByKey = + new Dictionary(StringComparer.Ordinal); + + /// Insertion order, for eviction. See . + public readonly Queue Order = new Queue(); + + public long Bytes; + public DateTime LastUsed; + } + + private static readonly object _sync = new object(); + private static Cache _cache; + + private static readonly TimeSpan IdleFor = TimeSpan.FromMinutes(5); + + // ── assets.fetch, the static and land half ─────────────────────────────────────────── + + /// + /// Both families' answer to assets.fetch. The correlation id, the operator's + /// consent, the key ceiling and the family decision were made by + /// ; every key here belongs to this reader. + /// + /// The paging envelope, the byte budget and the `catalog` guard are §3.4's and + /// phase 3's, unchanged — a caller that already walks the body catalogue walks this + /// with the same loop. + /// + private static void ReplyFetch(string reqId, List keys, string expected, string cursor) + { + string imagingReason; + + if (!BridgeAssets.ImagingOk(out imagingReason)) + { + // §17.9: a flat refusal, not a partial answer. Every picture in this family needs + // a decoder that goes through GDI+, so there is no half of it to serve. + BridgeAssets.Fail(reqId, "UNAVAILABLE", + "this shard host cannot render images - Mono's System.Drawing needs " + + "libgdiplus. (" + imagingReason + ")"); + return; + } + + string id = SourceId(); + + if (expected != null && expected != id) + { + BridgeAssets.Fail(reqId, "UNREADABLE", + "the shard's client files changed since that catalogue was read (catalog " + + expected + " is now " + id + "); ask again"); + return; + } + + Cache cache; + + lock (_sync) + { + if (_cache == null || _cache.Id != id) + _cache = new Cache { Id = id }; + + cache = _cache; + cache.LastUsed = DateTime.UtcNow; + } + + int from = ParseKeyCursor(cursor); + + var sb = BridgeJson.Begin("assets.fetch.ok"); + + sb.Str("reqId", reqId) + .Str("family", BridgeAssets.FamilyOfKey(keys[0])) + .Str("catalog", cache.Id) + .Num("extractorVersion", BridgeAssets.EXTRACTOR_VERSION) + .Num("asked", keys.Count) + .Num("from", from); + + var page = new BridgeAssets.PageBuilder(sb, "rows", BridgeConfig.AssetBatchBytes); + + using (var readers = new Readers()) + { + for (int i = from; i < keys.Count; i++) + { + string row = Row(cache, readers, keys[i]); + + if (!page.TryAdd(row, "k:" + (i + 1).ToString(CultureInfo.InvariantCulture))) + break; + } + } + + page.Close(); + + sb.Num("sent", page.Count); + + BridgeLink.Emit(sb.End()); + + Sweep(); + } + + /// + /// One key to one JSON row. + /// + /// A key this shard cannot serve is a **row**, never a failed request: an item id with + /// no art must not cost the other three hundred on the page. The three outcomes are the + /// ones phase 3 defined, and this family adds a `reason` beside them — additive, and + /// the only way an operator learns that eight of their records are damaged rather than + /// simply absent, which is a difference §4.5 spent a whole phase establishing. + /// + private static string Row(Cache cache, Readers readers, string key) + { + Rendered item = Resolve(cache, readers, key); + + var sb = new StringBuilder(2048); + + sb.Append("{\"key\":"); + BridgeJson.Text(sb, key); + + sb.Append(",\"status\":\"").Append(item.Status).Append('"'); + + if (item.Reason != null) + { + sb.Append(",\"reason\":"); + BridgeJson.Text(sb, item.Reason); + } + + if (item.Status != "ok") + { + sb.Append('}'); + return sb.ToString(); + } + + sb.Append(",\"sha256\":\"").Append(item.Sha256).Append('"'); + sb.Append(",\"bytes\":").Append(item.Png.Length.ToString(CultureInfo.InvariantCulture)); + sb.Append(",\"width\":").Append(item.Width.ToString(CultureInfo.InvariantCulture)); + sb.Append(",\"height\":").Append(item.Height.ToString(CultureInfo.InvariantCulture)); + + if (item.Hue > 0) + { + sb.Append(",\"hue\":").Append(item.Hue.ToString(CultureInfo.InvariantCulture)); + sb.Append(",\"partialHue\":").Append(item.PartialHue ? "true" : "false"); + } + + sb.Append(",\"source\":\"").Append(item.Source).Append('"'); + sb.Append(",\"png\":\"").Append(Convert.ToBase64String(item.Png)).Append("\"}"); + + return sb.ToString(); + } + + private static Rendered Resolve(Cache cache, Readers readers, string key) + { + lock (_sync) + { + Rendered cached; + + if (cache.ByKey.TryGetValue(key, out cached)) + return cached; + } + + Rendered item; + + try + { + item = Render(readers, key); + } + catch (Exception e) + { + Console.WriteLine("[Bridge] art: {0}: {1}: {2}", key, e.GetType().Name, e.Message); + + item = new Rendered + { + Key = key, + Status = "absent", + Reason = e.GetType().Name + }; + } + + if (item.Status == "ok") + Remember(cache, item); + + return item; + } + + /// + /// Holds the encoded bytes against a byte budget, evicting oldest-first. + /// + /// **Oldest-first rather than least-recently-used, deliberately.** The access pattern + /// this serves is a warm pass: the website asks for the keys it has never held, stores + /// them permanently, and does not ask again. What this cache is actually for is the + /// second page of a batch, a retry after a 425, and the same picture appearing in two + /// of a page's rows — all of which insertion order serves exactly as well as recency, + /// and with no bookkeeping on the hot path. A cache whose hit pattern has no recency in + /// it should not pretend to rank by it. + /// + /// Only successes are held. An absent key costs one index lookup, which is cheaper than + /// the dictionary entry that would remember it. + /// + private static void Remember(Cache cache, Rendered item) + { + lock (_sync) + { + if (cache.ByKey.ContainsKey(item.Key)) + return; + + cache.ByKey[item.Key] = item; + cache.Order.Enqueue(item.Key); + cache.Bytes += item.Weight; + + while (cache.Bytes > BridgeConfig.AssetArtCacheBytes && cache.Order.Count > 0) + { + string oldest = cache.Order.Dequeue(); + + Rendered evicted; + + if (!cache.ByKey.TryGetValue(oldest, out evicted)) + continue; + + cache.ByKey.Remove(oldest); + cache.Bytes -= evicted.Weight; + } + } + } + + // ── decode ─────────────────────────────────────────────────────────────────────────── + + /// + /// Validate, decode, hue, encode. In that order, and the order is the point. + /// + private static Rendered Render(Readers readers, string key) + { + bool land; + int id, hue; + + if (!TryParseKey(key, out land, out id, out hue)) + return Unsupported(key, "not a key this shard serves"); + + FileIndex index = readers.Index; + + if (index == null || index.Index == null) + return Absent(key, "this shard has no art file"); + + int at = land ? id : StaticBase + id; + + if (at < 0 || at >= index.Index.Length) + return Unsupported(key, "id " + id + " is past the end of this client's art index"); + + string reason; + + BridgeAssetValidator.Verdict verdict = + BridgeAssetValidator.CheckEntry(index, at, readers.DataLength, readers.VerdataLength, out reason); + + if (verdict == BridgeAssetValidator.Verdict.Absent) + { + // The 9,962 statics and 12,140 land tiles of §4.5: an index entry that reads + // `lookup 0, length 0`, which the library treats as a hit and answers with the + // previous asset's pixels. Absent is the true answer and the only safe one. + return Absent(key, reason); + } + + if (verdict != BridgeAssetValidator.Verdict.Ok) + { + // A damaged record rather than a missing one. Still absent to the website — there + // is no picture either way — but the reason is worth carrying, because this one an + // operator can act on. + Console.WriteLine("[Bridge] art: {0} refused: {1}", key, reason); + return Absent(key, reason); + } + + if (land) + { + if (!BridgeAssetValidator.LandLengthSane(index, at, out reason)) + { + Console.WriteLine("[Bridge] art: {0} refused: {1}", key, reason); + return Absent(key, reason); + } + } + else if (readers.Reader == null || !readers.Reader.StaticSane(index, at, out reason)) + { + Console.WriteLine("[Bridge] art: {0} refused: {1}", + key, reason ?? "the art record could not be read"); + + return Absent(key, reason ?? "the art record could not be read"); + } + + // A hue is resolved BEFORE anything is decoded, so a bad one costs no pixels and, more + // to the point, cannot half-apply to a picture that then gets cached and served. + Ultima.Hue applied = null; + bool partial = false; + + if (hue > 0) + { + if (!TryHue(id, hue, out applied, out partial, out reason)) + return Unsupported(key, reason); + } + + Bitmap bitmap = land + ? Art.GetLand(id) + // `checkmaxid: false` on purpose (§4.5): the default maps an out-of-range id to 0 + // and returns ITEM ZERO'S PICTURE. The id is already bounded against the index + // that was actually opened, so this can only be loud. + : Art.GetStatic(id, false); + + // **Whether this bitmap is ours to dispose is the same question as whether it is ours + // to hue**, and it has the same answer. With the library's cache off — which + // `BridgeAssets.Initialize` guarantees and `TryHue` re-checks — every call decodes a + // fresh instance that nothing else holds, so not disposing it would leak one bitmap per + // fetched key. With the cache on, that instance is the library's own copy and disposing + // it would leave a disposed `Bitmap` in a static array for the next caller to fault on. + // Both mistakes are silent; the flag decides, once, here. + bool owned = !Files.CacheData; + + try + { + if (bitmap == null || bitmap.Width <= 0 || bitmap.Height <= 0) + return Absent(key, "the decoder returned no picture"); + + if (applied != null) + applied.ApplyTo(bitmap, partial); + + byte[] png = BridgeAssets.BitmapToPng(bitmap); + + if (png == null) + return Absent(key, "the picture could not be encoded"); + + return new Rendered + { + Key = key, + Status = "ok", + Sha256 = BridgeAssets.Sha256Hex(png), + Png = png, + Width = bitmap.Width, + Height = bitmap.Height, + Hue = hue, + PartialHue = partial, + Source = readers.Source + }; + } + finally + { + if (owned && bitmap != null) + bitmap.Dispose(); + } + } + + /// + /// Resolves one wire hue onto a ramp, and decides whether it repaints the whole sprite + /// or only its grey pixels. + /// + /// Four things have to hold, and every one of them has a way of not holding that + /// produces a picture rather than an error: + /// + /// **The library's cache is off.** Re-checked here because ApplyTo repaints in + /// place: with the cache on, this would edit the copy Art hands to everyone + /// else. turns it off at boot and this refuses + /// if it somehow did not, because the failure is invisible and permanent. + /// + /// **`hues.mul` is present.** When it is missing Hues.Initialize does not throw + /// — it fills all 3,000 slots with a new Hue(index) whose ramp is **all zeroes**, + /// and applying one of those paints the sprite black. An all-zero ramp is therefore + /// refused whatever the reason for it; on this client there are none. + /// + /// **The index is in range.** The wire's hue is 1-based — Ultima.Map does the + /// same GetHue(hue - 1) at line 450 — and GetHue itself masks with + /// `0x3FFF` and falls back to slot 0 rather than failing, so an out-of-range hue would + /// silently become a different colour. Bound it here instead. + /// + /// **The PartialHue flag decides the mode**, per item id, out of + /// tiledata.mul. This is the one that is invisible: both modes decode, both are + /// the right size, and 13,259 of this client's item ids need the grey-only one. + /// **Land has no such flag**, which is why does not accept a + /// hue on a land key at all rather than guessing a mode for it. + /// + private static bool TryHue(int id, int hue, out Ultima.Hue applied, out bool partial, out string reason) + { + applied = null; + partial = false; + reason = null; + + if (Files.CacheData) + { + reason = "this shard's art cache is on, so a hue cannot be applied safely"; + Console.WriteLine("[Bridge] art: refusing hue {0}: {1}", hue, reason); + return false; + } + + if (hue < 1 || hue > MaxHue) + { + reason = "hue " + hue + " is outside 1-" + MaxHue; + return false; + } + + Ultima.Hue[] list = Ultima.Hues.List; + + if (list == null || hue - 1 >= list.Length || list[hue - 1] == null) + { + reason = "this client has no hue table"; + return false; + } + + Ultima.Hue candidate = list[hue - 1]; + + if (candidate.Colors == null || AllZero(candidate.Colors)) + { + reason = "hue " + hue + " has no colours in this client's hues.mul"; + return false; + } + + if (!TryPartialHue(id, out partial, out reason)) + return false; + + applied = candidate; + return true; + } + + private static bool AllZero(short[] colors) + { + for (int i = 0; i < colors.Length; i++) + { + if (colors[i] != 0) + return false; + } + + return true; + } + + /// + /// The PartialHue flag for one item id. + /// + /// Refuses rather than defaults when tiledata.mul cannot be read. Defaulting + /// either way would be a coin flip on 13,259 ids, and the losing side of it is a + /// picture that looks deliberate. + /// + /// **Every type here is spelled Ultima. on purpose, and it is not style.** + /// ServUO declares its own Server.TileData, Server.ItemData and + /// Server.TileFlag — with a PartialHue member — in + /// Server/TileData.cs. This file lives in Server.Custom.Bridge, so the + /// enclosing namespace beats the using Ultima; and the unqualified spelling + /// silently binds to the *server's* table: it compiles, the flag exists, and the answer + /// comes from a file resolved through Core.DataDirectories rather than through + /// Ultima.Files, which is the one thing §4.6 says never to do — decide a picture + /// with a file other than the one the pixels came out of. The first run of this reader + /// did exactly that and refused every hued key with a TypeInitializationException + /// from a class this code never meant to name. + /// + private static bool TryPartialHue(int id, out bool partial, out string reason) + { + partial = false; + reason = null; + + Ultima.ItemData[] table; + + try + { + table = Ultima.TileData.ItemTable; + } + catch (Exception e) + { + reason = "this client's tiledata could not be read (" + e.GetType().Name + ")"; + return false; + } + + if (table == null || id < 0 || id >= table.Length) + { + reason = "this client's tiledata does not describe item " + id; + return false; + } + + partial = (table[id].Flags & Ultima.TileFlag.PartialHue) != 0; + return true; + } + + private static Rendered Absent(string key, string reason) + { + return new Rendered { Key = key, Status = "absent", Reason = reason }; + } + + private static Rendered Unsupported(string key, string reason) + { + return new Rendered { Key = key, Status = "unsupported", Reason = reason }; + } + + // ── keys, cursors and the source id ────────────────────────────────────────────────── + + /// + /// static/<id>, static/<id>/h<hue> and + /// land/<id>. + /// + /// **A land key takes no hue segment.** The client can hue a land tile, but the mode + /// that decides how is an *item* flag and land has no equivalent — so the honest answer + /// to `land/3/h33` is that this shard does not serve it, rather than a picture produced + /// by guessing. Nothing on the wire carries a hued land tile today; if something ever + /// does, it arrives with a reason to choose. + /// + private static bool TryParseKey(string key, out bool land, out int id, out int hue) + { + land = false; + id = 0; + hue = 0; + + if (key == null) + return false; + + string[] parts = key.Split('/'); + + if (parts.Length < 2 || parts.Length > 3) + return false; + + if (parts[0] == LandFamily) + land = true; + else if (parts[0] != StaticFamily) + return false; + + if (!Int32.TryParse(parts[1], NumberStyles.None, CultureInfo.InvariantCulture, out id)) + return false; + + if (id < 0) + return false; + + if (land && id >= LandCount) + return false; + + if (parts.Length == 2) + return true; + + if (land) + return false; + + string segment = parts[2]; + + if (segment.Length < 2 || segment[0] != 'h') + return false; + + if (!Int32.TryParse(segment.Substring(1), NumberStyles.None, + CultureInfo.InvariantCulture, out hue)) + return false; + + // **`h0` is not a key.** Hue 0 on the wire means "this item is not hued", so the plain + // key already names its picture. Accepting `static/3922/h0` as a synonym would have + // the website store the identical PNG twice under two names, diff them separately on + // every Update, and show whichever row it happened to join against -- for a distinction + // that does not exist. The caller drops the segment instead. + return hue > 0; + } + + private static int ParseKeyCursor(string cursor) + { + if (cursor == null) + return 0; + + int value; + + if (cursor.StartsWith("k:", StringComparison.Ordinal) + && Int32.TryParse(cursor.Substring(2), NumberStyles.None, CultureInfo.InvariantCulture, out value)) + return Math.Max(0, value); + + return 0; + } + + /// + /// Everything that decides these bytes, hashed into one short id — the same guard + /// phase 3 built, over this family's inputs. + /// + /// Four files, and each earns its place: the art data file holds the pixels, + /// `hues.mul` holds the ramps, `tiledata.mul` decides which of the two hue modes an + /// item gets, and `verdata.mul` can patch any record in any of them. Leaving + /// `tiledata.mul` out would be the subtle one — a client patch that only flipped + /// PartialHue flags changes no pixel in any source file and every hued picture + /// derived from them. + /// + private static string SourceId() + { + var sb = new StringBuilder(256); + + sb.Append(BridgeAssets.EXTRACTOR_VERSION); + + foreach (string path in new[] + { + BridgeAssetValidator.ArtDataPath(), + FilePath("hues.mul"), + FilePath("tiledata.mul"), + FilePath("verdata.mul") + }) + { + sb.Append('|'); + + if (path == null) + continue; + + try + { + var info = new FileInfo(path); + + if (!info.Exists) + continue; + + sb.Append(info.Length).Append(',').Append(info.LastWriteTimeUtc.Ticks); + } + catch + { + // An unreadable file is itself a state, and one that must not change from page + // to page without being noticed. Leaving the slot empty does that. + } + } + + return BridgeAssets.Sha256Hex(Encoding.UTF8.GetBytes(sb.ToString())).Substring(0, 16); + } + + private static string FilePath(string name) + { + try + { + return Files.GetFilePath(name); + } + catch + { + return null; + } + } + + // ── shared plumbing ────────────────────────────────────────────────────────────────── + + /// + /// The art index and its record reader, opened for one reply and closed with it — the + /// same lifetime rule phase 3's Readers follows, and for the same reason: a page + /// decodes hundreds of sprites through them and opening them is microseconds, so + /// holding handles on the operator's client files for the life of a cache buys nothing. + /// + private sealed class Readers : IDisposable + { + public readonly FileIndex Index; + public readonly BridgeAssetValidator.RecordReader Reader; + public readonly long DataLength; + public readonly long VerdataLength; + + /// + /// Which file the pixels came out of — `uop` or `legacy` — carried on every row + /// beside the body catalogue's own `source` (§4.9). On this plane it answers §4.6's + /// operator question: art added to `art.mul` while `artLegacyMUL.uop` is present is + /// never read, and a row that says `uop` is what says so. + /// + public readonly string Source; + + public Readers() + { + string data = BridgeAssetValidator.ArtDataPath(); + string verdata = FilePath("verdata.mul"); + + DataLength = BridgeAssetValidator.MulLength(data); + VerdataLength = BridgeAssetValidator.MulLength(verdata); + + Source = data != null && data.EndsWith(".uop", StringComparison.OrdinalIgnoreCase) + ? "uop" + : "legacy"; + + try + { + Index = BridgeAssetValidator.OpenArtIndex(); + + if (data != null) + Reader = new BridgeAssetValidator.RecordReader(data, verdata); + } + catch (Exception e) + { + Console.WriteLine("[Bridge] art: could not open the art files: {0}", e.Message); + } + } + + public void Dispose() + { + if (Reader == null) + return; + + try + { + Reader.Dispose(); + } + catch + { + // Closing a read-only handle. Nothing useful is left to do. + } + } + } + + /// + /// Lets the held pictures go once nothing has asked for one in five minutes. The id is + /// derived from the client files rather than minted per build, so a walk that spans the + /// drop resumes against the same catalogue instead of starting over. + /// + private static void Sweep() + { + lock (_sync) + { + if (_cache == null) + return; + + if (DateTime.UtcNow - _cache.LastUsed > IdleFor) + _cache = null; + } + } + + public static string Status() + { + lock (_sync) + { + if (_cache == null) + return "art(empty)"; + + return String.Format("art(id={0} held={1} bytes={2} cap={3})", + _cache.Id, _cache.ByKey.Count, _cache.Bytes, BridgeConfig.AssetArtCacheBytes); + } + } + } +} diff --git a/overlay/Scripts/Custom/Bridge/BridgeAssets.cs b/overlay/Scripts/Custom/Bridge/BridgeAssets.cs index 9bde8cb..b6930ca 100644 --- a/overlay/Scripts/Custom/Bridge/BridgeAssets.cs +++ b/overlay/Scripts/Custom/Bridge/BridgeAssets.cs @@ -1,7 +1,10 @@ using System; using System.Collections.Generic; +using System.Drawing; +using System.Drawing.Imaging; using System.Globalization; using System.IO; +using System.Runtime.InteropServices; using System.Security.Cryptography; using System.Text; using System.Threading; @@ -123,7 +126,58 @@ namespace Server.Custom.Bridge if (!BridgeConfig.Enabled) return; + DisableTheLibraryCache(); + BridgeBoot.RegisterHandler("assets.sources", OnSources); + BridgeBoot.RegisterHandler("assets.fetch", OnFetch); + } + + /// + /// **Turns Ultima.Files.CacheData off for the life of the process** (phase 5, + /// §17.10). One line, and it answers two separate problems that both end in a + /// confident wrong picture or an out-of-memory shard. + /// + /// **The poisoning.** Art.GetStatic and Art.GetLand memoise into a + /// Bitmap[0xFFFF] and hand back **the same instance** on every call, while + /// Hue.ApplyTo repaints a bitmap **in place**. So hueing a static edits the + /// library's cached copy: measured on this client, hue item 3922 once and every later + /// request for the *plain* 3922 comes back hued, and a second hue stacks on the first. + /// Nothing downstream can see it — the row is the right size, the right shape and the + /// right id. It is §4.5's failure mode arriving through a completely different door. + /// + /// **The retention.** That array is never trimmed. Decoding this client's 39,189 + /// statics once would leave 74 MB of Bitmap in a static field of a game server, + /// kept for as long as the process lives, to serve pictures nobody asked for twice. + /// + /// The obvious alternative — copy each bitmap before hueing — was rejected, and not + /// only for the retention: new Bitmap(src) **throws** on the + /// Format16bppArgb1555 these decoders produce, so the copy has to name the + /// source pixel format explicitly, which is a subtlety on the wrong side of a + /// correctness boundary. + /// + /// **What it costs is nothing measurable here.** Animations — the whole of the + /// body catalogue — does not consult this flag at all, and + /// and each keep their own cache of + /// *encoded PNG bytes*, which is the thing worth holding: a tenth of the size, already + /// hashed, and released when it goes idle. + /// + /// It is a process-global on a library nothing else in this overlay reads, which is why + /// setting it here rather than saving and restoring it around each decode is safe — + /// and a save/restore would not be, because the asset worker is a thread. + /// + private static void DisableTheLibraryCache() + { + try + { + Files.CacheData = false; + } + catch (Exception e) + { + // A client this library cannot even open. The families report that for themselves, + // per key, with a reason; it must not stop the plugin booting. + Console.WriteLine("[Bridge] assets: could not disable the Ultima bitmap cache: {0}", + e.Message); + } } public static string Status() @@ -228,6 +282,248 @@ namespace Server.Custom.Bridge BridgeLink.Emit(sb.End()); } + // ── assets.fetch, and the families behind it (§5, phase 5) ─────────────────────────── + + /// + /// One family's answer to a fetch. Runs on the asset worker, never the Core thread. + /// + internal delegate void FamilyFetch(string reqId, List keys, string catalog, string cursor); + + private static readonly Dictionary _families = + new Dictionary(StringComparer.Ordinal); + + /// + /// Claims one §5 key family for a reader. + /// + /// Phase 3 gave assets.fetch to the body catalogue outright, which was right + /// while there was one family and wrong the moment there were three: the command is + /// the *transport*, and the family is a property of the key. So the shared parts — the + /// correlation id, the operator's consent, the key-count ceiling, and deciding which + /// reader a request belongs to — live here once, and a family only ever sees a list of + /// keys it owns. + /// + /// Registration is order-independent on purpose: ServUO calls every + /// Initialize in an order nothing here controls, and this fills a dictionary the + /// handler does not read until a request arrives. + /// + internal static void RegisterFamily(string name, FamilyFetch fetch) + { + lock (_families) + { + _families[name] = fetch; + } + } + + /// The families this shard can serve, for §6's stage 1 and for diagnostics. + internal static List Families() + { + lock (_families) + { + var names = new List(_families.Keys); + names.Sort(StringComparer.Ordinal); + return names; + } + } + + private static FamilyFetch FamilyFor(string name) + { + lock (_families) + { + FamilyFetch fetch; + return _families.TryGetValue(name, out fetch) ? fetch : null; + } + } + + /// + /// The family segment of a §5 key: everything before the first `/`. + /// + internal static string FamilyOfKey(string key) + { + if (String.IsNullOrEmpty(key)) + return null; + + int slash = key.IndexOf('/'); + + return slash <= 0 ? null : key.Substring(0, slash); + } + + /// + /// §14's `assets.fetch`, for every family. + /// + /// **The family is derived from the keys and is not a request field.** §5 made the key + /// the address of an asset, so a request that had to name its family as well would have + /// two places to be wrong and one of them silent. A batch must be of one family — + /// mixing them is refused rather than split — because the reply carries a single + /// `catalog` id, and that id is what stops an operator patching their client mid-import + /// from stitching one asset set out of two. Two families, two fingerprints, and a reply + /// that claimed one of them would be lying about the other. + /// + private static void OnFetch(Dictionary o) + { + var reqId = BridgeJson.GetString(o, "reqId"); + + if (reqId == null) + { + Fail(null, "BAD_REQUEST", "assets.fetch requires a reqId"); + return; + } + + if (!BridgeConfig.AssetsEnabled) + { + Fail(reqId, "DISABLED", "asset extraction is disabled on this shard"); + return; + } + + var keys = BridgeJson.GetStringList(o, "keys"); + + if (keys.Count == 0) + { + Fail(reqId, "BAD_REQUEST", "assets.fetch requires a non-empty `keys` array"); + return; + } + + if (keys.Count > BridgeConfig.AssetFetchKeys) + { + Fail(reqId, "BAD_REQUEST", + "assets.fetch takes at most " + BridgeConfig.AssetFetchKeys + + " keys per request (asked for " + keys.Count + ")"); + return; + } + + string family = FamilyOfKey(keys[0]); + + for (int i = 1; i < keys.Count; i++) + { + if (String.Equals(FamilyOfKey(keys[i]), family, StringComparison.Ordinal)) + continue; + + Fail(reqId, "BAD_REQUEST", + "assets.fetch takes keys of one family per request; this one mixes '" + + family + "' with '" + FamilyOfKey(keys[i]) + "'"); + return; + } + + FamilyFetch fetch = FamilyFor(family); + + if (fetch == null) + { + Fail(reqId, "BAD_REQUEST", + "this shard serves no '" + family + "' asset family (it serves " + + String.Join(", ", Families().ToArray()) + ")"); + return; + } + + var catalog = BridgeJson.GetString(o, "catalog"); + var cursor = BridgeJson.GetString(o, "cursor"); + + Accept(reqId, "assets.fetch", () => fetch(reqId, keys, catalog, cursor)); + } + + /// + /// ARGB1555 to a PNG with a transparent background. + /// + /// Frame writes 16-bit ARGB1555: a pixel the sprite does not cover is left as + /// zero and a pixel it does cover carries the top bit set. Saving that format straight + /// to PNG asks GDI+ to make the conversion, and what it does with a one-bit alpha + /// channel varies by platform — on Mono it is a different implementation entirely. A + /// sprite that came back with a black rectangle behind it would look fine in a test + /// that only checked the bytes decoded, and wrong on every page that showed it. + /// + /// So the expansion is done here, explicitly: alpha bit clear becomes fully + /// transparent, and each 5-bit channel is widened to 8 bits by repeating its high bits + /// ((c << 3) | (c >> 2)) rather than by shifting alone, which would + /// cap white at 248 and tint the whole catalogue. + /// + internal static byte[] BitmapToPng(Bitmap source) + { + var rect = new Rectangle(0, 0, source.Width, source.Height); + + if (source.PixelFormat != PixelFormat.Format16bppArgb1555) + { + // Not what this library has ever produced. Save it rather than reinterpret it: + // guessing at an unknown layout is how a catalogue fills with confident nonsense. + using (var ms = new MemoryStream()) + { + source.Save(ms, ImageFormat.Png); + return ms.ToArray(); + } + } + + using (var target = new Bitmap(source.Width, source.Height, PixelFormat.Format32bppArgb)) + { + BitmapData src = source.LockBits(rect, ImageLockMode.ReadOnly, PixelFormat.Format16bppArgb1555); + BitmapData dst = null; + + try + { + dst = target.LockBits(rect, ImageLockMode.WriteOnly, PixelFormat.Format32bppArgb); + + var line = new short[source.Width]; + var outLine = new int[source.Width]; + + for (int y = 0; y < source.Height; y++) + { + Marshal.Copy(new IntPtr(src.Scan0.ToInt64() + ((long)y * src.Stride)), + line, 0, source.Width); + + for (int x = 0; x < source.Width; x++) + { + int p = line[x] & 0xFFFF; + + if ((p & 0x8000) == 0) + { + outLine[x] = 0; + continue; + } + + int r = (p >> 10) & 0x1F; + int g = (p >> 5) & 0x1F; + int b = p & 0x1F; + + outLine[x] = unchecked((int)0xFF000000) + | (((r << 3) | (r >> 2)) << 16) + | (((g << 3) | (g >> 2)) << 8) + | ((b << 3) | (b >> 2)); + } + + Marshal.Copy(outLine, 0, new IntPtr(dst.Scan0.ToInt64() + ((long)y * dst.Stride)), + source.Width); + } + } + finally + { + if (dst != null) + target.UnlockBits(dst); + + source.UnlockBits(src); + } + + using (var ms = new MemoryStream()) + { + target.Save(ms, ImageFormat.Png); + return ms.ToArray(); + } + } + } + + /// + /// SHA-256, lowercase hex. Shared because the hash in a manifest row, the hash in a + /// fetch row and the hash the website stores must be one function. + /// + internal static string Sha256Hex(byte[] bytes) + { + using (var sha = SHA256.Create()) + { + byte[] digest = sha.ComputeHash(bytes); + var sb = new StringBuilder(digest.Length * 2); + + foreach (byte b in digest) + sb.Append(b.ToString("x2", CultureInfo.InvariantCulture)); + + return sb.ToString(); + } + } + /// /// The asset plane's one refusal frame, shared by every family on it. /// @@ -354,6 +650,24 @@ namespace Server.Custom.Bridge sb.Str("artDataFile", artData == null ? null : Path.GetFileName(artData)); + // Which §5 families this shard can be asked for. Additive, so the protocol stays 8: a + // consumer that does not read it behaves exactly as it did. One that does can tell an + // older overlay (bodies only) from this one without discovering it as a refused fetch + // halfway through a warm pass. + var families = Families(); + + sb.Append(",\"families\":["); + + for (int i = 0; i < families.Count; i++) + { + if (i > 0) + sb.Append(','); + + BridgeJson.Text(sb, families[i]); + } + + sb.Append(']'); + var page = new PageBuilder(sb, "files", BridgeConfig.AssetBatchBytes); bool anyMissingHash = false; diff --git a/overlay/Scripts/Custom/Bridge/BridgeBoot.cs b/overlay/Scripts/Custom/Bridge/BridgeBoot.cs index 1511e8f..48cb9ff 100644 --- a/overlay/Scripts/Custom/Bridge/BridgeBoot.cs +++ b/overlay/Scripts/Custom/Bridge/BridgeBoot.cs @@ -262,6 +262,7 @@ namespace Server.Custom.Bridge e.Mobile.SendMessage("Bridge: {0}", BridgeOneShots.Status()); e.Mobile.SendMessage("Bridge: {0}", BridgeAssets.Status()); e.Mobile.SendMessage("Bridge: {0}", BridgeCatalog.Status()); + e.Mobile.SendMessage("Bridge: {0}", BridgeArt.Status()); break; } } diff --git a/overlay/Scripts/Custom/Bridge/BridgeCatalog.cs b/overlay/Scripts/Custom/Bridge/BridgeCatalog.cs index 20f19fd..1591f23 100644 --- a/overlay/Scripts/Custom/Bridge/BridgeCatalog.cs +++ b/overlay/Scripts/Custom/Bridge/BridgeCatalog.cs @@ -1,11 +1,8 @@ using System; using System.Collections.Generic; using System.Drawing; -using System.Drawing.Imaging; using System.Globalization; using System.IO; -using System.Runtime.InteropServices; -using System.Security.Cryptography; using System.Text; using Ultima; @@ -101,7 +98,10 @@ namespace Server.Custom.Bridge return; BridgeBoot.RegisterHandler("assets.manifest", OnManifest); - BridgeBoot.RegisterHandler("assets.fetch", OnFetch); + + // `assets.fetch` is shared plumbing as of phase 5 (§5): BridgeAssets owns the command, + // decides which family a batch of keys belongs to, and calls the reader that owns it. + BridgeAssets.RegisterFamily(Family, ReplyFetch); } // ── the cache ──────────────────────────────────────────────────────────────────────── @@ -324,36 +324,11 @@ namespace Server.Custom.Bridge // ── assets.fetch ───────────────────────────────────────────────────────────────────── - private static void OnFetch(Dictionary o) - { - string reqId; - - if (!Admit(o, "assets.fetch", out reqId)) - return; - - var keys = BridgeJson.GetStringList(o, "keys"); - - if (keys.Count == 0) - { - BridgeAssets.Fail(reqId, "BAD_REQUEST", - "assets.fetch requires a non-empty `keys` array"); - return; - } - - if (keys.Count > BridgeConfig.AssetFetchKeys) - { - BridgeAssets.Fail(reqId, "BAD_REQUEST", - "assets.fetch takes at most " + BridgeConfig.AssetFetchKeys - + " keys per request (asked for " + keys.Count + ")"); - return; - } - - var catalog = BridgeJson.GetString(o, "catalog"); - var cursor = BridgeJson.GetString(o, "cursor"); - - BridgeAssets.Accept(reqId, "assets.fetch", () => ReplyFetch(reqId, keys, catalog, cursor)); - } - + /// + /// The `body` family's half of assets.fetch. The correlation id, the operator's + /// consent, the key ceiling and the family decision have already been made by + /// 's caller; every key here is this family's. + /// private static void ReplyFetch(string reqId, List keys, string expected, string cursor) { string imagingReason; @@ -651,7 +626,7 @@ namespace Server.Custom.Bridge Png = png, Width = pixels.Width, Height = pixels.Height, - Sha256 = Hash(png), + Sha256 = BridgeAssets.Sha256Hex(png), Source = "uop" }; } @@ -673,7 +648,7 @@ namespace Server.Custom.Bridge if (bitmap == null || bitmap.Width <= 0 || bitmap.Height <= 0) return null; - byte[] png = ToPng(bitmap); + byte[] png = BridgeAssets.BitmapToPng(bitmap); if (png == null) return null; @@ -687,112 +662,11 @@ namespace Server.Custom.Bridge Png = png, Width = bitmap.Width, Height = bitmap.Height, - Sha256 = Hash(png), + Sha256 = BridgeAssets.Sha256Hex(png), Source = "legacy" }; } - /// - /// ARGB1555 to a PNG with a transparent background. - /// - /// Frame writes 16-bit ARGB1555: a pixel the sprite does not cover is left as - /// zero and a pixel it does cover carries the top bit set. Saving that format straight - /// to PNG asks GDI+ to make the conversion, and what it does with a one-bit alpha - /// channel varies by platform — on Mono it is a different implementation entirely. A - /// sprite that came back with a black rectangle behind it would look fine in a test - /// that only checked the bytes decoded, and wrong on every page that showed it. - /// - /// So the expansion is done here, explicitly: alpha bit clear becomes fully - /// transparent, and each 5-bit channel is widened to 8 bits by repeating its high bits - /// ((c << 3) | (c >> 2)) rather than by shifting alone, which would - /// cap white at 248 and tint the whole catalogue. - /// - private static byte[] ToPng(Bitmap source) - { - var rect = new Rectangle(0, 0, source.Width, source.Height); - - if (source.PixelFormat != PixelFormat.Format16bppArgb1555) - { - // Not what this library has ever produced. Save it rather than reinterpret it: - // guessing at an unknown layout is how a catalogue fills with confident nonsense. - using (var ms = new MemoryStream()) - { - source.Save(ms, ImageFormat.Png); - return ms.ToArray(); - } - } - - using (var target = new Bitmap(source.Width, source.Height, PixelFormat.Format32bppArgb)) - { - BitmapData src = source.LockBits(rect, ImageLockMode.ReadOnly, PixelFormat.Format16bppArgb1555); - BitmapData dst = null; - - try - { - dst = target.LockBits(rect, ImageLockMode.WriteOnly, PixelFormat.Format32bppArgb); - - var line = new short[source.Width]; - var outLine = new int[source.Width]; - - for (int y = 0; y < source.Height; y++) - { - Marshal.Copy(new IntPtr(src.Scan0.ToInt64() + ((long)y * src.Stride)), - line, 0, source.Width); - - for (int x = 0; x < source.Width; x++) - { - int p = line[x] & 0xFFFF; - - if ((p & 0x8000) == 0) - { - outLine[x] = 0; - continue; - } - - int r = (p >> 10) & 0x1F; - int g = (p >> 5) & 0x1F; - int b = p & 0x1F; - - outLine[x] = unchecked((int)0xFF000000) - | (((r << 3) | (r >> 2)) << 16) - | (((g << 3) | (g >> 2)) << 8) - | ((b << 3) | (b >> 2)); - } - - Marshal.Copy(outLine, 0, new IntPtr(dst.Scan0.ToInt64() + ((long)y * dst.Stride)), - source.Width); - } - } - finally - { - if (dst != null) - target.UnlockBits(dst); - - source.UnlockBits(src); - } - - using (var ms = new MemoryStream()) - { - target.Save(ms, ImageFormat.Png); - return ms.ToArray(); - } - } - } - - private static string Hash(byte[] bytes) - { - using (var sha = SHA256.Create()) - { - byte[] digest = sha.ComputeHash(bytes); - var sb = new StringBuilder(digest.Length * 2); - - foreach (byte b in digest) - sb.Append(b.ToString("x2", CultureInfo.InvariantCulture)); - - return sb.ToString(); - } - } - // ── player bodies (§5.2) ───────────────────────────────────────────────────────────── /// @@ -997,7 +871,7 @@ namespace Server.Custom.Bridge } } - return Hash(Encoding.UTF8.GetBytes(sb.ToString())).Substring(0, 16); + return BridgeAssets.Sha256Hex(Encoding.UTF8.GetBytes(sb.ToString())).Substring(0, 16); } // ── shared plumbing ────────────────────────────────────────────────────────────────── diff --git a/overlay/Scripts/Custom/Bridge/BridgeConfig.cs b/overlay/Scripts/Custom/Bridge/BridgeConfig.cs index dede1bd..60adaa7 100644 --- a/overlay/Scripts/Custom/Bridge/BridgeConfig.cs +++ b/overlay/Scripts/Custom/Bridge/BridgeConfig.cs @@ -124,6 +124,13 @@ namespace Server.Custom.Bridge public static int AssetPlayerDirection { get; private set; } public static int AssetCreatureDirection { get; private set; } + // How many bytes of rendered item and land art the shard holds between requests (§11, + // phase 5). This is a convenience, not a store: the website keeps every picture it fetches + // and does not ask twice, so what this actually buys is the second page of a batch, a + // retry after a 425, and the same item appearing in two rows of one page. Sized so a + // full 512 KB batch and the one before it both fit with room over. + public static int AssetArtCacheBytes { get; private set; } + public static int LeaseMaxDurationSec { get; private set; } public static int LeaseGraceSec { get; private set; } @@ -222,6 +229,16 @@ namespace Server.Custom.Bridge AssetPlayerDirection = Clamp(Config.Get("Bridge.AssetPlayerDirection", 0), 0, 4); AssetCreatureDirection = Clamp(Config.Get("Bridge.AssetCreatureDirection", 1), 0, 4); + // The floor is one batch: a cache that cannot hold the page being built evicts rows + // while they are still being written, which is a cache that costs and never pays. The + // ceiling is a game server's memory, and 64 MB of PNG is already ~34,000 sprites -- + // most of this client's art, held for a working set that is measured in hundreds. + AssetArtCacheBytes = Config.Get("Bridge.AssetArtCacheBytes", 16 * 1024 * 1024); + if (AssetArtCacheBytes < AssetBatchBytes) + AssetArtCacheBytes = AssetBatchBytes; + if (AssetArtCacheBytes > 64 * 1024 * 1024) + AssetArtCacheBytes = 64 * 1024 * 1024; + StatSweepSeconds = Config.Get("Bridge.StatSweepSeconds", 30); DecaySweepSeconds = Config.Get("Bridge.DecaySweepSeconds", 60); EconomySweepSeconds = Config.Get("Bridge.EconomySweepSeconds", 300); -- 2.49.1 From a9bd18e48ee75ed2fcde31c49309c5a7f2d9701f Mon Sep 17 00:00:00 2001 From: wtclaude Date: Mon, 14 Sep 2026 01:09:38 -0500 Subject: [PATCH 08/11] feat(asset-bridge): the 73 bodies action 0 could not see, and the ceiling that makes looking safe (Phase 6) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The catalogue asked every body for action 0 and reported the rest absent. 73 of this client's bodies have no art there and real art deeper — body 820's first drawn action is 23, and it is a horse — so they rendered as text on the bestiary. The catalogue now falls back to the first action that has art, and the key names that action (`body/820/a23`). 1,022 -> 1,095 rows. Walking the action axis is the one thing that can walk off the end of a body's slots, and the slots after a body's band are the NEXT BODY'S. Measured here: one action past the band, 643 of 795 legacy bodies return a fully validated picture and 452 of those are byte-identical to body+1's action 0 (body 1 action 22 is an ettin; body 3's is an imp, both confirmed by rendering them). Phase 0's validator cannot catch that — the record is real — so the ceiling refuses the ADDRESS, in ResolveAnimation where every caller already goes. The ceiling is the index banding, never `Animations.GetAnimLength`: for a body reaching file type 5 as id 34 that function answers 22 while the arithmetic gives 13, and the difference is nine actions of another creature's art. A fetch serves only the key the catalogue chose for that body. `body/820/a0` and `body/400/a2` come back `unsupported` with the chosen action alongside, never by decoding what was asked for. `EXTRACTOR_VERSION` 2 -> 3 (unchanged input, a different answer). Protocol stays 8 — `action` on a manifest/fetch row is additive. Deep frame keys and the bulk-fill switch that §16 planned for this phase were NOT built: the site displays still pictures, and a complete one-direction animation set measures 174,453 frames / 281.5 MB against no consumer (docs §11.2, org lead 2026-09-11). Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4 --- .../Custom/Bridge/BridgeAssetValidator.cs | 112 ++++++++++ overlay/Scripts/Custom/Bridge/BridgeAssets.cs | 7 +- .../Scripts/Custom/Bridge/BridgeCatalog.cs | 208 ++++++++++++++---- 3 files changed, 282 insertions(+), 45 deletions(-) diff --git a/overlay/Scripts/Custom/Bridge/BridgeAssetValidator.cs b/overlay/Scripts/Custom/Bridge/BridgeAssetValidator.cs index e4ed921..e1ecacc 100644 --- a/overlay/Scripts/Custom/Bridge/BridgeAssetValidator.cs +++ b/overlay/Scripts/Custom/Bridge/BridgeAssetValidator.cs @@ -406,11 +406,123 @@ namespace Server.Custom.Bridge return false; } + int actions = ActionsOf(translated, fileType); + + if (action >= actions) + { + // §4.10, measured in phase 6: this is the never-sweep rule again, one axis over. + // A body's slots are contiguous and the next body's begin immediately after them, + // so `index + action * 5` past the ceiling addresses ANOTHER BODY'S action — a + // real record, at a real offset, that every check below passes. Measured on this + // client: of 795 legacy bodies, 643 return a fully validated picture one action + // past their band and **452 of those are byte-identical to body+1's action 0**. + // Body 1 action 22 is an ettin; body 3 action 22 is an imp. Nothing downstream + // can tell, which is why the refusal has to be here. + reason = "body " + body + " has " + actions + " actions in file type " + fileType + + "; action " + action + " belongs to the next body"; + return false; + } + index = AnimIndexOf(translated, fileType) + (action * 5) + direction; return true; } + /// + /// How many actions the index reserves for a body — the only safe ceiling, and it is + /// the banding rather than the library's own answer. + /// + /// Animations.GetAnimLength exists and looks like the right source. It is not: + /// for a body reaching file type 5 as id 34 it answers **22** while + /// puts that body in the 65-slot band, which is **13**. The + /// two disagree on exactly one body of this client (reached by translation from body + /// 276), and taking the larger number is nine actions of somebody else's art. So the + /// count is derived from the same arithmetic that produces the offset, in the same + /// file, where the two cannot drift apart. + /// + public static bool ActionCount(int body, out int actions, out int fileType, out string reason) + { + reason = null; + actions = 0; + fileType = 0; + + if (body <= 0) + { + reason = "body " + body + " is not addressable"; + return false; + } + + int translated = body; + int hue = 0; + + try + { + Animations.Translate(ref translated, ref hue); + fileType = BodyConverter.Convert(ref translated); + } + catch (Exception e) + { + reason = "body.def/bodyconv.def lookup failed: " + e.GetType().Name; + return false; + } + + if (AnimDataPath(fileType) == null) + { + reason = "bodyconv sends body " + body + " to file type " + fileType + + ", which this client does not have"; + return false; + } + + actions = ActionsOf(translated, fileType); + + return true; + } + + /// + /// The banding of , read as an action count: a body's slots + /// are five directions per action, so the band size divided by five is how many + /// actions it owns. + /// + private static int ActionsOf(int body, int fileType) + { + return SlotsOf(body, fileType) / 5; + } + + /// + /// How many index slots 's arithmetic gives this body. The + /// bands are transcribed there and their sizes here, from the same source and in the + /// same order, because a ceiling that disagrees with an offset is worse than no + /// ceiling at all. + /// + private static int SlotsOf(int body, int fileType) + { + switch (fileType) + { + case 2: + return body < 200 ? 110 : 65; + + case 3: + if (body < 300) + return 65; + + return body < 400 ? 110 : 175; + + case 5: + // Body 34's exclusion again — it is in the second band here, so it owns 13 + // actions and not 22. This is the one body `GetAnimLength` is wrong about. + if (body < 200 && body != 34) + return 110; + + return body < 400 ? 65 : 175; + + default: // 1 and 4 share their banding + if (body < 200) + return 110; + + return body < 400 ? 65 : 175; + } + } + /// /// Animations.GetFileIndex's own arithmetic, which is private. The banding is /// per file type and the boundaries differ between them, so this is transcribed rather diff --git a/overlay/Scripts/Custom/Bridge/BridgeAssets.cs b/overlay/Scripts/Custom/Bridge/BridgeAssets.cs index b6930ca..5cb2417 100644 --- a/overlay/Scripts/Custom/Bridge/BridgeAssets.cs +++ b/overlay/Scripts/Custom/Bridge/BridgeAssets.cs @@ -85,8 +85,13 @@ namespace Server.Custom.Bridge /// stock client is 235 new sprites and two of them player-character bodies; and the /// player-body set no longer carries ghost ids. Every client file is byte-identical /// and the answer is different, which is precisely what this number exists to say. + /// + /// **3** — phase 6 (§4.10, §11.2). A body with no art at action 0 is catalogued at + /// the first action that has any, and its key names that action. 74 more bodies on a + /// stock client, no existing key's bytes changed — but a body that was absent is now + /// a row, which is the same "unchanged input, different answer" this number covers. /// - public const int EXTRACTOR_VERSION = 2; + public const int EXTRACTOR_VERSION = 3; // ── the one slot (§3.2) ────────────────────────────────────────────────────────────── diff --git a/overlay/Scripts/Custom/Bridge/BridgeCatalog.cs b/overlay/Scripts/Custom/Bridge/BridgeCatalog.cs index 1591f23..6cd82c6 100644 --- a/overlay/Scripts/Custom/Bridge/BridgeCatalog.cs +++ b/overlay/Scripts/Custom/Bridge/BridgeCatalog.cs @@ -14,11 +14,20 @@ namespace Server.Custom.Bridge /// /// One thumbnail per creature body: the working set that makes a bestiary, a marketplace /// listing and a character sheet render. Everything deeper — every action, every frame — - /// is the same addressing scheme at a deeper key, fetched on demand in a later phase; this - /// is the set that is worth importing before anything asks for it, because on this - /// machine's client it is **1,022 sprites at about a kilobyte each** — 787 out of the - /// legacy `anim*.mul` files and, since phase 4, 235 more out of `AnimationFrame*.uop`, - /// which ServUO's vendored decoder never opens (§4.3, §4.9). + /// is the same addressing scheme at a deeper key and is **not served** (§11.2, phase 6: + /// the site shows still pictures, so frames wait for a consumer that wants them). This is + /// the set that is worth importing before anything asks for it, because on this machine's + /// client it is **1,096 sprites at about a kilobyte each** — 787 out of the legacy + /// `anim*.mul` files, 235 more out of `AnimationFrame*.uop` since phase 4 (§4.3, §4.9), + /// and 74 more since phase 6, which have no art at action 0 and real art deeper. + /// + /// ── **One picture per body, at the first action that has one** ── + /// + /// A key carries the action it came from — `body/820/a23` for a horse whose action 0 is + /// empty — so the catalogue is still exactly one row per body, and the row says which + /// picture it is. What it never does is decode an action the walk did not choose: a fetch + /// for `body/820/a0` is `unsupported`, not a second attempt, because the slots past a + /// body's band belong to the next body and every check passes on them (§4.10). /// /// Two request kinds, which are §6's two stages for assets rather than for sources: /// @@ -89,8 +98,22 @@ namespace Server.Custom.Bridge /// Bodies are addressable to 2047; the sweep behind §4.8 covered exactly this. private const int MaxBody = 2047; - /// The catalogue is first frames only. Deep keys are phase 6. - private const int CatalogAction = 0; + /// + /// The action a thumbnail comes from when the body has one, which is nearly always. + /// Everything deeper than a first frame is deferred — see §11.2. + /// + private const int PreferredAction = 0; + + /// + /// How far the fallback looks for a body with no art at . + /// + /// 35 because that is the largest band any file type gives a body, so an action beyond + /// it is not something the client's own layout can name. The legacy arm is bounded + /// tighter still and per body, by — + /// this is only the scan's outer stop, and it is the UOP arm's real one, where an + /// action is a named entry rather than an offset. + /// + private const int MaxAction = 35; public static void Initialize() { @@ -110,6 +133,14 @@ namespace Server.Custom.Bridge { public string Key; public int Body; + + /// + /// Which action this body's thumbnail came from — + /// for all but 74 bodies on this client, and on the wire because the key names it + /// (§5, §11.2). A consumer that assumes `a0` would build a dead URL for a horse. + /// + public int Action; + public int Direction; public int FileType; public string Sha256; @@ -130,8 +161,15 @@ namespace Server.Custom.Bridge private sealed class Catalog { public string Id; - public readonly Dictionary ByKey = - new Dictionary(StringComparer.Ordinal); + + /// + /// Keyed by **body**, not by asset key, since phase 6: a body's key now carries + /// the action its picture came from, so the key cannot be spelled until the body + /// has been resolved. A fetch arrives holding a key and has to reach the same + /// sprite, which it does by parsing the body out of it and comparing. + /// + public readonly Dictionary ByBody = new Dictionary(); + public readonly List Order = new List(); /// The next body the scan has yet to look at. @@ -264,6 +302,7 @@ namespace Server.Custom.Bridge item.Append(",\"width\":").Append(sprite.Width.ToString(CultureInfo.InvariantCulture)); item.Append(",\"height\":").Append(sprite.Height.ToString(CultureInfo.InvariantCulture)); item.Append(",\"body\":").Append(sprite.Body.ToString(CultureInfo.InvariantCulture)); + item.Append(",\"action\":").Append(sprite.Action.ToString(CultureInfo.InvariantCulture)); item.Append(",\"direction\":").Append(sprite.Direction.ToString(CultureInfo.InvariantCulture)); item.Append(",\"source\":\"").Append(sprite.Source).Append('"'); item.Append('}'); @@ -413,9 +452,9 @@ namespace Server.Custom.Bridge /// private static string Render(Catalog catalog, Readers readers, string key) { - int body; + int body, action; - if (!TryParseKey(key, out body)) + if (!TryParseKey(key, out body, out action)) { var bad = new StringBuilder(96); bad.Append("{\"key\":"); @@ -437,12 +476,26 @@ namespace Server.Custom.Bridge return item.ToString(); } + if (sprite.Action != action) + { + // The body has a picture, but not at the action this key names. Two ways to get + // here and both are the caller's: an old manifest that catalogued this body at + // `a0` before the client was patched, or a key someone built by assuming the + // action. Neither is served — decoding the asked-for action instead would be + // §4.10's wrong picture, arrived at politely. + item.Append(",\"status\":\"unsupported\""); + item.Append(",\"action\":").Append(sprite.Action.ToString(CultureInfo.InvariantCulture)); + item.Append('}'); + return item.ToString(); + } + item.Append(",\"status\":\"ok\""); item.Append(",\"sha256\":\"").Append(sprite.Sha256).Append('"'); item.Append(",\"bytes\":").Append(sprite.Png.Length.ToString(CultureInfo.InvariantCulture)); item.Append(",\"width\":").Append(sprite.Width.ToString(CultureInfo.InvariantCulture)); item.Append(",\"height\":").Append(sprite.Height.ToString(CultureInfo.InvariantCulture)); item.Append(",\"body\":").Append(sprite.Body.ToString(CultureInfo.InvariantCulture)); + item.Append(",\"action\":").Append(sprite.Action.ToString(CultureInfo.InvariantCulture)); item.Append(",\"direction\":").Append(sprite.Direction.ToString(CultureInfo.InvariantCulture)); item.Append(",\"source\":\"").Append(sprite.Source).Append('"'); item.Append(",\"png\":\"").Append(Convert.ToBase64String(sprite.Png)).Append("\"}"); @@ -459,13 +512,11 @@ namespace Server.Custom.Bridge /// private static Sprite Resolve(Catalog catalog, Readers readers, int body) { - string key = Key(body); - lock (_sync) { Sprite cached; - if (catalog.ByKey.TryGetValue(key, out cached)) + if (catalog.ByBody.TryGetValue(body, out cached)) return cached; } @@ -473,38 +524,88 @@ namespace Server.Custom.Bridge ? BridgeConfig.AssetPlayerDirection : BridgeConfig.AssetCreatureDirection; - // Legacy first, always. The vendored decoder is what 787 of this client's bodies come - // out of, it is what phase 3 measured, and the UOP packages hold a different and - // mostly disjoint set (measured: of the 244 bodies they carry, 8 also have legacy - // art). So this is a fallback rather than a choice, and no body changes reader while - // a client sits still. - Sprite sprite = ResolveLegacy(key, readers, body, direction) - ?? ResolveUop(key, readers, body, direction); + Sprite sprite = ResolveAny(readers, body, direction); if (sprite == null) return null; lock (_sync) { - if (!catalog.ByKey.ContainsKey(key)) + if (!catalog.ByBody.ContainsKey(body)) { - catalog.ByKey[key] = sprite; + catalog.ByBody[body] = sprite; catalog.Order.Add(sprite); } - return catalog.ByKey[key]; + return catalog.ByBody[body]; } } + /// + /// One body's thumbnail: action 0 if it has one, otherwise the first action that does. + /// + /// ── **Why there is a fallback at all** ── + /// + /// Through phase 5 a body with no art at action 0 was simply absent, and on this + /// client **74 bodies are in exactly that state while carrying real art deeper** — 66 + /// of them UOP, 8 legacy. Body 820's first drawn action is 23, and it is a horse. + /// They rendered as text on the bestiary for want of looking one action further. + /// + /// ── **Why the key says which action it is** ── + /// + /// The fallback's picture is `body/820/a23`, not `body/820/a0`. Naming it `a0` would + /// have been fewer changes downstream and a key that lies about its content, which is + /// the failure this protocol keeps meeting from other directions (§4.5, §4.8, §11.1). + /// + /// ── **Why the ceiling is not a detail** ── + /// + /// Scanning actions is the one thing that can walk off the end of a body's slots, and + /// the slots immediately after a body's are the **next body's**. Measured in phase 6: + /// 643 of 795 legacy bodies return a fully validated, correctly-sized picture one + /// action past their band, and 452 of those are byte-identical to body+1's action 0. + /// refuses past the ceiling, so + /// this walk cannot produce one — see §4.10. + /// + private static Sprite ResolveAny(Readers readers, int body, int direction) + { + int actions, fileType; + string reason; + + // The legacy ceiling. A body the legacy path cannot place at all still gets the UOP + // arm below, where an action is a named entry rather than an offset into a band. + if (!BridgeAssetValidator.ActionCount(body, out actions, out fileType, out reason)) + actions = 0; + + for (int action = PreferredAction; action < MaxAction; action++) + { + // Legacy first, always. The vendored decoder is what 787 of this client's bodies + // come out of, it is what phase 3 measured, and the UOP packages hold a different + // and mostly disjoint set (measured: of the 244 bodies they carry, 8 also have + // legacy art). So this is a fallback rather than a choice, and no body changes + // reader while a client sits still. + Sprite sprite = action < actions + ? ResolveLegacy(Key(body, action), readers, body, action, direction) + : null; + + if (sprite == null) + sprite = ResolveUop(Key(body, action), readers, body, action, direction); + + if (sprite != null) + return sprite; + } + + return null; + } + /// /// ServUO's vendored Animations over anim*.mul, behind §4.5's validator. /// - private static Sprite ResolveLegacy(string key, Readers readers, int body, int direction) + private static Sprite ResolveLegacy(string key, Readers readers, int body, int action, int direction) { int fileType, at; string reason; - if (!BridgeAssetValidator.ResolveAnimation(body, CatalogAction, direction, + if (!BridgeAssetValidator.ResolveAnimation(body, action, direction, out fileType, out at, out reason)) return null; @@ -534,12 +635,12 @@ namespace Server.Custom.Bridge try { - return Decode(key, body, direction, fileType); + return Decode(key, body, action, direction, fileType); } catch (Exception e) { - Console.WriteLine("[Bridge] catalogue: body {0}: {1}: {2}", - body, e.GetType().Name, e.Message); + Console.WriteLine("[Bridge] catalogue: body {0} action {1}: {2}: {3}", + body, action, e.GetType().Name, e.Message); return null; } } @@ -558,9 +659,9 @@ namespace Server.Custom.Bridge /// hash of a name carrying the body id, and the payload repeats that id in its own /// header for to check. A miss is a miss. /// - private static Sprite ResolveUop(string key, Readers readers, int body, int direction) + private static Sprite ResolveUop(string key, Readers readers, int body, int action, int direction) { - ulong hash = BridgeUop.HashOf(body, CatalogAction); + ulong hash = BridgeUop.HashOf(body, action); byte[] payload = null; string reason = null; @@ -574,8 +675,8 @@ namespace Server.Custom.Bridge if (!package.TryRead(hash, out payload, out reason)) { - Console.WriteLine("[Bridge] catalogue: body {0} in {1}: {2}", - body, BridgeUop.PackageName(n), reason); + Console.WriteLine("[Bridge] catalogue: body {0} action {1} in {2}: {3}", + body, action, BridgeUop.PackageName(n), reason); return null; } @@ -589,7 +690,8 @@ namespace Server.Custom.Bridge if (!BridgeUop.Group.TryOpen(payload, body, out group, out reason)) { - Console.WriteLine("[Bridge] catalogue: body {0} uop: {1}", body, reason); + Console.WriteLine("[Bridge] catalogue: body {0} action {1} uop: {2}", + body, action, reason); return null; } @@ -607,7 +709,8 @@ namespace Server.Custom.Bridge // exactly the same condition — so it is absent, silently. Anything else is a // record this reader refused, and that is worth a line. if (!empty) - Console.WriteLine("[Bridge] catalogue: body {0} uop: {1}", body, reason); + Console.WriteLine("[Bridge] catalogue: body {0} action {1} uop: {2}", + body, action, reason); return null; } @@ -621,6 +724,7 @@ namespace Server.Custom.Bridge { Key = key, Body = body, + Action = action, Direction = direction, FileType = 0, Png = png, @@ -631,14 +735,14 @@ namespace Server.Custom.Bridge }; } - private static Sprite Decode(string key, int body, int direction, int fileType) + private static Sprite Decode(string key, int body, int action, int direction, int fileType) { int hue = 0; // `preserveHue: false` — the catalogue is the creature's own art, and a body-level hue // from Body.def belongs to a specific mob rather than to the species. §5's key scheme // is where a hued variant is expressed (`static/3922/h33`), not here. - Frame[] frames = Animations.GetAnimation(body, CatalogAction, direction, ref hue, false, true); + Frame[] frames = Animations.GetAnimation(body, action, direction, ref hue, false, true); if (frames == null || frames.Length == 0 || frames[0] == null) return null; @@ -657,6 +761,7 @@ namespace Server.Custom.Bridge { Key = key, Body = body, + Action = action, Direction = direction, FileType = fileType, Png = png, @@ -758,20 +863,28 @@ namespace Server.Custom.Bridge // ── keys, cursors and the source id ────────────────────────────────────────────────── - private static string Key(int body) + private static string Key(int body, int action) { return "body/" + body.ToString(CultureInfo.InvariantCulture) - + "/a" + CatalogAction.ToString(CultureInfo.InvariantCulture); + + "/a" + action.ToString(CultureInfo.InvariantCulture); } /// - /// `body/<id>/a0`, and nothing else in this phase. A deeper key - /// (`body/400/a2/f3`) is well-formed under §5 and simply not served yet, so it comes - /// back `unsupported` rather than being silently read as its own first frame. + /// `body/<id>/a<n>`, and nothing else in this phase. A deeper key + /// (`body/400/a2/f3`) is well-formed under §5 and simply not served, so it comes back + /// `unsupported` rather than being silently read as its own first frame. + /// + /// The action is parsed rather than required to be zero — 74 of this client's bodies + /// are catalogued at a different one (§11.2) — but a parsed action is not an accepted + /// one. serves a key only when it is the key the catalogue itself + /// chose for that body, which is what keeps §4.10's ceiling from being reachable + /// through a request: nothing the website can ask makes this decode an action the + /// catalogue did not already pick. /// - private static bool TryParseKey(string key, out int body) + private static bool TryParseKey(string key, out int body, out int action) { body = 0; + action = -1; if (key == null) return false; @@ -787,7 +900,14 @@ namespace Server.Custom.Bridge if (body < 1 || body > MaxBody) return false; - return parts[2] == "a" + CatalogAction.ToString(CultureInfo.InvariantCulture); + if (parts[2].Length < 2 || parts[2][0] != 'a') + return false; + + if (!Int32.TryParse(parts[2].Substring(1), NumberStyles.None, + CultureInfo.InvariantCulture, out action)) + return false; + + return action >= 0 && action < MaxAction; } private static int ParseBodyCursor(string cursor) -- 2.49.1 From 13b6fc02a48373119cc828fbd89d489ffe838d3e Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 14 Sep 2026 02:00:22 -0500 Subject: [PATCH 09/11] feat(asset-bridge): the shard's own files stop needing a shared filesystem (Phase 7) The spawn atlas was the one place the platform's rule -- only the sidecar bridges the shard -- was broken, and it was broken by the component that faces the internet: SPAWN_ATLAS.md required the website to read the ServUO tree off a bind mount or a shared volume. This serves those files over the loopback link instead (docs/link/v8.md 10). The measurement came first and changed the shape. 10 said the shard would serve `tree/