diff --git a/tools/patch_client.ps1 b/tools/patch_client.ps1 new file mode 100644 index 0000000..d1baf90 --- /dev/null +++ b/tools/patch_client.ps1 @@ -0,0 +1,580 @@ +<# +.SYNOPSIS + Builds a deliberately patched UO client for the Asset Bridge phase 0 spike. + +.DESCRIPTION + docs/link/v8.md section 16 phase 0 drives ServUO's vendored `Ultima` decoders "over a deliberately + patched client". Stock clients are not the interesting case: they are the case the library was + written against, and the whole reason phase 0 exists is that section 4 chose to call code that can + take the shard down if it is wrong. A shard operator's client is patched -- custom art, a + verdata.mul, a hand-edited Bodyconv.def -- and that is what has to be survived. + + This copies a client and then breaks the copy in four deliberate, catalogued ways. It NEVER + writes to the source: every file it patches is hashed before and after, and a changed source + hash aborts the run. + + Each defect is recorded in `patched-client.manifest.json` next to the copy, so the probe's + report can be read against what was actually done rather than against a memory of it. The + manifest is the answer to "is a nonzero REFUSED-BUT-DECODED count a bug or the point?". + +.PARAMETER Source + The client to copy. Defaults to this machine's. + +.PARAMETER Dest + Where to build the patched copy. Needs ~3.5 GB. + +.PARAMETER Tiers + Which defects to apply. Default: all four. + + verdata Author a verdata.mul, which this client does not have. Ultima consults Verdata on + EVERY art and anim lookup (Art's FileIndex is built with verdata file id 4, + Animations' with 6), so on a client with no verdata.mul that entire branch is + dead code that has never been exercised -- the largest untested surface in the + library we are about to depend on. Includes one legitimate patch and one whose + lookup points past verdata.mul's own end, because `FileIndex.Seek` bounds-checks + the mul and does not bounds-check verdata. + + customart Fill unused artidx.mul slots with real records appended to art.mul, the way a + custom-art shard does. Tests that our out-of-range accounting comes from the + file rather than from a constant someone wrote down. + + corrupt Rewrite index entries and record headers into the shapes that reading Art.cs + says are reachable: a lookup past EOF, a record that starts inside the file and + ends outside it, a length too small for a header, absurd dimensions, a row table + pointing outside its own record, and a land tile shorter than the fixed 2,024 + bytes LoadLand always reads. + + bodyconv Add Bodyconv.def lines pointing bodies at an anim file that holds nothing, and at + an index in another file that holds something unrelated -- the gargoyle-666 spider + case, reproduced on purpose. Proves the extractor takes BodyConverter.Convert's + answer and stops (v8.md section 4.3). + + nouop Move artLegacyMUL.uop aside, so art is read from art.mul/artidx.mul. + + This is not cosmetic and it is not optional if you want the customart or corrupt + tiers to mean anything. FileIndex's UOP constructor ends with a bare + `MulPath = uopPath`: when artLegacyMUL.uop is present it wins OUTRIGHT and + art.mul / artidx.mul are never opened. Every index-level defect below writes to + files the library does not read on a modern client, so without this tier those + two tiers are inert while still reporting that they applied. + + It is also a real configuration in its own right: plenty of shards run mul-only + clients, and a custom-art shard that adds graphics to art.mul while the UOP is + still there gets nothing at all -- an operator trap worth knowing about. + +.PARAMETER SkipCopy + Re-patch an existing copy without re-copying 3.5 GB. Only safe on a copy this script made and + has not patched yet -- patching twice compounds the defects and invalidates the manifest. + +.EXAMPLE + .\tools\patch_client.ps1 -Dest D:\uo-patched-client + +.NOTES + Test scaffolding. Never deployed. The copy contains EA's client art -- like every other + extraction in this project it stays on the machine that made it and is never committed. +#> + +[CmdletBinding()] +param( + [string] $Source = 'D:\Games\Electronic Arts\Ultima Online Classic', + [Parameter(Mandatory = $true)] + [string] $Dest, + [ValidateSet('verdata', 'customart', 'corrupt', 'bodyconv', 'nouop')] + [string[]] $Tiers = @('nouop', 'verdata', 'customart', 'corrupt', 'bodyconv'), + [switch] $SkipCopy, + [switch] $Force +) + +$ErrorActionPreference = 'Stop' + +# Files this script may write to in the copy. Anything not on this list is a bug in the script, +# and the source-hash check at the end is what proves it. +$PatchTargets = @('artidx.mul', 'art.mul', 'verdata.mul', 'Bodyconv.def', 'artLegacyMUL.uop') + +# -- Little-endian helpers (BitConverter is fine, but the intent reads better named) ---------- + +function Read-Int32LE([byte[]] $Bytes, [int] $Offset) { + return [BitConverter]::ToInt32($Bytes, $Offset) +} + +function Write-Int32LE([byte[]] $Bytes, [int] $Offset, [int] $Value) { + [Array]::Copy([BitConverter]::GetBytes([int] $Value), 0, $Bytes, $Offset, 4) +} + +function Get-ArtEntry([byte[]] $Idx, [int] $Index) { + $at = $Index * 12 + return [pscustomobject]@{ + Index = $Index + Lookup = Read-Int32LE $Idx $at + Length = Read-Int32LE $Idx ($at + 4) + Extra = Read-Int32LE $Idx ($at + 8) + } +} + +function Set-ArtEntry([byte[]] $Idx, [int] $Index, [int] $Lookup, [int] $Length, [int] $Extra) { + $at = $Index * 12 + Write-Int32LE $Idx $at $Lookup + Write-Int32LE $Idx ($at + 4) $Length + Write-Int32LE $Idx ($at + 8) $Extra +} + +# The defect catalogue. Every mutation appends to this, and it is written out as the manifest. +$script:Defects = New-Object System.Collections.ArrayList + +function Add-Defect([string] $Tier, [string] $Key, [string] $What, [string] $Expect) { + [void] $script:Defects.Add([pscustomobject]@{ + tier = $Tier + key = $Key + what = $What + expect = $Expect + }) + Write-Host (" {0,-22} {1}" -f $Key, $What) +} + +# -- Preflight -------------------------------------------------------------------------------- + +if (-not (Test-Path -LiteralPath $Source)) { + throw "source client not found: $Source" +} + +$sourceFull = (Resolve-Path -LiteralPath $Source).Path + +if (Test-Path -LiteralPath $Dest) { + $destFull = (Resolve-Path -LiteralPath $Dest).Path + if ($destFull -eq $sourceFull) { + throw "Dest is the source client. Refusing -- this script destroys what it points at." + } + if (-not $SkipCopy -and -not $Force) { + throw "$Dest already exists. Pass -Force to overwrite it, or -SkipCopy to patch it in place." + } +} + +Write-Host "source: $sourceFull" +Write-Host "dest: $Dest" +Write-Host "tiers: $($Tiers -join ', ')" +Write-Host "" + +# Hash the source files we are about to touch, so "it never writes to the source" is checked and +# not merely asserted. +$before = @{} +foreach ($name in $PatchTargets) { + $path = Join-Path $sourceFull $name + if (Test-Path -LiteralPath $path) { + $before[$name] = (Get-FileHash -LiteralPath $path -Algorithm SHA256).Hash + } +} + +# -- Copy ------------------------------------------------------------------------------------- + +if ($SkipCopy) { + Write-Host "skipping copy (-SkipCopy)" + if (-not (Test-Path -LiteralPath $Dest)) { throw "-SkipCopy but $Dest does not exist" } +} else { + Write-Host "copying (this is ~3.5 GB; a few minutes)..." + # /MIR so a -Force re-run starts clean rather than merging into an already-patched tree. + # /NJH /NJS /NDL /NFL keep robocopy's output to the errors. + $null = robocopy $sourceFull $Dest /MIR /R:1 /W:1 /NJH /NJS /NDL /NFL /NP + # Robocopy exit codes below 8 are success; 8 and above are real failures. + if ($LASTEXITCODE -ge 8) { throw "robocopy failed with exit code $LASTEXITCODE" } + # Robocopy's "1 = files were copied" would otherwise become this script's exit code and read + # as a failure to anything checking it. + $global:LASTEXITCODE = 0 + Write-Host "copied." +} + +Write-Host "" + +$destFull = (Resolve-Path -LiteralPath $Dest).Path + +$artIdxPath = Join-Path $destFull 'artidx.mul' +$artMulPath = Join-Path $destFull 'art.mul' + +if (-not (Test-Path -LiteralPath $artIdxPath)) { throw "no artidx.mul in the copy" } + +$idx = [System.IO.File]::ReadAllBytes($artIdxPath) +$entryCount = [int] ($idx.Length / 12) +$artMulLength = (Get-Item -LiteralPath $artMulPath).Length + +Write-Host ("artidx.mul holds {0:N0} entries; art.mul is {1:N0} bytes" -f $entryCount, $artMulLength) +Write-Host "" + +# -- Tier: nouop ------------------------------------------------------------------------------ + +$uopPath = Join-Path $destFull 'artLegacyMUL.uop' +$uopPresent = Test-Path -LiteralPath $uopPath + +if ($Tiers -contains 'nouop') { + Write-Host "tier nouop" + + if (-not $uopPresent) { + Write-Host " no artLegacyMUL.uop in the copy -- already a mul-only client" + } else { + Move-Item -LiteralPath $uopPath -Destination "$uopPath.disabled" -Force + $uopPresent = $false + Add-Defect 'nouop' 'artLegacyMUL.uop' 'moved aside so art is read from art.mul/artidx.mul' ` + 'every index-level defect below becomes reachable; without this they are inert' + } + + Write-Host "" +} elseif ($uopPresent -and (($Tiers -contains 'corrupt') -or ($Tiers -contains 'customart'))) { + Write-Host " WARNING: artLegacyMUL.uop is present and the nouop tier was not selected." + Write-Host " FileIndex prefers the UOP outright, so the corrupt and customart tiers" + Write-Host " will write to files the library never opens. Add -Tiers nouop." + Write-Host "" +} + +# Static ids are offset by 0x4000 in the index; land tiles occupy 0..0x3FFF. +$StaticBase = 0x4000 + +# Find donor records to copy and victims to corrupt: real, modestly sized statics, so the defects +# are applied to entries that genuinely work today. Picking arbitrary ids risks landing on slots +# that are already empty, where a "defect" would prove nothing. +$donors = New-Object System.Collections.ArrayList +for ($id = 0x1000; $id -lt 0x3000 -and $donors.Count -lt 24; $id++) { + $e = Get-ArtEntry $idx ($id + $StaticBase) + if ($e.Lookup -ge 0 -and $e.Length -gt 200 -and $e.Length -lt 4000 -and ($e.Lookup + $e.Length) -le $artMulLength) { + [void] $donors.Add([pscustomobject]@{ Id = $id; Entry = $e }) + } +} + +if ($donors.Count -lt 12) { throw "found only $($donors.Count) usable donor statics -- the copy looks wrong" } + +Write-Host "using donor statics: $(($donors | Select-Object -First 12 | ForEach-Object { $_.Id }) -join ', ')" +Write-Host "" + +$idxDirty = $false + +# -- Tier: customart -------------------------------------------------------------------------- + +if ($Tiers -contains 'customart') { + Write-Host "tier customart" + + # A custom-art client does not fill spare slots -- artidx.mul is exactly sized (62,692 + # entries here, not one to spare), so adding art means GROWING the index. `Art` builds its + # FileIndex with length 0x10000, so there is room for 2,844 more ids before the library stops + # looking, and the stock ceiling turns out to be nothing more than the size of a file. + $idxCeiling = 0x10000 + + if ($entryCount -ge $idxCeiling) { + Write-Host " artidx.mul is already at the 0x10000 ceiling -- skipping tier" + } else { + $addCount = 8 + $grown = New-Object byte[] (($entryCount + $addCount) * 12) + [Array]::Copy($idx, 0, $grown, 0, $idx.Length) + $idx = $grown + + # Read every donor record BEFORE opening the append handle. Append mode takes an + # exclusive lock, so reading the same file while appending to it fails outright. + $buffers = @() + $reader = [System.IO.File]::Open($artMulPath, 'Open', 'Read', 'ReadWrite') + + try { + for ($n = 0; $n -lt $addCount; $n++) { + $donor = $donors[$n] + $buffer = New-Object byte[] $donor.Entry.Length + [void] $reader.Seek($donor.Entry.Lookup, 'Begin') + [void] $reader.Read($buffer, 0, $buffer.Length) + $buffers += , $buffer + } + } finally { $reader.Dispose() } + + $appendAt = $artMulLength + $stream = [System.IO.File]::Open($artMulPath, 'Append', 'Write') + + try { + for ($n = 0; $n -lt $addCount; $n++) { + $buffer = $buffers[$n] + $stream.Write($buffer, 0, $buffer.Length) + + $slot = $entryCount + $n + $newId = $slot - $StaticBase + Set-ArtEntry $idx $slot $appendAt $buffer.Length $donors[$n].Entry.Extra + $appendAt += $buffer.Length + + Add-Defect 'customart' "static/$newId" ` + "custom art appended past the stock ceiling (a copy of static/$($donors[$n].Id))" ` + 'decodes cleanly; proves the ceiling is read from the file, not from a constant' + } + } finally { $stream.Dispose() } + + $entryCount += $addCount + $idxDirty = $true + } + + Write-Host "" +} + +# -- Tier: corrupt ---------------------------------------------------------------------------- + +if ($Tiers -contains 'corrupt') { + Write-Host "tier corrupt" + + $artMulLength = (Get-Item -LiteralPath $artMulPath).Length + $v = 8 # donors 0..7 may have been consumed by customart as sources; they are unmodified + + # 1. A lookup past the end of art.mul. FileIndex.Seek DOES check this one + # (`Stream.Length < e.lookup`), so the library and the validator should agree. + $victim = $donors[$v++].Id + Set-ArtEntry $idx ($victim + $StaticBase) ([int]($artMulLength + 4096)) 512 0 + Add-Defect 'corrupt' "static/$victim" 'lookup 4 KB past the end of art.mul' ` + 'refused by the validator; Seek also catches this one, so no picture' + + # 2. A record that STARTS inside the file and ENDS outside it. This is the gap: Seek checks + # the start and never the end, stream.Read returns short, the decoders ignore the count, + # and m_StreamBuffer still holds the PREVIOUS asset. The expected outcome is a picture of + # something else entirely, reported as a success by every count in the library. + $victim = $donors[$v++].Id + Set-ArtEntry $idx ($victim + $StaticBase) ([int]($artMulLength - 64)) 8192 0 + Add-Defect 'corrupt' "static/$victim" 'record starts 64 bytes before EOF and declares 8,192' ` + 'REFUSED BUT DECODED -- the stale-buffer wrong picture' + + # 3. A length too small to hold even the 8-byte header. + $victim = $donors[$v++].Id + $donorEntry = $donors[$v - 1].Entry + Set-ArtEntry $idx ($victim + $StaticBase) $donorEntry.Lookup 4 0 + Add-Defect 'corrupt' "static/$victim" 'declared length 4 -- smaller than the static header' ` + 'refused by the validator' + + # 4/5/6 rewrite the record BODY, so they need their own bytes rather than an index edit. + # Appended to art.mul and pointed at, which leaves the donor's real record intact. + $stream = [System.IO.File]::Open($artMulPath, 'Append', 'Write') + try { + $appendAt = (Get-Item -LiteralPath $artMulPath).Length + + # 4. Absurd dimensions. LoadStatic allocates new Bitmap(width, height) straight from two + # ushorts in the file. 8000x8000 is ~128 MB -- survivable, and the point is made; the + # same field can ask for 65535x65535, which is 8 GB from a two-byte edit. + $victim = $donors[$v++].Id + $rec = New-Object byte[] 2048 + [Array]::Copy([BitConverter]::GetBytes([uint16] 8000), 0, $rec, 4, 2) + [Array]::Copy([BitConverter]::GetBytes([uint16] 8000), 0, $rec, 6, 2) + $stream.Write($rec, 0, $rec.Length) + Set-ArtEntry $idx ($victim + $StaticBase) $appendAt $rec.Length 0 + $appendAt += $rec.Length + Add-Defect 'corrupt' "static/$victim" 'header declares 8000x8000 (a ~128 MB allocation from two bytes)' ` + 'refused by the validator; the library would allocate it' + + # 5. A row-lookup table pointing outside the record. This is what LoadStatic's unbounded + # read cursor was written to walk off the end of. + $victim = $donors[$v++].Id + $rec = New-Object byte[] 512 + [Array]::Copy([BitConverter]::GetBytes([uint16] 32), 0, $rec, 4, 2) # width + [Array]::Copy([BitConverter]::GetBytes([uint16] 32), 0, $rec, 6, 2) # height + for ($row = 0; $row -lt 32; $row++) { + # Each row's offset is added to (height + 4); 60000 puts every row far outside. + [Array]::Copy([BitConverter]::GetBytes([uint16] 60000), 0, $rec, (8 + $row * 2), 2) + } + $stream.Write($rec, 0, $rec.Length) + Set-ArtEntry $idx ($victim + $StaticBase) $appendAt $rec.Length 0 + $appendAt += $rec.Length + Add-Defect 'corrupt' "static/$victim" 'row table points 60,000 words outside a 512-byte record' ` + 'refused by the validator; the library reads adjacent heap' + + # 6. A well-formed row table whose run length overruns the record. + $victim = $donors[$v++].Id + $rec = New-Object byte[] 256 + [Array]::Copy([BitConverter]::GetBytes([uint16] 16), 0, $rec, 4, 2) + [Array]::Copy([BitConverter]::GetBytes([uint16] 2), 0, $rec, 6, 2) + [Array]::Copy([BitConverter]::GetBytes([uint16] 0), 0, $rec, 8, 2) # row 0 offset + [Array]::Copy([BitConverter]::GetBytes([uint16] 0), 0, $rec, 10, 2) # row 1 offset + $runAt = (2 + 4) * 2 # (height + 4) words + [Array]::Copy([BitConverter]::GetBytes([uint16] 0), 0, $rec, $runAt, 2) # xOffset + [Array]::Copy([BitConverter]::GetBytes([uint16] 16), 0, $rec, ($runAt + 2), 2) # xRun, but + # the record has nowhere near 16 pixels left after this point. + $stream.Write($rec, 0, $rec.Length) + Set-ArtEntry $idx ($victim + $StaticBase) $appendAt 20 0 + $appendAt += $rec.Length + Add-Defect 'corrupt' "static/$victim" 'a 16-pixel run declared in a 20-byte record' ` + 'refused by the validator' + } finally { $stream.Dispose() } + + # 7. A land tile shorter than the 2,024 bytes LoadLand reads unconditionally. + $landVictim = 0x0100 + $landEntry = Get-ArtEntry $idx $landVictim + if ($landEntry.Lookup -ge 0 -and $landEntry.Length -gt 0) { + Set-ArtEntry $idx $landVictim $landEntry.Lookup 512 0 + Add-Defect 'corrupt' "land/$landVictim" 'land record declared 512 bytes; LoadLand always reads 2,024' ` + 'refused by the validator; the library reads past the buffer' + } + + $idxDirty = $true + Write-Host "" +} + +if ($idxDirty) { + [System.IO.File]::WriteAllBytes($artIdxPath, $idx) + Write-Host "wrote artidx.mul" + Write-Host "" +} + +# -- Tier: verdata ---------------------------------------------------------------------------- + +if ($Tiers -contains 'verdata') { + Write-Host "tier verdata" + + # Layout: int32 count, then count * 5 int32 (file, index, lookup, length, extra), then the + # payloads. `lookup` is an absolute offset into this file. + $entries = New-Object System.Collections.ArrayList + $payloads = New-Object System.Collections.ArrayList + + $donorA = $donors[$donors.Count - 1] + $donorB = $donors[$donors.Count - 2] + + $artSource = [System.IO.File]::Open($artMulPath, 'Open', 'Read', 'ReadWrite') + try { + $bufferA = New-Object byte[] $donorA.Entry.Length + [void] $artSource.Seek($donorA.Entry.Lookup, 'Begin') + [void] $artSource.Read($bufferA, 0, $bufferA.Length) + } finally { $artSource.Dispose() } + + # The victims: ids whose art will now come from verdata.mul rather than art.mul. + $legitVictim = $donors[$donors.Count - 3].Id + $pastEofVictim = $donors[$donors.Count - 4].Id + + # A legitimate patch -- the branch working as designed. Without this the tier only proves the + # failure case, and "verdata is broken" and "verdata is never reached" look identical. + [void] $payloads.Add($bufferA) + [void] $entries.Add([pscustomobject]@{ + File = 4; Index = ($legitVictim + $StaticBase); Length = $bufferA.Length; Extra = $donorA.Entry.Extra + PayloadIndex = 0; PastEof = $false + }) + + # The failure case. FileIndex.Seek bounds-checks the mul stream and calls Verdata.Seek with no + # check at all; seeking a FileStream past EOF is legal, the read returns nothing, and the + # shared decode buffer still holds the previous asset. + [void] $entries.Add([pscustomobject]@{ + File = 4; Index = ($pastEofVictim + $StaticBase); Length = 900; Extra = 0 + PayloadIndex = -1; PastEof = $true + }) + + # An anim patch, so the tier covers the other file the verdata branch serves. anim.mul is + # verdata file 6; for body < 200 the record index is body*110 + action*5 + direction. + $animBody = 34 # wolf -- decodes on this client, so a patch to it is observable + $animIndex = ($animBody * 110) + (0 * 5) + 1 + [void] $entries.Add([pscustomobject]@{ + File = 6; Index = $animIndex; Length = 700; Extra = 0 + PayloadIndex = -1; PastEof = $true + }) + + $headerSize = 4 + ($entries.Count * 20) + $offset = $headerSize + foreach ($entry in $entries) { + if ($entry.PayloadIndex -ge 0) { + $entry | Add-Member -NotePropertyName Lookup -NotePropertyValue $offset -Force + $offset += $payloads[$entry.PayloadIndex].Length + } + } + + $totalSize = $offset + + # Past-EOF lookups are resolved last, because "past the end" is only meaningful once the end + # is known. + foreach ($entry in $entries) { + if ($entry.PastEof) { + $entry | Add-Member -NotePropertyName Lookup -NotePropertyValue ($totalSize + 8192) -Force + } + } + + $verdata = New-Object byte[] $totalSize + Write-Int32LE $verdata 0 $entries.Count + + $at = 4 + foreach ($entry in $entries) { + Write-Int32LE $verdata $at $entry.File + Write-Int32LE $verdata ($at + 4) $entry.Index + Write-Int32LE $verdata ($at + 8) $entry.Lookup + Write-Int32LE $verdata ($at + 12) $entry.Length + Write-Int32LE $verdata ($at + 16) $entry.Extra + $at += 20 + } + + foreach ($entry in $entries) { + if ($entry.PayloadIndex -ge 0) { + $payload = $payloads[$entry.PayloadIndex] + [Array]::Copy($payload, 0, $verdata, $entry.Lookup, $payload.Length) + } + } + + [System.IO.File]::WriteAllBytes((Join-Path $destFull 'verdata.mul'), $verdata) + + Add-Defect 'verdata' "static/$legitVictim" ` + "legitimately patched to static/$($donorA.Id)'s art via verdata.mul" ` + 'decodes; the picture must CHANGE, which is how we know the branch ran' + Add-Defect 'verdata' "static/$pastEofVictim" ` + 'verdata entry whose lookup is 8 KB past the end of verdata.mul' ` + 'REFUSED BUT DECODED -- Verdata.Seek is not bounds-checked' + Add-Defect 'verdata' "body/$animBody" ` + "anim.mul record $animIndex patched to a verdata offset past EOF" ` + 'the wolf must not silently become another creature' + + Write-Host (" wrote verdata.mul: {0} entries, {1:N0} bytes" -f $entries.Count, $totalSize) + Write-Host "" +} + +# -- Tier: bodyconv --------------------------------------------------------------------------- + +if ($Tiers -contains 'bodyconv') { + Write-Host "tier bodyconv" + + $bodyconvPath = Join-Path $destFull 'Bodyconv.def' + + if (-not (Test-Path -LiteralPath $bodyconvPath)) { + Write-Host " no Bodyconv.def in the copy -- skipping tier" + } else { + # Columns are tab-separated: original, anim2, anim3, anim4, anim5. -1 means "not in that + # file". BodyConverter.Convert returns the file type of the FIRST column that is not -1, + # and the extractor must take that answer and stop. + $lines = @( + "", + "# Asset Bridge phase 0 -- deliberate defects (tools/patch_client.ps1)", + "1900`t-1`t-1`t-1`t60000", + "1901`t666`t-1`t-1`t-1" + ) + + Add-Content -LiteralPath $bodyconvPath -Value ($lines -join "`r`n") -Encoding ASCII + + Add-Defect 'bodyconv' 'body/1900' 'mapped to anim5 index 60,000, which does not exist' ` + 'reports nothing -- and must NOT fall back to another anim file' + Add-Defect 'bodyconv' 'body/1901' 'mapped to anim2 index 666, where something unrelated lives' ` + 'decodes a picture of the WRONG creature -- the spider case, on purpose' + } + + Write-Host "" +} + +# -- The source must be untouched ------------------------------------------------------------- + +$tampered = @() +foreach ($name in $before.Keys) { + $path = Join-Path $sourceFull $name + $now = (Get-FileHash -LiteralPath $path -Algorithm SHA256).Hash + if ($now -ne $before[$name]) { $tampered += $name } +} + +if ($tampered.Count -gt 0) { + throw "THE SOURCE CLIENT WAS MODIFIED: $($tampered -join ', '). Restore it from the installer before doing anything else." +} + +Write-Host "source client verified unchanged ($($before.Count) files hashed before and after)" + +# -- Manifest --------------------------------------------------------------------------------- + +$manifest = [pscustomobject]@{ + built = (Get-Date).ToUniversalTime().ToString('u') + source = $sourceFull + dest = $destFull + tiers = $Tiers + defects = @($script:Defects) +} + +$manifestPath = Join-Path $destFull 'patched-client.manifest.json' +$manifest | ConvertTo-Json -Depth 5 | Set-Content -LiteralPath $manifestPath -Encoding utf8 + +Write-Host "" +Write-Host ("{0} deliberate defects; manifest at {1}" -f $script:Defects.Count, $manifestPath) +Write-Host "" +Write-Host "Point the probe at it by adding to the shard's Config/Bridge.cfg:" +Write-Host "" +Write-Host " AssetProbeClient=$destFull" +Write-Host "" +Write-Host "then, in game or from the rig driver: [assetprobe all patched" diff --git a/tools/scaffolding/BridgeAssetProbe.cs b/tools/scaffolding/BridgeAssetProbe.cs new file mode 100644 index 0000000..6f0bfb9 --- /dev/null +++ b/tools/scaffolding/BridgeAssetProbe.cs @@ -0,0 +1,1346 @@ +using System; +using System.Collections.Generic; +using System.Drawing; +using System.Globalization; +using System.IO; +using System.Text; +using System.Threading; + +using Server.Commands; + +using Ultima; + +namespace Server.Custom +{ + /// + /// **Asset Bridge phase 0 — the spike** (docs/link/v8.md §16). + /// + /// §4 decided to call ServUO's own vendored Ultima decoders rather than reimplement + /// them. That decision rests on a probe run from **PowerShell** against a **stock** client, + /// and neither of those is the environment the extractor will actually live in. This runs + /// the same decoders from **inside a running ServUO**, against a **deliberately patched** + /// client, and its whole job is to find a fault on a path we call before eight phases are + /// built on top of one. + /// + /// What "a fault" means here is wider than a crash, and the wider half is the dangerous + /// half. `Ultima`'s decoders take their bounds from the files they are reading, so a + /// malformed record does not usually throw — it produces **a confident, wrong picture**. + /// Four such shapes are known from reading the source and each has its own counter below: + /// + /// * LoadStatic walks bindata[count++] with no bound on count. The + /// two guards in that loop bound the *write* into the bitmap and not the *read* out of + /// the record, so a record whose row table points outside itself reads adjacent heap. + /// * stream.Read(m_StreamBuffer, 0, length) ignores its return value, and + /// m_StreamBuffer is reused and only ever grown. A short read therefore decodes + /// **the previous asset's bytes** under this asset's id. + /// * LoadLand reads a fixed 2,024 bytes whatever length says. + /// * Art.GetLegalItemID returns **0** for an out-of-range id, so GetStatic + /// of a nonexistent id can hand back item 0's picture instead of nothing. + /// + /// So every sweep here compares two answers: what says + /// about the index entry *before* the call, and what Ultima does *with* it. The + /// interesting cell is not the error count. It is **REFUSED-BUT-DECODED** — a record the + /// validator rejects and the library cheerfully returns a bitmap for. Those are the wrong + /// pictures, and they are invisible to any success count. + /// + /// **Nothing here calls Ultima.Gumps**, which is a safety rule and not a preference + /// (v8.md §4.1) — with one deliberate exception, the opt-in gump section, whose + /// entire purpose is to reproduce the process-killing access violation from inside ServUO + /// so the rule has evidence behind it. It is off by default and it **takes the shard down**. + /// + /// Test scaffolding. Never deployed; deploy.ps1 copies only overlay/. + /// In game / from BridgeRigDriver: [assetprobe [section] [stock|patched]. + /// Flag: AssetProbeOnStart. Build the patched client with + /// tools/patch_client.ps1. + /// + public static class BridgeAssetProbe + { + // Where the run writes. The checkpoint is the point of the whole arrangement: some of + // these faults are corrupted-state exceptions that no catch block sees, so the last id + // written to disk is the only evidence of where the process died. + private static readonly string OutputDir = Path.Combine(Core.BaseDirectory, "Logs", "AssetProbe"); + + private static string _checkpointPath; + private static StreamWriter _report; + private static readonly object _sync = new object(); + private static bool _running; + + // Snapshot of the player-character body ids, taken on the Core thread (§5.2). Not + // hardcoded: RaceDefinitions.cs passes the gargoyle's ghost bodies in the opposite order + // to the other races, and a shard that calls RegisterRace adds ids no table of ours holds. + private static List _playerBodies; + + private struct PlayerBody + { + public string Race; + public string Slot; + public int Body; + + public PlayerBody(string race, string slot, int body) + { + Race = race; + Slot = slot; + Body = body; + } + } + + public static void Initialize() + { + CommandSystem.Register("assetprobe", AccessLevel.Administrator, Probe_OnCommand); + + if (Config.Get("Bridge.AssetProbeOnStart", false)) + EventSink.ServerStarted += () => Timer.DelayCall(TimeSpan.FromSeconds(10.0), () => Begin(null, "all", null)); + } + + [Usage("assetprobe [all|paths|statics|land|bodies|players|cliloc|gump] [stock|patched]")] + [Description("Drives ServUO's vendored Ultima decoders against a client and reports every fault and every wrong picture.")] + private static void Probe_OnCommand(CommandEventArgs e) + { + var section = e.Arguments.Length > 0 ? e.Arguments[0].ToLowerInvariant() : "all"; + var which = e.Arguments.Length > 1 ? e.Arguments[1].ToLowerInvariant() : null; + + Begin(e.Mobile, section, which); + } + + // ── Entry ──────────────────────────────────────────────────────────────────────────── + + /// + /// Reads what only the Core thread may read, then hands the sweep to a background + /// thread. That split is not tidiness — it is the §8 threading shape this protocol + /// introduces, rehearsed here: the decode must run OFF the Core thread (a 66,000-id + /// sweep would freeze the shard), and the world reads it depends on must run ON it. + /// + public static void Begin(Mobile from, string section, string which) + { + lock (_sync) + { + if (_running) + { + Tell(from, "already running — one sweep at a time, so the checkpoint means something"); + return; + } + + _running = true; + } + + // Core-thread reads first. + _playerBodies = ReadPlayerBodies(); + + var thread = new Thread(() => Run(from, section, which)); + thread.IsBackground = true; + thread.Name = "BridgeAssetProbe"; + thread.Priority = ThreadPriority.BelowNormal; + thread.Start(); + + Tell(from, "started on a background thread — output in Logs/AssetProbe"); + } + + /// §5.2: ask the shard which bodies are player characters; never hardcode them. + private static List ReadPlayerBodies() + { + var list = new List(); + + foreach (var race in Race.AllRaces) + { + if (race == null) + continue; + + list.Add(new PlayerBody(race.Name, "male", race.MaleBody)); + list.Add(new PlayerBody(race.Name, "female", race.FemaleBody)); + list.Add(new PlayerBody(race.Name, "male ghost", race.MaleGhostBody)); + list.Add(new PlayerBody(race.Name, "female ghost", race.FemaleGhostBody)); + } + + return list; + } + + private static void Run(Mobile from, string section, string which) + { + try + { + Directory.CreateDirectory(OutputDir); + + var stamp = DateTime.UtcNow.ToString("yyyyMMdd-HHmmss", CultureInfo.InvariantCulture); + _checkpointPath = Path.Combine(OutputDir, "checkpoint.txt"); + _report = new StreamWriter(Path.Combine(OutputDir, "report-" + stamp + ".txt"), false, new UTF8Encoding(false)); + _report.AutoFlush = true; + + Say("Asset Bridge phase 0 probe — " + DateTime.UtcNow.ToString("u", CultureInfo.InvariantCulture)); + Say("section=" + section + " client=" + (which ?? "(config default)")); + Say(""); + + // Caching off, or a full sweep holds ~49,000 live Bitmaps. That is not only memory: + // every Bitmap is a GDI object and Windows caps a process at 10,000 of them, so a + // cached sweep fails partway through for a reason that has nothing to do with the + // files being read. Each bitmap below is disposed as soon as it is measured. + Files.CacheData = false; + + if (!PointAtClient(which)) + return; + + bool all = section == "all"; + + if (all || section == "paths") SectionPaths(); + if (all || section == "statics") SectionStatics(); + if (all || section == "land") SectionLand(); + if (all || section == "bodies") SectionBodies(); + if (all || section == "players") SectionPlayers(); + if (all || section == "cliloc") SectionCliloc(); + + // Never part of "all". This one is expected to kill the process. + if (section == "gump") SectionGump(); + + Checkpoint("done", 0); + Say(""); + Say("complete."); + } + catch (Exception e) + { + Say("PROBE FAILED: " + e); + } + finally + { + if (_report != null) + { + _report.Dispose(); + _report = null; + } + + lock (_sync) + _running = false; + } + } + + // ── Pointing Ultima at a client ────────────────────────────────────────────────────── + + /// + /// Re-points Ultima.Files at the stock or the patched client and reloads every + /// index built from it. + /// + /// **Not Files.SetMulPath(string).** That overload keeps any entry already + /// holding an absolute path — its own comment reads // absolut dir ignore — and + /// it *writes* absolute paths. So it works exactly once: the second call, and every + /// call that would switch back, is a silent no-op, and the probe would report a run + /// against the patched client while reading the stock one. The two-argument overload + /// writes the key directly and is the only one that can be called twice. + /// + private static bool PointAtClient(string which) + { + var patched = Config.Get("Bridge.AssetProbeClient", (string)null); + + if (which == null) + which = string.IsNullOrEmpty(patched) ? "stock" : "patched"; + + string root; + + if (which == "patched") + { + if (string.IsNullOrEmpty(patched)) + { + Say("no Bridge.AssetProbeClient configured — build one with tools/patch_client.ps1"); + return false; + } + + root = patched; + } + else + { + // What the shard itself resolved at boot: the §1 premise, read rather than assumed. + root = Core.DataDirectories.Count > 0 ? Core.DataDirectories[0] : Files.Directory; + } + + if (string.IsNullOrEmpty(root) || !Directory.Exists(root)) + { + Say("client directory does not exist: " + (root ?? "(null)")); + return false; + } + + foreach (var file in InterestingFiles) + { + var full = Path.Combine(root, file); + // An absent file must resolve to nothing, NOT fall through to the stock client — + // otherwise a patched tree missing a file quietly borrows the real one and the + // run proves nothing. + Files.SetMulPath(File.Exists(full) ? full : string.Empty, file); + } + + Verdata.Initialize(); + Art.Reload(); + Animations.Reload(); + BodyConverter.Initialize(); + Hues.Initialize(); + + Say("pointed at the " + which + " client: " + root); + Say("verdata patches loaded: " + (Verdata.Patches == null ? 0 : Verdata.Patches.Length)); + Say(""); + + return true; + } + + /// + /// Every key this protocol reads, lowercase because that is how Files.MulPath is + /// keyed. Gump keys are absent on purpose (§4.1). + /// + private static readonly string[] InterestingFiles = + { + "anim.idx", "anim.mul", "anim2.idx", "anim2.mul", "anim3.idx", "anim3.mul", + "anim4.idx", "anim4.mul", "anim5.idx", "anim5.mul", + "art.mul", "artidx.mul", "artlegacymul.uop", + "body.def", "bodyconv.def", "hues.mul", "verdata.mul", "cliloc.enu" + }; + + // ── paths ──────────────────────────────────────────────────────────────────────────── + + /// + /// §1's premise, verified from inside the shard rather than argued: the client files + /// are already here, and the shard already knows where. + /// + private static void SectionPaths() + { + Head("paths"); + + Say("Core.DataDirectories (" + Core.DataDirectories.Count + "):"); + + foreach (var dir in Core.DataDirectories) + Say(" " + dir); + + Say("Ultima.Files.Directory: " + (Files.Directory ?? "(null)")); + Say(""); + + foreach (var file in InterestingFiles) + { + var path = Files.GetFilePath(file); + + if (path == null) + { + Say(string.Format(" {0,-22} ABSENT", file)); + continue; + } + + var info = new FileInfo(path); + Say(string.Format( + " {0,-22} {1,14:N0} bytes {2}", file, info.Length, + info.LastWriteTimeUtc.ToString("u", CultureInfo.InvariantCulture))); + } + + Say(""); + } + + // ── statics ────────────────────────────────────────────────────────────────────────── + + private static void SectionStatics() + { + Head("statics — Art.GetStatic"); + + var index = BridgeAssetValidator.OpenArtIndex(); + + if (index == null) + { + Say("no art index — artidx.mul/art.mul did not resolve"); + return; + } + + var dataPath = BridgeAssetValidator.ArtDataPath(); + long mulLength = BridgeAssetValidator.MulLength(dataPath); + long verdataLength = BridgeAssetValidator.MulLength(Files.GetFilePath("verdata.mul")); + + Say(" index offsets are into: " + dataPath); + + var tally = new Tally(); + int max = Config.Get("Bridge.AssetProbeMaxStatic", 0xFFFF); + + using (var reader = new BridgeAssetValidator.RecordReader( + dataPath, Files.GetFilePath("verdata.mul"))) + { + for (int id = 0; id <= max; id++) + { + Checkpoint("statics", id); + + // The validator's verdict, taken from the index entry BEFORE the library is + // asked. 0x4000 is the static offset Art applies internally. + string reason; + var verdict = BridgeAssetValidator.CheckEntry(index, id + 0x4000, mulLength, verdataLength, out reason); + + // The entry can be well-formed and the record inside it still hostile, so a + // passing entry gets its row table walked before the library sees the id. + // This is the check with teeth — LoadStatic's read cursor is unbounded. + if (verdict == BridgeAssetValidator.Verdict.Ok) + { + string deepReason; + + if (!reader.StaticSane(index, id + 0x4000, out deepReason)) + { + verdict = BridgeAssetValidator.Verdict.Refused; + reason = deepReason; + } + } + + // checkmaxid:false deliberately. With it true, GetLegalItemID maps an + // out-of-range id to 0 and the call returns ITEM 0's picture — the + // out-of-range answer would be a real bitmap of the wrong thing, which is + // precisely the confusion being counted. + Bitmap bmp = null; + string thrown = null; + + try + { + bmp = Art.GetStatic(id, false); + } + catch (Exception e) + { + thrown = e.GetType().Name + ": " + e.Message; + } + + Record(tally, verdict, reason, bmp != null, thrown, "static/" + id); + + if (bmp != null) + bmp.Dispose(); + } + } + + tally.Report("statics 0.." + max); + } + + // ── land ───────────────────────────────────────────────────────────────────────────── + + private static void SectionLand() + { + Head("land — Art.GetLand"); + + var index = BridgeAssetValidator.OpenArtIndex(); + + if (index == null) + { + Say("no art index"); + return; + } + + long mulLength = BridgeAssetValidator.MulLength(BridgeAssetValidator.ArtDataPath()); + long verdataLength = BridgeAssetValidator.MulLength(Files.GetFilePath("verdata.mul")); + + var tally = new Tally(); + + for (int id = 0; id < 0x4000; id++) + { + Checkpoint("land", id); + + string reason; + var verdict = BridgeAssetValidator.CheckEntry(index, id, mulLength, verdataLength, out reason); + + // LoadLand reads a fixed 2,024 bytes whatever the record says, so a short record + // reads past the buffer. The validator's land rule is the only thing standing + // between that and an out-of-bounds read. + if (verdict == BridgeAssetValidator.Verdict.Ok) + { + string landReason; + + if (!BridgeAssetValidator.LandLengthSane(index, id, out landReason)) + { + verdict = BridgeAssetValidator.Verdict.Refused; + reason = landReason; + } + } + + Bitmap bmp = null; + string thrown = null; + + try + { + bmp = Art.GetLand(id); + } + catch (Exception e) + { + thrown = e.GetType().Name + ": " + e.Message; + } + + Record(tally, verdict, reason, bmp != null, thrown, "land/" + id); + + if (bmp != null) + bmp.Dispose(); + } + + tally.Report("land 0..16383"); + } + + // ── bodies ─────────────────────────────────────────────────────────────────────────── + + /// + /// Sweeps every body id, taking BodyConverter.Convert's answer and stopping + /// there. + /// + /// **It must never ask the other anim files when that answer yields nothing** (v8.md + /// §4.3). Doing so does not find missing art: gargoyle 666 maps to anim5, where + /// this client has nothing, and asking anim2 for index 666 returns 175 + /// decodable frames of a giant spider. Every one of those reads reports success, and + /// nothing downstream can tell. A "0 rows" outcome is the correct answer. + /// + /// So the sweep records the file type each body resolved to and whether that file + /// answered — and never a second opinion. + /// + private static void SectionBodies() + { + Head("bodies — Animations.GetAnimation, one direction, first frame"); + + int direction = Config.Get("Bridge.AssetProbeCreatureDirection", 1); + int decoded = 0, empty = 0, faulted = 0; + var byFileType = new int[8]; + var faults = new List(); + + for (int body = 0; body < 2048; body++) + { + Checkpoint("bodies", body); + + int translated = body; + int fileType; + + try + { + fileType = BodyConverter.Convert(ref translated); + } + catch (Exception e) + { + faulted++; + faults.Add("body " + body + " BodyConverter.Convert: " + e.GetType().Name + ": " + e.Message); + continue; + } + + if (fileType >= 0 && fileType < byFileType.Length) + byFileType[fileType]++; + + try + { + int hue = 0; + var frames = Animations.GetAnimation(body, 0, direction, ref hue, false, true); + + if (frames != null && frames.Length > 0 && frames[0] != null && frames[0].Bitmap != null) + { + decoded++; + frames[0].Bitmap.Dispose(); + } + else + { + empty++; + } + } + catch (Exception e) + { + faulted++; + + if (faults.Count < 40) + faults.Add("body " + body + " (fileType " + fileType + "): " + e.GetType().Name + ": " + e.Message); + } + } + + Say("direction " + direction + " (creature default — §5.1)"); + Say(string.Format(" decoded {0} empty {1} FAULTED {2}", decoded, empty, faulted)); + Say(" by file type: " + string.Join(", ", FileTypeCounts(byFileType))); + + if (faults.Count > 0) + { + Say(""); + Say(" faults:"); + + foreach (var f in faults) + Say(" " + f); + } + + Say(""); + } + + private static string[] FileTypeCounts(int[] byFileType) + { + var parts = new List(); + + for (int i = 0; i < byFileType.Length; i++) + { + if (byFileType[i] > 0) + parts.Add(i + "=" + byFileType[i]); + } + + return parts.ToArray(); + } + + // ── players ────────────────────────────────────────────────────────────────────────── + + /// + /// The twelve (on stock 57.4) player-character bodies, each at direction 0 — head-on, + /// because a character is a portrait and should look at you (§5.1). + /// + /// Six of them are expected to report nothing on the legacy path: both human ghosts and + /// every gargoyle body are UOP-only. **That is the measurement, not a failure** — it is + /// what phase 4's UOP reader exists for, and a probe that flagged it red would teach an + /// operator to ignore the panel. + /// + private static void SectionPlayers() + { + Head("player bodies — Race.AllRaces, direction 0"); + + if (_playerBodies == null || _playerBodies.Count == 0) + { + Say("no races registered (was the Core-thread snapshot taken?)"); + return; + } + + int direction = Config.Get("Bridge.AssetProbePlayerDirection", 0); + int decoded = 0, absent = 0; + + foreach (var pb in _playerBodies) + { + Checkpoint("players", pb.Body); + + int translated = pb.Body; + int fileType = BodyConverter.Convert(ref translated); + + string outcome; + + try + { + int hue = 0; + var frames = Animations.GetAnimation(pb.Body, 0, direction, ref hue, false, true); + + if (frames != null && frames.Length > 0 && frames[0] != null && frames[0].Bitmap != null) + { + var bmp = frames[0].Bitmap; + outcome = "decoded " + bmp.Width + "x" + bmp.Height; + bmp.Dispose(); + decoded++; + } + else + { + outcome = "no art on the legacy path (UOP-only — phase 4)"; + absent++; + } + } + catch (Exception e) + { + outcome = "FAULTED " + e.GetType().Name + ": " + e.Message; + } + + Say(string.Format(" {0,-10} {1,-14} body {2,-5} fileType {3,-3} {4}", + pb.Race, pb.Slot, pb.Body, fileType, outcome)); + } + + Say(""); + Say(string.Format(" {0} decoded, {1} absent, of {2}", decoded, absent, _playerBodies.Count)); + Say(""); + } + + // ── cliloc ─────────────────────────────────────────────────────────────────────────── + + /// + /// Runs against the client's own Cliloc.enu and, when + /// a reference is configured, diffs it against UOFiddler's output entry by entry. + /// + /// The reference is what makes this a test rather than a demonstration. A decompressor + /// that is subtly wrong still produces a plausible table — mostly-right strings with a + /// few mangled ones is the expected shape of a bug in an inverse-BWT coder, and a row + /// count alone would pass it. Produce the reference with + /// website/server/tools/cliloc-export --tsv. + /// + private static void SectionCliloc() + { + Head("cliloc — the ported Mythic reader (§9)"); + + var path = Files.GetFilePath("cliloc.enu"); + + if (path == null) + { + Say("cliloc.enu did not resolve"); + return; + } + + Checkpoint("cliloc", 0); + + var started = DateTime.UtcNow; + + List entries; + string warning, error; + + if (!BridgeMythicCliloc.TryLoadFile(path, out entries, out warning, out error)) + { + Say("FAILED: " + error); + return; + } + + var elapsed = DateTime.UtcNow - started; + + int blank = 0; + + foreach (var entry in entries) + { + if (string.IsNullOrEmpty(entry.Text)) + blank++; + } + + Say(string.Format(" {0:N0} entries in {1:N0} ms ({2:N0} blank, {3:N0} would be stored)", + entries.Count, elapsed.TotalMilliseconds, blank, entries.Count - blank)); + + if (warning != null) + Say(" WARNING: " + warning); + + var reference = Config.Get("Bridge.AssetProbeClilocRef", (string)null); + + if (string.IsNullOrEmpty(reference)) + { + Say(" no Bridge.AssetProbeClilocRef set — row count only, which proves nothing about the strings"); + Say(""); + return; + } + + CompareToReference(entries, reference); + } + + /// + /// Diffs against a UOFiddler-produced tab-separated table. The comparison is + /// deliberately two-sided: an id we produced and it did not is as much a defect as a + /// mismatched string, and only checking the ids we happen to hold would hide a table + /// that stopped early. + /// + private static void CompareToReference(List entries, string reference) + { + if (!File.Exists(reference)) + { + Say(" reference not found: " + reference); + return; + } + + var theirs = new Dictionary(); + + foreach (var line in File.ReadAllLines(reference)) + { + var tab = line.IndexOf('\t'); + + if (tab <= 0) + continue; + + int number; + + if (!int.TryParse(line.Substring(0, tab), NumberStyles.Integer, CultureInfo.InvariantCulture, out number)) + continue; + + theirs[number] = line.Substring(tab + 1); + } + + var ours = new Dictionary(); + + foreach (var entry in entries) + ours[entry.Number] = entry.Text; + + int matched = 0, differed = 0, onlyOurs = 0, onlyTheirs = 0; + var examples = new List(); + + foreach (var pair in ours) + { + string theirText; + + if (!theirs.TryGetValue(pair.Key, out theirText)) + { + onlyOurs++; + continue; + } + + // The reference is written by a tool that collapses tabs and newlines to spaces, + // so compare on the same footing rather than reporting whitespace as a defect. + if (Flatten(pair.Value) == theirText) + { + matched++; + } + else + { + differed++; + + if (examples.Count < 10) + { + examples.Add(" #" + pair.Key + + "\n ours: " + Truncate(Flatten(pair.Value)) + + "\n theirs: " + Truncate(theirText)); + } + } + } + + foreach (var key in theirs.Keys) + { + if (!ours.ContainsKey(key)) + onlyTheirs++; + } + + Say(string.Format(" vs UOFiddler: {0:N0} identical, {1:N0} differ, {2:N0} only ours, {3:N0} only theirs", + matched, differed, onlyOurs, onlyTheirs)); + + if (differed == 0 && onlyOurs == 0 && onlyTheirs == 0) + Say(" IDENTICAL — the port reproduces UOFiddler's table exactly"); + + foreach (var example in examples) + Say(example); + + Say(""); + } + + private static string Flatten(string s) + { + return s.Replace('\t', ' ').Replace('\r', ' ').Replace('\n', ' '); + } + + private static string Truncate(string s) + { + return s.Length <= 90 ? s : s.Substring(0, 90) + "…"; + } + + // ── gump: the deliberate crash ─────────────────────────────────────────────────────── + + /// + /// Reproduces §4.1's access violation **from inside a running ServUO**, which is the + /// only place the claim actually matters. `Gumps` is the one decoder that builds its + /// `FileIndex` with hasExtra: true, and FileIndex.cs's own comment says + /// that branch exists for gumpartlegacy.uop. + /// + /// AccessViolationException is a corrupted-state exception and .NET Framework + /// 4.8 does not deliver it to an ordinary catch, so **this takes the shard down** and + /// there is no in-process defence. That is the finding, and the reason "nothing calls + /// Ultima.Gumps" is a safety rule rather than a scoping preference. Never part + /// of "all"; never run on anything but a rig. + /// + private static void SectionGump() + { + Head("gump — DELIBERATE CRASH (§4.1)"); + Say(" This is expected to kill the process. Nothing in Protocol 8 calls Ultima.Gumps."); + Say(" If the shard survives this section, that is itself the finding — record it."); + + Checkpoint("gump", 2); + + try + { + var bmp = Ultima.Gumps.GetGump(2); + Say(" SURVIVED: GetGump(2) returned " + (bmp == null ? "null" : bmp.Width + "x" + bmp.Height)); + + if (bmp != null) + bmp.Dispose(); + } + catch (Exception e) + { + Say(" caught (so it was not a corrupted-state exception): " + e.GetType().Name + ": " + e.Message); + } + + Say(""); + } + + // ── Tally ──────────────────────────────────────────────────────────────────────────── + + /// + /// The four counts that matter, and one of them is the point of the whole probe. + /// + /// RefusedButDecoded is a record the validator rejects and the library returned + /// a picture for anyway. On a stock client that number should be zero. On a patched + /// one it is the population of wrong pictures — the failure this protocol most needs + /// to avoid, because it raises no error anywhere and no success count can see it. + /// + private sealed class Tally + { + public int Ok; // validator passed, decoded + public int Absent; // validator says nothing there, library agreed + public int AbsentButDecoded; // NOTHING is there, and the library returned a picture + public int Refused; // validator refused, library also returned nothing + public int RefusedButDecoded; // validator refused, library returned a picture anyway + public int OkButNothing; // validator passed, library returned nothing + public int Threw; // the library threw + public readonly List Examples = new List(); + + public void Report(string label) + { + Say(string.Format(" {0}:", label)); + Say(string.Format(" ok {0:N0} absent {1:N0} refused {2:N0} threw {3:N0}", Ok, Absent, Refused, Threw)); + Say(string.Format(" validator passed but nothing decoded: {0:N0}", OkButNothing)); + Say(string.Format(" WRONG PICTURES, empty record: {0:N0}", AbsentButDecoded)); + Say(string.Format(" WRONG PICTURES, bad record: {0:N0}", RefusedButDecoded)); + + if (Examples.Count > 0) + { + Say(" examples:"); + + foreach (var example in Examples) + Say(" " + example); + } + + Say(""); + } + } + + private static void Record(Tally tally, BridgeAssetValidator.Verdict verdict, string reason, bool decoded, string thrown, string key) + { + if (thrown != null) + { + tally.Threw++; + + if (tally.Examples.Count < 20) + tally.Examples.Add(key + " THREW " + thrown + (reason == null ? "" : " [validator: " + reason + "]")); + + return; + } + + switch (verdict) + { + case BridgeAssetValidator.Verdict.Ok: + if (decoded) + tally.Ok++; + else + tally.OkButNothing++; + + break; + + case BridgeAssetValidator.Verdict.Absent: + // An empty record that still yields a bitmap is not a disagreement about + // strictness. It is the shared-buffer defect: LoadStatic reuses + // m_StreamBuffer, only ever grows it, and discards stream.Read's return, so + // a zero-length record decodes whatever the PREVIOUS asset left behind. + if (decoded) + { + tally.AbsentButDecoded++; + + if (tally.Examples.Count < 20) + tally.Examples.Add(key + " has no record (" + reason + ") — the library returned a picture"); + } + else + { + tally.Absent++; + } + + break; + + case BridgeAssetValidator.Verdict.Refused: + if (decoded) + { + tally.RefusedButDecoded++; + + if (tally.Examples.Count < 20) + tally.Examples.Add(key + " REFUSED (" + reason + ") — library returned a picture anyway"); + } + else + { + tally.Refused++; + } + + break; + } + } + + // ── Output ─────────────────────────────────────────────────────────────────────────── + + /// + /// Writes the id the probe is **about to** touch, then flushes. + /// + /// Learned the expensive way from the PowerShell probes and it matters more here: an + /// access violation is not catchable and does not unwind, so nothing in this file runs + /// after one. The last line in this file is the only evidence of which id killed the + /// shard. + /// + private static void Checkpoint(string section, int id) + { + try + { + File.WriteAllText(_checkpointPath, section + " " + id + " @ " + + DateTime.UtcNow.ToString("u", CultureInfo.InvariantCulture) + Environment.NewLine); + } + catch + { + // A checkpoint that cannot be written must not stop the sweep. + } + } + + private static void Head(string title) + { + Say("── " + title + " " + new string('─', Math.Max(0, 70 - title.Length))); + } + + private static void Say(string text) + { + Console.WriteLine("[assetprobe] {0}", text); + + var report = _report; + + if (report != null) + { + try + { + report.WriteLine(text); + } + catch + { + // Reporting must never be the thing that fails the run. + } + } + } + + private static void Tell(Mobile to, string text) + { + Console.WriteLine("[assetprobe] {0}", text); + + if (to != null) + to.SendMessage(text); + } + } + + /// + /// **Validate before calling** — the response the org lead chose for §4.2's residual risk, + /// prototyped here so phase 1 adopts it with measurements rather than on faith. + /// + /// The principle: `Ultima`'s decoders take their bounds from the file they are reading, so + /// the extractor must decide whether a record is worth handing over *before* handing it + /// over. Every check below is against the index entry and the record header — cheap, and + /// enough to turn an uncatchable corrupted-state exception into a skipped asset. + /// + /// It cannot be complete and does not claim to be. It closes the shapes that reading the + /// source showed are reachable; the probe's REFUSED-BUT-DECODED count is what says whether + /// the boundary is drawn in the right place. + /// + /// Promoted into the overlay in phase 1. + /// + public static class BridgeAssetValidator + { + public enum Verdict + { + /// Nothing at this id, and the index says so honestly. + Absent, + + /// The entry is self-consistent and inside its file. + Ok, + + /// The entry claims something the file cannot support. Do not decode it. + Refused + } + + /// Land tiles decode a fixed 44×44 diamond: 2 × (2+4+…+44) ushorts. + public const int LandRecordBytes = 2024; + + /// + /// A ceiling on decoded art dimensions. `LoadStatic` allocates + /// new Bitmap(width, height) straight from two ushorts in the record, so a + /// corrupt header asks for up to 65535×65535 — an 8 GB allocation, from a file. Real + /// art is a couple of hundred pixels at most. + /// + public const int MaxArtDimension = 1024; + + /// + /// Builds our own index over the same files, with the same constructor arguments + /// Art uses — including hasExtra: false, which is the whole reason the + /// art path is safe where the gump path is not (§4.1). + /// + public static FileIndex OpenArtIndex() + { + if (ArtDataPath() == null) + return null; + + return new FileIndex("Artidx.mul", "Art.mul", "artLegacyMUL.uop", 0x10000, 4, ".tga", 0x13FDC, false); + } + + /// + /// The file an art index entry's lookup is an offset **into** — which is not + /// art.mul on any current client. + /// + /// This cost a whole probe run to learn and it is the single most important thing + /// phase 1 must not get wrong. FileIndex's UOP constructor ends with a bare + /// MulPath = uopPath: **when artLegacyMUL.uop exists it wins outright**, + /// and art.mul / artidx.mul are never opened at all. A validator that + /// bounds offsets against art.mul while the index holds UOP offsets is not + /// merely approximate, it is nonsense — the first run of this probe refused 34,299 + /// perfectly good statics for "declaring 10533x2085" because it was reading UOP + /// offsets into the wrong file. + /// + /// So the resolution order here mirrors FileIndex's exactly, and anything that + /// needs the bytes behind an entry must ask this rather than assume. + /// + public static string ArtDataPath() + { + var uop = Files.GetFilePath("artlegacymul.uop"); + + if (uop != null) + return uop; + + return Files.GetFilePath("art.mul"); + } + + public static long MulLength(string path) + { + if (path == null) + return 0; + + try + { + return new FileInfo(path).Length; + } + catch + { + return 0; + } + } + + /// + /// Judges one index entry. + /// + /// The check FileIndex.Seek is missing is the last one: it tests + /// Stream.Length < e.lookup — that the record *starts* inside the file — and + /// never that it *ends* inside it. A record that begins two bytes before EOF and + /// declares a length of 4,000 passes, and stream.Read then returns a short count + /// that the decoders discard, leaving the previous asset's bytes in the shared buffer. + /// + public static Verdict CheckEntry(FileIndex index, int at, long mulLength, long verdataLength, out string reason) + { + reason = null; + + if (index == null || index.Index == null || at < 0 || at >= index.Index.Length) + { + reason = "index " + at + " out of range"; + return Verdict.Absent; + } + + Entry3D e = index.Index[at]; + + if (e.lookup < 0) + { + reason = "lookup " + e.lookup; + return Verdict.Absent; + } + + bool patched = (e.length & (1 << 31)) != 0; + int length = e.length & 0x7FFFFFFF; + + if (!patched && e.length < 0) + { + reason = "length " + e.length; + return Verdict.Absent; + } + + if (length == 0) + { + reason = "lookup " + e.lookup + ", length 0"; + return Verdict.Absent; + } + + long ceiling = patched ? verdataLength : mulLength; + + if (ceiling <= 0) + { + reason = (patched ? "verdata.mul" : "the art data file") + " has no length"; + return Verdict.Refused; + } + + if (e.lookup >= ceiling) + { + reason = "lookup " + e.lookup + " past the end of " + + (patched ? "verdata.mul" : "the mul") + " (" + ceiling + ")"; + return Verdict.Refused; + } + + // The missing check. A short read is silent, and its consequence is the PREVIOUS + // asset's picture served under this id. + if (e.lookup + (long)length > ceiling) + { + reason = "record runs " + (e.lookup + (long)length - ceiling) + " bytes past the end of " + + (patched ? "verdata.mul" : "the mul"); + return Verdict.Refused; + } + + return Verdict.Ok; + } + + /// + /// `LoadLand` reads 2,024 bytes regardless of the declared length, so a shorter record + /// reads past the end of a buffer sized from that length. + /// + public static bool LandLengthSane(FileIndex index, int at, out string reason) + { + reason = null; + + if (index == null || index.Index == null || at < 0 || at >= index.Index.Length) + return true; + + int length = index.Index[at].length & 0x7FFFFFFF; + + if (length > 0 && length < LandRecordBytes) + { + reason = "land record is " + length + " bytes; LoadLand always reads " + LandRecordBytes; + return false; + } + + return true; + } + + /// + /// Walks a static record's own row table the way LoadStatic will, and refuses + /// it if that walk would read outside the record. + /// + /// This is the check with teeth. LoadStatic's inner loop guards the write into + /// the bitmap (xOffset > delta, xOffset + xRun > delta) and does + /// nothing at all about the read cursor, which advances until it happens to find a + /// zero pair — potentially far outside a pinned array. Simulating the same walk with + /// a bound is the cheapest way to know whether handing the id over is safe. + /// + public static bool StaticRecordSane(byte[] record, int length, out string reason) + { + reason = null; + + if (length < 8) + { + reason = "record is " + length + " bytes; a static header needs 8"; + return false; + } + + int words = length / 2; + int width = ReadUInt16(record, 4); + int height = ReadUInt16(record, 6); + + // LoadStatic returns null for these rather than misbehaving, so it is not a refusal. + if (width <= 0 || height <= 0) + return true; + + if (width > MaxArtDimension || height > MaxArtDimension) + { + reason = "declares " + width + "x" + height + ", past the " + MaxArtDimension + "px ceiling"; + return false; + } + + // The row-lookup table: height ushorts starting at word 4. + if (4 + height > words) + { + reason = "row table (" + height + " entries) does not fit in a " + length + "-byte record"; + return false; + } + + int start = height + 4; + + for (int y = 0; y < height; y++) + { + int cursor = start + ReadUInt16(record, (4 + y) * 2); + + while (true) + { + // Two ushorts for the run header, and they must both be inside the record. + if (cursor < 0 || cursor + 1 >= words) + { + reason = "row " + y + " reads at word " + cursor + ", past the record's " + words; + return false; + } + + int xOffset = ReadUInt16(record, cursor * 2); + int xRun = ReadUInt16(record, (cursor + 1) * 2); + cursor += 2; + + if (xOffset + xRun == 0) + break; + + // LoadStatic stops the row here, so the read cursor stops with it. + if (xOffset > width || xOffset + xRun > width) + break; + + if (cursor + xRun > words) + { + reason = "row " + y + " declares a " + xRun + "-pixel run running past the record"; + return false; + } + + cursor += xRun; + } + } + + return true; + } + + private static int ReadUInt16(byte[] b, int at) + { + return b[at] | (b[at + 1] << 8); + } + + /// + /// Reads a record's actual bytes so can walk it. + /// + /// Holds its own handles rather than borrowing the library's, because FileIndex + /// hands out the stream it decodes from and moving that stream's position underneath + /// the decoder would be its own bug. Opened FileShare.ReadWrite to match how + /// FileIndex opens the same files. + /// + public sealed class RecordReader : IDisposable + { + private readonly FileStream _mul; + private readonly FileStream _verdata; + private byte[] _scratch = new byte[64 * 1024]; + + public RecordReader(string mulPath, string verdataPath) + { + _mul = Open(mulPath); + _verdata = Open(verdataPath); + } + + private static FileStream Open(string path) + { + if (path == null || !File.Exists(path)) + return null; + + try + { + return new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite); + } + catch + { + return null; + } + } + + /// + /// True when the record at is safe to hand to + /// Art.GetStatic. A record that cannot be read at all is reported sane — + /// has already judged the entry, and this must not + /// invent a second reason to refuse. + /// + public bool StaticSane(FileIndex index, int at, out string reason) + { + reason = null; + + if (index == null || index.Index == null || at < 0 || at >= index.Index.Length) + return true; + + Entry3D e = index.Index[at]; + bool patched = (e.length & (1 << 31)) != 0; + int length = e.length & 0x7FFFFFFF; + + var stream = patched ? _verdata : _mul; + + if (stream == null || length <= 0 || e.lookup < 0) + return true; + + if (_scratch.Length < length) + _scratch = new byte[length]; + + int read; + + try + { + stream.Seek(e.lookup, SeekOrigin.Begin); + read = stream.Read(_scratch, 0, length); + } + catch (Exception ex) + { + reason = "cannot read the record: " + ex.GetType().Name; + return false; + } + + // The short read the decoders discard. Refusing here is the whole point: the + // library would decode whatever the shared buffer happened to hold. + if (read < length) + { + reason = "short read — " + read + " of " + length + " bytes available"; + return false; + } + + return StaticRecordSane(_scratch, length, out reason); + } + + public void Dispose() + { + if (_mul != null) + _mul.Dispose(); + + if (_verdata != null) + _verdata.Dispose(); + } + } + } +} diff --git a/tools/scaffolding/BridgeMythicCliloc.cs b/tools/scaffolding/BridgeMythicCliloc.cs new file mode 100644 index 0000000..c77974d --- /dev/null +++ b/tools/scaffolding/BridgeMythicCliloc.cs @@ -0,0 +1,532 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Text; + +namespace Server.Custom +{ + /// + /// A reader for the **Mythic compressed** cliloc container, in plain .NET Framework 4.8 C#. + /// + /// This is the Asset Bridge's §9 decoder — the ONE decoder Protocol 8 writes rather than + /// calls (docs/link/v8.md §4, §9). ServUO's bundled Ultima.StringList implements only + /// the plain layout and throws Non-negative number required on every modern client's + /// file, which is also why the shard's own VendorSearch.GetItemName is already inert. + /// + /// **Provenance.** Ported from UOFiddler's Ultima/Helpers/MythicDecompress.cs, + /// MoveToFront.cs and StringList.TryParse (polserver/UOFiddler). UOFiddler is + /// released under the **Beerware** licence, so carrying its algorithm into this + /// GPL-3.0-or-later tree is clean — see v8.md §9. + /// + /// **What the port had to change**, and why the differences are not cosmetic: + /// + /// * UOFiddler targets net10.0 and its implementation is written in Span<T>, + /// stackalloc, ArrayPool and BinaryPrimitives. ServUO compiles the + /// overlay against net48 with no package feed, so all of that becomes plain arrays. + /// * Every read of the compressed payload is **bounds-checked here and is not there**. + /// Upstream indexes input[m + 1024] and input[firstVal + 1024] with + /// offsets derived from the file's own frequency header, inside a + /// try { } catch (Exception) { return false; }. That is adequate for a desktop + /// tool and is not adequate for us: this runs inside a live shard, and a corrupt or + /// hostile Cliloc.enu must produce a refusal, not an exception unwinding through the + /// bridge. Every such index is tested before use and returns false instead. + /// + /// Phase 0 uses this from to prove the port reproduces + /// UOFiddler's own output exactly. **Phase 2 promotes this file into + /// overlay/Scripts/Custom/Bridge/** — it lives in scaffolding only for as long as it + /// is a spike. + /// + public static class BridgeMythicCliloc + { + /// The first DWORD of a compressed file is the decompressed length, XORed with this. + private const uint HeaderXorKey = 0x8E2C9A3D; + + /// 256 little-endian int32 symbol frequencies precede the coded payload. + private const int FrequencyHeaderSize = 1024; + + /// One decoded cliloc row. Mirrors Ultima.StringEntry's three fields. + public struct Entry + { + public int Number; + public byte Flag; + public string Text; + + public Entry(int number, byte flag, string text) + { + Number = number; + Flag = flag; + Text = text; + } + } + + // ── Container detection ────────────────────────────────────────────────────────────── + + /// + /// True when the file looks like the Mythic container. The marker is the high byte of + /// the first DWORD being 0x8E — which is not a magic number in the file so much + /// as a consequence of : a plausible decompressed length is + /// small enough that its top byte is zero, so the XOR leaves 0x8E showing. + /// + public static bool LooksCompressed(byte[] buffer) + { + return buffer != null && buffer.Length >= 4 && buffer[3] == 0x8E; + } + + // ── The public entry point ─────────────────────────────────────────────────────────── + + /// + /// Reads a cliloc file, compressed or plain, and returns its entries. + /// + /// Tries the layout the header suggests first and the other one second — the same + /// fallback UOFiddler performs, and the reason an already-converted file passes + /// straight through. is non-null when a layout parsed + /// *partially*: that is the case a caller must surface rather than swallow, because a + /// quietly short table is the failure mode the website's importer refuses. + /// + public static bool TryLoadFile(string path, out List entries, out string warning, out string error) + { + entries = null; + warning = null; + error = null; + + byte[] buffer; + + try + { + buffer = File.ReadAllBytes(path); + } + catch (Exception e) + { + error = "cannot read " + path + ": " + e.Message; + return false; + } + + return TryLoad(buffer, out entries, out warning, out error); + } + + /// Reads an in-memory cliloc file. See . + public static bool TryLoad(byte[] buffer, out List entries, out string warning, out string error) + { + entries = null; + warning = null; + error = null; + + bool compressedFirst = LooksCompressed(buffer); + + List primary; + string primaryError; + bool primaryComplete; + + if (TryParse(buffer, compressedFirst, out primary, out primaryComplete, out primaryError) && primaryComplete) + { + entries = primary; + return true; + } + + List fallback; + string fallbackError; + bool fallbackComplete; + + if (TryParse(buffer, !compressedFirst, out fallback, out fallbackComplete, out fallbackError) && fallbackComplete) + { + entries = fallback; + return true; + } + + // Neither layout parsed to the end. Take whichever salvaged more rows and say so. + int primaryCount = primary == null ? 0 : primary.Count; + int fallbackCount = fallback == null ? 0 : fallback.Count; + + if (primaryCount == 0 && fallbackCount == 0) + { + error = "as " + Label(compressedFirst) + ": " + primaryError + + "; as " + Label(!compressedFirst) + ": " + fallbackError; + return false; + } + + if (primaryCount >= fallbackCount) + { + entries = primary; + warning = "parsed partially as " + Label(compressedFirst) + ": " + primaryError + + " (" + primaryCount + " entries salvaged)"; + } + else + { + entries = fallback; + warning = "parsed partially as " + Label(!compressedFirst) + ": " + fallbackError + + " (" + fallbackCount + " entries salvaged)"; + } + + return true; + } + + private static string Label(bool compressed) + { + return compressed ? "compressed" : "uncompressed"; + } + + // ── Record layout ──────────────────────────────────────────────────────────────────── + + /// + /// Walks the plain record layout: a 4-byte and a 2-byte header, then repeating + /// [int32 number][byte flag][uint16 length][length bytes of UTF-8]. + /// + /// distinguishes "parsed to the end of the file" from + /// "stopped early but salvaged rows", which is the distinction the caller needs and + /// an exception would destroy. + /// + private static bool TryParse(byte[] buffer, bool decompress, out List entries, out bool complete, out string error) + { + entries = new List(); + complete = false; + error = null; + + byte[] data; + + if (decompress) + { + if (!TryDecompress(buffer, out data, out error)) + return false; + } + else + { + data = buffer; + } + + if (data.Length < 6) + { + error = "file is " + data.Length + " bytes, smaller than the 6-byte header"; + return false; + } + + int cursor = 6; // int32 version marker + int16 language marker + int lastNumber = -1; + + while (cursor < data.Length) + { + int entryStart = cursor; + int remaining = data.Length - cursor; + + if (remaining < 7) + { + error = "unexpected " + remaining + " trailing byte(s) at 0x" + entryStart.ToString("X") + + " after entry #" + lastNumber + "; an entry header needs 7"; + return true; + } + + int number = ReadInt32(data, cursor); + byte flag = data[cursor + 4]; + // Deliberately UNSIGNED. Read as Int16, a string of 32768 bytes or more comes back + // negative and corrupts every record after it. + int length = data[cursor + 5] | (data[cursor + 6] << 8); + cursor += 7; + + if (length > data.Length - cursor) + { + error = "entry #" + number + " at 0x" + entryStart.ToString("X") + " declares length " + + length + " but only " + (data.Length - cursor) + " byte(s) remain (parsed " + + entries.Count + " so far)"; + return true; + } + + string text; + + try + { + text = Encoding.UTF8.GetString(data, cursor, length); + } + catch (Exception e) + { + error = "entry #" + number + " at 0x" + entryStart.ToString("X") + " has " + length + + " body bytes that are not valid UTF-8: " + e.Message; + return true; + } + + cursor += length; + + entries.Add(new Entry(number, flag, text)); + lastNumber = number; + } + + complete = true; + return true; + } + + // ── Mythic stage 1: the XOR header and the move-to-front code ──────────────────────── + + /// + /// Reads the obfuscated decompressed length from the first DWORD. Public so a caller + /// can size a buffer before committing to the decode. + /// + public static uint PeekDecompressedLength(byte[] source) + { + if (source == null || source.Length < 4) + return 0; + + return ReadUInt32(source, 0) ^ HeaderXorKey; + } + + /// + /// Decompresses the Mythic container: strip the 4-byte length header, undo the + /// move-to-front coding, then run stage 2. + /// + public static bool TryDecompress(byte[] source, out byte[] output, out string error) + { + output = null; + error = null; + + if (source == null || source.Length < 4) + { + error = "payload shorter than the 4-byte length header"; + return false; + } + + uint dataLength = ReadUInt32(source, 0) ^ HeaderXorKey; + + // A wrong guess about the container makes this astronomically large, which is the + // cheapest possible rejection and must happen before any allocation. + if (dataLength == 0 || dataLength > int.MaxValue) + { + error = "implausible decompressed length " + dataLength + " — not the compressed layout"; + return false; + } + + var mtf = new byte[source.Length - 4]; + MoveToFrontDecode(source, 4, mtf); + + var destination = new byte[(int)dataLength]; + int written; + + if (!TryInternalDecompress(mtf, destination, out written, out error)) + return false; + + if (written != (int)dataLength) + { + error = "decompressed " + written + " bytes, header declared " + dataLength; + return false; + } + + output = destination; + return true; + } + + /// + /// Move-to-front decode. Each input byte is an index into a 256-symbol table; the + /// symbol found there is emitted and moved to the front. + /// + private static void MoveToFrontDecode(byte[] input, int offset, byte[] output) + { + var symbols = new byte[256]; + + for (int i = 0; i < 256; i++) + symbols[i] = (byte)i; + + for (int i = 0; i < output.Length; i++) + { + int index = input[offset + i]; + byte symbol = symbols[index]; + output[i] = symbol; + + for (int j = index; j > 0; j--) + symbols[j] = symbols[j - 1]; + + symbols[0] = symbol; + } + } + + // ── Mythic stage 2 ─────────────────────────────────────────────────────────────────── + + /// + /// Turns the MTF-decoded payload back into the original bytes. + /// + /// The payload is a 1024-byte frequency header (256 little-endian int32 symbol counts) + /// followed by the coded stream. The counts partition the stream into one run per + /// symbol; cursor[] holds each run's read position and limit[] its end, + /// and the walk emits a symbol, advances that symbol's run, and re-orders the symbol + /// table by the index it reads. + /// + /// Every index derived from file content is checked. Upstream's equivalent is wrapped + /// in a blanket catch; here a malformed file is a false with a reason. + /// + private static bool TryInternalDecompress(byte[] input, byte[] destination, out int written, out string error) + { + written = 0; + error = null; + + if (input.Length < FrequencyHeaderSize) + { + error = "payload (" + input.Length + " bytes) is smaller than the 1024-byte frequency header"; + return false; + } + + var counts = new int[256]; // symbol → number of occurrences + var cursor = new int[256]; // symbol → next unread position in its run + var limit = new int[256]; // symbol → one past the end of its run + + int sum = 0; + + for (int i = 0; i < 256; i++) + { + counts[i] = ReadInt32(input, i * 4); + + if (counts[i] < 0) + { + error = "frequency header declares a negative count for symbol " + i; + return false; + } + + sum += counts[i]; + + if (sum < 0) + { + error = "frequency header sums past int range at symbol " + i; + return false; + } + } + + if (sum == 0) + { + written = 0; + return true; + } + + if (destination.Length < sum) + { + error = "destination holds " + destination.Length + " bytes, payload needs " + sum; + return false; + } + + int nonZeroCount = 0; + + for (int i = 0; i < 256; i++) + { + if (counts[i] != 0) + nonZeroCount++; + } + + // The coded stream must be long enough to hold one index per emitted byte. + if (input.Length - FrequencyHeaderSize < sum) + { + error = "coded stream holds " + (input.Length - FrequencyHeaderSize) + " bytes, frequency header claims " + sum; + return false; + } + + var order = new byte[256]; + FrequencyOrder(counts, order); + + var symbolTable = new byte[256]; + + for (int i = 0; i < 256; i++) + symbolTable[i] = (byte)i; + + for (int i = 0, m = 0; i < nonZeroCount; ++i) + { + byte symbol = order[i]; + + // m indexes the coded stream and comes from the file's own counts. + if (m < 0 || m >= input.Length - FrequencyHeaderSize) + { + error = "run table for symbol " + symbol + " starts at " + m + ", past the coded stream"; + return false; + } + + symbolTable[input[m + FrequencyHeaderSize]] = symbol; + cursor[symbol] = m + 1; + m += counts[symbol]; + limit[symbol] = m; + } + + byte val = symbolTable[0]; + int count = 0; + int liveSymbols = nonZeroCount; + + do + { + destination[count] = val; + + if (cursor[val] < limit[val]) + { + int at = cursor[val] + FrequencyHeaderSize; + + if (at < FrequencyHeaderSize || at >= input.Length) + { + error = "run for symbol " + val + " reads at " + at + ", past the " + input.Length + "-byte payload"; + return false; + } + + byte index = input[at]; + cursor[val]++; + + if (index != 0) + { + ShiftLeft(symbolTable, index); + symbolTable[index] = val; + val = symbolTable[0]; + } + } + else if (liveSymbols-- > 0) + { + ShiftLeft(symbolTable, liveSymbols); + val = symbolTable[0]; + } + + count++; + } + while (count < sum); + + written = sum; + return true; + } + + /// + /// Orders symbols by descending frequency: repeatedly take the largest remaining count + /// and record its symbol. Ties go to the lower symbol, because the scan keeps the first + /// strictly-greater value — matching upstream, and the tie-break is load-bearing. + /// + private static void FrequencyOrder(int[] counts, byte[] output) + { + var tmp = new int[256]; + Array.Copy(counts, tmp, 256); + + for (int i = 0; i < 256; i++) + { + int best = 0; + byte index = 0; + + for (int j = 0; j < 256; j++) + { + if (tmp[j] > best) + { + index = (byte)j; + best = tmp[j]; + } + } + + if (best == 0) + break; + + output[i] = index; + tmp[index] = 0; + } + } + + /// Shifts [1..element] down one slot, dropping element 0. + private static void ShiftLeft(byte[] input, int element) + { + for (int i = 0; i < element; ++i) + input[i] = input[i + 1]; + } + + // ── Little-endian readers (BinaryPrimitives is not available on net48) ─────────────── + + private static int ReadInt32(byte[] b, int at) + { + return b[at] | (b[at + 1] << 8) | (b[at + 2] << 16) | (b[at + 3] << 24); + } + + private static uint ReadUInt32(byte[] b, int at) + { + return (uint)(b[at] | (b[at + 1] << 8) | (b[at + 2] << 16) | (b[at + 3] << 24)); + } + } +} diff --git a/tools/scaffolding/BridgeRigDriver.cs b/tools/scaffolding/BridgeRigDriver.cs index 8bb61c1..116efc1 100644 --- a/tools/scaffolding/BridgeRigDriver.cs +++ b/tools/scaffolding/BridgeRigDriver.cs @@ -155,6 +155,14 @@ namespace Server.Custom case "partprobe": BridgeParticipationProbe.Run(null, Arg(parts, 1), Int(Arg(parts, 2)), Int(Arg(parts, 3))); break; + // Asset Bridge phase 0. Here for the same reason as partprobe, and for one more: + // the point of that spike is comparing the STOCK client's answers with a patched + // client's, and `AssetProbeOnStart` can only ever run whichever one the config + // names. Driving it from here runs both against a single boot, so a difference + // between them cannot be a difference between two shard processes. + case "assetprobe": + BridgeAssetProbe.Begin(null, Arg(parts, 1) ?? "all", Arg(parts, 2)); + break; // Phase 12a. `world.despawn` answering `gone` rather than `removed` is the // path a player takes every time they kill an event creature, and it is the one // outcome the rig cannot reach by asking the bridge: every bridge verb that diff --git a/tools/scaffolding/README.md b/tools/scaffolding/README.md index ed1d281..95a79cc 100644 --- a/tools/scaffolding/README.md +++ b/tools/scaffolding/README.md @@ -14,10 +14,12 @@ These two scripts produced the measured budget in [PLAN.md](https://gitea.whitlo | `BridgeCrierProbe.cs` | `Scripts/Custom/BridgeCrierProbe.cs` | Logs the global town-crier entry list every 3s so `towncrier.add` / `remove` can be seen landing in game state. Flag: `CrierProbeOnStart`. | | `BridgeVendorSaleProbe.cs` | `Scripts/Custom/BridgeVendorSaleProbe.cs` | Fires `PlayerVendorSale` (Phase 7) with real seeded-vendor data so `vendor.sale` can be verified without a live buy. Requires the Phase 7 patches applied. Flag: `VendorSaleProbeOnStart`. | | `BridgeDemoDress.cs` | `Scripts/Custom/BridgeDemoDress.cs` | Renames a seeded world so it is presentable in a screenshot: shop signs, vendor and character names, house signs. Also stages a few condemned houses back into IDOC, and sets a known password on `seed_000` so a character can be logged in. Flags: `DemoDressOnStart`, `DemoDressPassword`. In game: `[demodress`. | -| `BridgeRigDriver.cs` | `Scripts/Custom/BridgeRigDriver.cs` | Drives the shard from OUTSIDE the game, one verb per line in `Config/rigcmd.txt`, which the driver polls and truncates. Written for the engagement Phase 11b acceptance walk, where each step's assertion is what happened BETWEEN two steps, so the steps have to be separated by the observer rather than by a hard-coded delay -- and ServUO's console takes a fixed verb set (`Scripts/Misc/ConsoleCommands.cs`), so `[p5probe` cannot be typed at a headless shard at all. Verbs: `decaylist`, `decay`, `vendorlist`, `vendorfunds`, `citylist`, `governor`, `election`, `activate`, `password`, `configset`, `configread`, `partprobe`, `save`, `shutdown`. Flag: `RigDriverEnabled`. `configset` exists because **`Config.Get` is written by exactly ONE caller in the whole of ServUO 57.4** (`Server/ScriptCompiler.cs`): no in-game command, gump or console verb writes a config key, so on a stock shard a GM cannot drift a configuration lease even deliberately, and a lease's compare-and-set restore would have no way to be proved. `configread` reads a key back through `Config.Get` long after every type initialiser has run, which is how a key that TOOK is told from one that only appeared to. **Sets passwords, writes live config and mutates the world.** | +| `BridgeRigDriver.cs` | `Scripts/Custom/BridgeRigDriver.cs` | Drives the shard from OUTSIDE the game, one verb per line in `Config/rigcmd.txt`, which the driver polls and truncates. Written for the engagement Phase 11b acceptance walk, where each step's assertion is what happened BETWEEN two steps, so the steps have to be separated by the observer rather than by a hard-coded delay -- and ServUO's console takes a fixed verb set (`Scripts/Misc/ConsoleCommands.cs`), so `[p5probe` cannot be typed at a headless shard at all. Verbs: `decaylist`, `decay`, `vendorlist`, `vendorfunds`, `citylist`, `governor`, `election`, `activate`, `password`, `configset`, `configread`, `partprobe`, `assetprobe`, `save`, `shutdown`. Flag: `RigDriverEnabled`. `configset` exists because **`Config.Get` is written by exactly ONE caller in the whole of ServUO 57.4** (`Server/ScriptCompiler.cs`): no in-game command, gump or console verb writes a config key, so on a stock shard a GM cannot drift a configuration lease even deliberately, and a lease's compare-and-set restore would have no way to be proved. `configread` reads a key back through `Config.Get` long after every type initialiser has run, which is how a key that TOOK is told from one that only appeared to. **Sets passwords, writes live config and mutates the world.** | | `BridgeProtocol5Probe.cs` | `Scripts/Custom/BridgeProtocol5Probe.cs` | Drives all three Protocol 5 enrichments so their frames can be observed: walks one house Fairly -> Greatly -> IDOC (the PAIR is the assertion -- `estimatedCollapse` must appear only on the IDOC frame), reports each player vendor's fee state straight off the `PlayerVendor` so the emitted `fees` block can be checked against the shard's own numbers, and fires `EventSink.AccountLogin`. Flags: `Protocol5ProbeOnStart`, `Protocol5ProbeAccount`, `Protocol5ProbePassword`. In game: `[p5probe`. **Sets a password on the named account.** | | `BridgeProtocol6Probe.cs` | `Scripts/Custom/BridgeProtocol6Probe.cs` | Spawns a real champion boss through the shard's own `SpawnChampion()`, waits two champ sweeps so the boss is attributed to its altar, registers unequal damage from two seeded players and kills it -- so `champ.boss.killed` can be observed with a real damage table. **The wait is the assertion**: without it the kill still emits, but with no `serial`/`type`/`level`, which is the documented fallback rather than the case being tested. The altar is placed inside a NAMED region on purpose (see below). Flag: `Protocol6ProbeOnStart`. In game: `[p6probe`. **Spawns and kills a champion boss; rig only.** Protocol 6's other half, the idempotency key, needs no probe -- it is driven from outside with two identical POSTs to the sidecar. | | `BridgeParticipationProbe.cs` | `Scripts/Custom/BridgeParticipationProbe.cs` | Produces real kill credit inside a participation area with no game client: moves two player mobiles to the venue, spawns a creature there, damages it unequally from both and kills it. **Presence is the half it cannot drive** -- the sweep credits players with a live `NetState`, which is the correct test and not one a probe should loosen, so presence accrual needs a real login. In game: `[partprobe `; from a headless rig, through `BridgeRigDriver`'s `partprobe` verb (the two ship together for that reason). **Moves players and spawns and kills a creature; rig only.** | +| `BridgeAssetProbe.cs` | `Scripts/Custom/BridgeAssetProbe.cs` | **Asset Bridge phase 0** (docs/link/v8.md §16). Drives ServUO's vendored `Ultima` decoders from inside a running shard against a deliberately patched client, and compares every answer with what a pre-flight validator says about the index entry *before* the call. The interesting column is not the error count, it is **WRONG PICTURES** -- records the validator rejects and the library renders anyway. Sweeps statics, land, all 2,048 bodies, the player-character bodies from `Race.AllRaces`, and the ported Mythic cliloc reader against UOFiddler's own output. In game / from `BridgeRigDriver`: `[assetprobe [section] [stock|patched]`. Flag: `AssetProbeOnStart`. **Its `gump` section deliberately kills the shard** and is never part of `all`. | +| `BridgeMythicCliloc.cs` | `Scripts/Custom/BridgeMythicCliloc.cs` | The §9 reader for the **Mythic compressed** cliloc container -- the one decoder Protocol 8 writes rather than calls. Ported from UOFiddler (Beerware) into net48 C# with every file-derived index bounds-checked, which upstream's blanket `catch` does not do. Reproduces UOFiddler's 123,490-entry table exactly. **Phase 2 promotes this file into `overlay/`**; it is scaffolding only for as long as it is a spike. | ## Deploy overwrites Bridge.cfg @@ -189,3 +191,127 @@ value that lies, printed next to a frame that disagrees with it. Emitting from a named region is therefore the test. At a dungeon altar the field is legitimately absent and the probe proves nothing about it. + +## What phase 0 found + +`BridgeAssetProbe` exists because [v8.md](https://gitea.whitlocktech.com/RunicGateway/docs/src/branch/main/link/v8.md) §4 chose to **call** ServUO's vendored `Ultima` rather than reimplement it, and the evidence for that choice was a PowerShell probe against a stock client — neither the process nor the client the extractor will actually run in. These are its results, from inside a running ServUO 57.4 against this machine's client, and against a copy broken in 21 catalogued ways by `tools/patch_client.ps1`. + +### The UOP wins outright, and it took a whole run to notice + +`FileIndex`'s UOP constructor ends with a bare `MulPath = uopPath`. **When `artLegacyMUL.uop` is present it wins, and `art.mul` / `artidx.mul` are never opened at all.** Every current client ships the UOP, so: + +- A validator that bounds an index offset against `art.mul` while the index holds UOP offsets is not approximate, it is nonsense. The first run of this probe refused **34,299 perfectly good statics** for "declaring 10533x2085" — and every one of those refusals looked like a real finding. `BridgeAssetValidator.ArtDataPath()` now mirrors `FileIndex`'s own resolution order, and phase 1 must too. +- A custom-art shard that adds graphics to `art.mul` while the UOP is still in place **gets nothing**, silently. That is an operator trap rather than a bug in this protocol, but the extractor is where it will be noticed. +- The `corrupt` and `customart` tiers of `patch_client.ps1` therefore need its `nouop` tier to mean anything at all. Without it they report that they applied, and change nothing. + +### 22,102 wrong pictures on a stock, unmodified client + +The counts that matter, `assetprobe all stock`: + +``` +statics 0..65535 ok 39,189 WRONG PICTURES (empty record) 9,962 threw 16,385 +land 0..16383 ok 4,244 WRONG PICTURES (empty record) 12,140 +``` + +Those 22,102 ids have an index entry of `lookup 0, length 0` — **no record at all**. `FileIndex.Seek` treats that as a hit (it rejects `lookup < 0` and `length < 0`, and zero is neither), hands back the stream, and `LoadStatic` decodes `length` = 0 bytes into `m_StreamBuffer` — which is **reused, only ever grown, and filled by a `stream.Read` whose return value is discarded**. So the id renders whatever the previously-decoded asset left in the buffer. + +**It is specific to the UOP path.** Run the same sweep against the mul path and those ids come back empty and honest, because `artidx.mul` stores `-1` for an absent record while unmapped UOP slots are simply zeroed structs. That is also why the earlier PowerShell probe counted 32,766 of these as "ok": they decode, they raise nothing, and no success count can tell them from art. + +A bulk import that trusted the library would have written 22,102 duplicate images into the site under ids that have no art. This one measurement is the argument for validate-before-calling. + +### Every deliberate defect was caught by the validator and rendered by the library + +`assetprobe all patched`, against the 21-defect client: + +``` +statics ok 39,190 absent 9,954 refused 1 WRONG PICTURES (bad record) 6 threw 16,385 +land ok 4,243 absent 12,140 WRONG PICTURES (bad record) 1 +``` + +| id | the defect | what the library did | +|---|---|---| +| `static/4104` | lookup 4 KB past the end of `art.mul` | returns nothing — `Seek` does check the record's **start** | +| `static/4105` | starts 64 bytes before EOF, declares 8,192 | **renders the previous asset** — `Seek` never checks the record's **end** | +| `static/4108` | declared length 4, smaller than the header | renders something | +| `static/4109` | header declares 8000x8000 | **allocates it** — a ~128 MB bitmap from two bytes in a file, and the same field can ask for 65535×65535 | +| `static/4111` | row table points 60,000 words outside a 512-byte record | renders — `LoadStatic`'s two guards bound the *write* into the bitmap, and nothing bounds the *read* | +| `static/4112` | a 16-pixel run declared in a 20-byte record | renders | +| `static/4131` | verdata entry whose lookup is past verdata.mul's own end | renders — **`Verdata.Seek` has no bounds check whatsoever** | +| `land/256` | 512-byte land record | renders — `LoadLand` reads a fixed 2,024 bytes whatever the length says | + +Seven of the eight produce a confident, wrong picture and raise nothing anywhere. + +The validator refused all eight, and refused **nothing** on the stock client across 49,151 statics and 16,384 land tiles. That second number is the one that matters: a checker that refuses real art is worse than no checker, so "zero false refusals on a clean client" is what makes validate-before-calling more than a hopeful phrase. + +The eight `customart` ids appended past the stock ceiling all decode cleanly, which is that tier's whole point — the ceiling is a property of a file, not a constant anyone should write down. + +### Two more ways to get a wrong answer out of an id that has no art + +- **`Art.GetStatic(id, false)` throws `IndexOutOfRangeException` for `id >= 49,152`** rather than returning null — 16,385 of them in a full sweep. +- **`Art.GetStatic(id)` with the default `checkmaxid: true` is worse**: `GetLegalItemID` maps an out-of-range id to **0**, so the call returns **item 0's picture**. An exception is recoverable; a picture of the wrong item is not even detectable. + +So the extractor takes its id ceiling from the index it opened, and passes `checkmaxid: false` so an overrun is loud rather than plausible. + +### The gump crash reproduces in-process, and nothing catches it + +`assetprobe gump` called `Ultima.Gumps.GetGump(2)` once. **The ServUO process disappeared** — no exception line in the report, no `catch` reached, no shutdown, nothing in the console. The report ends mid-section, and `checkpoint.txt` reading `gump 2` is the entire record of what happened. That is exactly why the checkpoint is written *before* the call and flushed. + +`AccessViolationException` is a corrupted-state exception and .NET Framework 4.8 does not deliver it to ordinary handlers, so **there is no in-process defence** — on a live shard this is a crash with players on it. "Nothing calls `Ultima.Gumps`" is a safety rule, and phase 0's job was to make sure that sentence had been earned rather than assumed. It has. + +### The cliloc port is byte-identical to UOFiddler + +``` +123,490 entries in 218 ms (55,986 blank, 67,504 would be stored) +vs UOFiddler: 123,490 identical, 0 differ, 0 only ours, 0 only theirs +``` + +§9 is proven: the shard can produce the whole table with no UOFiddler installed, no `dotnet build`, and no 5 MB file copied to a server. + +The reference is what makes this a test rather than a demonstration. A subtly wrong inverse-BWT coder still produces a plausible table — mostly-right strings with a few mangled ones is the *expected* shape of a bug in this algorithm, and a row count alone would sail past it. + +Note the blank count is **55,986**, not the 55,994 recorded from the manual pipeline. The difference is eight whitespace-only entries, blank to a `trim()` and not to `IsNullOrEmpty` — a definition rather than a defect, but exactly the sort of eight-row drift that gets investigated as one. + +### What phase 0 did not cover, and phase 1 must + +**The animation path has no validator.** The patched client's verdata entry for body 34 points past verdata.mul's end and the wolf still "decoded" — counted among the 1,144 successes, silently rendering something else, with nothing in the report to say so. `GetAnimation` also allocates `new int[frameCount]` straight from a file-supplied int. Everything above about statics applies here and none of it is implemented yet. + +The deliberate `Bodyconv.def` mis-mappings (bodies 1900 and 1901) produced **nothing** rather than a wrong creature on this client, so they did not reproduce the spider. The gargoyle rows remain the real evidence for the never-sweep-file-types rule: 666, 667, 694 and 695 report nothing, and nothing is the correct answer. + +### Reference: the rest of the run + +``` +bodies 0..2047, direction 1 decoded 1,144 empty 904 faulted 0 + by file type: 1=1222, 2=140, 3=244, 4=150, 5=292 + +player bodies (Race.AllRaces, direction 0) 6 decoded, 6 absent, of 12 + Human 400 / 401 decode; ghosts 402 / 403 absent + Elf 605 / 606 / 607 / 608 all decode + Gargoyle 666 / 667 / 694 / 695 all absent +``` + +Two details worth keeping. The body counts reproduce the PowerShell probe **exactly**, from a different process against the same files, which is what makes the two runs comparable at all. And the gargoyle *ghost* bodies resolve to file type **1**, not 5 like the living gargoyle bodies — so "the gargoyle is an anim5 problem" is not quite the shape of it. + +## Building the patched client + +```powershell +.\tools\patch_client.ps1 -Dest D:\uo-patched-client +``` + +Copies a client (~3.5 GB) and breaks the copy in five catalogued tiers — `nouop`, `verdata`, `customart`, `corrupt`, `bodyconv`. **It never writes to the source**: every file it touches is hashed in the source before and after, and a changed hash aborts the run. Each defect is recorded in `patched-client.manifest.json` beside the copy, which is what makes a nonzero WRONG PICTURES count readable as "the tier worked" instead of "something broke". + +Then point the shard at it and drive the probe: + +```ini +RigDriverEnabled=true +AssetProbeClient=D:\uo-patched-client +AssetProbeClilocRef= +``` + +``` +assetprobe all stock # the baseline: the validator must refuse nothing here +assetprobe all patched # the experiment +``` + +Run both against **one boot**, through `rigcmd.txt`, so a difference between them cannot be a difference between two shard processes. Without `AssetProbeClilocRef` the cliloc section reports a row count, which proves nothing about the strings. + +**The copy is EA's client art.** It stays on the machine that made it, exactly like every other extraction in this project, and is never committed.