Merge pull request 'feat(asset-bridge): phase 0 spike — the decoders, from inside a live shard' (#27) from feat/asset-bridge-p0 into edge
Reviewed-on: #27
This commit is contained in:
580
tools/patch_client.ps1
Normal file
580
tools/patch_client.ps1
Normal file
@@ -0,0 +1,580 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Builds a deliberately patched UO client for the Asset Bridge phase 0 spike.
|
||||
|
||||
.DESCRIPTION
|
||||
docs/link/v8.md section 16 phase 0 drives ServUO's vendored `Ultima` decoders "over a deliberately
|
||||
patched client". Stock clients are not the interesting case: they are the case the library was
|
||||
written against, and the whole reason phase 0 exists is that section 4 chose to call code that can
|
||||
take the shard down if it is wrong. A shard operator's client is patched -- custom art, a
|
||||
verdata.mul, a hand-edited Bodyconv.def -- and that is what has to be survived.
|
||||
|
||||
This copies a client and then breaks the copy in four deliberate, catalogued ways. It NEVER
|
||||
writes to the source: every file it patches is hashed before and after, and a changed source
|
||||
hash aborts the run.
|
||||
|
||||
Each defect is recorded in `patched-client.manifest.json` next to the copy, so the probe's
|
||||
report can be read against what was actually done rather than against a memory of it. The
|
||||
manifest is the answer to "is a nonzero REFUSED-BUT-DECODED count a bug or the point?".
|
||||
|
||||
.PARAMETER Source
|
||||
The client to copy. Defaults to this machine's.
|
||||
|
||||
.PARAMETER Dest
|
||||
Where to build the patched copy. Needs ~3.5 GB.
|
||||
|
||||
.PARAMETER Tiers
|
||||
Which defects to apply. Default: all four.
|
||||
|
||||
verdata Author a verdata.mul, which this client does not have. Ultima consults Verdata on
|
||||
EVERY art and anim lookup (Art's FileIndex is built with verdata file id 4,
|
||||
Animations' with 6), so on a client with no verdata.mul that entire branch is
|
||||
dead code that has never been exercised -- the largest untested surface in the
|
||||
library we are about to depend on. Includes one legitimate patch and one whose
|
||||
lookup points past verdata.mul's own end, because `FileIndex.Seek` bounds-checks
|
||||
the mul and does not bounds-check verdata.
|
||||
|
||||
customart Fill unused artidx.mul slots with real records appended to art.mul, the way a
|
||||
custom-art shard does. Tests that our out-of-range accounting comes from the
|
||||
file rather than from a constant someone wrote down.
|
||||
|
||||
corrupt Rewrite index entries and record headers into the shapes that reading Art.cs
|
||||
says are reachable: a lookup past EOF, a record that starts inside the file and
|
||||
ends outside it, a length too small for a header, absurd dimensions, a row table
|
||||
pointing outside its own record, and a land tile shorter than the fixed 2,024
|
||||
bytes LoadLand always reads.
|
||||
|
||||
bodyconv Add Bodyconv.def lines pointing bodies at an anim file that holds nothing, and at
|
||||
an index in another file that holds something unrelated -- the gargoyle-666 spider
|
||||
case, reproduced on purpose. Proves the extractor takes BodyConverter.Convert's
|
||||
answer and stops (v8.md section 4.3).
|
||||
|
||||
nouop Move artLegacyMUL.uop aside, so art is read from art.mul/artidx.mul.
|
||||
|
||||
This is not cosmetic and it is not optional if you want the customart or corrupt
|
||||
tiers to mean anything. FileIndex's UOP constructor ends with a bare
|
||||
`MulPath = uopPath`: when artLegacyMUL.uop is present it wins OUTRIGHT and
|
||||
art.mul / artidx.mul are never opened. Every index-level defect below writes to
|
||||
files the library does not read on a modern client, so without this tier those
|
||||
two tiers are inert while still reporting that they applied.
|
||||
|
||||
It is also a real configuration in its own right: plenty of shards run mul-only
|
||||
clients, and a custom-art shard that adds graphics to art.mul while the UOP is
|
||||
still there gets nothing at all -- an operator trap worth knowing about.
|
||||
|
||||
.PARAMETER SkipCopy
|
||||
Re-patch an existing copy without re-copying 3.5 GB. Only safe on a copy this script made and
|
||||
has not patched yet -- patching twice compounds the defects and invalidates the manifest.
|
||||
|
||||
.EXAMPLE
|
||||
.\tools\patch_client.ps1 -Dest D:\uo-patched-client
|
||||
|
||||
.NOTES
|
||||
Test scaffolding. Never deployed. The copy contains EA's client art -- like every other
|
||||
extraction in this project it stays on the machine that made it and is never committed.
|
||||
#>
|
||||
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[string] $Source = 'D:\Games\Electronic Arts\Ultima Online Classic',
|
||||
[Parameter(Mandatory = $true)]
|
||||
[string] $Dest,
|
||||
[ValidateSet('verdata', 'customart', 'corrupt', 'bodyconv', 'nouop')]
|
||||
[string[]] $Tiers = @('nouop', 'verdata', 'customart', 'corrupt', 'bodyconv'),
|
||||
[switch] $SkipCopy,
|
||||
[switch] $Force
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
|
||||
# Files this script may write to in the copy. Anything not on this list is a bug in the script,
|
||||
# and the source-hash check at the end is what proves it.
|
||||
$PatchTargets = @('artidx.mul', 'art.mul', 'verdata.mul', 'Bodyconv.def', 'artLegacyMUL.uop')
|
||||
|
||||
# -- Little-endian helpers (BitConverter is fine, but the intent reads better named) ----------
|
||||
|
||||
function Read-Int32LE([byte[]] $Bytes, [int] $Offset) {
|
||||
return [BitConverter]::ToInt32($Bytes, $Offset)
|
||||
}
|
||||
|
||||
function Write-Int32LE([byte[]] $Bytes, [int] $Offset, [int] $Value) {
|
||||
[Array]::Copy([BitConverter]::GetBytes([int] $Value), 0, $Bytes, $Offset, 4)
|
||||
}
|
||||
|
||||
function Get-ArtEntry([byte[]] $Idx, [int] $Index) {
|
||||
$at = $Index * 12
|
||||
return [pscustomobject]@{
|
||||
Index = $Index
|
||||
Lookup = Read-Int32LE $Idx $at
|
||||
Length = Read-Int32LE $Idx ($at + 4)
|
||||
Extra = Read-Int32LE $Idx ($at + 8)
|
||||
}
|
||||
}
|
||||
|
||||
function Set-ArtEntry([byte[]] $Idx, [int] $Index, [int] $Lookup, [int] $Length, [int] $Extra) {
|
||||
$at = $Index * 12
|
||||
Write-Int32LE $Idx $at $Lookup
|
||||
Write-Int32LE $Idx ($at + 4) $Length
|
||||
Write-Int32LE $Idx ($at + 8) $Extra
|
||||
}
|
||||
|
||||
# The defect catalogue. Every mutation appends to this, and it is written out as the manifest.
|
||||
$script:Defects = New-Object System.Collections.ArrayList
|
||||
|
||||
function Add-Defect([string] $Tier, [string] $Key, [string] $What, [string] $Expect) {
|
||||
[void] $script:Defects.Add([pscustomobject]@{
|
||||
tier = $Tier
|
||||
key = $Key
|
||||
what = $What
|
||||
expect = $Expect
|
||||
})
|
||||
Write-Host (" {0,-22} {1}" -f $Key, $What)
|
||||
}
|
||||
|
||||
# -- Preflight --------------------------------------------------------------------------------
|
||||
|
||||
if (-not (Test-Path -LiteralPath $Source)) {
|
||||
throw "source client not found: $Source"
|
||||
}
|
||||
|
||||
$sourceFull = (Resolve-Path -LiteralPath $Source).Path
|
||||
|
||||
if (Test-Path -LiteralPath $Dest) {
|
||||
$destFull = (Resolve-Path -LiteralPath $Dest).Path
|
||||
if ($destFull -eq $sourceFull) {
|
||||
throw "Dest is the source client. Refusing -- this script destroys what it points at."
|
||||
}
|
||||
if (-not $SkipCopy -and -not $Force) {
|
||||
throw "$Dest already exists. Pass -Force to overwrite it, or -SkipCopy to patch it in place."
|
||||
}
|
||||
}
|
||||
|
||||
Write-Host "source: $sourceFull"
|
||||
Write-Host "dest: $Dest"
|
||||
Write-Host "tiers: $($Tiers -join ', ')"
|
||||
Write-Host ""
|
||||
|
||||
# Hash the source files we are about to touch, so "it never writes to the source" is checked and
|
||||
# not merely asserted.
|
||||
$before = @{}
|
||||
foreach ($name in $PatchTargets) {
|
||||
$path = Join-Path $sourceFull $name
|
||||
if (Test-Path -LiteralPath $path) {
|
||||
$before[$name] = (Get-FileHash -LiteralPath $path -Algorithm SHA256).Hash
|
||||
}
|
||||
}
|
||||
|
||||
# -- Copy -------------------------------------------------------------------------------------
|
||||
|
||||
if ($SkipCopy) {
|
||||
Write-Host "skipping copy (-SkipCopy)"
|
||||
if (-not (Test-Path -LiteralPath $Dest)) { throw "-SkipCopy but $Dest does not exist" }
|
||||
} else {
|
||||
Write-Host "copying (this is ~3.5 GB; a few minutes)..."
|
||||
# /MIR so a -Force re-run starts clean rather than merging into an already-patched tree.
|
||||
# /NJH /NJS /NDL /NFL keep robocopy's output to the errors.
|
||||
$null = robocopy $sourceFull $Dest /MIR /R:1 /W:1 /NJH /NJS /NDL /NFL /NP
|
||||
# Robocopy exit codes below 8 are success; 8 and above are real failures.
|
||||
if ($LASTEXITCODE -ge 8) { throw "robocopy failed with exit code $LASTEXITCODE" }
|
||||
# Robocopy's "1 = files were copied" would otherwise become this script's exit code and read
|
||||
# as a failure to anything checking it.
|
||||
$global:LASTEXITCODE = 0
|
||||
Write-Host "copied."
|
||||
}
|
||||
|
||||
Write-Host ""
|
||||
|
||||
$destFull = (Resolve-Path -LiteralPath $Dest).Path
|
||||
|
||||
$artIdxPath = Join-Path $destFull 'artidx.mul'
|
||||
$artMulPath = Join-Path $destFull 'art.mul'
|
||||
|
||||
if (-not (Test-Path -LiteralPath $artIdxPath)) { throw "no artidx.mul in the copy" }
|
||||
|
||||
$idx = [System.IO.File]::ReadAllBytes($artIdxPath)
|
||||
$entryCount = [int] ($idx.Length / 12)
|
||||
$artMulLength = (Get-Item -LiteralPath $artMulPath).Length
|
||||
|
||||
Write-Host ("artidx.mul holds {0:N0} entries; art.mul is {1:N0} bytes" -f $entryCount, $artMulLength)
|
||||
Write-Host ""
|
||||
|
||||
# -- Tier: nouop ------------------------------------------------------------------------------
|
||||
|
||||
$uopPath = Join-Path $destFull 'artLegacyMUL.uop'
|
||||
$uopPresent = Test-Path -LiteralPath $uopPath
|
||||
|
||||
if ($Tiers -contains 'nouop') {
|
||||
Write-Host "tier nouop"
|
||||
|
||||
if (-not $uopPresent) {
|
||||
Write-Host " no artLegacyMUL.uop in the copy -- already a mul-only client"
|
||||
} else {
|
||||
Move-Item -LiteralPath $uopPath -Destination "$uopPath.disabled" -Force
|
||||
$uopPresent = $false
|
||||
Add-Defect 'nouop' 'artLegacyMUL.uop' 'moved aside so art is read from art.mul/artidx.mul' `
|
||||
'every index-level defect below becomes reachable; without this they are inert'
|
||||
}
|
||||
|
||||
Write-Host ""
|
||||
} elseif ($uopPresent -and (($Tiers -contains 'corrupt') -or ($Tiers -contains 'customart'))) {
|
||||
Write-Host " WARNING: artLegacyMUL.uop is present and the nouop tier was not selected."
|
||||
Write-Host " FileIndex prefers the UOP outright, so the corrupt and customart tiers"
|
||||
Write-Host " will write to files the library never opens. Add -Tiers nouop."
|
||||
Write-Host ""
|
||||
}
|
||||
|
||||
# Static ids are offset by 0x4000 in the index; land tiles occupy 0..0x3FFF.
|
||||
$StaticBase = 0x4000
|
||||
|
||||
# Find donor records to copy and victims to corrupt: real, modestly sized statics, so the defects
|
||||
# are applied to entries that genuinely work today. Picking arbitrary ids risks landing on slots
|
||||
# that are already empty, where a "defect" would prove nothing.
|
||||
$donors = New-Object System.Collections.ArrayList
|
||||
for ($id = 0x1000; $id -lt 0x3000 -and $donors.Count -lt 24; $id++) {
|
||||
$e = Get-ArtEntry $idx ($id + $StaticBase)
|
||||
if ($e.Lookup -ge 0 -and $e.Length -gt 200 -and $e.Length -lt 4000 -and ($e.Lookup + $e.Length) -le $artMulLength) {
|
||||
[void] $donors.Add([pscustomobject]@{ Id = $id; Entry = $e })
|
||||
}
|
||||
}
|
||||
|
||||
if ($donors.Count -lt 12) { throw "found only $($donors.Count) usable donor statics -- the copy looks wrong" }
|
||||
|
||||
Write-Host "using donor statics: $(($donors | Select-Object -First 12 | ForEach-Object { $_.Id }) -join ', ')"
|
||||
Write-Host ""
|
||||
|
||||
$idxDirty = $false
|
||||
|
||||
# -- Tier: customart --------------------------------------------------------------------------
|
||||
|
||||
if ($Tiers -contains 'customart') {
|
||||
Write-Host "tier customart"
|
||||
|
||||
# A custom-art client does not fill spare slots -- artidx.mul is exactly sized (62,692
|
||||
# entries here, not one to spare), so adding art means GROWING the index. `Art` builds its
|
||||
# FileIndex with length 0x10000, so there is room for 2,844 more ids before the library stops
|
||||
# looking, and the stock ceiling turns out to be nothing more than the size of a file.
|
||||
$idxCeiling = 0x10000
|
||||
|
||||
if ($entryCount -ge $idxCeiling) {
|
||||
Write-Host " artidx.mul is already at the 0x10000 ceiling -- skipping tier"
|
||||
} else {
|
||||
$addCount = 8
|
||||
$grown = New-Object byte[] (($entryCount + $addCount) * 12)
|
||||
[Array]::Copy($idx, 0, $grown, 0, $idx.Length)
|
||||
$idx = $grown
|
||||
|
||||
# Read every donor record BEFORE opening the append handle. Append mode takes an
|
||||
# exclusive lock, so reading the same file while appending to it fails outright.
|
||||
$buffers = @()
|
||||
$reader = [System.IO.File]::Open($artMulPath, 'Open', 'Read', 'ReadWrite')
|
||||
|
||||
try {
|
||||
for ($n = 0; $n -lt $addCount; $n++) {
|
||||
$donor = $donors[$n]
|
||||
$buffer = New-Object byte[] $donor.Entry.Length
|
||||
[void] $reader.Seek($donor.Entry.Lookup, 'Begin')
|
||||
[void] $reader.Read($buffer, 0, $buffer.Length)
|
||||
$buffers += , $buffer
|
||||
}
|
||||
} finally { $reader.Dispose() }
|
||||
|
||||
$appendAt = $artMulLength
|
||||
$stream = [System.IO.File]::Open($artMulPath, 'Append', 'Write')
|
||||
|
||||
try {
|
||||
for ($n = 0; $n -lt $addCount; $n++) {
|
||||
$buffer = $buffers[$n]
|
||||
$stream.Write($buffer, 0, $buffer.Length)
|
||||
|
||||
$slot = $entryCount + $n
|
||||
$newId = $slot - $StaticBase
|
||||
Set-ArtEntry $idx $slot $appendAt $buffer.Length $donors[$n].Entry.Extra
|
||||
$appendAt += $buffer.Length
|
||||
|
||||
Add-Defect 'customart' "static/$newId" `
|
||||
"custom art appended past the stock ceiling (a copy of static/$($donors[$n].Id))" `
|
||||
'decodes cleanly; proves the ceiling is read from the file, not from a constant'
|
||||
}
|
||||
} finally { $stream.Dispose() }
|
||||
|
||||
$entryCount += $addCount
|
||||
$idxDirty = $true
|
||||
}
|
||||
|
||||
Write-Host ""
|
||||
}
|
||||
|
||||
# -- Tier: corrupt ----------------------------------------------------------------------------
|
||||
|
||||
if ($Tiers -contains 'corrupt') {
|
||||
Write-Host "tier corrupt"
|
||||
|
||||
$artMulLength = (Get-Item -LiteralPath $artMulPath).Length
|
||||
$v = 8 # donors 0..7 may have been consumed by customart as sources; they are unmodified
|
||||
|
||||
# 1. A lookup past the end of art.mul. FileIndex.Seek DOES check this one
|
||||
# (`Stream.Length < e.lookup`), so the library and the validator should agree.
|
||||
$victim = $donors[$v++].Id
|
||||
Set-ArtEntry $idx ($victim + $StaticBase) ([int]($artMulLength + 4096)) 512 0
|
||||
Add-Defect 'corrupt' "static/$victim" 'lookup 4 KB past the end of art.mul' `
|
||||
'refused by the validator; Seek also catches this one, so no picture'
|
||||
|
||||
# 2. A record that STARTS inside the file and ENDS outside it. This is the gap: Seek checks
|
||||
# the start and never the end, stream.Read returns short, the decoders ignore the count,
|
||||
# and m_StreamBuffer still holds the PREVIOUS asset. The expected outcome is a picture of
|
||||
# something else entirely, reported as a success by every count in the library.
|
||||
$victim = $donors[$v++].Id
|
||||
Set-ArtEntry $idx ($victim + $StaticBase) ([int]($artMulLength - 64)) 8192 0
|
||||
Add-Defect 'corrupt' "static/$victim" 'record starts 64 bytes before EOF and declares 8,192' `
|
||||
'REFUSED BUT DECODED -- the stale-buffer wrong picture'
|
||||
|
||||
# 3. A length too small to hold even the 8-byte header.
|
||||
$victim = $donors[$v++].Id
|
||||
$donorEntry = $donors[$v - 1].Entry
|
||||
Set-ArtEntry $idx ($victim + $StaticBase) $donorEntry.Lookup 4 0
|
||||
Add-Defect 'corrupt' "static/$victim" 'declared length 4 -- smaller than the static header' `
|
||||
'refused by the validator'
|
||||
|
||||
# 4/5/6 rewrite the record BODY, so they need their own bytes rather than an index edit.
|
||||
# Appended to art.mul and pointed at, which leaves the donor's real record intact.
|
||||
$stream = [System.IO.File]::Open($artMulPath, 'Append', 'Write')
|
||||
try {
|
||||
$appendAt = (Get-Item -LiteralPath $artMulPath).Length
|
||||
|
||||
# 4. Absurd dimensions. LoadStatic allocates new Bitmap(width, height) straight from two
|
||||
# ushorts in the file. 8000x8000 is ~128 MB -- survivable, and the point is made; the
|
||||
# same field can ask for 65535x65535, which is 8 GB from a two-byte edit.
|
||||
$victim = $donors[$v++].Id
|
||||
$rec = New-Object byte[] 2048
|
||||
[Array]::Copy([BitConverter]::GetBytes([uint16] 8000), 0, $rec, 4, 2)
|
||||
[Array]::Copy([BitConverter]::GetBytes([uint16] 8000), 0, $rec, 6, 2)
|
||||
$stream.Write($rec, 0, $rec.Length)
|
||||
Set-ArtEntry $idx ($victim + $StaticBase) $appendAt $rec.Length 0
|
||||
$appendAt += $rec.Length
|
||||
Add-Defect 'corrupt' "static/$victim" 'header declares 8000x8000 (a ~128 MB allocation from two bytes)' `
|
||||
'refused by the validator; the library would allocate it'
|
||||
|
||||
# 5. A row-lookup table pointing outside the record. This is what LoadStatic's unbounded
|
||||
# read cursor was written to walk off the end of.
|
||||
$victim = $donors[$v++].Id
|
||||
$rec = New-Object byte[] 512
|
||||
[Array]::Copy([BitConverter]::GetBytes([uint16] 32), 0, $rec, 4, 2) # width
|
||||
[Array]::Copy([BitConverter]::GetBytes([uint16] 32), 0, $rec, 6, 2) # height
|
||||
for ($row = 0; $row -lt 32; $row++) {
|
||||
# Each row's offset is added to (height + 4); 60000 puts every row far outside.
|
||||
[Array]::Copy([BitConverter]::GetBytes([uint16] 60000), 0, $rec, (8 + $row * 2), 2)
|
||||
}
|
||||
$stream.Write($rec, 0, $rec.Length)
|
||||
Set-ArtEntry $idx ($victim + $StaticBase) $appendAt $rec.Length 0
|
||||
$appendAt += $rec.Length
|
||||
Add-Defect 'corrupt' "static/$victim" 'row table points 60,000 words outside a 512-byte record' `
|
||||
'refused by the validator; the library reads adjacent heap'
|
||||
|
||||
# 6. A well-formed row table whose run length overruns the record.
|
||||
$victim = $donors[$v++].Id
|
||||
$rec = New-Object byte[] 256
|
||||
[Array]::Copy([BitConverter]::GetBytes([uint16] 16), 0, $rec, 4, 2)
|
||||
[Array]::Copy([BitConverter]::GetBytes([uint16] 2), 0, $rec, 6, 2)
|
||||
[Array]::Copy([BitConverter]::GetBytes([uint16] 0), 0, $rec, 8, 2) # row 0 offset
|
||||
[Array]::Copy([BitConverter]::GetBytes([uint16] 0), 0, $rec, 10, 2) # row 1 offset
|
||||
$runAt = (2 + 4) * 2 # (height + 4) words
|
||||
[Array]::Copy([BitConverter]::GetBytes([uint16] 0), 0, $rec, $runAt, 2) # xOffset
|
||||
[Array]::Copy([BitConverter]::GetBytes([uint16] 16), 0, $rec, ($runAt + 2), 2) # xRun, but
|
||||
# the record has nowhere near 16 pixels left after this point.
|
||||
$stream.Write($rec, 0, $rec.Length)
|
||||
Set-ArtEntry $idx ($victim + $StaticBase) $appendAt 20 0
|
||||
$appendAt += $rec.Length
|
||||
Add-Defect 'corrupt' "static/$victim" 'a 16-pixel run declared in a 20-byte record' `
|
||||
'refused by the validator'
|
||||
} finally { $stream.Dispose() }
|
||||
|
||||
# 7. A land tile shorter than the 2,024 bytes LoadLand reads unconditionally.
|
||||
$landVictim = 0x0100
|
||||
$landEntry = Get-ArtEntry $idx $landVictim
|
||||
if ($landEntry.Lookup -ge 0 -and $landEntry.Length -gt 0) {
|
||||
Set-ArtEntry $idx $landVictim $landEntry.Lookup 512 0
|
||||
Add-Defect 'corrupt' "land/$landVictim" 'land record declared 512 bytes; LoadLand always reads 2,024' `
|
||||
'refused by the validator; the library reads past the buffer'
|
||||
}
|
||||
|
||||
$idxDirty = $true
|
||||
Write-Host ""
|
||||
}
|
||||
|
||||
if ($idxDirty) {
|
||||
[System.IO.File]::WriteAllBytes($artIdxPath, $idx)
|
||||
Write-Host "wrote artidx.mul"
|
||||
Write-Host ""
|
||||
}
|
||||
|
||||
# -- Tier: verdata ----------------------------------------------------------------------------
|
||||
|
||||
if ($Tiers -contains 'verdata') {
|
||||
Write-Host "tier verdata"
|
||||
|
||||
# Layout: int32 count, then count * 5 int32 (file, index, lookup, length, extra), then the
|
||||
# payloads. `lookup` is an absolute offset into this file.
|
||||
$entries = New-Object System.Collections.ArrayList
|
||||
$payloads = New-Object System.Collections.ArrayList
|
||||
|
||||
$donorA = $donors[$donors.Count - 1]
|
||||
$donorB = $donors[$donors.Count - 2]
|
||||
|
||||
$artSource = [System.IO.File]::Open($artMulPath, 'Open', 'Read', 'ReadWrite')
|
||||
try {
|
||||
$bufferA = New-Object byte[] $donorA.Entry.Length
|
||||
[void] $artSource.Seek($donorA.Entry.Lookup, 'Begin')
|
||||
[void] $artSource.Read($bufferA, 0, $bufferA.Length)
|
||||
} finally { $artSource.Dispose() }
|
||||
|
||||
# The victims: ids whose art will now come from verdata.mul rather than art.mul.
|
||||
$legitVictim = $donors[$donors.Count - 3].Id
|
||||
$pastEofVictim = $donors[$donors.Count - 4].Id
|
||||
|
||||
# A legitimate patch -- the branch working as designed. Without this the tier only proves the
|
||||
# failure case, and "verdata is broken" and "verdata is never reached" look identical.
|
||||
[void] $payloads.Add($bufferA)
|
||||
[void] $entries.Add([pscustomobject]@{
|
||||
File = 4; Index = ($legitVictim + $StaticBase); Length = $bufferA.Length; Extra = $donorA.Entry.Extra
|
||||
PayloadIndex = 0; PastEof = $false
|
||||
})
|
||||
|
||||
# The failure case. FileIndex.Seek bounds-checks the mul stream and calls Verdata.Seek with no
|
||||
# check at all; seeking a FileStream past EOF is legal, the read returns nothing, and the
|
||||
# shared decode buffer still holds the previous asset.
|
||||
[void] $entries.Add([pscustomobject]@{
|
||||
File = 4; Index = ($pastEofVictim + $StaticBase); Length = 900; Extra = 0
|
||||
PayloadIndex = -1; PastEof = $true
|
||||
})
|
||||
|
||||
# An anim patch, so the tier covers the other file the verdata branch serves. anim.mul is
|
||||
# verdata file 6; for body < 200 the record index is body*110 + action*5 + direction.
|
||||
$animBody = 34 # wolf -- decodes on this client, so a patch to it is observable
|
||||
$animIndex = ($animBody * 110) + (0 * 5) + 1
|
||||
[void] $entries.Add([pscustomobject]@{
|
||||
File = 6; Index = $animIndex; Length = 700; Extra = 0
|
||||
PayloadIndex = -1; PastEof = $true
|
||||
})
|
||||
|
||||
$headerSize = 4 + ($entries.Count * 20)
|
||||
$offset = $headerSize
|
||||
foreach ($entry in $entries) {
|
||||
if ($entry.PayloadIndex -ge 0) {
|
||||
$entry | Add-Member -NotePropertyName Lookup -NotePropertyValue $offset -Force
|
||||
$offset += $payloads[$entry.PayloadIndex].Length
|
||||
}
|
||||
}
|
||||
|
||||
$totalSize = $offset
|
||||
|
||||
# Past-EOF lookups are resolved last, because "past the end" is only meaningful once the end
|
||||
# is known.
|
||||
foreach ($entry in $entries) {
|
||||
if ($entry.PastEof) {
|
||||
$entry | Add-Member -NotePropertyName Lookup -NotePropertyValue ($totalSize + 8192) -Force
|
||||
}
|
||||
}
|
||||
|
||||
$verdata = New-Object byte[] $totalSize
|
||||
Write-Int32LE $verdata 0 $entries.Count
|
||||
|
||||
$at = 4
|
||||
foreach ($entry in $entries) {
|
||||
Write-Int32LE $verdata $at $entry.File
|
||||
Write-Int32LE $verdata ($at + 4) $entry.Index
|
||||
Write-Int32LE $verdata ($at + 8) $entry.Lookup
|
||||
Write-Int32LE $verdata ($at + 12) $entry.Length
|
||||
Write-Int32LE $verdata ($at + 16) $entry.Extra
|
||||
$at += 20
|
||||
}
|
||||
|
||||
foreach ($entry in $entries) {
|
||||
if ($entry.PayloadIndex -ge 0) {
|
||||
$payload = $payloads[$entry.PayloadIndex]
|
||||
[Array]::Copy($payload, 0, $verdata, $entry.Lookup, $payload.Length)
|
||||
}
|
||||
}
|
||||
|
||||
[System.IO.File]::WriteAllBytes((Join-Path $destFull 'verdata.mul'), $verdata)
|
||||
|
||||
Add-Defect 'verdata' "static/$legitVictim" `
|
||||
"legitimately patched to static/$($donorA.Id)'s art via verdata.mul" `
|
||||
'decodes; the picture must CHANGE, which is how we know the branch ran'
|
||||
Add-Defect 'verdata' "static/$pastEofVictim" `
|
||||
'verdata entry whose lookup is 8 KB past the end of verdata.mul' `
|
||||
'REFUSED BUT DECODED -- Verdata.Seek is not bounds-checked'
|
||||
Add-Defect 'verdata' "body/$animBody" `
|
||||
"anim.mul record $animIndex patched to a verdata offset past EOF" `
|
||||
'the wolf must not silently become another creature'
|
||||
|
||||
Write-Host (" wrote verdata.mul: {0} entries, {1:N0} bytes" -f $entries.Count, $totalSize)
|
||||
Write-Host ""
|
||||
}
|
||||
|
||||
# -- Tier: bodyconv ---------------------------------------------------------------------------
|
||||
|
||||
if ($Tiers -contains 'bodyconv') {
|
||||
Write-Host "tier bodyconv"
|
||||
|
||||
$bodyconvPath = Join-Path $destFull 'Bodyconv.def'
|
||||
|
||||
if (-not (Test-Path -LiteralPath $bodyconvPath)) {
|
||||
Write-Host " no Bodyconv.def in the copy -- skipping tier"
|
||||
} else {
|
||||
# Columns are tab-separated: original, anim2, anim3, anim4, anim5. -1 means "not in that
|
||||
# file". BodyConverter.Convert returns the file type of the FIRST column that is not -1,
|
||||
# and the extractor must take that answer and stop.
|
||||
$lines = @(
|
||||
"",
|
||||
"# Asset Bridge phase 0 -- deliberate defects (tools/patch_client.ps1)",
|
||||
"1900`t-1`t-1`t-1`t60000",
|
||||
"1901`t666`t-1`t-1`t-1"
|
||||
)
|
||||
|
||||
Add-Content -LiteralPath $bodyconvPath -Value ($lines -join "`r`n") -Encoding ASCII
|
||||
|
||||
Add-Defect 'bodyconv' 'body/1900' 'mapped to anim5 index 60,000, which does not exist' `
|
||||
'reports nothing -- and must NOT fall back to another anim file'
|
||||
Add-Defect 'bodyconv' 'body/1901' 'mapped to anim2 index 666, where something unrelated lives' `
|
||||
'decodes a picture of the WRONG creature -- the spider case, on purpose'
|
||||
}
|
||||
|
||||
Write-Host ""
|
||||
}
|
||||
|
||||
# -- The source must be untouched -------------------------------------------------------------
|
||||
|
||||
$tampered = @()
|
||||
foreach ($name in $before.Keys) {
|
||||
$path = Join-Path $sourceFull $name
|
||||
$now = (Get-FileHash -LiteralPath $path -Algorithm SHA256).Hash
|
||||
if ($now -ne $before[$name]) { $tampered += $name }
|
||||
}
|
||||
|
||||
if ($tampered.Count -gt 0) {
|
||||
throw "THE SOURCE CLIENT WAS MODIFIED: $($tampered -join ', '). Restore it from the installer before doing anything else."
|
||||
}
|
||||
|
||||
Write-Host "source client verified unchanged ($($before.Count) files hashed before and after)"
|
||||
|
||||
# -- Manifest ---------------------------------------------------------------------------------
|
||||
|
||||
$manifest = [pscustomobject]@{
|
||||
built = (Get-Date).ToUniversalTime().ToString('u')
|
||||
source = $sourceFull
|
||||
dest = $destFull
|
||||
tiers = $Tiers
|
||||
defects = @($script:Defects)
|
||||
}
|
||||
|
||||
$manifestPath = Join-Path $destFull 'patched-client.manifest.json'
|
||||
$manifest | ConvertTo-Json -Depth 5 | Set-Content -LiteralPath $manifestPath -Encoding utf8
|
||||
|
||||
Write-Host ""
|
||||
Write-Host ("{0} deliberate defects; manifest at {1}" -f $script:Defects.Count, $manifestPath)
|
||||
Write-Host ""
|
||||
Write-Host "Point the probe at it by adding to the shard's Config/Bridge.cfg:"
|
||||
Write-Host ""
|
||||
Write-Host " AssetProbeClient=$destFull"
|
||||
Write-Host ""
|
||||
Write-Host "then, in game or from the rig driver: [assetprobe all patched"
|
||||
1346
tools/scaffolding/BridgeAssetProbe.cs
Normal file
1346
tools/scaffolding/BridgeAssetProbe.cs
Normal file
File diff suppressed because it is too large
Load Diff
532
tools/scaffolding/BridgeMythicCliloc.cs
Normal file
532
tools/scaffolding/BridgeMythicCliloc.cs
Normal file
@@ -0,0 +1,532 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.IO;
|
||||
using System.Text;
|
||||
|
||||
namespace Server.Custom
|
||||
{
|
||||
/// <summary>
|
||||
/// A reader for the **Mythic compressed** cliloc container, in plain .NET Framework 4.8 C#.
|
||||
///
|
||||
/// This is the Asset Bridge's §9 decoder — the ONE decoder Protocol 8 writes rather than
|
||||
/// calls (docs/link/v8.md §4, §9). ServUO's bundled <c>Ultima.StringList</c> implements only
|
||||
/// the plain layout and throws <c>Non-negative number required</c> on every modern client's
|
||||
/// file, which is also why the shard's own <c>VendorSearch.GetItemName</c> is already inert.
|
||||
///
|
||||
/// **Provenance.** Ported from UOFiddler's <c>Ultima/Helpers/MythicDecompress.cs</c>,
|
||||
/// <c>MoveToFront.cs</c> and <c>StringList.TryParse</c> (polserver/UOFiddler). UOFiddler is
|
||||
/// released under the **Beerware** licence, so carrying its algorithm into this
|
||||
/// GPL-3.0-or-later tree is clean — see v8.md §9.
|
||||
///
|
||||
/// **What the port had to change**, and why the differences are not cosmetic:
|
||||
///
|
||||
/// * UOFiddler targets net10.0 and its implementation is written in <c>Span<T></c>,
|
||||
/// <c>stackalloc</c>, <c>ArrayPool</c> and <c>BinaryPrimitives</c>. ServUO compiles the
|
||||
/// overlay against net48 with no package feed, so all of that becomes plain arrays.
|
||||
/// * Every read of the compressed payload is **bounds-checked here and is not there**.
|
||||
/// Upstream indexes <c>input[m + 1024]</c> and <c>input[firstVal + 1024]</c> with
|
||||
/// offsets derived from the file's own frequency header, inside a
|
||||
/// <c>try { } catch (Exception) { return false; }</c>. That is adequate for a desktop
|
||||
/// tool and is not adequate for us: this runs inside a live shard, and a corrupt or
|
||||
/// hostile Cliloc.enu must produce a refusal, not an exception unwinding through the
|
||||
/// bridge. Every such index is tested before use and returns <c>false</c> instead.
|
||||
///
|
||||
/// Phase 0 uses this from <see cref="BridgeAssetProbe"/> to prove the port reproduces
|
||||
/// UOFiddler's own output exactly. **Phase 2 promotes this file into
|
||||
/// <c>overlay/Scripts/Custom/Bridge/</c>** — it lives in scaffolding only for as long as it
|
||||
/// is a spike.
|
||||
/// </summary>
|
||||
public static class BridgeMythicCliloc
|
||||
{
|
||||
/// <summary>The first DWORD of a compressed file is the decompressed length, XORed with this.</summary>
|
||||
private const uint HeaderXorKey = 0x8E2C9A3D;
|
||||
|
||||
/// <summary>256 little-endian int32 symbol frequencies precede the coded payload.</summary>
|
||||
private const int FrequencyHeaderSize = 1024;
|
||||
|
||||
/// <summary>One decoded cliloc row. Mirrors <c>Ultima.StringEntry</c>'s three fields.</summary>
|
||||
public struct Entry
|
||||
{
|
||||
public int Number;
|
||||
public byte Flag;
|
||||
public string Text;
|
||||
|
||||
public Entry(int number, byte flag, string text)
|
||||
{
|
||||
Number = number;
|
||||
Flag = flag;
|
||||
Text = text;
|
||||
}
|
||||
}
|
||||
|
||||
// ── Container detection ──────────────────────────────────────────────────────────────
|
||||
|
||||
/// <summary>
|
||||
/// True when the file looks like the Mythic container. The marker is the high byte of
|
||||
/// the first DWORD being <c>0x8E</c> — which is not a magic number in the file so much
|
||||
/// as a consequence of <see cref="HeaderXorKey"/>: a plausible decompressed length is
|
||||
/// small enough that its top byte is zero, so the XOR leaves 0x8E showing.
|
||||
/// </summary>
|
||||
public static bool LooksCompressed(byte[] buffer)
|
||||
{
|
||||
return buffer != null && buffer.Length >= 4 && buffer[3] == 0x8E;
|
||||
}
|
||||
|
||||
// ── The public entry point ───────────────────────────────────────────────────────────
|
||||
|
||||
/// <summary>
|
||||
/// Reads a cliloc file, compressed or plain, and returns its entries.
|
||||
///
|
||||
/// Tries the layout the header suggests first and the other one second — the same
|
||||
/// fallback UOFiddler performs, and the reason an already-converted file passes
|
||||
/// straight through. <paramref name="warning"/> is non-null when a layout parsed
|
||||
/// *partially*: that is the case a caller must surface rather than swallow, because a
|
||||
/// quietly short table is the failure mode the website's importer refuses.
|
||||
/// </summary>
|
||||
public static bool TryLoadFile(string path, out List<Entry> entries, out string warning, out string error)
|
||||
{
|
||||
entries = null;
|
||||
warning = null;
|
||||
error = null;
|
||||
|
||||
byte[] buffer;
|
||||
|
||||
try
|
||||
{
|
||||
buffer = File.ReadAllBytes(path);
|
||||
}
|
||||
catch (Exception e)
|
||||
{
|
||||
error = "cannot read " + path + ": " + e.Message;
|
||||
return false;
|
||||
}
|
||||
|
||||
return TryLoad(buffer, out entries, out warning, out error);
|
||||
}
|
||||
|
||||
/// <summary>Reads an in-memory cliloc file. See <see cref="TryLoadFile"/>.</summary>
|
||||
public static bool TryLoad(byte[] buffer, out List<Entry> entries, out string warning, out string error)
|
||||
{
|
||||
entries = null;
|
||||
warning = null;
|
||||
error = null;
|
||||
|
||||
bool compressedFirst = LooksCompressed(buffer);
|
||||
|
||||
List<Entry> primary;
|
||||
string primaryError;
|
||||
bool primaryComplete;
|
||||
|
||||
if (TryParse(buffer, compressedFirst, out primary, out primaryComplete, out primaryError) && primaryComplete)
|
||||
{
|
||||
entries = primary;
|
||||
return true;
|
||||
}
|
||||
|
||||
List<Entry> fallback;
|
||||
string fallbackError;
|
||||
bool fallbackComplete;
|
||||
|
||||
if (TryParse(buffer, !compressedFirst, out fallback, out fallbackComplete, out fallbackError) && fallbackComplete)
|
||||
{
|
||||
entries = fallback;
|
||||
return true;
|
||||
}
|
||||
|
||||
// Neither layout parsed to the end. Take whichever salvaged more rows and say so.
|
||||
int primaryCount = primary == null ? 0 : primary.Count;
|
||||
int fallbackCount = fallback == null ? 0 : fallback.Count;
|
||||
|
||||
if (primaryCount == 0 && fallbackCount == 0)
|
||||
{
|
||||
error = "as " + Label(compressedFirst) + ": " + primaryError
|
||||
+ "; as " + Label(!compressedFirst) + ": " + fallbackError;
|
||||
return false;
|
||||
}
|
||||
|
||||
if (primaryCount >= fallbackCount)
|
||||
{
|
||||
entries = primary;
|
||||
warning = "parsed partially as " + Label(compressedFirst) + ": " + primaryError
|
||||
+ " (" + primaryCount + " entries salvaged)";
|
||||
}
|
||||
else
|
||||
{
|
||||
entries = fallback;
|
||||
warning = "parsed partially as " + Label(!compressedFirst) + ": " + fallbackError
|
||||
+ " (" + fallbackCount + " entries salvaged)";
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
private static string Label(bool compressed)
|
||||
{
|
||||
return compressed ? "compressed" : "uncompressed";
|
||||
}
|
||||
|
||||
// ── Record layout ────────────────────────────────────────────────────────────────────
|
||||
|
||||
/// <summary>
|
||||
/// Walks the plain record layout: a 4-byte and a 2-byte header, then repeating
|
||||
/// [int32 number][byte flag][uint16 length][length bytes of UTF-8].
|
||||
///
|
||||
/// <paramref name="complete"/> distinguishes "parsed to the end of the file" from
|
||||
/// "stopped early but salvaged rows", which is the distinction the caller needs and
|
||||
/// an exception would destroy.
|
||||
/// </summary>
|
||||
private static bool TryParse(byte[] buffer, bool decompress, out List<Entry> entries, out bool complete, out string error)
|
||||
{
|
||||
entries = new List<Entry>();
|
||||
complete = false;
|
||||
error = null;
|
||||
|
||||
byte[] data;
|
||||
|
||||
if (decompress)
|
||||
{
|
||||
if (!TryDecompress(buffer, out data, out error))
|
||||
return false;
|
||||
}
|
||||
else
|
||||
{
|
||||
data = buffer;
|
||||
}
|
||||
|
||||
if (data.Length < 6)
|
||||
{
|
||||
error = "file is " + data.Length + " bytes, smaller than the 6-byte header";
|
||||
return false;
|
||||
}
|
||||
|
||||
int cursor = 6; // int32 version marker + int16 language marker
|
||||
int lastNumber = -1;
|
||||
|
||||
while (cursor < data.Length)
|
||||
{
|
||||
int entryStart = cursor;
|
||||
int remaining = data.Length - cursor;
|
||||
|
||||
if (remaining < 7)
|
||||
{
|
||||
error = "unexpected " + remaining + " trailing byte(s) at 0x" + entryStart.ToString("X")
|
||||
+ " after entry #" + lastNumber + "; an entry header needs 7";
|
||||
return true;
|
||||
}
|
||||
|
||||
int number = ReadInt32(data, cursor);
|
||||
byte flag = data[cursor + 4];
|
||||
// Deliberately UNSIGNED. Read as Int16, a string of 32768 bytes or more comes back
|
||||
// negative and corrupts every record after it.
|
||||
int length = data[cursor + 5] | (data[cursor + 6] << 8);
|
||||
cursor += 7;
|
||||
|
||||
if (length > data.Length - cursor)
|
||||
{
|
||||
error = "entry #" + number + " at 0x" + entryStart.ToString("X") + " declares length "
|
||||
+ length + " but only " + (data.Length - cursor) + " byte(s) remain (parsed "
|
||||
+ entries.Count + " so far)";
|
||||
return true;
|
||||
}
|
||||
|
||||
string text;
|
||||
|
||||
try
|
||||
{
|
||||
text = Encoding.UTF8.GetString(data, cursor, length);
|
||||
}
|
||||
catch (Exception e)
|
||||
{
|
||||
error = "entry #" + number + " at 0x" + entryStart.ToString("X") + " has " + length
|
||||
+ " body bytes that are not valid UTF-8: " + e.Message;
|
||||
return true;
|
||||
}
|
||||
|
||||
cursor += length;
|
||||
|
||||
entries.Add(new Entry(number, flag, text));
|
||||
lastNumber = number;
|
||||
}
|
||||
|
||||
complete = true;
|
||||
return true;
|
||||
}
|
||||
|
||||
// ── Mythic stage 1: the XOR header and the move-to-front code ────────────────────────
|
||||
|
||||
/// <summary>
|
||||
/// Reads the obfuscated decompressed length from the first DWORD. Public so a caller
|
||||
/// can size a buffer before committing to the decode.
|
||||
/// </summary>
|
||||
public static uint PeekDecompressedLength(byte[] source)
|
||||
{
|
||||
if (source == null || source.Length < 4)
|
||||
return 0;
|
||||
|
||||
return ReadUInt32(source, 0) ^ HeaderXorKey;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Decompresses the Mythic container: strip the 4-byte length header, undo the
|
||||
/// move-to-front coding, then run stage 2.
|
||||
/// </summary>
|
||||
public static bool TryDecompress(byte[] source, out byte[] output, out string error)
|
||||
{
|
||||
output = null;
|
||||
error = null;
|
||||
|
||||
if (source == null || source.Length < 4)
|
||||
{
|
||||
error = "payload shorter than the 4-byte length header";
|
||||
return false;
|
||||
}
|
||||
|
||||
uint dataLength = ReadUInt32(source, 0) ^ HeaderXorKey;
|
||||
|
||||
// A wrong guess about the container makes this astronomically large, which is the
|
||||
// cheapest possible rejection and must happen before any allocation.
|
||||
if (dataLength == 0 || dataLength > int.MaxValue)
|
||||
{
|
||||
error = "implausible decompressed length " + dataLength + " — not the compressed layout";
|
||||
return false;
|
||||
}
|
||||
|
||||
var mtf = new byte[source.Length - 4];
|
||||
MoveToFrontDecode(source, 4, mtf);
|
||||
|
||||
var destination = new byte[(int)dataLength];
|
||||
int written;
|
||||
|
||||
if (!TryInternalDecompress(mtf, destination, out written, out error))
|
||||
return false;
|
||||
|
||||
if (written != (int)dataLength)
|
||||
{
|
||||
error = "decompressed " + written + " bytes, header declared " + dataLength;
|
||||
return false;
|
||||
}
|
||||
|
||||
output = destination;
|
||||
return true;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Move-to-front decode. Each input byte is an index into a 256-symbol table; the
|
||||
/// symbol found there is emitted and moved to the front.
|
||||
/// </summary>
|
||||
private static void MoveToFrontDecode(byte[] input, int offset, byte[] output)
|
||||
{
|
||||
var symbols = new byte[256];
|
||||
|
||||
for (int i = 0; i < 256; i++)
|
||||
symbols[i] = (byte)i;
|
||||
|
||||
for (int i = 0; i < output.Length; i++)
|
||||
{
|
||||
int index = input[offset + i];
|
||||
byte symbol = symbols[index];
|
||||
output[i] = symbol;
|
||||
|
||||
for (int j = index; j > 0; j--)
|
||||
symbols[j] = symbols[j - 1];
|
||||
|
||||
symbols[0] = symbol;
|
||||
}
|
||||
}
|
||||
|
||||
// ── Mythic stage 2 ───────────────────────────────────────────────────────────────────
|
||||
|
||||
/// <summary>
|
||||
/// Turns the MTF-decoded payload back into the original bytes.
|
||||
///
|
||||
/// The payload is a 1024-byte frequency header (256 little-endian int32 symbol counts)
|
||||
/// followed by the coded stream. The counts partition the stream into one run per
|
||||
/// symbol; <c>cursor[]</c> holds each run's read position and <c>limit[]</c> its end,
|
||||
/// and the walk emits a symbol, advances that symbol's run, and re-orders the symbol
|
||||
/// table by the index it reads.
|
||||
///
|
||||
/// Every index derived from file content is checked. Upstream's equivalent is wrapped
|
||||
/// in a blanket catch; here a malformed file is a <c>false</c> with a reason.
|
||||
/// </summary>
|
||||
private static bool TryInternalDecompress(byte[] input, byte[] destination, out int written, out string error)
|
||||
{
|
||||
written = 0;
|
||||
error = null;
|
||||
|
||||
if (input.Length < FrequencyHeaderSize)
|
||||
{
|
||||
error = "payload (" + input.Length + " bytes) is smaller than the 1024-byte frequency header";
|
||||
return false;
|
||||
}
|
||||
|
||||
var counts = new int[256]; // symbol → number of occurrences
|
||||
var cursor = new int[256]; // symbol → next unread position in its run
|
||||
var limit = new int[256]; // symbol → one past the end of its run
|
||||
|
||||
int sum = 0;
|
||||
|
||||
for (int i = 0; i < 256; i++)
|
||||
{
|
||||
counts[i] = ReadInt32(input, i * 4);
|
||||
|
||||
if (counts[i] < 0)
|
||||
{
|
||||
error = "frequency header declares a negative count for symbol " + i;
|
||||
return false;
|
||||
}
|
||||
|
||||
sum += counts[i];
|
||||
|
||||
if (sum < 0)
|
||||
{
|
||||
error = "frequency header sums past int range at symbol " + i;
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
if (sum == 0)
|
||||
{
|
||||
written = 0;
|
||||
return true;
|
||||
}
|
||||
|
||||
if (destination.Length < sum)
|
||||
{
|
||||
error = "destination holds " + destination.Length + " bytes, payload needs " + sum;
|
||||
return false;
|
||||
}
|
||||
|
||||
int nonZeroCount = 0;
|
||||
|
||||
for (int i = 0; i < 256; i++)
|
||||
{
|
||||
if (counts[i] != 0)
|
||||
nonZeroCount++;
|
||||
}
|
||||
|
||||
// The coded stream must be long enough to hold one index per emitted byte.
|
||||
if (input.Length - FrequencyHeaderSize < sum)
|
||||
{
|
||||
error = "coded stream holds " + (input.Length - FrequencyHeaderSize) + " bytes, frequency header claims " + sum;
|
||||
return false;
|
||||
}
|
||||
|
||||
var order = new byte[256];
|
||||
FrequencyOrder(counts, order);
|
||||
|
||||
var symbolTable = new byte[256];
|
||||
|
||||
for (int i = 0; i < 256; i++)
|
||||
symbolTable[i] = (byte)i;
|
||||
|
||||
for (int i = 0, m = 0; i < nonZeroCount; ++i)
|
||||
{
|
||||
byte symbol = order[i];
|
||||
|
||||
// m indexes the coded stream and comes from the file's own counts.
|
||||
if (m < 0 || m >= input.Length - FrequencyHeaderSize)
|
||||
{
|
||||
error = "run table for symbol " + symbol + " starts at " + m + ", past the coded stream";
|
||||
return false;
|
||||
}
|
||||
|
||||
symbolTable[input[m + FrequencyHeaderSize]] = symbol;
|
||||
cursor[symbol] = m + 1;
|
||||
m += counts[symbol];
|
||||
limit[symbol] = m;
|
||||
}
|
||||
|
||||
byte val = symbolTable[0];
|
||||
int count = 0;
|
||||
int liveSymbols = nonZeroCount;
|
||||
|
||||
do
|
||||
{
|
||||
destination[count] = val;
|
||||
|
||||
if (cursor[val] < limit[val])
|
||||
{
|
||||
int at = cursor[val] + FrequencyHeaderSize;
|
||||
|
||||
if (at < FrequencyHeaderSize || at >= input.Length)
|
||||
{
|
||||
error = "run for symbol " + val + " reads at " + at + ", past the " + input.Length + "-byte payload";
|
||||
return false;
|
||||
}
|
||||
|
||||
byte index = input[at];
|
||||
cursor[val]++;
|
||||
|
||||
if (index != 0)
|
||||
{
|
||||
ShiftLeft(symbolTable, index);
|
||||
symbolTable[index] = val;
|
||||
val = symbolTable[0];
|
||||
}
|
||||
}
|
||||
else if (liveSymbols-- > 0)
|
||||
{
|
||||
ShiftLeft(symbolTable, liveSymbols);
|
||||
val = symbolTable[0];
|
||||
}
|
||||
|
||||
count++;
|
||||
}
|
||||
while (count < sum);
|
||||
|
||||
written = sum;
|
||||
return true;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Orders symbols by descending frequency: repeatedly take the largest remaining count
|
||||
/// and record its symbol. Ties go to the lower symbol, because the scan keeps the first
|
||||
/// strictly-greater value — matching upstream, and the tie-break is load-bearing.
|
||||
/// </summary>
|
||||
private static void FrequencyOrder(int[] counts, byte[] output)
|
||||
{
|
||||
var tmp = new int[256];
|
||||
Array.Copy(counts, tmp, 256);
|
||||
|
||||
for (int i = 0; i < 256; i++)
|
||||
{
|
||||
int best = 0;
|
||||
byte index = 0;
|
||||
|
||||
for (int j = 0; j < 256; j++)
|
||||
{
|
||||
if (tmp[j] > best)
|
||||
{
|
||||
index = (byte)j;
|
||||
best = tmp[j];
|
||||
}
|
||||
}
|
||||
|
||||
if (best == 0)
|
||||
break;
|
||||
|
||||
output[i] = index;
|
||||
tmp[index] = 0;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Shifts <c>[1..element]</c> down one slot, dropping element 0.</summary>
|
||||
private static void ShiftLeft(byte[] input, int element)
|
||||
{
|
||||
for (int i = 0; i < element; ++i)
|
||||
input[i] = input[i + 1];
|
||||
}
|
||||
|
||||
// ── Little-endian readers (BinaryPrimitives is not available on net48) ───────────────
|
||||
|
||||
private static int ReadInt32(byte[] b, int at)
|
||||
{
|
||||
return b[at] | (b[at + 1] << 8) | (b[at + 2] << 16) | (b[at + 3] << 24);
|
||||
}
|
||||
|
||||
private static uint ReadUInt32(byte[] b, int at)
|
||||
{
|
||||
return (uint)(b[at] | (b[at + 1] << 8) | (b[at + 2] << 16) | (b[at + 3] << 24));
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -155,6 +155,14 @@ namespace Server.Custom
|
||||
case "partprobe":
|
||||
BridgeParticipationProbe.Run(null, Arg(parts, 1), Int(Arg(parts, 2)), Int(Arg(parts, 3)));
|
||||
break;
|
||||
// Asset Bridge phase 0. Here for the same reason as partprobe, and for one more:
|
||||
// the point of that spike is comparing the STOCK client's answers with a patched
|
||||
// client's, and `AssetProbeOnStart` can only ever run whichever one the config
|
||||
// names. Driving it from here runs both against a single boot, so a difference
|
||||
// between them cannot be a difference between two shard processes.
|
||||
case "assetprobe":
|
||||
BridgeAssetProbe.Begin(null, Arg(parts, 1) ?? "all", Arg(parts, 2));
|
||||
break;
|
||||
// Phase 12a. `world.despawn` answering `gone` rather than `removed` is the
|
||||
// path a player takes every time they kill an event creature, and it is the one
|
||||
// outcome the rig cannot reach by asking the bridge: every bridge verb that
|
||||
|
||||
@@ -14,10 +14,12 @@ These two scripts produced the measured budget in [PLAN.md](https://gitea.whitlo
|
||||
| `BridgeCrierProbe.cs` | `Scripts/Custom/BridgeCrierProbe.cs` | Logs the global town-crier entry list every 3s so `towncrier.add` / `remove` can be seen landing in game state. Flag: `CrierProbeOnStart`. |
|
||||
| `BridgeVendorSaleProbe.cs` | `Scripts/Custom/BridgeVendorSaleProbe.cs` | Fires `PlayerVendorSale` (Phase 7) with real seeded-vendor data so `vendor.sale` can be verified without a live buy. Requires the Phase 7 patches applied. Flag: `VendorSaleProbeOnStart`. |
|
||||
| `BridgeDemoDress.cs` | `Scripts/Custom/BridgeDemoDress.cs` | Renames a seeded world so it is presentable in a screenshot: shop signs, vendor and character names, house signs. Also stages a few condemned houses back into IDOC, and sets a known password on `seed_000` so a character can be logged in. Flags: `DemoDressOnStart`, `DemoDressPassword`. In game: `[demodress`. |
|
||||
| `BridgeRigDriver.cs` | `Scripts/Custom/BridgeRigDriver.cs` | Drives the shard from OUTSIDE the game, one verb per line in `Config/rigcmd.txt`, which the driver polls and truncates. Written for the engagement Phase 11b acceptance walk, where each step's assertion is what happened BETWEEN two steps, so the steps have to be separated by the observer rather than by a hard-coded delay -- and ServUO's console takes a fixed verb set (`Scripts/Misc/ConsoleCommands.cs`), so `[p5probe` cannot be typed at a headless shard at all. Verbs: `decaylist`, `decay`, `vendorlist`, `vendorfunds`, `citylist`, `governor`, `election`, `activate`, `password`, `configset`, `configread`, `partprobe`, `save`, `shutdown`. Flag: `RigDriverEnabled`. `configset` exists because **`Config.Get` is written by exactly ONE caller in the whole of ServUO 57.4** (`Server/ScriptCompiler.cs`): no in-game command, gump or console verb writes a config key, so on a stock shard a GM cannot drift a configuration lease even deliberately, and a lease's compare-and-set restore would have no way to be proved. `configread` reads a key back through `Config.Get` long after every type initialiser has run, which is how a key that TOOK is told from one that only appeared to. **Sets passwords, writes live config and mutates the world.** |
|
||||
| `BridgeRigDriver.cs` | `Scripts/Custom/BridgeRigDriver.cs` | Drives the shard from OUTSIDE the game, one verb per line in `Config/rigcmd.txt`, which the driver polls and truncates. Written for the engagement Phase 11b acceptance walk, where each step's assertion is what happened BETWEEN two steps, so the steps have to be separated by the observer rather than by a hard-coded delay -- and ServUO's console takes a fixed verb set (`Scripts/Misc/ConsoleCommands.cs`), so `[p5probe` cannot be typed at a headless shard at all. Verbs: `decaylist`, `decay`, `vendorlist`, `vendorfunds`, `citylist`, `governor`, `election`, `activate`, `password`, `configset`, `configread`, `partprobe`, `assetprobe`, `save`, `shutdown`. Flag: `RigDriverEnabled`. `configset` exists because **`Config.Get` is written by exactly ONE caller in the whole of ServUO 57.4** (`Server/ScriptCompiler.cs`): no in-game command, gump or console verb writes a config key, so on a stock shard a GM cannot drift a configuration lease even deliberately, and a lease's compare-and-set restore would have no way to be proved. `configread` reads a key back through `Config.Get` long after every type initialiser has run, which is how a key that TOOK is told from one that only appeared to. **Sets passwords, writes live config and mutates the world.** |
|
||||
| `BridgeProtocol5Probe.cs` | `Scripts/Custom/BridgeProtocol5Probe.cs` | Drives all three Protocol 5 enrichments so their frames can be observed: walks one house Fairly -> Greatly -> IDOC (the PAIR is the assertion -- `estimatedCollapse` must appear only on the IDOC frame), reports each player vendor's fee state straight off the `PlayerVendor` so the emitted `fees` block can be checked against the shard's own numbers, and fires `EventSink.AccountLogin`. Flags: `Protocol5ProbeOnStart`, `Protocol5ProbeAccount`, `Protocol5ProbePassword`. In game: `[p5probe`. **Sets a password on the named account.** |
|
||||
| `BridgeProtocol6Probe.cs` | `Scripts/Custom/BridgeProtocol6Probe.cs` | Spawns a real champion boss through the shard's own `SpawnChampion()`, waits two champ sweeps so the boss is attributed to its altar, registers unequal damage from two seeded players and kills it -- so `champ.boss.killed` can be observed with a real damage table. **The wait is the assertion**: without it the kill still emits, but with no `serial`/`type`/`level`, which is the documented fallback rather than the case being tested. The altar is placed inside a NAMED region on purpose (see below). Flag: `Protocol6ProbeOnStart`. In game: `[p6probe`. **Spawns and kills a champion boss; rig only.** Protocol 6's other half, the idempotency key, needs no probe -- it is driven from outside with two identical POSTs to the sidecar. |
|
||||
| `BridgeParticipationProbe.cs` | `Scripts/Custom/BridgeParticipationProbe.cs` | Produces real kill credit inside a participation area with no game client: moves two player mobiles to the venue, spawns a creature there, damages it unequally from both and kills it. **Presence is the half it cannot drive** -- the sweep credits players with a live `NetState`, which is the correct test and not one a probe should loosen, so presence accrual needs a real login. In game: `[partprobe <map> <x> <y>`; from a headless rig, through `BridgeRigDriver`'s `partprobe` verb (the two ship together for that reason). **Moves players and spawns and kills a creature; rig only.** |
|
||||
| `BridgeAssetProbe.cs` | `Scripts/Custom/BridgeAssetProbe.cs` | **Asset Bridge phase 0** (docs/link/v8.md §16). Drives ServUO's vendored `Ultima` decoders from inside a running shard against a deliberately patched client, and compares every answer with what a pre-flight validator says about the index entry *before* the call. The interesting column is not the error count, it is **WRONG PICTURES** -- records the validator rejects and the library renders anyway. Sweeps statics, land, all 2,048 bodies, the player-character bodies from `Race.AllRaces`, and the ported Mythic cliloc reader against UOFiddler's own output. In game / from `BridgeRigDriver`: `[assetprobe [section] [stock|patched]`. Flag: `AssetProbeOnStart`. **Its `gump` section deliberately kills the shard** and is never part of `all`. |
|
||||
| `BridgeMythicCliloc.cs` | `Scripts/Custom/BridgeMythicCliloc.cs` | The §9 reader for the **Mythic compressed** cliloc container -- the one decoder Protocol 8 writes rather than calls. Ported from UOFiddler (Beerware) into net48 C# with every file-derived index bounds-checked, which upstream's blanket `catch` does not do. Reproduces UOFiddler's 123,490-entry table exactly. **Phase 2 promotes this file into `overlay/`**; it is scaffolding only for as long as it is a spike. |
|
||||
|
||||
## Deploy overwrites Bridge.cfg
|
||||
|
||||
@@ -189,3 +191,127 @@ value that lies, printed next to a frame that disagrees with it.
|
||||
|
||||
Emitting from a named region is therefore the test. At a dungeon altar the field is legitimately
|
||||
absent and the probe proves nothing about it.
|
||||
|
||||
## What phase 0 found
|
||||
|
||||
`BridgeAssetProbe` exists because [v8.md](https://gitea.whitlocktech.com/RunicGateway/docs/src/branch/main/link/v8.md) §4 chose to **call** ServUO's vendored `Ultima` rather than reimplement it, and the evidence for that choice was a PowerShell probe against a stock client — neither the process nor the client the extractor will actually run in. These are its results, from inside a running ServUO 57.4 against this machine's client, and against a copy broken in 21 catalogued ways by `tools/patch_client.ps1`.
|
||||
|
||||
### The UOP wins outright, and it took a whole run to notice
|
||||
|
||||
`FileIndex`'s UOP constructor ends with a bare `MulPath = uopPath`. **When `artLegacyMUL.uop` is present it wins, and `art.mul` / `artidx.mul` are never opened at all.** Every current client ships the UOP, so:
|
||||
|
||||
- A validator that bounds an index offset against `art.mul` while the index holds UOP offsets is not approximate, it is nonsense. The first run of this probe refused **34,299 perfectly good statics** for "declaring 10533x2085" — and every one of those refusals looked like a real finding. `BridgeAssetValidator.ArtDataPath()` now mirrors `FileIndex`'s own resolution order, and phase 1 must too.
|
||||
- A custom-art shard that adds graphics to `art.mul` while the UOP is still in place **gets nothing**, silently. That is an operator trap rather than a bug in this protocol, but the extractor is where it will be noticed.
|
||||
- The `corrupt` and `customart` tiers of `patch_client.ps1` therefore need its `nouop` tier to mean anything at all. Without it they report that they applied, and change nothing.
|
||||
|
||||
### 22,102 wrong pictures on a stock, unmodified client
|
||||
|
||||
The counts that matter, `assetprobe all stock`:
|
||||
|
||||
```
|
||||
statics 0..65535 ok 39,189 WRONG PICTURES (empty record) 9,962 threw 16,385
|
||||
land 0..16383 ok 4,244 WRONG PICTURES (empty record) 12,140
|
||||
```
|
||||
|
||||
Those 22,102 ids have an index entry of `lookup 0, length 0` — **no record at all**. `FileIndex.Seek` treats that as a hit (it rejects `lookup < 0` and `length < 0`, and zero is neither), hands back the stream, and `LoadStatic` decodes `length` = 0 bytes into `m_StreamBuffer` — which is **reused, only ever grown, and filled by a `stream.Read` whose return value is discarded**. So the id renders whatever the previously-decoded asset left in the buffer.
|
||||
|
||||
**It is specific to the UOP path.** Run the same sweep against the mul path and those ids come back empty and honest, because `artidx.mul` stores `-1` for an absent record while unmapped UOP slots are simply zeroed structs. That is also why the earlier PowerShell probe counted 32,766 of these as "ok": they decode, they raise nothing, and no success count can tell them from art.
|
||||
|
||||
A bulk import that trusted the library would have written 22,102 duplicate images into the site under ids that have no art. This one measurement is the argument for validate-before-calling.
|
||||
|
||||
### Every deliberate defect was caught by the validator and rendered by the library
|
||||
|
||||
`assetprobe all patched`, against the 21-defect client:
|
||||
|
||||
```
|
||||
statics ok 39,190 absent 9,954 refused 1 WRONG PICTURES (bad record) 6 threw 16,385
|
||||
land ok 4,243 absent 12,140 WRONG PICTURES (bad record) 1
|
||||
```
|
||||
|
||||
| id | the defect | what the library did |
|
||||
|---|---|---|
|
||||
| `static/4104` | lookup 4 KB past the end of `art.mul` | returns nothing — `Seek` does check the record's **start** |
|
||||
| `static/4105` | starts 64 bytes before EOF, declares 8,192 | **renders the previous asset** — `Seek` never checks the record's **end** |
|
||||
| `static/4108` | declared length 4, smaller than the header | renders something |
|
||||
| `static/4109` | header declares 8000x8000 | **allocates it** — a ~128 MB bitmap from two bytes in a file, and the same field can ask for 65535×65535 |
|
||||
| `static/4111` | row table points 60,000 words outside a 512-byte record | renders — `LoadStatic`'s two guards bound the *write* into the bitmap, and nothing bounds the *read* |
|
||||
| `static/4112` | a 16-pixel run declared in a 20-byte record | renders |
|
||||
| `static/4131` | verdata entry whose lookup is past verdata.mul's own end | renders — **`Verdata.Seek` has no bounds check whatsoever** |
|
||||
| `land/256` | 512-byte land record | renders — `LoadLand` reads a fixed 2,024 bytes whatever the length says |
|
||||
|
||||
Seven of the eight produce a confident, wrong picture and raise nothing anywhere.
|
||||
|
||||
The validator refused all eight, and refused **nothing** on the stock client across 49,151 statics and 16,384 land tiles. That second number is the one that matters: a checker that refuses real art is worse than no checker, so "zero false refusals on a clean client" is what makes validate-before-calling more than a hopeful phrase.
|
||||
|
||||
The eight `customart` ids appended past the stock ceiling all decode cleanly, which is that tier's whole point — the ceiling is a property of a file, not a constant anyone should write down.
|
||||
|
||||
### Two more ways to get a wrong answer out of an id that has no art
|
||||
|
||||
- **`Art.GetStatic(id, false)` throws `IndexOutOfRangeException` for `id >= 49,152`** rather than returning null — 16,385 of them in a full sweep.
|
||||
- **`Art.GetStatic(id)` with the default `checkmaxid: true` is worse**: `GetLegalItemID` maps an out-of-range id to **0**, so the call returns **item 0's picture**. An exception is recoverable; a picture of the wrong item is not even detectable.
|
||||
|
||||
So the extractor takes its id ceiling from the index it opened, and passes `checkmaxid: false` so an overrun is loud rather than plausible.
|
||||
|
||||
### The gump crash reproduces in-process, and nothing catches it
|
||||
|
||||
`assetprobe gump` called `Ultima.Gumps.GetGump(2)` once. **The ServUO process disappeared** — no exception line in the report, no `catch` reached, no shutdown, nothing in the console. The report ends mid-section, and `checkpoint.txt` reading `gump 2` is the entire record of what happened. That is exactly why the checkpoint is written *before* the call and flushed.
|
||||
|
||||
`AccessViolationException` is a corrupted-state exception and .NET Framework 4.8 does not deliver it to ordinary handlers, so **there is no in-process defence** — on a live shard this is a crash with players on it. "Nothing calls `Ultima.Gumps`" is a safety rule, and phase 0's job was to make sure that sentence had been earned rather than assumed. It has.
|
||||
|
||||
### The cliloc port is byte-identical to UOFiddler
|
||||
|
||||
```
|
||||
123,490 entries in 218 ms (55,986 blank, 67,504 would be stored)
|
||||
vs UOFiddler: 123,490 identical, 0 differ, 0 only ours, 0 only theirs
|
||||
```
|
||||
|
||||
§9 is proven: the shard can produce the whole table with no UOFiddler installed, no `dotnet build`, and no 5 MB file copied to a server.
|
||||
|
||||
The reference is what makes this a test rather than a demonstration. A subtly wrong inverse-BWT coder still produces a plausible table — mostly-right strings with a few mangled ones is the *expected* shape of a bug in this algorithm, and a row count alone would sail past it.
|
||||
|
||||
Note the blank count is **55,986**, not the 55,994 recorded from the manual pipeline. The difference is eight whitespace-only entries, blank to a `trim()` and not to `IsNullOrEmpty` — a definition rather than a defect, but exactly the sort of eight-row drift that gets investigated as one.
|
||||
|
||||
### What phase 0 did not cover, and phase 1 must
|
||||
|
||||
**The animation path has no validator.** The patched client's verdata entry for body 34 points past verdata.mul's end and the wolf still "decoded" — counted among the 1,144 successes, silently rendering something else, with nothing in the report to say so. `GetAnimation` also allocates `new int[frameCount]` straight from a file-supplied int. Everything above about statics applies here and none of it is implemented yet.
|
||||
|
||||
The deliberate `Bodyconv.def` mis-mappings (bodies 1900 and 1901) produced **nothing** rather than a wrong creature on this client, so they did not reproduce the spider. The gargoyle rows remain the real evidence for the never-sweep-file-types rule: 666, 667, 694 and 695 report nothing, and nothing is the correct answer.
|
||||
|
||||
### Reference: the rest of the run
|
||||
|
||||
```
|
||||
bodies 0..2047, direction 1 decoded 1,144 empty 904 faulted 0
|
||||
by file type: 1=1222, 2=140, 3=244, 4=150, 5=292
|
||||
|
||||
player bodies (Race.AllRaces, direction 0) 6 decoded, 6 absent, of 12
|
||||
Human 400 / 401 decode; ghosts 402 / 403 absent
|
||||
Elf 605 / 606 / 607 / 608 all decode
|
||||
Gargoyle 666 / 667 / 694 / 695 all absent
|
||||
```
|
||||
|
||||
Two details worth keeping. The body counts reproduce the PowerShell probe **exactly**, from a different process against the same files, which is what makes the two runs comparable at all. And the gargoyle *ghost* bodies resolve to file type **1**, not 5 like the living gargoyle bodies — so "the gargoyle is an anim5 problem" is not quite the shape of it.
|
||||
|
||||
## Building the patched client
|
||||
|
||||
```powershell
|
||||
.\tools\patch_client.ps1 -Dest D:\uo-patched-client
|
||||
```
|
||||
|
||||
Copies a client (~3.5 GB) and breaks the copy in five catalogued tiers — `nouop`, `verdata`, `customart`, `corrupt`, `bodyconv`. **It never writes to the source**: every file it touches is hashed in the source before and after, and a changed hash aborts the run. Each defect is recorded in `patched-client.manifest.json` beside the copy, which is what makes a nonzero WRONG PICTURES count readable as "the tier worked" instead of "something broke".
|
||||
|
||||
Then point the shard at it and drive the probe:
|
||||
|
||||
```ini
|
||||
RigDriverEnabled=true
|
||||
AssetProbeClient=D:\uo-patched-client
|
||||
AssetProbeClilocRef=<a clilocs.tsv from website/server/tools/cliloc-export --tsv>
|
||||
```
|
||||
|
||||
```
|
||||
assetprobe all stock # the baseline: the validator must refuse nothing here
|
||||
assetprobe all patched # the experiment
|
||||
```
|
||||
|
||||
Run both against **one boot**, through `rigcmd.txt`, so a difference between them cannot be a difference between two shard processes. Without `AssetProbeClilocRef` the cliloc section reports a row count, which proves nothing about the strings.
|
||||
|
||||
**The copy is EA's client art.** It stays on the machine that made it, exactly like every other extraction in this project, and is never committed.
|
||||
|
||||
Reference in New Issue
Block a user