Files
runicgateway.com/src/content/docs/docs/administration/troubleshooting.mdx
wtclaude 4e1ce2316a
All checks were successful
PR checks / checks (pull_request) Successful in 1m41s
docs(site): the Asset Bridge — protocol 8, a screen for client files, and pictures in the screenshots (Phase 9c)
The last leg of the Asset Bridge (docs/link/v8.md §16, phase 9c). The cutover put
protocol 8 on `main` in every declaration site, which turned this repo red by
construction: `checkFacts.mjs` reads the protocol from `link`'s `main`.

Two checks were failing, and only one of them was expected.

**checkFacts (10)** — protocol 7 → 8 in all three reads (sidecar, overlay, bundle),
the bundle triple to 2026.09.15 / sidecar v2.3.0 / overlay v1.3.0, and the releases
to link v2.3.0, installer v0.2.0, Module-uo v1.3.0. `moduleApi` and the capability
list did not move: core's whole share of eight phases was a deletion.

**checkReference (9)** — nobody had planned for this one. `Bridge.cfg` grew nine keys
across phases 5, 6 and 7 and the reference page had gone on not mentioning them,
which is the failure mode that check exists for. They are listed as two new groups,
`Client assets` and `Spawn files`, because they are two separate consents.

## The prose the bridge changed

- **New page, Administration → Client files.** `Admin → Client files` is a screen an
  operator has to press a button on and the site had no page for it. It carries the
  rule nothing else states: **nothing here happens on a restart**, so a patched client
  keeps serving the old artwork until somebody says so.
- **Protocol versions** said the most recent bump touched five repositories and named
  `website`. Protocol 8 touched four and did not reach core — its absence is the
  interesting half, and the page now says why. The store-migration paragraph gains 8,
  which changed no line of the sidecar's store despite moving megabytes.
- **The bridge** said "two ways in". There are three: bulk reads go over the
  request/reply path in pages, one in flight at a time, and the reason they must not
  ride the event stream is structural rather than about speed.
- **Requirements** gains `libgdiplus` for Linux shard hosts, and the note that no game
  client has to be installed for any of this — a ServUO shard cannot boot without one.
- **Troubleshooting** gains the failure mode that did not exist before: a bestiary with
  no pictures, or items reading as numbers.
- **Maintenance** gains the button to press after patching a client, and **Verify the
  whole stack** the `doctor` check that landed with it.
- The spawn atlas and the marketplace both carry artwork now, so both capability
  descriptions say so, and `canonicalDocs` points at `link/v8.md`.

## Screenshots, retaken against the real thing

The atlas and marketplace shots predate the asset plane, so both were retaken on the
demo deployment against a real shard and a real sidecar, plus the new admin screen:
1,095 portraits imported in 2.1 s, 67,496 names in 1.4 s, 313 item pictures in 0.8 s.
Nothing here is a mock-up. The atlas shot moved 36px down its page because portraits
made the rows taller.

## One defect found on the way

`seedDemo.mjs` pinned the demo deployment at **protocol 4** with a comment explaining
a `module-uo` debt that has since been paid. Its default now comes from
`platform.json`, so the rig cannot drift two protocols behind the platform again.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4
2026-09-15 05:45:38 -05:00

193 lines
11 KiB
Plaintext

---
title: Troubleshooting
description: The failures a deployment actually hits, what each one looks like, and the fix.
---
import { Aside } from '@astrojs/starlight/components';
Grouped by where the problem is, because the first useful question is always *which half is
broken*.
## The site will not start
Read the log first — `docker compose logs app` — because the server says exactly why.
| What the log says | What it means |
|---|---|
| `SECRET_ENC_KEY must be set in production` | The key that encrypts stored secrets is missing. Set it in `.env` and start again. The container crash-loops until you do. |
| A `BOT_INTERNAL_KEY` complaint | Blank, still a placeholder, or shorter than 16 characters. Required in production even when the bot is not running. |
| A database connection error, repeatedly | The app came up before the database was ready, or `DB_*` is wrong. The Compose file's health check handles the first case; check the values for the second. |
| Nothing at all, container restarting | The image did not pull. `docker compose pull` on its own shows the error. |
<Aside type="caution" title="Both of those key errors happen on the FIRST boot, not later">
They are checked at require time, before the server listens. A deployment that has ever
served a request has both of them set.
</Aside>
## Nobody can sign in
- **Your address is rate-limited or bot-banned.** Both are working as designed. Check
**Admin → Web Bot Activity** from another network, or restart the app container — the
scoring state is in memory and resets with it.
- **The password is right and the form still fails.** Check the log for the actual status:
a `429` is the rate limiter, a `403` is usually the honeypot, and a `401` really is the
password.
- **SSO returns to the login page.** SSO is link-only: an identity that is not already
linked to an account cannot sign in, and that is the expected outcome rather than a
misconfiguration. Link it from the account screen first.
- **Two-factor is lost.** Recovery codes, or another admin's **Reset two-factor** on
**Users → View**. See [Authentication](/docs/administration/authentication/).
## A module will not start
**Admin → Modules** names the stage and the reason. The usual three:
| Reason | Fix |
|---|---|
| `module directory not present on the volume` | The row exists and the files do not — someone deleted the directory by hand. Reinstall, or remove the row with an uninstall. |
| A schema failure | The module's schema fragment could not be applied. The log carries the SQL error. |
| A version refusal | The module wants a newer core API than this image. Upgrade the site. |
Whatever the reason, **the site is up and the module's routes are absent** — that is by
design, and it is why a broken module is an inconvenience rather than an outage. Fix the
cause and restart: failed modules are retried on every boot.
**The install button rejects a URL.** The host must be in the allowlist on the same screen,
and the URL must be HTTPS. An empty allowlist forbids every install.
**The install succeeds and nothing appears.** It needs a restart. The banner says so, and
the row reads *Restart to start* until then.
## The Restart button did not bring the site back
The button exits the process and relies on a supervisor to start it again. If your
deployment has nothing supervising it — `npm start` in a terminal, a container without
`restart:` — the site stays down until you start it yourself. Compose with
`restart: unless-stopped` is the supported shape.
## The game screens are empty or say offline
Work outwards from the game, and stop at the first check that fails.
1. **In game:** `[bridge status` — `connected=False` means the shard cannot reach the
sidecar.
2. **On the shard host:** `curl -s http://127.0.0.1:8080/health` — `plugin_connected: true`
is the value that matters.
3. **On the shard host:** `runicgateway doctor` — checks the install record, every overlay
file hash, the service, and that the sidecar and overlay agree on a protocol.
4. **On the site:** the [shard connection screen](/docs/administration/the-shard-connection/)
— its four indicators say which link is broken.
Two log lines with specific meanings: **`409`** is a protocol mismatch (set the Protocol
field to what `/health` reports), and **`401`** is the auth token (read the live one back
with `uo-link-sidecar --print-config`; do not retype it from a screenshot).
<Aside type="note" title="“Nothing changed and it stopped working” usually means a ServUO update">
An update to the server tree can revert `Scripts.csproj`, at which point the plugin sits in
the tree and never compiles — and ServUO ignores the script build's exit code, so the boot
looks clean. `doctor` catches it by comparing file hashes against the install record.
</Aside>
## The bestiary has no pictures, or items show numbers
Those come out of the UO client on the shard host, and **nothing imports them on a
restart** — a button on **Admin → Client files** is the only thing that does. Check that
page first: it reports why rather than failing.
| What it says | What to do |
|---|---|
| Counts are zero and no import is recorded | Press *Update*. On a shard that was linked before this existed, nobody ever has. |
| *…set `AssetsEnabled` on the shard* | The asset plane is off in `Bridge.cfg`. It is a separate switch on purpose. |
| *The shard host cannot render images* | A Linux host with no `libgdiplus`. Install it and press *Update* again. Names are unaffected either way. |
| *The shard is busy with another client-file request* | Not a fault. Something ordinary holds the slot; it frees itself. |
| Pictures were fine and went blank | Check the uploads volume before anything else — the database still remembers pictures that are no longer on disk, and *Re-import everything* is the button for exactly that. |
Items reading as numbers rather than names is the same page, different section: it means the
cliloc table has not been imported. See [Client files](/docs/administration/client-files/).
## Teams are missing
Check the sync panel on **Admin → Teams** before anything else: *last success: never* with
`no uo-link configured` means the shard connection, not the Team machinery. And on a site
with **no module installed**, an empty Team list is correct and final — core cannot create
a Team. See [Teams](/docs/administration/teams/).
## Email and announcements never arrive
- **The contact form opens a mail client.** Email delivery is not configured; that is the
documented fallback. Enter SMTP credentials in **Settings → Email delivery**. If this site
used to send mail and stopped, the Gmail connect flow was removed — the admin dashboard
says so, and [Notifications and email](/docs/administration/notifications-and-email/) has
the migration.
- **Mail is configured but nothing arrives, and there is no error.** Two usual causes, both
invisible without a test send. *Implicit TLS* left on for port 587 hangs rather than
failing; and a **Send from** address the server will not let you send as is rejected for
SPF/DMARC reasons. Press **Send test** — its failure message names both cases.
- **Email was working and the toggle is still on.** *Enable email sending* now gates every
message, not just some of them. If it is off, nothing is sent, including the contact
form.
- **A published post announced nothing.** The Discord bot is a separate container. If the
Discord Bot screen says *bot unreachable*, it is not running.
- **A missed announcement does not come back.** Nothing retries; the post itself is still
on the site.
## Nothing is sent for one particular event
Mail works, other notifications arrive, but this one thing never produces anything. The
answer is almost always in **Engagement → Rules**, and it is one of four:
- **The rule is off.** Every rule ships disabled, including the ones your modules bring
with them, so "installed" is not "on".
- **The rule is badged *dormant*.** It is switched on but cannot fire — usually because the
module that declared its trigger, or the audience it points at, is no longer installed.
- **It fired and was held back by its own cooldown**, which is per person, per subject and
per channel. The Send Log shows nothing for a message that was never queued.
- **The audience resolves to nobody.** The rule editor's reach preview is the fastest way
to find that out — it will tell you the count is zero and why.
[Engagement rules](/docs/administration/engagement-rules/) walks through all four.
## One person stopped receiving email
Everyone else is getting mail, so the transport is fine. Check
**Engagement → Suppressions**, then **Engagement → Send Log**.
- **They are on the suppression list.** The site stops mailing an address once the
receiving server says the mailbox does not exist. Addresses are stored one way and
shown masked (`d***@example.com`), so search by their domain to find the row. If they
have since fixed their mailbox, press **Lift a suppression** and type the full
address — the screen genuinely does not have it, which is why you are asked.
- **Suppression only affects engagement rules.** Password resets, invites and address
verification still go out to a suppressed address, because those are things the person
asked for themselves. So "they can reset their password but get no notifications" is
the expected shape of this problem, not a contradiction.
- **The Send Log says *Not sent*.** That is a suppression: nothing was sent to the mail
server at all. *Bounced* means it was sent and the mailbox does not exist. *Failed*
means the relay refused it for some other reason — that one is about your
configuration, not about them.
- **The Send Log has no row for them at all.** They were excluded before anything was
queued. Either they have not opted in on **Notifications** for that stream, or
*Require a verified email address* is on in **Settings** and they have not confirmed
theirs. The rule editor's audience preview shows how many people each of those removes.
## Everyone stopped receiving email at once
Do **not** start clearing the suppression list — it is almost certainly not the cause.
A whole-deployment stop is a transport problem: an expired password, a relay that has
started refusing you, or *Enable email sending* switched off. The Send Log will show
*Failed* rather than *Bounced* or *Not sent*, and **Settings → Email delivery** shows the
last error. A wrong password never suppresses anybody; only the receiving server saying a
specific mailbox does not exist does that.
## Uploads and modules fail with permission errors
Docker created a bind-mount source that the container user cannot write — usually because
the directory was deleted and recreated by Docker as `root`. `chown 1000:1000 modules` (or
`logs`, or `brand`) on the host fixes it. Do not delete those directories.
## When you need to ask for help
Bring three things: the relevant lines from `docker compose logs app`, the output of
`runicgateway doctor` if a game server is involved, and what you changed last. The
[community page](/community/) has where to ask.