feat(auth): unique, changeable, verifiable email addresses (engagement Phase 1b) #167

wtclaude · 2026-08-29T06:55:08Z

wtclaude commented

2026-08-29 06:55:08 +00:00

Engagement Phase 1b — design of record: docs/website/ENGAGEMENT.md Phase 1b (companion PR). Lands alone, between Phase 1 and Phase 2, and before any engagement mail exists — retrofitting uniqueness after a send log and a suppression list hold rows is strictly worse.

Discharges §0.6.

The plan's index design was wrong, and would have destroyed data

Step 2 said to pin the UNIQUE index to a case-insensitive collation "for the same reason username was". The reasoning is right — Foo@x.com and foo@x.com are one mailbox, and folding belongs in the index rather than in bypassable application code. The collation is not.

Tested against the deployment's own MariaDB 11.8: under both utf8mb4_general_ci and the server-default utf8mb4_uca1400_ai_ci, josé@x.com and jose@x.com compare EQUAL. Every _ci collation available here is also accent-insensitive. Those are two different mailboxes. That index would refuse the second address forever, and the de-duplication below would have nulled a legitimate account's address and reported it as a duplicate that never was.

The collations that would be exactly right (utf8mb4_general1400_as_ci, utf8mb4_0900_as_ci) are MariaDB 11.4+ only, so pinning one just moves §0.6's "a UNIQUE email can stop a boot" to a different trigger.

What shipped instead:

email      VARCHAR(255) NULL,
email_norm VARCHAR(255) COLLATE utf8mb4_bin AS (LOWER(email)) STORED,
UNIQUE KEY uq_users_email_norm (email_norm)

LOWER() under _bin folds case without folding accents — verified, not assumed. The fold still lives in the schema, which was the point of the original rule. Multiple NULLs stay legal, which is what lets the de-dupe clear an address without deleting an account. No FK references users.email, so the STORED-generated-column trap from TEAMS.md phase 2 (ER_GENERATED_COLUMN_FUNCTION_IS_NOT_ALLOWED on ON DELETE SET NULL) does not apply.

The same bug, one statement later

Written the obvious way, the de-dupe compares LOWER(u2.email) = LOWER(u.email) — which uses the column's collation, and so over-folds even though the index does not. A seeded fixture caught it nulling jose@x.com as a "duplicate" of josé@x.com: the exact defect the index change was made to prevent, reintroduced immediately.

So the migration adds email_norm before de-duplicating and groups on it — the two agree by construction rather than by a hand-matched COLLATE clause a later edit can get wrong. Order in schema.sql is load-bearing and commented as such.

What changed

Schema (all idempotent; after the first boot every statement matches zero rows):

UPDATE users SET email = NULL WHERE email = '' — '' is a value, not an absence, so two accounts holding it would collide and stop the boot. Unreachable through today's routes; this runs against databases whose history we do not control.
add email_pending + email_norm — no index yet; a UNIQUE index here is precisely the ALTER that fails and takes the site down.
record every account about to lose its address, before nulling it — the report is the only place the value survives. Oldest-wins, ties broken by id. Verified status deliberately does not arbitrate: SSO set it from an address merely existing.
clear the losers. Never deletes a row. (The extra derived table is not decoration — MariaDB refuses a subquery on the table being updated, error 1093.)
now the index can go on.
seed the verification gate: on fresh / off upgrade.

Telling the two constraints apart. isDuplicateUsername() was a bare ER_DUP_ENTRY test. The violated index name is available only in the driver's message text — no structured field — so this reads it back out, with its own test. That message also embeds the bound parameters, so on an email collision it contains the address: a second reason these never reach a client. isDuplicateUsername is now "duplicate and NOT email", so no call site newly falls through to a 500 on a database whose index carries an unexpected name.

§0.6 named two call sites. There are five, and the three it omits fail worse:

Site	Before	Now
`auth.controller` register	"That username is already taken" for an email collision	generic 400, real reason logged, not fed to the bot scorer
`sso.controller` provision	retried usernames against a conflict no username can clear, burning all 25 tries	stops at the first attempt, returns a distinguishable reason
`invite.controller` accept	blamed the username	409 that explains, after 409 at creation (decision 1)
`admin.controller` createUser	no catch at all → opaque 500	409 naming the field
`admin.controller` updateUser	no catch at all → opaque 500	409 naming the field

The answers differ on purpose: a public form, an authenticated IdP callback, a half-completed invite and an admin screen do not owe the same person the same amount of truth.

Change-and-verify. A requested address is staged in email_pending; only the tokened link installs it. The account keeps receiving password-reset mail at the address it already has, so a typo cannot silently redirect account recovery. currentPassword required when the account has one, with the SSO carve-out changePassword already makes.

SSO now reads the IdP's actual email_verified / verified claim. Forward-only — existing rows keep their flag; retroactively demoting live users is the G22 mistake.

Deliberate deviation

The plan says a "signed" link. Every comparable flow here (user_invites, password_resets, mobile_refresh_tokens) uses an opaque random token with only its sha256 at rest. email_verifications matches them rather than introducing a second token mechanism for one caller.

provisionSsoPlayer now returns { user } or { error } rather than the user or a bare null, and is exported for a test — the behaviour that matters is a count, not observable through the route handlers without stubbing most of the OAuth flow.

Anti-enumeration, which is load-bearing

Confirming answers with the byte-identical 404 for: expired, already-used, superseded, and an address another account confirmed first. Distinguishing them would make the endpoint an oracle for which addresses hold accounts. There is a test per case asserting the same string.

Verification

1239 server tests, 288 client tests, all green. Swagger regenerated (6 paths added, 0 removed, 0 surviving path definitions changed — verified by set-comparison, not by reading the diff); routes:manifest --check clean at 210 routes.

routeManifest.test.js fails locally unless modules/uo is excluded (MODULES_DIR=<empty dir>) — the committed manifest is core-only. Known environment trap, not a regression.

Walked on a live rig — real MariaDB 11.8 seeded with the pre-upgrade schema plus three accounts sharing an address, upgraded by booting the real server, with a real SMTP send into a mail catcher (which also discharges Phase 1's outstanding "no live SMTP send"):

the upgrade boots clean; oldest kept the address, two nulled and reported with the exact addresses lost; josé@ and jose@ both survived
gate seeded off on the upgrade, on on a fresh install
dashboard warning fired with the right count, cleared on acknowledge
a change request staged the address and left the live one receiving mail; the link went only to the new address; opening it from a session-less client installed it and set no cookie
a replayed link, and a second account confirming an address the first had just taken, both returned the identical generic 404 — real reason logged, never returned
NEWMAIL@RIG.TEST refused at registration as a duplicate of newmail@rig.test, while néwmail@rig.test registered successfully beside it — which is the whole argument for the index change, demonstrated
password reset still works, and its lookup folds case through the same column

Reviewer notes

Reset mail is deliberately not gated on email_verified. That gate governs opt-in engagement mail; applying it to account recovery would lock out every user carrying an address from before verification existed.
Nothing consumes email_verification_required yet (Phase 4/9 do). It is seeded now because the fresh-vs-upgrade distinction is only knowable at the migration that adds it.
A pending address is deliberately not unique — it reserves nothing, and two users may both be pending on one address. The second to confirm loses, with the same generic failure.
Not fixed here, flagged instead: CLAUDE.md still says "identities are never auto-provisioned", which provisionSsoPlayer contradicts (§0.6 finding 3 records this). Out of this phase's scope — say the word and I will take it separately.
docs/website/api-route-inventory.json is regenerated in the companion PR and is still ungated; it will drift again.

AI disclosure

Written with Claude Code (Opus 5).

Engagement **Phase 1b** — design of record: [`docs/website/ENGAGEMENT.md` Phase 1b](https://gitea.whitlocktech.com/RunicGateway/docs) (companion PR). Lands alone, between Phase 1 and Phase 2, and **before any engagement mail exists** — retrofitting uniqueness after a send log and a suppression list hold rows is strictly worse. Discharges §0.6. ## The plan's index design was wrong, and would have destroyed data Step 2 said to pin the UNIQUE index to a case-insensitive collation *"for the same reason `username` was"*. The reasoning is right — `Foo@x.com` and `foo@x.com` are one mailbox, and folding belongs in the index rather than in bypassable application code. The collation is not. Tested against the deployment's own MariaDB 11.8: under **both** `utf8mb4_general_ci` and the server-default `utf8mb4_uca1400_ai_ci`, **`josé@x.com` and `jose@x.com` compare EQUAL**. Every `_ci` collation available here is also accent-**in**sensitive. Those are two different mailboxes. That index would refuse the second address forever, and the de-duplication below would have **nulled a legitimate account's address** and reported it as a duplicate that never was. The collations that would be exactly right (`utf8mb4_general1400_as_ci`, `utf8mb4_0900_as_ci`) are MariaDB 11.4+ only, so pinning one just moves §0.6's "a UNIQUE email can stop a boot" to a different trigger. **What shipped instead:** ```sql email VARCHAR(255) NULL, email_norm VARCHAR(255) COLLATE utf8mb4_bin AS (LOWER(email)) STORED, UNIQUE KEY uq_users_email_norm (email_norm) ``` `LOWER()` under `_bin` folds case without folding accents — verified, not assumed. The fold still lives in the schema, which was the point of the original rule. Multiple NULLs stay legal, which is what lets the de-dupe clear an address without deleting an account. No FK references `users.email`, so the STORED-generated-column trap from TEAMS.md phase 2 (`ER_GENERATED_COLUMN_FUNCTION_IS_NOT_ALLOWED` on `ON DELETE SET NULL`) does not apply. ### The same bug, one statement later Written the obvious way, the de-dupe compares `LOWER(u2.email) = LOWER(u.email)` — which uses the **column's** collation, and so over-folds even though the index does not. A seeded fixture caught it nulling `jose@x.com` as a "duplicate" of `josé@x.com`: the exact defect the index change was made to prevent, reintroduced immediately. So the migration **adds `email_norm` before de-duplicating and groups on it** — the two agree by construction rather than by a hand-matched `COLLATE` clause a later edit can get wrong. Order in `schema.sql` is load-bearing and commented as such. ## What changed **Schema** (all idempotent; after the first boot every statement matches zero rows): 1. `UPDATE users SET email = NULL WHERE email = ''` — `''` is a value, not an absence, so two accounts holding it would collide and stop the boot. Unreachable through today's routes; this runs against databases whose history we do not control. 2. add `email_pending` + `email_norm` — **no index yet**; a UNIQUE index here is precisely the ALTER that fails and takes the site down. 3. record every account about to lose its address, **before** nulling it — the report is the only place the value survives. **Oldest-wins**, ties broken by id. Verified status deliberately does not arbitrate: SSO set it from an address merely existing. 4. clear the losers. **Never deletes a row.** (The extra derived table is not decoration — MariaDB refuses a subquery on the table being updated, error 1093.) 5. now the index can go on. 6. seed the verification gate: `on` fresh / `off` upgrade. **Telling the two constraints apart.** `isDuplicateUsername()` was a bare `ER_DUP_ENTRY` test. The violated index name is available **only in the driver's message text** — no structured field — so this reads it back out, with its own test. That message also embeds the bound parameters, so on an email collision it *contains the address*: a second reason these never reach a client. `isDuplicateUsername` is now "duplicate and NOT email", so no call site newly falls through to a 500 on a database whose index carries an unexpected name. **§0.6 named two call sites. There are five**, and the three it omits fail worse: | Site | Before | Now | |---|---|---| | `auth.controller` register | *"That username is already taken"* for an **email** collision | generic 400, real reason logged, **not** fed to the bot scorer | | `sso.controller` provision | retried **usernames** against a conflict no username can clear, burning all 25 tries | stops at the first attempt, returns a distinguishable reason | | `invite.controller` accept | blamed the username | 409 that explains, **after** 409 at *creation* (decision 1) | | `admin.controller` createUser | **no catch at all** → opaque 500 | 409 naming the field | | `admin.controller` updateUser | **no catch at all** → opaque 500 | 409 naming the field | The answers differ on purpose: a public form, an authenticated IdP callback, a half-completed invite and an admin screen do not owe the same person the same amount of truth. **Change-and-verify.** A requested address is **staged** in `email_pending`; only the tokened link installs it. The account keeps receiving password-reset mail at the address it already has, so a typo cannot silently redirect account recovery. `currentPassword` required when the account has one, with the SSO carve-out `changePassword` already makes. **SSO** now reads the IdP's actual `email_verified` / `verified` claim. Forward-only — existing rows keep their flag; retroactively demoting live users is the G22 mistake. ## Deliberate deviation The plan says a *"signed"* link. Every comparable flow here (`user_invites`, `password_resets`, `mobile_refresh_tokens`) uses an opaque random token with only its sha256 at rest. `email_verifications` matches them rather than introducing a second token mechanism for one caller. `provisionSsoPlayer` now returns `{ user }` or `{ error }` rather than the user or a bare null, and is exported for a test — the behaviour that matters is a *count*, not observable through the route handlers without stubbing most of the OAuth flow. ## Anti-enumeration, which is load-bearing Confirming answers with the **byte-identical** 404 for: expired, already-used, superseded, **and an address another account confirmed first**. Distinguishing them would make the endpoint an oracle for which addresses hold accounts. There is a test per case asserting the same string. ## Verification **1239 server tests, 288 client tests, all green.** Swagger regenerated (**6 paths added, 0 removed, 0 surviving path definitions changed** — verified by set-comparison, not by reading the diff); `routes:manifest --check` clean at 210 routes. > `routeManifest.test.js` fails locally unless `modules/uo` is excluded (`MODULES_DIR=<empty dir>`) — the committed manifest is core-only. Known environment trap, not a regression. **Walked on a live rig** — real MariaDB 11.8 seeded with the *pre-upgrade* schema plus three accounts sharing an address, upgraded by booting the real server, with a **real SMTP send** into a mail catcher (which also discharges Phase 1's outstanding *"no live SMTP send"*): - the upgrade **boots clean**; oldest kept the address, two nulled and reported with the exact addresses lost; `josé@` and `jose@` both survived - gate seeded **`off`** on the upgrade, **`on`** on a fresh install - dashboard warning fired with the right count, cleared on acknowledge - a change request staged the address and **left the live one receiving mail**; the link went only to the new address; opening it from a session-less client installed it and **set no cookie** - a replayed link, and a second account confirming an address the first had just taken, both returned the identical generic 404 — real reason logged, never returned - `NEWMAIL@RIG.TEST` refused at registration as a duplicate of `newmail@rig.test`, while **`néwmail@rig.test` registered successfully beside it** — which is the whole argument for the index change, demonstrated - password reset still works, and its lookup folds case through the same column ## Reviewer notes - Reset mail is deliberately **not** gated on `email_verified`. That gate governs opt-in engagement mail; applying it to account recovery would lock out every user carrying an address from before verification existed. - **Nothing consumes `email_verification_required` yet** (Phase 4/9 do). It is seeded now because the fresh-vs-upgrade distinction is only knowable at the migration that adds it. - A pending address is deliberately **not** unique — it reserves nothing, and two users may both be pending on one address. The second to confirm loses, with the same generic failure. - **Not fixed here, flagged instead:** CLAUDE.md still says *"identities are never auto-provisioned"*, which `provisionSsoPlayer` contradicts (§0.6 finding 3 records this). Out of this phase's scope — say the word and I will take it separately. - `docs/website/api-route-inventory.json` is regenerated in the companion PR and is **still ungated**; it will drift again. ## AI disclosure Written with Claude Code (Opus 5).

wtclaude added 1 commit 2026-08-29 06:55:11 +00:00

feat(auth): unique, changeable, verifiable email addresses (engagement Phase 1b)

PR Checks / bot-tests (pull_request) Successful in 29s

Details

PR Checks / client-build (pull_request) Successful in 31s

Details

PR Checks / server-tests (pull_request) Successful in 10m34s

Details

fbb4b0bd91

Makes `users.email` unique, de-duplicates the addresses an upgrade will find,
and builds the self-service change-and-verify flow that did not exist.

The uniqueness index is on a generated `email_norm AS (LOWER(email)) STORED`
column under `utf8mb4_bin`, NOT on `email` under a `_ci` collation as the plan
specified. Every case-insensitive collation this server offers is also
accent-insensitive: `josé@x.com` and `jose@x.com` compare equal, and those are
two different mailboxes. The plan's index would have refused the second address
forever and the de-duplication would have nulled a legitimate account's.

A requested address is STAGED in `email_pending` and only a tokened link
installs it, so a typo cannot silently redirect account-recovery mail.

`isDuplicateUsername()` now distinguishes the two indexes. All five call sites
branch on it; each answers differently on purpose, because a public form, an
IdP callback, a half-completed invite and an admin screen do not owe the same
person the same amount of truth.

SSO reads the IdP's actual `email_verified`/`verified` claim instead of
inferring verification from an address merely being present.

Co-Authored-By: Claude <noreply@anthropic.com>

wtclaude referenced this pull request from RunicGateway/docs

2026-08-29 06:55:33 +00:00

docs(website): unique, changeable, verifiable email addresses (engagement Phase 1b) #180

whitlocktech merged commit 6016b325bb into edge

2026-08-29 07:08:46 +00:00

whitlocktech deleted branch feature/unique-verifiable-email

2026-08-29 07:08:47 +00:00

whitlocktech referenced this issue from a commit

2026-08-29 07:08:48 +00:00

Merge pull request 'feat(auth): unique, changeable, verifiable email addresses (engagement Phase 1b)' (#167) from feature/unique-verifiable-email into edge

Sign in to join this conversation.