AI Implementation feature(919): Admin Area System: Database Backup/Restore and Danger Zone 1.02 (#64)

docs/api/admin-system.md

@@ -4,7 +4,7 @@ title: Admin System Endpoints
 description: Admin-only destructive System operations: backup download, restore validate/commit, re-authenticated confirmation tokens, reset-scores, and wipe-challenges.
 resource: backend/src/modules/admin/system/admin-system.controller.ts
 tags: [api, admin, system, backup, restore, danger-zone, confirmation-tokens]
-timestamp: 2026-07-23T13:27:25Z
+timestamp: 2026-07-23T15:46:55Z
 ---
 
 # Overview
@@ -127,9 +127,15 @@ Flow:
    by cloning the live DB, then clearing every application table and
    inserting the archived rows in FK-safe order. Uploads are copied
    aside as `<UPLOAD_DIR>.restore-stage-<stamp>` and the live
-   database + uploads are swapped atomically via
-   `FilesystemTransactionService.stageSwap()` (with a `restore-backup`
-   rollback handle).
+   database + uploads are swapped as one recoverable generation via
+   `FilesystemTransactionService.stageSwap(..., { stagingDir })` (with a
+   `restore-backup` rollback handle). Before moving either live path, the
+   service writes `<SYSTEM_OP_STAGING_DIR>/<transactionId>/manifest.json`.
+   The manifest advances through `prepared`, `live-snapshotted`,
+   `replacements-installed`, and `verified`; commit removes rollback and
+   manifest artifacts. SQLite `-wal` and `-shm` sidecars travel with the
+   database rollback generation and stale sidecars are removed from the
+   replacement.
 
    The offline clone rebuild begins by capturing the exact SQL of the
    deployment-wide `trg_user_last_admin_update` and
@@ -161,6 +167,25 @@ Flow:
    revoked via `AuthService.revokeAllRefreshSessions()` so the SPA is
    forced back to `/login`.
 
+### Startup recovery contract
+
+Before TypeORM opens SQLite, `DatabaseInitService.init()` calls
+`FilesystemTransactionService.recoverTransactions()` using
+`SYSTEM_OP_STAGING_DIR` (default `<DATA_DIR>/.system-staging`):
+
+| Manifest phase | Required complete generation | Startup action |
+|---|---|---|
+| `prepared` | Existing live paths | Keep the old generation and remove the transaction manifest. |
+| `live-snapshotted` | Every rollback path exists and corresponding live path is absent | Restore all rollback paths, including tracked SQLite sidecars. |
+| `replacements-installed` or `verified` | Every replacement live path and rollback path exists | Keep the complete new generation and remove rollback/staged artifacts. |
+| Any ambiguous or mixed state | Neither generation is complete | Throw `SYSTEM_RESTORE_FAILED`, preserve the manifest and artifacts, and abort startup. |
+| `committed` or `rolled-back` | Terminal | Remove the transaction directory. |
+
+After manifest recovery, startup best-effort removes stale unowned
+`.rollback-*`, `.restore-stage-*`, and `.wipe-stage-*` artifacts near the
+configured database/uploads paths. Fresh artifacts and paths referenced by a
+manifest are retained.
+
 Returns `{ ok: true, operation: 'restore-backup' }`.
 
 ## POST /confirmations
@@ -301,7 +326,9 @@ curl -sS ... -d '{"operation":"reset-scores","token":"<raw>"}'
 | Restore                      | `backend/src/modules/admin/system/restore.service.ts`                           |
 | Confirmation tokens          | `backend/src/modules/admin/system/confirmation-token.service.ts`                |
 | Danger zone                  | `backend/src/modules/admin/system/danger-zone.service.ts`                       |
-| Filesystem swap              | `backend/src/modules/admin/system/filesystem-transaction.service.ts`            |
+| Filesystem swap + recovery   | `backend/src/modules/admin/system/filesystem-transaction.service.ts`            |
+| Shared filesystem module     | `backend/src/modules/admin/system/filesystem-transaction.module.ts`             |
+| Startup recovery wiring      | `backend/src/database/database-init.service.ts`, `backend/src/database/database.module.ts` |
 | Staging dir helper           | `backend/src/common/utils/upload.ts` (`resolveSystemStagingDir`)                |
 | Password re-auth / revoke    | `backend/src/modules/auth/auth.service.ts` (`reauthenticateAdmin`, `revokeAllRefreshSessions`) |
 | Error codes                  | `backend/src/common/errors/error-codes.ts`                                      |
@@ -313,4 +340,4 @@ curl -sS ... -d '{"operation":"reset-scores","token":"<raw>"}'
 | Front-end route              | `frontend/src/app/app.routes.ts` (admin child `system`)                         |
 | Cross-store invalidation     | `frontend/src/app/core/services/system-data-change.service.ts`                  |
 | Forced logout helper         | `frontend/src/app/core/services/auth.service.ts` (`forceServerInvalidation`)    |
-| Tests                        | `tests/backend/admin-system-authorization.spec.ts`, `admin-system-backup.spec.ts`, `admin-system-confirmation-token.spec.ts`, `admin-system-danger.spec.ts`, `admin-system-restore-validation.spec.ts`, `admin-system-restore-commit.spec.ts`; `tests/frontend/admin-system.pure.spec.ts` |
+| Tests                        | `tests/backend/admin-system-authorization.spec.ts`, `admin-system-backup.spec.ts`, `admin-system-confirmation-token.spec.ts`, `admin-system-danger.spec.ts`, `admin-system-restore-validation.spec.ts`, `admin-system-restore-commit.spec.ts`, `database-init-recovery.spec.ts`, `filesystem-transaction.spec.ts`; `tests/frontend/admin-system.pure.spec.ts` |

docs/architecture/backend-modules.md

@@ -3,7 +3,7 @@ type: architecture
 title: Backend Module Map
 description: NestJS modules, controllers, services, and how they are wired together.
 tags: [architecture, backend, nestjs, modules]
-timestamp: 2026-07-23T10:12:24Z
+timestamp: 2026-07-23T15:46:55Z
 ---
 
 # Module Map
@@ -20,7 +20,8 @@ also registers two global providers:
 
 | Module           | Path                                          | Responsibility                                                                                  |
 |------------------|-----------------------------------------------|-------------------------------------------------------------------------------------------------|
-| `DatabaseModule` | `backend/src/database/database.module.ts`     | Configures TypeORM with `better-sqlite3`, registers entities, runs migrations, enforces WAL.    |
+| `DatabaseModule` | `backend/src/database/database.module.ts`     | Configures TypeORM with `better-sqlite3`, registers entities, runs migrations, enforces WAL, and imports `FilesystemTransactionModule` so startup recovery runs before the database is opened.    |
+| `FilesystemTransactionModule` | `backend/src/modules/admin/system/filesystem-transaction.module.ts` | Provides the shared durable filesystem swap service to both `DatabaseModule` startup recovery and `AdminSystemModule` restore operations without a circular dependency. |
 | `CommonModule`   | `backend/src/common/common.module.ts`         | Provides shared services (CSRF middleware class, backoff, registration rate limit, SSE hub, theme loader, etc.). |
 | `SettingsModule` | `backend/src/modules/settings/settings.module.ts` | Exposes `SettingsService` (get/set/getAll over `setting` table).                              |
 | `AuthModule`     | `backend/src/modules/auth/auth.module.ts`     | `AuthController` (`/api/v1/auth/{register,login,refresh,logout,me,change-password,csrf}`) + `AuthService` (login/refresh/logout/register-first-admin + public `register` + `getMe` + `changePassword`). Imports `SettingsModule` so the public-register flow can read the `registrationsEnabled` flag, and `UsersModule` (forwardRef) for `UsersRankService`. |
@@ -28,7 +29,7 @@ also registers two global providers:
 | `SetupModule`    | `backend/src/modules/setup/setup.module.ts`   | `SetupController` (`POST /api/v1/setup/create-admin`) + `SetupService`. Returns 409 `SYSTEM_INITIALIZED` once any admin exists; serializes concurrent first-admin attempts via an in-process promise chain. |
 | `SystemModule`   | `backend/src/modules/system/system.module.ts` | `SystemController` (`/api/v1/bootstrap`, `/event/status`, SSE streams) + `SystemService`.       |
 | `AdminModule`    | `backend/src/modules/admin/admin.module.ts`   | Five controllers (`AdminController` for `/api/v1/admin/users*`, `AdminGeneralController` for `/api/v1/admin/general/*`, `AdminCategoriesController` for `/api/v1/admin/categories*`, `AdminChallengesController` for `/api/v1/admin/challenges*`, `AdminSystemController` for `/api/v1/admin/system/*`) plus their services (`AdminService`, `AdminGeneralService`, `AdminCategoriesService`, `AdminChallengesService`, `ChallengeFilesService`, `BackupService`, `RestoreService`, `DangerZoneService`, `FilesystemTransactionService`, `ConfirmationTokenService`). Gated by `AdminGuard` + `@Roles('admin')`. Imports `TypeOrmModule.forFeature([UserEntity, CategoryEntity, ChallengeEntity, ChallengeFileEntity, SolveEntity])`, `AuthModule`, `UsersModule`, `SettingsModule`, `CommonModule` (for `SseHubService`, `ThemeLoaderService`), and `AdminSystemModule` (which adds `AdminOperationTokenEntity` + `AuthModule`). |
-| `AdminSystemModule` | `backend/src/modules/admin/system/admin-system.module.ts` | Hosts the destructive System operations: `AdminSystemController` + `BackupService` + `RestoreService` + `DangerZoneService` + `FilesystemTransactionService` + `ConfirmationTokenService`. Imports `TypeOrmModule.forFeature([AdminOperationTokenEntity])` and `AuthModule` (for `reauthenticateAdmin`/`revokeAllRefreshSessions`). |
+| `AdminSystemModule` | `backend/src/modules/admin/system/admin-system.module.ts` | Hosts the destructive System operations: `AdminSystemController` + `BackupService` + `RestoreService` + `DangerZoneService` + `ConfirmationTokenService`. Imports shared `FilesystemTransactionModule`, `TypeOrmModule.forFeature([AdminOperationTokenEntity])`, and `AuthModule` (for `reauthenticateAdmin`/`revokeAllRefreshSessions`). |
 | `UploadsModule`  | `backend/src/modules/uploads/uploads.module.ts` | `UploadsController` (`/api/v1/uploads/*`). Admin-only multipart.                            |
 | `BlogModule`     | `backend/src/modules/blog/blog.module.ts`     | `BlogController`/`BlogService` expose published posts at `GET /api/v1/blog/posts`; `AdminBlogController`/`AdminBlogService` provide guarded CRUD at `/api/v1/admin/blog/posts` using `BlogPostEntity`. |
 | `ChallengesModule` | `backend/src/modules/challenges/challenges.module.ts` | `ChallengesController` (`/api/v1/challenges/{board,status,:id,:id/solves}`) + `ChallengesEventsController` (authenticated SSE `/api/v1/events`) + `ChallengesService` (board, detail, submit, scoring util). |
@@ -82,7 +83,9 @@ also registers two global providers:
 # Startup chain
 
 `main.ts` → `AppModule` (loads `ConfigModule`, then all feature modules)
-→ `DatabaseInitService.init()` (runs migrations + WAL)
+→ `DatabaseInitService.init()` → `FilesystemTransactionService.recoverTransactions()`
+(resolves interrupted restore manifests before TypeORM opens the live SQLite file;
+ambiguous generations abort startup) → stale unowned swap-artifact sweep → migrations + WAL
 → `app.use(...)` middlewares (helmet, parsers, CSRF, static)
 → `SwaggerModule.setup(...)` (OpenAPI 3.1)
 → `SpaFallbackMiddleware` → `app.listen()`.

docs/architecture/key-files.md

@@ -3,7 +3,7 @@ type: architecture
 title: Key Files Index
 description: One-line responsibility for important source and contract-test files, including event-window, challenge, scoreboard, session, notification, and blog flows.
 tags: [architecture, key-files, event-window, validation, default-challenge-ip, sse, bootstrap, migrations, challenges, import, board, notifications, per-user, session, blog]
-timestamp: 2026-07-23T13:27:25Z
+timestamp: 2026-07-23T15:46:55Z
 ---
 
 # Backend
@@ -13,8 +13,9 @@ timestamp: 2026-07-23T13:27:25Z
 | `backend/src/main.ts` | Bootstraps Nest, middleware, OpenAPI, static assets, and SPA fallback. |
 | `backend/src/app.module.ts` | Wires feature modules and global providers. |
 | `backend/src/config/env.schema.ts` | Zod-validated runtime config; canonical defaults for `DATABASE_PATH` (`./data/db.sqlite`) and `UPLOAD_DIR` (`./data/uploads`). |
-| `backend/src/database/database.module.ts` | Configures TypeORM with SQLite. |
-| `backend/src/database/database-init.service.ts` | Initializes the database and runs migrations; `verifySeed()` also asserts the canonical six system-category keys and reports missing/duplicate rows. |
+| `backend/src/database/database.module.ts` | Configures TypeORM with SQLite and imports the shared `FilesystemTransactionModule` used by pre-TypeORM startup recovery. |
+| `backend/src/database/database-init.service.ts` | Before opening SQLite, resolves durable swap manifests to a coherent old or verified-new database/uploads generation, fails startup closed on ambiguity, best-effort sweeps stale unowned swap artifacts, then runs migrations, enforces WAL, and verifies seeds. |
+| `backend/src/modules/admin/system/filesystem-transaction.module.ts` | Exports one shared `FilesystemTransactionService` provider to `DatabaseModule` and `AdminSystemModule`, avoiding a circular dependency. |
 | `backend/src/database/migrations/1700000000400-RepairCategorySchemaAndSystemCategories.ts` | Forward-only repair migration: adds `created_at`/`updated_at` to legacy six-column `category` tables, deletes obsolete `FOR`/`OSI` system rows, rewrites canonical row metadata, inserts any missing canonical key, and re-asserts unique indexes. Exports `CANONICAL_SYSTEM_CATEGORIES`. |
 | `backend/src/common/guards/jwt-auth.guard.ts` | Global JWT authorization guard. |
 | `backend/src/common/middleware/csrf.middleware.ts` | Double-submit CSRF protection. |
@@ -124,11 +125,11 @@ timestamp: 2026-07-23T13:27:25Z
 | `tests/frontend/scoreboard.pure.spec.ts` | Pure helpers: `applyRankingSort` (competition-rank numbering), `parseSolveEventIntoRanking` (add-or-increment + sort), `dedupEventLogBySolveId`, `applySolveToGraph` (anchors + plateau + top-10 trim), `mutateMatrixFromSolve`, `stablePlayerColorIndex`, `formatSolveDateTime`. |
 | `tests/frontend/scoreboard.store.spec.ts` | Store lifecycle + live merge: `loadAll` aggregation + idempotency, SSE `solve` frame applying to all four tabs, `stop()` teardown, exponential reconnect. |
 | `backend/src/modules/admin/system/admin-system.controller.ts` | Mounts `/api/v1/admin/system/*` (backup download, restore validate/commit, confirmations, scores/reset, challenges/wipe); all handlers require `AdminGuard` + `@Roles('admin')` and the destructive ones also consume a confirmation token. |
-| `backend/src/modules/admin/system/admin-system.module.ts` | Wires `AdminSystemController`, `BackupService`, `RestoreService`, `DangerZoneService`, `FilesystemTransactionService`, `ConfirmationTokenService`; imports `TypeOrmModule.forFeature([AdminOperationTokenEntity])` and `AuthModule`. |
+| `backend/src/modules/admin/system/admin-system.module.ts` | Wires `AdminSystemController`, `BackupService`, `RestoreService`, `DangerZoneService`, and `ConfirmationTokenService`; imports the shared `FilesystemTransactionModule`, `TypeOrmModule.forFeature([AdminOperationTokenEntity])`, and `AuthModule`. |
 | `backend/src/modules/admin/system/backup.service.ts` | Builds the full backup JSON: every application table (`sqlite_master` discovery, excluding `admin_operation_token`/`migrations`/`sqlite_*`), with base64-encoded uploads walked recursively from `UPLOAD_DIR` (skipping `.staging`). Exposes `discoverTables()`, `getUploadDir()`, and `BackupService.stringify(doc)`. |
-| `backend/src/modules/admin/system/restore.service.ts` | Two-phase restore: `stageArchive(rawText, userId)` validates the archive (zod), decodes uploads into `<DATA_DIR>/.system-staging/restore-<id>/`, and returns a summary; `commitRestore(stagingId)` clones the live DB, captures and temporarily drops the `trg_user_last_admin_update` / `trg_user_last_admin_delete` triggers in the offline candidate, clears every application table, re-inserts archived rows in FK-safe order, asserts the archive still contains at least one admin, re-creates the captured triggers, then atomically swaps the live DB + uploads via `FilesystemTransactionService.stageSwap({ name: 'restore-backup', ... })`. On any failure the swap is rolled back and `SYSTEM_RESTORE_ROLLED_BACK` is returned; the rebuild also best-effort re-creates any missing trigger before rethrowing so a rolled-back candidate never leaks missing the `LAST_ADMIN` safety net. |
+| `backend/src/modules/admin/system/restore.service.ts` | Two-phase restore: validates/stages the archive, rebuilds an offline SQLite candidate while preserving the last-admin triggers, then swaps database + uploads as one durable manifest-backed generation; tracks SQLite `-wal`/`-shm` sidecars and reconnects TypeORM only after verification. |
 | `backend/src/modules/admin/system/danger-zone.service.ts` | `resetScores()` deletes every `solve` row transactionally; `wipeChallenges()` snapshots `<UPLOAD_DIR>/challenges` to a side directory, deletes every challenge inside a transaction, physically removes the live `-challenges` directory post-commit, and either deletes the snapshot or restores from it on failure. |
-| `backend/src/modules/admin/system/filesystem-transaction.service.ts` | Generic file/directory swap primitive: `stageSwap({ pairs, hooks })` renames each live path to a rollback location, then renames the staged path into place (with copy+unlink cross-device fallback); `commit(handle)` removes rollback artifacts; `rollback(handle)` restores the original live paths. Provides static helpers `rmSafe`, `copyDirSync`, `ensureDir`. |
+| `backend/src/modules/admin/system/filesystem-transaction.service.ts` | Crash-safe multi-path swap primitive. Persists per-transaction phase/path manifests under `SYSTEM_OP_STAGING_DIR`, moves configured sidecars with their generation, commits or rolls back idempotently, restores a complete old generation after an interrupted snapshot, finalizes a complete verified new generation, fails closed on mixed/ambiguous state, and sweeps stale unowned rollback/restore/wipe artifacts. |
 | `backend/src/modules/admin/system/confirmation-token.service.ts` | Issues SHA-256-hashed single-use tokens with `SYSTEM_OP_CONFIRM_TOKEN_TTL_MS` TTL; `consume()` runs in a transaction with conditional `WHERE consumedAt IS NULL` updates so only the first concurrent caller succeeds. `purgeExpired()` deletes expired and >24h-old consumed rows. |
 | `backend/src/modules/admin/system/dto/admin-system.dto.ts` | Re-exports `ADMIN_OPERATION_KINDS` and zod contracts for the re-auth body, danger confirm body, and restore-commit body; declares the backup format identifier + version constants. |
 | `backend/src/database/entities/admin-operation-token.entity.ts` | TypeORM entity and the canonical `ADMIN_OPERATION_KINDS` literal union (`restore-backup`, `reset-scores`, `wipe-challenges`). |
@@ -142,7 +143,9 @@ timestamp: 2026-07-23T13:27:25Z
 | `tests/backend/admin-system-confirmation-token.spec.ts` | Issue/consume/expire/reuse/mismatch flow and TTL config. |
 | `tests/backend/admin-system-danger.spec.ts` | Reset-scores and wipe-challenges happy path + DB/file rollback. |
 | `tests/backend/admin-system-restore-validation.spec.ts` | Archive schema validation, base64 size mismatch, sha256 mismatch, safe-path rejection, and duplicate-path rejection. |
-| `tests/backend/admin-system-restore-commit.spec.ts` | Controller forwards `stagingId` to `RestoreService.commitRestore`, plus a real-file service-level regression that restores a backup containing the same admin as the live system, asserts swapped table settings + uploads + trigger preservation, and verifies an admin-less archive is rejected with `SYSTEM_RESTORE_ROLLED_BACK`. |
+| `tests/backend/admin-system-restore-commit.spec.ts` | Controller forwarding, real-file restore, trigger preservation, admin-less archive rejection, and kill-during-swap regression proving restart recovers one coherent database/uploads generation. |
+| `tests/backend/database-init-recovery.spec.ts` | Startup recovery contracts: restores old generations from interrupted manifests, fails closed without mutation on hybrid state, and removes stale unowned artifacts. |
+| `tests/backend/filesystem-transaction.spec.ts` | Durable manifest phase, commit/rollback, old/new generation recovery, ambiguous fail-closed, and stale artifact sweep contracts. |
 | `frontend/src/app/features/admin/system/system.component.ts` | `/admin/system` smart page: two panels (Database + Danger Zone), backup download, restore pick + validate, confirm modal trigger, and per-operation success/error surfacing + cross-store invalidation. |
 | `frontend/src/app/features/admin/system/system-confirm-modal.component.ts` | Re-authentication + confirmation-phrase modal with operation-specific copy from `system.pure.ts`. |
 | `frontend/src/app/features/admin/system/system.service.ts` | HTTP client for the six `/api/v1/admin/system/*` endpoints. |

docs/guides/admin-system-page.md

@@ -3,7 +3,7 @@ type: guide
 title: Admin — System (Backup, Restore, Danger Zone)
 description: How an admin navigates to /admin/system, creates a backup, stages and commits a restore, resets all scores, and wipes challenges — including the re-authentication confirmation, progress, and side effects.
 tags: [guide, admin, system, backup, restore, danger-zone, confirmation]
-timestamp: 2026-07-23T12:04:50Z
+timestamp: 2026-07-23T15:46:55Z
 ---
 
 # Navigation
@@ -157,7 +157,13 @@ messages. Notable mappings:
    password, type `RESTORE BACKUP`, click **Confirm**.
 6. The modal closes, the page shows a brief *Restore complete.
    Logging out…* toast, and the SPA navigates to `/login`.
-7. Sign in again with the same admin credentials. The data on the
+7. If the server process is terminated during the database/uploads swap,
+   restart the server. Before accepting requests, startup recovery uses the
+   durable restore manifest to restore the complete old generation or finalize
+   the complete verified new generation; it never intentionally boots with a
+   mixed database/uploads generation. An ambiguous incomplete generation fails
+   startup closed and preserves the manifest and artifacts for diagnosis.
+8. Sign in again with the same admin credentials. The data on the
    target environment now matches the source backup.
 
 ## Manual reset happy path

docs/index.md

@@ -12,7 +12,7 @@ controls, administrator-managed Markdown blog publishing, and public and
 signed-in blog views.
 
 The docs below are organized by purpose so agents can pull just the slice
-they need. Last regenerated 2026-07-23T13:27:25Z.
+they need. Last regenerated 2026-07-23T15:46:55Z.
 
 # Architecture