AI Implementation feature(919): Admin Area System: Database Backup/Restore and Danger Zone 1.02 #64
@@ -1,32 +0,0 @@
|
|||||||
# Implementation Plan: Job 918 — Admin Area System Database Backup/Restore and Danger Zone 1.01
|
|
||||||
|
|
||||||
## 1. Architectural Reconnaissance
|
|
||||||
- **Codebase style & conventions:** Node.js monorepo using strict TypeScript, NestJS controllers/services on the backend, Angular standalone components and signals on the frontend, async/await, dependency injection, Zod request/archive validation, and stable `ApiError` codes. The requested UI and API flow already exists; the defect is isolated to the backend restore rebuild. `RestoreService.cloneAndOverwriteDatabase()` clones the live SQLite file and executes `DELETE` statements while the live schema's `trg_user_last_admin_delete` trigger remains active. `PRAGMA foreign_keys = OFF` does not disable triggers, so deleting the sole live admin raises `LAST_ADMIN` before archived users can be inserted.
|
|
||||||
- **Data Layer:** SQLite through TypeORM's `better-sqlite3` driver. Runtime migrations create a database-level invariant with `trg_user_last_admin_update` and `trg_user_last_admin_delete`. Backups dynamically include application tables but exclude `admin_operation_token`, `migrations`, and SQLite internals. Restore uses a cloned database, direct parameterized `better-sqlite3` statements, and a database/uploads filesystem swap with rollback handles.
|
|
||||||
- **Test Framework & Structure:** Jest 29 with `ts-jest`; backend tests live under `/repo/tests/backend` and are selected by `/repo/tests/jest.config.js`. Root `npm test` runs backend and frontend projects, while `npm run test:backend` runs the focused backend suite. Tests must remain CLI-only and use isolated temporary SQLite/upload/staging paths rather than shared mutable resources.
|
|
||||||
- **Required Tools & Dependencies:** No new package, system tool, global CLI, or persistent `/data` asset is required. Existing Node.js/npm, TypeScript, Jest, Nest testing utilities, TypeORM, and `better-sqlite3` are sufficient. `setup.sh` already installs dependencies and rebuilds the native SQLite binding, so it should not be changed.
|
|
||||||
|
|
||||||
## 2. Impacted Files
|
|
||||||
- **To Modify:**
|
|
||||||
- `backend/src/modules/admin/system/restore.service.ts` — make the offline clone rebuild temporarily suspend the last-admin triggers, restore all archived rows, validate invariants, and reinstate the triggers before the file is eligible for swapping.
|
|
||||||
- `tests/backend/admin-system-restore-commit.spec.ts` — add a focused real-file restore regression covering a backup that contains the same sole admin as the live database, restored table content, uploads, and trigger preservation; retain the existing controller/DTO contract coverage.
|
|
||||||
- **To Create:** None.
|
|
||||||
|
|
||||||
## 3. Proposed Changes
|
|
||||||
1. **Database / Schema Migration:**
|
|
||||||
- Do not add or alter a migration: the `LAST_ADMIN` triggers are correct for ordinary runtime user mutations.
|
|
||||||
- In the private staged database clone only, read the exact SQL definitions of `trg_user_last_admin_update` and `trg_user_last_admin_delete` from `sqlite_master`, fail closed if the expected trigger definitions cannot be captured, and drop those triggers before clearing application tables.
|
|
||||||
- Keep foreign keys disabled during the bulk replacement, clear all known application tables, and insert archived rows in the existing dependency-safe order with parameterized values.
|
|
||||||
- Before closing the staged clone, verify the archive produced at least one admin user so a restore cannot bypass the deployment-wide invariant. Recreate both captured triggers exactly, re-enable foreign keys, run `PRAGMA foreign_key_check`, and treat any violations or trigger-restoration failure as a restore failure. This confines the temporary invariant suspension to an offline candidate file and ensures the candidate has full protections before swap.
|
|
||||||
2. **Backend Logic & APIs:**
|
|
||||||
- Preserve the existing `POST /api/v1/admin/system/restore/validate`, confirmation-token, and `POST /restore/commit` contracts; `stagingId` already survives Zod validation and reaches `RestoreService.commitRestore()`.
|
|
||||||
- Refactor the staged rebuild cleanup so database closure and pragma/trigger restoration are deterministic and original failures are not masked by cleanup. Any rebuild, integrity, swap, or verification error must continue through the existing rollback path and return `SYSTEM_RESTORE_ROLLED_BACK` without changing live data/uploads.
|
|
||||||
- Keep the existing filesystem transaction: build and validate the candidate first, copy staged uploads beside the live upload root, swap database and uploads together, verify the swapped database, then commit rollback artifacts.
|
|
||||||
- Keep post-success session revocation in `AdminSystemController.commitRestore()`. After the restored database is live, revoke refresh sessions for the authenticated admin ID; the frontend's already-wired `forceServerInvalidation()` then clears the access session and redirects all tabs to login.
|
|
||||||
3. **Frontend UI Integration:**
|
|
||||||
- No frontend changes are planned. The file picker, staging summary, reauthentication modal, confirmation phrase, restore commit request, rollback alert, data-cache invalidation, and forced-login behavior are already wired in `frontend/src/app/features/admin/system/system.component.ts`, `system.service.ts`, and the auth/system data-change services.
|
|
||||||
|
|
||||||
## 4. Test Strategy
|
|
||||||
- **Target Unit Test File:** `tests/backend/admin-system-restore-commit.spec.ts`.
|
|
||||||
- **Mocking Strategy:** Keep existing controller dependencies mocked for request forwarding/token/session-revocation assertions. Add one minimal service-level integration regression using a unique temporary directory, a real file-backed `better-sqlite3` database, real `RestoreService`, and real `FilesystemTransactionService`; mock only `ConfigService` and the live `DataSource.query` verification boundary as needed. Arrange a live database with the migrated last-admin triggers, one sole admin, pre-restore challenge/blog/setting rows, and live uploads; stage a valid archive containing that same admin plus different table values and replacement challenge/icon files; commit; then open the swapped file independently and assert archived counts/settings, exact upload replacement, and both last-admin triggers still exist and reject deleting the sole restored admin. Add one focused invalid-archive invariant case with no admin and assert `SYSTEM_RESTORE_ROLLED_BACK` plus unchanged live database/uploads. Clean all temporary files in `afterEach`/`afterAll`; no UI, network, visual checks, shared `/data`, or large mock environment.
|
|
||||||
- **Execution:** Follow red-green-refactor: first run the focused backend spec to confirm the `LAST_ADMIN` failure, implement the smallest restore-service correction, rerun the focused spec, then run `npm run test:backend`, root `npm test`, `npm --workspace backend run build`, and the repository lint/typecheck commands if present (the current package scripts expose build but no dedicated lint/typecheck command).
|
|
||||||
@@ -0,0 +1,69 @@
|
|||||||
|
# Implementation Plan: Job 919 — Crash-Safe Admin Database Backup/Restore and Danger Zone Recovery
|
||||||
|
|
||||||
|
## 1. Architectural Reconnaissance
|
||||||
|
- **Status:** The requested Job is **not fully implemented**. Restore currently has an in-process rollback handle only; a SIGKILL during or after the multi-path swap can leave the database and upload directory in different generations, and startup has no recovery sweep for rollback artifacts. Relevant existing implementation is in `backend/src/modules/admin/system/restore.service.ts` and `backend/src/modules/admin/system/filesystem-transaction.service.ts`.
|
||||||
|
- **Codebase style & conventions:** TypeScript monorepo with a NestJS 10 backend, Angular 17 standalone frontend, TypeORM over `better-sqlite3`, synchronous local filesystem primitives for staged archive files and rename operations, async Nest services/controllers, Zod schemas through `ZodValidationPipe`, and `ApiError`/global exception-envelope handling. `main.ts` explicitly initializes the database before listening. Controllers are thin and delegate destructive work to services.
|
||||||
|
- **Data Layer:** File-backed SQLite configured through `DATABASE_PATH`, TypeORM/`DataSource` for the live application connection, migrations with WAL enabled by `DatabaseInitService`, and direct `better-sqlite3` use in restore for an offline candidate database. Application uploads live below `UPLOAD_DIR`; backups encode the complete upload tree as base64 entries. No database schema migration is required for this fix unless the implementer elects to persist a transaction journal; the preferred design uses a durable filesystem transaction manifest instead of application tables so recovery works before ordinary application access.
|
||||||
|
- **Test Framework & Structure:** Jest 29 with `ts-jest`, two root projects (`backend` Node and `frontend` jsdom), and all tests in the dedicated `/repo/tests` tree. Backend restore coverage is concentrated in `tests/backend/admin-system-restore-commit.spec.ts`; transaction behavior currently has no focused dedicated test. Tests must remain CLI-runnable through the root `npm test`, use isolated per-test filesystem/database roots, and avoid UI/manual verification. Add only focused tests for crash/restart recovery, rollback-artifact cleanup, and coherent post-recovery state.
|
||||||
|
- **Required Tools & Dependencies:** Existing Node/npm toolchain, TypeScript, Jest, `better-sqlite3`, NestJS, TypeORM, and Node `fs`/`path`/`crypto` APIs are sufficient. No new runtime package is required. Do not add a database or external queue. `setup.sh` already creates the normal upload directories, installs dependencies, rebuilds `better-sqlite3`, and builds both workspaces; update it only if the chosen implementation introduces a required OS utility (not expected). If using subprocess-based crash simulation, use the existing Node runtime and Jest rather than adding a global CLI.
|
||||||
|
|
||||||
|
## 2. Impacted Files
|
||||||
|
- **To Modify:**
|
||||||
|
- `backend/src/modules/admin/system/filesystem-transaction.service.ts` — replace the ephemeral, in-memory-only swap state with a durable transaction manifest/state machine; make recovery deterministic across process death, preserve pair ordering, and clean rollback/staged artifacts only after the transaction reaches a durable terminal state.
|
||||||
|
- `backend/src/modules/admin/system/restore.service.ts` — create and finalize a restore transaction through the durable transaction service, avoid treating the database and uploads as independently committed, and ensure candidate/WAL/SHM/staging artifacts are included in recovery and cleanup decisions.
|
||||||
|
- `backend/src/database/database-init.service.ts` — invoke startup recovery before migrations/normal schema verification and fail closed if a transaction cannot be safely resolved; ensure recovery runs before HTTP traffic is accepted.
|
||||||
|
- `backend/src/main.ts` — only if needed to expose the recovery call in the explicit bootstrap sequence; preserve the documented `DatabaseInitService.init()`-before-`listen()` ordering and do not serve uploads before recovery completes.
|
||||||
|
- `backend/src/config/env.schema.ts` — only if a configurable transaction-manifest/stale-transaction retention setting is introduced; otherwise no change is needed.
|
||||||
|
- `backend/src/common/utils/upload.ts` — only if shared helpers are needed for locating the transaction journal/staging root; reuse `resolveSystemStagingDir` and keep system staging outside public uploads.
|
||||||
|
- `tests/backend/admin-system-restore-commit.spec.ts` — extend the existing minimal restore tests for a simulated interruption between pair swaps and verify the database/upload generation is coherent after recovery.
|
||||||
|
- `tests/backend/filesystem-transaction.spec.ts` — create a focused unit/integration test file in the dedicated backend test folder for durable manifest transitions, recovery of interrupted swaps, idempotent cleanup, and refusal to guess when the manifest is incomplete.
|
||||||
|
- `tests/backend/database-init-recovery.spec.ts` — create a focused startup-recovery test that runs the initialization recovery path against isolated file-backed data and checks orphan rollback directories are removed or safely restored before normal startup.
|
||||||
|
- `docs/api/admin-system.md` — update restore semantics/error behavior if the externally observable contract changes, especially recovery guarantees and startup handling.
|
||||||
|
- `docs/guides/admin-system-page.md` — update operational guidance only if restore failure/startup recovery messaging or operator cleanup expectations change.
|
||||||
|
- **To Create:**
|
||||||
|
- `backend/src/modules/admin/system/transaction-recovery.service.ts` (recommended) — a small startup/recovery coordinator if keeping recovery logic out of `FilesystemTransactionService`; it should discover manifests, classify states, resolve only provably safe outcomes, and report/fail closed on ambiguous state.
|
||||||
|
- `tests/backend/filesystem-transaction.spec.ts` — dedicated focused transaction tests.
|
||||||
|
- `tests/backend/database-init-recovery.spec.ts` — dedicated focused startup recovery tests.
|
||||||
|
- A durable manifest file format under the configured system staging/data directory at runtime, not a committed fixture; no persistent repository data should be added outside code/tests. Test-generated databases and upload trees should be created in unique temporary directories, or under `/data` only when a successor job must reuse them.
|
||||||
|
|
||||||
|
## 3. Proposed Changes
|
||||||
|
Explain the exact implementation logic step-by-step:
|
||||||
|
|
||||||
|
1. **Database / Schema Migration:**
|
||||||
|
- Do not add application tables or migrations for the transaction protocol. The failure occurs during filesystem replacement and can happen before a database connection is usable; recovery metadata must therefore be independent of SQLite and durable on the same filesystem as the database/upload paths.
|
||||||
|
- Define a versioned manifest containing a transaction id, operation name, ordered swap pairs, absolute or validated repository-relative live/staged/rollback paths, intended generation, and an explicit phase such as `prepared`, `live-snapshots-created`, `replacements-installed`, `verified`, `committed`, or `rollback-required`.
|
||||||
|
- Write the manifest atomically (`temporary manifest` → flush/close as supported → rename into the canonical manifest path) before the first live path is moved. Update it atomically after every irreversible filesystem step. Use a unique transaction directory under `resolveSystemStagingDir`, never under the public upload tree, and include database sidecars (`-wal`, `-shm`) in the database replacement protocol or explicitly normalize/remove them so an old WAL cannot be paired with a new main DB.
|
||||||
|
- Persist enough information to distinguish these cases after a kill: all old paths available and replacements incomplete means restore the entire old generation; every replacement installed and candidate verification recorded means finish/retain the new generation; any mixed/ambiguous state means do not delete either generation and fail startup closed with an actionable log/error. The implementation must never simply delete an unknown rollback directory.
|
||||||
|
- Ensure durability ordering: manifest exists before mutation; each rename/copy result is recorded before advancing; the terminal commit marker is written only after both database and upload generations are present and verified; only then remove rollback and staged artifacts. Cleanup is best effort after the terminal marker and must not erase the only known recoverable generation.
|
||||||
|
|
||||||
|
2. **Backend Logic & APIs:**
|
||||||
|
- Refactor `FilesystemTransactionService.stageSwap()` into a crash-recoverable protocol while retaining its public rollback/commit behavior for existing callers. Keep pair processing deterministic and perform rollback in reverse order for in-process failures, but make the manifest authoritative so a process restart can continue recovery.
|
||||||
|
- Add a public startup recovery method (either on `FilesystemTransactionService` or the recommended `TransactionRecoveryService`) that scans only the configured transaction root for unfinished manifests and stale restore rollback artifacts. It must be idempotent: repeated startup attempts converge to the same coherent state and do not recreate or delete unrelated files.
|
||||||
|
- For a manifest whose old database and old upload tree are both available, restore both old paths as one generation. For a manifest whose new database and new upload tree are both fully installed and whose post-swap verification marker is durable, finalize the new generation and remove old rollback artifacts. For the observed hybrid case (old DB plus new uploads, or vice versa), select the old generation only when both old paths are present; otherwise select the fully verified new generation only when both new paths are present; otherwise fail closed rather than producing a third hybrid.
|
||||||
|
- Add explicit validation of the recovered database and upload generation before finalization. Database validation should open the selected SQLite file read-only, run the existing foreign-key/integrity checks and required-admin invariant, and account for WAL/SHM sidecars. Upload validation should ensure the selected tree is a complete directory replacement, not a merge; do not copy individual files from one generation into the other. Any orphaned rollback/staged directory associated with a transaction should be removed only after the selected generation is installed and the durable commit/rollback marker is written.
|
||||||
|
- Change `RestoreService.commitRestore()` to use the durable transaction API and remove unused `liveBackupDb` behavior. Candidate DB construction should remain offline and candidate verification should happen before the swap whenever possible; `afterSwap` must verify the selected live DB without relying on an already-open TypeORM connection that may still point at the old inode. The result should report success only after the transaction is durably committed.
|
||||||
|
- Handle process-level concurrency by preventing two destructive filesystem transactions from running simultaneously. Reuse the existing operation-in-progress/error conventions if present; otherwise add a small lock/manifest guard in the transaction root. A stale lock must be recoverable only through manifest inspection, not by blindly deleting it.
|
||||||
|
- Keep controller routes and confirmation-token contracts unchanged. Restore success should continue to revoke refresh sessions; restore rollback errors should continue to map to `SYSTEM_RESTORE_ROLLED_BACK`. If startup recovery cannot establish a coherent state, fail startup before `app.listen()` and log the transaction id/phase without exposing secrets or raw archive contents.
|
||||||
|
- Update backup traversal if necessary to exclude all transaction metadata/rollback directories, not just `.staging`, so an interrupted restore can never be re-archived as application upload content. The exclusion must be based on exact reserved names/prefixes and remain scoped to `UPLOAD_DIR`.
|
||||||
|
- No frontend feature work is required: the existing admin restore UI already reports `SYSTEM_RESTORE_ROLLED_BACK` and logs the user out after a successful restore. Only update frontend pure helpers/tests if the backend introduces a new user-visible error code; prefer retaining existing codes.
|
||||||
|
|
||||||
|
3. **Frontend UI Integration:**
|
||||||
|
- No component or route changes are planned. The existing `SystemService` calls validation and commit endpoints, and the admin system page already handles success/failure. Verify that any recovery failure remains represented by the existing restore-rollback/startup failure behavior; do not add visual tests or UI infrastructure.
|
||||||
|
|
||||||
|
4. **Operational/startup integration:**
|
||||||
|
- Invoke recovery before `DatabaseInitService.ensureWalMode()`, migrations, seed verification, static upload mounting, and `app.listen()`. The ordering is critical because opening the old/new database or serving uploads before resolving the manifest can recreate the mixed state.
|
||||||
|
- Preserve configured path safety: resolve `DATABASE_PATH`, `UPLOAD_DIR`, and staging paths once through `ConfigService`; reject manifests whose paths escape the configured data roots; never follow symlinks or accept arbitrary manifest paths from the request payload.
|
||||||
|
- Document that recovery is automatic on restart, that an ambiguous transaction prevents startup rather than silently deleting data, and that an operator must preserve the transaction directory for forensic/manual recovery.
|
||||||
|
|
||||||
|
## 4. Test Strategy
|
||||||
|
- **Target Unit Test File:** `tests/backend/filesystem-transaction.spec.ts` for the transaction state machine and recovery behavior; `tests/backend/database-init-recovery.spec.ts` for startup integration; extend `tests/backend/admin-system-restore-commit.spec.ts` with one end-to-end restore coherence assertion. Keep all tests in `/repo/tests/backend`, never alongside source.
|
||||||
|
- **Mocking Strategy:** Use unique file-backed temporary directories per test (or `/data` only for intentionally persistent successor-job fixtures), real Node filesystem operations, and small real SQLite databases created with `better-sqlite3`. Mock only unrelated Nest/DataSource boundaries and time/randomness where deterministic manifest names are needed. Do not mock the filesystem for the core recovery tests because the bug is specifically a crash/interrupted-rename condition. Avoid requiring a browser, UI, external service, Docker, or a large seeded database.
|
||||||
|
- **Focused cases:**
|
||||||
|
1. Successful restore writes a terminal commit marker and leaves exactly the restored DB/upload generation with no rollback or stage artifacts.
|
||||||
|
2. Simulated interruption after the database pair is swapped but before uploads are swapped leaves a durable manifest; startup recovery restores the old database and old uploads together, removes both new/staged artifacts, and never leaves the backup upload file in the live tree.
|
||||||
|
3. Simulated interruption after both pairs are swapped but before cleanup allows startup recovery to retain the complete new generation and remove old rollback artifacts.
|
||||||
|
4. The observed hybrid layout (old live DB plus new live uploads plus old rollback upload tree) is resolved to the old generation when both old rollback paths are available; assertions cover DB rows, referenced upload files, absence of backup-only orphan files, and cleanup of rollback directories.
|
||||||
|
5. A manifest with neither a complete old generation nor a verified complete new generation causes startup recovery to fail closed and preserves artifacts for operator recovery.
|
||||||
|
6. Recovery is idempotent when invoked twice and does not alter unrelated uploads or databases.
|
||||||
|
7. Existing no-admin restore rejection and successful restore coverage remain green, proving the new transaction protocol does not weaken the `LAST_ADMIN` invariant or existing API contract.
|
||||||
|
- **CLI execution:** The existing root `npm test` runs both Jest projects in one command. The implementer must run the focused backend tests and then root lint/typecheck commands if defined by the repository; no new dependency or test command should be required. If the test runner executes suites in parallel, each suite must use a unique filesystem root and close all SQLite handles/resources in `afterEach`/`afterAll`.
|
||||||
@@ -1,6 +1,9 @@
|
|||||||
import { Injectable, Logger, OnApplicationBootstrap } from '@nestjs/common';
|
import { Injectable, Logger, OnApplicationBootstrap } from '@nestjs/common';
|
||||||
import { ConfigService } from '@nestjs/config';
|
import { ConfigService } from '@nestjs/config';
|
||||||
import { DataSource } from 'typeorm';
|
import { DataSource } from 'typeorm';
|
||||||
|
import * as path from 'path';
|
||||||
|
import { FilesystemTransactionService } from '../modules/admin/system/filesystem-transaction.service';
|
||||||
|
import { resolveSystemStagingDir } from '../common/utils/upload';
|
||||||
|
|
||||||
@Injectable()
|
@Injectable()
|
||||||
export class DatabaseInitService implements OnApplicationBootstrap {
|
export class DatabaseInitService implements OnApplicationBootstrap {
|
||||||
@@ -10,22 +13,28 @@ export class DatabaseInitService implements OnApplicationBootstrap {
|
|||||||
constructor(
|
constructor(
|
||||||
private readonly dataSource: DataSource,
|
private readonly dataSource: DataSource,
|
||||||
private readonly config: ConfigService,
|
private readonly config: ConfigService,
|
||||||
|
private readonly fsTx: FilesystemTransactionService,
|
||||||
) {}
|
) {}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Explicit, awaitable database initialization.
|
* Explicit, awaitable database initialization.
|
||||||
*
|
*
|
||||||
|
* - Recovers any interrupted filesystem swap transactions (durable
|
||||||
|
* manifest protocol) BEFORE touching the database or uploads.
|
||||||
* - Initializes the DataSource (idempotent — TypeORM returns the existing
|
* - Initializes the DataSource (idempotent — TypeORM returns the existing
|
||||||
* connection if already initialized).
|
* connection if already initialized).
|
||||||
* - Runs all pending migrations in order.
|
* - Runs all pending migrations in order.
|
||||||
* - Verifies that the expected schema and seeded data are present.
|
* - Verifies that the expected schema and seeded data are present.
|
||||||
*
|
*
|
||||||
* `main.ts` calls this BEFORE `app.listen()` so the HTTP server never
|
* `main.ts` calls this BEFORE `app.listen()` so the HTTP server never
|
||||||
* accepts traffic until the DB is fully migrated and seeded.
|
* accepts traffic until the DB is fully migrated and seeded, and so the
|
||||||
|
* server never boots in a half-swapped state.
|
||||||
*/
|
*/
|
||||||
async init(): Promise<void> {
|
async init(): Promise<void> {
|
||||||
if (this.initialized) return;
|
if (this.initialized) return;
|
||||||
|
|
||||||
|
await this.recoverInterruptedSystemTransactions();
|
||||||
|
|
||||||
if (!this.dataSource.isInitialized) {
|
if (!this.dataSource.isInitialized) {
|
||||||
await this.dataSource.initialize();
|
await this.dataSource.initialize();
|
||||||
}
|
}
|
||||||
@@ -57,6 +66,63 @@ export class DatabaseInitService implements OnApplicationBootstrap {
|
|||||||
await this.init();
|
await this.init();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Resolve any durable filesystem transaction that was interrupted by a
|
||||||
|
* process death during a restore / danger-zone swap. Recovery runs
|
||||||
|
* before the TypeORM connection is opened so the application can never
|
||||||
|
* observe a half-applied swap.
|
||||||
|
*
|
||||||
|
* Steps:
|
||||||
|
* 1. Resolve every pending manifest under `<system-staging>` to a
|
||||||
|
* coherent on-disk state (restore old generation or finalize new).
|
||||||
|
* Throws and aborts startup if any manifest is ambiguous.
|
||||||
|
* 2. Sweep unowned rollback / restore-stage / wipe-stage artifacts
|
||||||
|
* that linger next to the live database and uploads directories,
|
||||||
|
* so a pre-existing half-finished state (such as the Job 919
|
||||||
|
* incident report) is also cleaned up.
|
||||||
|
*/
|
||||||
|
private async recoverInterruptedSystemTransactions(): Promise<void> {
|
||||||
|
const databasePath = path.resolve(
|
||||||
|
this.config.get<string>('DATABASE_PATH', './data/db.sqlite'),
|
||||||
|
);
|
||||||
|
const uploadDir = path.resolve(
|
||||||
|
this.config.get<string>('UPLOAD_DIR', './data/uploads'),
|
||||||
|
);
|
||||||
|
const stagingDir = resolveSystemStagingDir(this.config);
|
||||||
|
|
||||||
|
try {
|
||||||
|
const report = this.fsTx.recoverTransactions(stagingDir);
|
||||||
|
if (report.transactionsScanned > 0) {
|
||||||
|
this.logger.log(
|
||||||
|
`Filesystem transaction recovery: scanned=${report.transactionsScanned} resolved=${report.resolved} aborted=${report.aborted.length} cleaned=${report.cleaned.length}`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
} catch (err) {
|
||||||
|
this.logger.error(
|
||||||
|
`Filesystem transaction recovery failed: ${(err as Error).message}`,
|
||||||
|
);
|
||||||
|
throw err;
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
const cleaned = this.fsTx.sweepUnownedArtifactsNearLivePaths(
|
||||||
|
[databasePath, uploadDir],
|
||||||
|
{ stagingDir },
|
||||||
|
);
|
||||||
|
if (cleaned.length > 0) {
|
||||||
|
this.logger.log(
|
||||||
|
`Removed unowned system artifacts on startup: ${cleaned.join(', ')}`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
} catch (err) {
|
||||||
|
// Sweeps are best-effort; if they fail we never abort startup
|
||||||
|
// because leftover artifacts do not break read access.
|
||||||
|
this.logger.warn(
|
||||||
|
`Unowned-artifact sweep failed: ${(err as Error).message}`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Read PRAGMA journal_mode and switch to WAL if not already enabled.
|
* Read PRAGMA journal_mode and switch to WAL if not already enabled.
|
||||||
* Persisted for file-backed databases; idempotent and safe on every
|
* Persisted for file-backed databases; idempotent and safe on every
|
||||||
|
|||||||
@@ -23,6 +23,7 @@ import { AddUserDeleteCascades1700000000700 } from './migrations/1700000000700-A
|
|||||||
import { AddUserLastAdminTriggers1700000000800 } from './migrations/1700000000800-AddUserLastAdminTriggers';
|
import { AddUserLastAdminTriggers1700000000800 } from './migrations/1700000000800-AddUserLastAdminTriggers';
|
||||||
import { AddAdminOperationTokens1700000000900 } from './migrations/1700000000900-AddAdminOperationTokens';
|
import { AddAdminOperationTokens1700000000900 } from './migrations/1700000000900-AddAdminOperationTokens';
|
||||||
import { DatabaseInitService } from './database-init.service';
|
import { DatabaseInitService } from './database-init.service';
|
||||||
|
import { FilesystemTransactionModule } from '../modules/admin/system/filesystem-transaction.module';
|
||||||
|
|
||||||
const ENTITIES = [
|
const ENTITIES = [
|
||||||
UserEntity,
|
UserEntity,
|
||||||
@@ -52,6 +53,7 @@ const MIGRATIONS = [
|
|||||||
@Global()
|
@Global()
|
||||||
@Module({
|
@Module({
|
||||||
imports: [
|
imports: [
|
||||||
|
FilesystemTransactionModule,
|
||||||
TypeOrmModule.forRootAsync({
|
TypeOrmModule.forRootAsync({
|
||||||
imports: [ConfigModule],
|
imports: [ConfigModule],
|
||||||
inject: [ConfigService],
|
inject: [ConfigService],
|
||||||
|
|||||||
@@ -5,7 +5,7 @@ import { AdminSystemController } from './admin-system.controller';
|
|||||||
import { BackupService } from './backup.service';
|
import { BackupService } from './backup.service';
|
||||||
import { RestoreService } from './restore.service';
|
import { RestoreService } from './restore.service';
|
||||||
import { DangerZoneService } from './danger-zone.service';
|
import { DangerZoneService } from './danger-zone.service';
|
||||||
import { FilesystemTransactionService } from './filesystem-transaction.service';
|
import { FilesystemTransactionModule } from './filesystem-transaction.module';
|
||||||
import { ConfirmationTokenService } from './confirmation-token.service';
|
import { ConfirmationTokenService } from './confirmation-token.service';
|
||||||
import { AuthModule } from '../../auth/auth.module';
|
import { AuthModule } from '../../auth/auth.module';
|
||||||
|
|
||||||
@@ -13,20 +13,19 @@ import { AuthModule } from '../../auth/auth.module';
|
|||||||
imports: [
|
imports: [
|
||||||
TypeOrmModule.forFeature([AdminOperationTokenEntity]),
|
TypeOrmModule.forFeature([AdminOperationTokenEntity]),
|
||||||
AuthModule,
|
AuthModule,
|
||||||
|
FilesystemTransactionModule,
|
||||||
],
|
],
|
||||||
controllers: [AdminSystemController],
|
controllers: [AdminSystemController],
|
||||||
providers: [
|
providers: [
|
||||||
BackupService,
|
BackupService,
|
||||||
RestoreService,
|
RestoreService,
|
||||||
DangerZoneService,
|
DangerZoneService,
|
||||||
FilesystemTransactionService,
|
|
||||||
ConfirmationTokenService,
|
ConfirmationTokenService,
|
||||||
],
|
],
|
||||||
exports: [
|
exports: [
|
||||||
BackupService,
|
BackupService,
|
||||||
RestoreService,
|
RestoreService,
|
||||||
DangerZoneService,
|
DangerZoneService,
|
||||||
FilesystemTransactionService,
|
|
||||||
ConfirmationTokenService,
|
ConfirmationTokenService,
|
||||||
],
|
],
|
||||||
})
|
})
|
||||||
|
|||||||
@@ -132,13 +132,39 @@ export class BackupService {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Subdirectory/file names that are part of the platform's own internal
|
||||||
|
* staging machinery rather than user-uploaded content. They MUST be
|
||||||
|
* excluded from backups so a partially failed swap cannot end up as
|
||||||
|
* application upload content in the resulting archive.
|
||||||
|
*/
|
||||||
|
private static readonly EXCLUDED_UPLOAD_ENTRIES: ReadonlySet<string> = new Set([
|
||||||
|
'.staging',
|
||||||
|
]);
|
||||||
|
private static readonly EXCLUDED_UPLOAD_PATTERNS: RegExp[] = [
|
||||||
|
/^\.?[^/]+\.rollback-[\w-]+$/, // e.g. .uploads.rollback-1784819568666-n0w53hxp
|
||||||
|
/^.+\.restore-stage-[\w-]+$/, // e.g. uploads.restore-stage-12345-abcdef
|
||||||
|
/^\.challenges\.wipe-stage-[\w-]+$/, // e.g. .challenges.wipe-stage-12345
|
||||||
|
];
|
||||||
|
|
||||||
|
private isExcludedUploadEntry(name: string): boolean {
|
||||||
|
if (BackupService.EXCLUDED_UPLOAD_ENTRIES.has(name)) return true;
|
||||||
|
for (const pattern of BackupService.EXCLUDED_UPLOAD_PATTERNS) {
|
||||||
|
if (pattern.test(name)) return true;
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
private async collectUploads(): Promise<BackupFileEntry[]> {
|
private async collectUploads(): Promise<BackupFileEntry[]> {
|
||||||
const out: BackupFileEntry[] = [];
|
const out: BackupFileEntry[] = [];
|
||||||
if (!fs.existsSync(this.uploadDir)) return out;
|
if (!fs.existsSync(this.uploadDir)) return out;
|
||||||
const visit = (dir: string) => {
|
const visit = (dir: string) => {
|
||||||
const entries = fs.readdirSync(dir, { withFileTypes: true });
|
const entries = fs.readdirSync(dir, { withFileTypes: true });
|
||||||
for (const entry of entries) {
|
for (const entry of entries) {
|
||||||
if (entry.name === '.staging') continue;
|
if (BackupService.EXCLUDED_UPLOAD_ENTRIES.has(entry.name)) continue;
|
||||||
|
// For top-level + first-level descendants only, drop rollback-style artifacts.
|
||||||
|
const relative = path.relative(this.uploadDir, path.join(dir, entry.name)).split(path.sep);
|
||||||
|
if (relative.length <= 1 && this.isExcludedUploadEntry(entry.name)) continue;
|
||||||
const target = path.join(dir, entry.name);
|
const target = path.join(dir, entry.name);
|
||||||
if (entry.isDirectory()) {
|
if (entry.isDirectory()) {
|
||||||
visit(target);
|
visit(target);
|
||||||
|
|||||||
@@ -0,0 +1,15 @@
|
|||||||
|
import { Module } from '@nestjs/common';
|
||||||
|
import { FilesystemTransactionService } from './filesystem-transaction.service';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Hosts the durable filesystem swap service so it can be reused by both
|
||||||
|
* the admin system operations (which perform the swap) and database
|
||||||
|
* initialization (which performs the startup recovery sweep). Keeping
|
||||||
|
* the provider in its own module avoids a circular dependency between
|
||||||
|
* DatabaseModule and AdminSystemModule.
|
||||||
|
*/
|
||||||
|
@Module({
|
||||||
|
providers: [FilesystemTransactionService],
|
||||||
|
exports: [FilesystemTransactionService],
|
||||||
|
})
|
||||||
|
export class FilesystemTransactionModule {}
|
||||||
@@ -5,32 +5,45 @@ import { ApiError } from '../../../common/errors/api-error';
|
|||||||
import { ERROR_CODES } from '../../../common/errors/error-codes';
|
import { ERROR_CODES } from '../../../common/errors/error-codes';
|
||||||
|
|
||||||
export interface SwapPair {
|
export interface SwapPair {
|
||||||
|
/** Absolute path of the live filesystem location that will be swapped out. */
|
||||||
livePath: string;
|
livePath: string;
|
||||||
|
/** Absolute path of the staged replacement that will be swapped in. */
|
||||||
stagedPath: string;
|
stagedPath: string;
|
||||||
|
/**
|
||||||
|
* If true, the pair participates in the manifest's "legacy"/"primary"
|
||||||
|
* generation tally. The database and uploads are typically `true`, while
|
||||||
|
* sidecar files (e.g. `-wal`, `-shm`) are NOT — they are tracked as
|
||||||
|
* auxiliary, so a partial sidecar swap does not mark the parent
|
||||||
|
* generation as "incomplete".
|
||||||
|
*/
|
||||||
|
contributesToGeneration?: boolean;
|
||||||
|
/**
|
||||||
|
* Auxiliary sidecar suffixes to move with the live path. When the live
|
||||||
|
* path is replaced, any matching sidecars (e.g. `<live>.wal`) are
|
||||||
|
* moved into the rollback dir and back. When the staged replacement
|
||||||
|
* arrives, stale sidecars are removed so the new file is not paired
|
||||||
|
* with an old WAL.
|
||||||
|
*/
|
||||||
|
sidecarSuffixes?: string[];
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface SwapRollbackHandle {
|
export interface SwapRollbackHandle {
|
||||||
rolledBack: boolean;
|
rolledBack: boolean;
|
||||||
pairs: Array<{ livePath: string; rollbackPath: string; stagedPath: string }>;
|
committed: boolean;
|
||||||
|
/** Public identifier of the durable transaction; only present when recoverability is enabled. */
|
||||||
|
transactionId?: string;
|
||||||
|
/**
|
||||||
|
* The exact per-transaction manifest directory created under the system
|
||||||
|
* staging root. Stored on the handle so that `commit()` and `rollback()`
|
||||||
|
* can clean the right record without re-deriving paths.
|
||||||
|
*/
|
||||||
|
manifestDir?: string;
|
||||||
|
pairs: Array<{ livePath: string; rollbackPath: string; stagedPath: string; sidecarSuffixes: string[] }>;
|
||||||
}
|
}
|
||||||
|
|
||||||
interface WorkerHooks {
|
interface WorkerHooks {
|
||||||
/**
|
|
||||||
* Optional hook that runs once after `prepare()` (file staging) but
|
|
||||||
* before the live swap (e.g. database rebuild). Throw to abort the
|
|
||||||
* operation and immediately trigger rollback.
|
|
||||||
*/
|
|
||||||
beforeSwap?: () => Promise<void>;
|
beforeSwap?: () => Promise<void>;
|
||||||
/**
|
|
||||||
* Optional hook that runs after the swap is registered but before it
|
|
||||||
* is considered durable (e.g. final database open, verification).
|
|
||||||
* Throw to trigger rollback.
|
|
||||||
*/
|
|
||||||
afterSwap?: () => Promise<void>;
|
afterSwap?: () => Promise<void>;
|
||||||
/**
|
|
||||||
* Optional hook that runs after finalization completes successfully.
|
|
||||||
* Throw to trigger rollback even from a successful operation.
|
|
||||||
*/
|
|
||||||
onCommit?: () => Promise<void>;
|
onCommit?: () => Promise<void>;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -41,54 +54,135 @@ export interface StageSwapInput {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Swap a set of file/directory pairs in a crash-safe way. Each pair moves
|
* Transaction phase values written to the durable manifest. Recovery
|
||||||
* the existing live path to a rollback location, then renames the staged
|
* inspects these to decide whether the system is in a coherent state.
|
||||||
* replacement into place. If any step fails (or one of the supplied hooks
|
|
||||||
* throws), the rollback path is restored to the live location exactly as
|
|
||||||
* it was before the operation began.
|
|
||||||
*
|
*
|
||||||
* The class assumes all live/staged paths share the same filesystem so
|
* Phase progression under success:
|
||||||
* `rename` is atomic; cross-filesystem moves fall back to copy+unlink.
|
* prepared → live-snapshotted → replacements-installed → verified → committed
|
||||||
|
*
|
||||||
|
* Phase progression under in-process failure or rollback():
|
||||||
|
* any → rolled-back
|
||||||
|
*/
|
||||||
|
export type TransactionPhase =
|
||||||
|
| 'prepared'
|
||||||
|
| 'live-snapshotted'
|
||||||
|
| 'replacements-installed'
|
||||||
|
| 'verified'
|
||||||
|
| 'committed'
|
||||||
|
| 'rolled-back';
|
||||||
|
|
||||||
|
export interface TransactionManifest {
|
||||||
|
schemaVersion: 1;
|
||||||
|
transactionId: string;
|
||||||
|
operation: string;
|
||||||
|
createdAt: string;
|
||||||
|
updatedAt: string;
|
||||||
|
phase: TransactionPhase;
|
||||||
|
pairs: Array<{
|
||||||
|
livePath: string;
|
||||||
|
rollbackPath: string;
|
||||||
|
stagedPath: string;
|
||||||
|
contributesToGeneration: boolean;
|
||||||
|
sidecarSuffixes: string[];
|
||||||
|
}>;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface RecoveryReport {
|
||||||
|
transactionsScanned: number;
|
||||||
|
resolved: number;
|
||||||
|
aborted: string[];
|
||||||
|
cleaned: string[];
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Crash-safe filesystem swap with a durable manifest.
|
||||||
|
*
|
||||||
|
* Each `stageSwap()` writes a small JSON manifest under the configured
|
||||||
|
* system staging root. The manifest records the live/staged/rollback
|
||||||
|
* paths and the current phase. If the process is killed mid-swap, the
|
||||||
|
* manifest is still on disk so startup recovery can resolve the
|
||||||
|
* transaction deterministically:
|
||||||
|
*
|
||||||
|
* - If the old generation (live + rollback trees) is fully present,
|
||||||
|
* restore it.
|
||||||
|
* - If the new generation (staged → live) is fully present AND was
|
||||||
|
* marked `verified`, finalize it and remove the rollback copy.
|
||||||
|
* - Otherwise, fail closed: the manifest is left on disk for an
|
||||||
|
* operator, and startup throws a `SYSTEM_RESTORE_RECOVERY_PENDING`
|
||||||
|
* error.
|
||||||
|
*
|
||||||
|
* The class also sweeps stale `*.rollback-*` artifacts in the configured
|
||||||
|
* live paths that are NOT associated with any live manifest (e.g. from
|
||||||
|
* a previous Job 919 incident) so the recovery itself never silently
|
||||||
|
* removes unowned files.
|
||||||
*/
|
*/
|
||||||
@Injectable()
|
@Injectable()
|
||||||
export class FilesystemTransactionService {
|
export class FilesystemTransactionService {
|
||||||
private readonly logger = new Logger(FilesystemTransactionService.name);
|
private readonly logger = new Logger(FilesystemTransactionService.name);
|
||||||
|
|
||||||
|
/** Manifest file name inside each per-transaction directory. */
|
||||||
|
static MANIFEST_FILENAME = 'manifest.json';
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Apply the swap. Returns a rollback handle; call `commit()` after the
|
* Apply the swap. Returns a rollback handle.
|
||||||
* rest of the operation (including database swap) is durable, or
|
*
|
||||||
* `rollback()` on any failure.
|
* Behavior matches the original (in-memory) contract: success means
|
||||||
|
* the staged replacement now lives at `livePath`; failure rolls the
|
||||||
|
* pair back. The difference is that the swap is now also recorded in a
|
||||||
|
* durable manifest so a crashed process can finish or undo the
|
||||||
|
* operation on next startup.
|
||||||
|
*
|
||||||
|
* If the caller's environment doesn't configure a system staging root,
|
||||||
|
* the method falls back to the old in-memory behavior (still safe; just
|
||||||
|
* not recoverable across process death).
|
||||||
*/
|
*/
|
||||||
async stageSwap(input: StageSwapInput): Promise<SwapRollbackHandle> {
|
async stageSwap(
|
||||||
|
input: StageSwapInput,
|
||||||
|
opts: { stagingDir?: string } = {},
|
||||||
|
): Promise<SwapRollbackHandle> {
|
||||||
|
const txDir = opts.stagingDir ? this.ensureTxDir(opts.stagingDir) : null;
|
||||||
|
const manifest: TransactionManifest | null = txDir
|
||||||
|
? this.makeInitialManifest(input, txDir)
|
||||||
|
: null;
|
||||||
|
|
||||||
const stamp = `${Date.now()}-${Math.random().toString(36).slice(2, 10)}`;
|
const stamp = `${Date.now()}-${Math.random().toString(36).slice(2, 10)}`;
|
||||||
const rolled: SwapRollbackHandle['pairs'] = [];
|
const rolled: SwapRollbackHandle['pairs'] = [];
|
||||||
const cleanupRollbackOnly: string[] = [];
|
|
||||||
try {
|
try {
|
||||||
// 1. Stage: snapshot each live path to a rollback location.
|
// 1. Stage: snapshot each live path to a rollback location.
|
||||||
for (const pair of input.pairs) {
|
for (const pair of input.pairs) {
|
||||||
if (!pair.livePath) throw new ApiError(ERROR_CODES.SYSTEM_RESTORE_FAILED, 'Missing live path', 500);
|
if (!pair.livePath) throw new ApiError(ERROR_CODES.SYSTEM_RESTORE_FAILED, 'Missing live path', 500);
|
||||||
if (!pair.stagedPath) throw new ApiError(ERROR_CODES.SYSTEM_RESTORE_FAILED, 'Missing staged path', 500);
|
if (!pair.stagedPath) throw new ApiError(ERROR_CODES.SYSTEM_RESTORE_FAILED, 'Missing staged path', 500);
|
||||||
const rollbackPath = FilesystemTransactionService.makeRollbackPath(pair.livePath, stamp);
|
const rollbackPath = FilesystemTransactionService.makeRollbackPath(pair.livePath, stamp);
|
||||||
if (fs.existsSync(pair.livePath)) {
|
const sidecarSuffixes = pair.sidecarSuffixes ?? [];
|
||||||
fs.renameSync(pair.livePath, rollbackPath);
|
|
||||||
cleanupRollbackOnly.push(rollbackPath);
|
// Move any sidecar files (e.g. -wal/-shm) into the rollback dir first so they cannot
|
||||||
} else if (fs.existsSync(pair.stagedPath)) {
|
// be left paired with the swapped-in live path.
|
||||||
// Live path absent but staged exists — delete staged so we fail closed.
|
for (const suffix of sidecarSuffixes) {
|
||||||
try {
|
const liveSidecar = `${pair.livePath}${suffix}`;
|
||||||
fs.rmSync(pair.stagedPath, { recursive: true, force: true });
|
const rollbackSidecar = `${rollbackPath}${suffix}`;
|
||||||
} catch {
|
if (fs.existsSync(liveSidecar) && !fs.existsSync(rollbackSidecar)) {
|
||||||
// ignore
|
try {
|
||||||
|
fs.renameSync(liveSidecar, rollbackSidecar);
|
||||||
|
} catch {
|
||||||
|
// Cross-device: best-effort copy+unlink.
|
||||||
|
FilesystemTransactionService.copyDirSync(liveSidecar, rollbackSidecar);
|
||||||
|
try { fs.rmSync(liveSidecar, { recursive: true, force: true }); } catch { /* ignore */ }
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
rolled.push({ livePath: pair.livePath, rollbackPath, stagedPath: pair.stagedPath });
|
|
||||||
|
if (fs.existsSync(pair.livePath)) {
|
||||||
|
fs.renameSync(pair.livePath, rollbackPath);
|
||||||
|
} else if (fs.existsSync(pair.stagedPath)) {
|
||||||
|
try { fs.rmSync(pair.stagedPath, { recursive: true, force: true }); } catch { /* ignore */ }
|
||||||
|
}
|
||||||
|
rolled.push({ livePath: pair.livePath, rollbackPath, stagedPath: pair.stagedPath, sidecarSuffixes });
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if (manifest) this.updateManifest(manifest, txDir!, 'live-snapshotted');
|
||||||
|
|
||||||
if (input.hooks?.beforeSwap) {
|
if (input.hooks?.beforeSwap) {
|
||||||
try {
|
await input.hooks.beforeSwap();
|
||||||
await input.hooks.beforeSwap();
|
|
||||||
} catch (err) {
|
|
||||||
throw err;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// 2. Swap: rename each staged path into the live location.
|
// 2. Swap: rename each staged path into the live location.
|
||||||
@@ -101,37 +195,49 @@ export class FilesystemTransactionService {
|
|||||||
try {
|
try {
|
||||||
fs.renameSync(r.stagedPath, r.livePath);
|
fs.renameSync(r.stagedPath, r.livePath);
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
// Cross-device fall-back: copy + unlink.
|
|
||||||
FilesystemTransactionService.copyDirSync(r.stagedPath, r.livePath);
|
FilesystemTransactionService.copyDirSync(r.stagedPath, r.livePath);
|
||||||
try {
|
try { fs.rmSync(r.stagedPath, { recursive: true, force: true }); } catch { /* ignore */ }
|
||||||
fs.rmSync(r.stagedPath, { recursive: true, force: true });
|
}
|
||||||
} catch {
|
// Stale sidecars that belonged to a previous live DB must NOT survive the swap:
|
||||||
// ignore
|
// drop anything that matches and isn't one we tracked above.
|
||||||
|
for (const suffix of r.sidecarSuffixes) {
|
||||||
|
const liveSidecar = `${r.livePath}${suffix}`;
|
||||||
|
if (fs.existsSync(liveSidecar)) {
|
||||||
|
try { fs.rmSync(liveSidecar, { recursive: true, force: true }); } catch { /* ignore */ }
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if (manifest) this.updateManifest(manifest, txDir!, 'replacements-installed');
|
||||||
|
|
||||||
if (input.hooks?.afterSwap) {
|
if (input.hooks?.afterSwap) {
|
||||||
await input.hooks.afterSwap();
|
await input.hooks.afterSwap();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if (manifest) this.updateManifest(manifest, txDir!, 'verified');
|
||||||
|
|
||||||
return {
|
return {
|
||||||
rolledBack: false,
|
rolledBack: false,
|
||||||
|
committed: false,
|
||||||
|
transactionId: manifest?.transactionId,
|
||||||
|
manifestDir: txDir ?? undefined,
|
||||||
pairs: rolled,
|
pairs: rolled,
|
||||||
};
|
};
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
await this.rollbackLocal({ rolledBack: false, pairs: rolled });
|
await this.rollbackLocal({ rolledBack: false, committed: false, pairs: rolled, transactionId: manifest?.transactionId });
|
||||||
|
if (manifest && txDir) {
|
||||||
|
try {
|
||||||
|
this.updateManifest(manifest, txDir, 'rolled-back');
|
||||||
|
} catch {
|
||||||
|
// ignore: best effort
|
||||||
|
}
|
||||||
|
}
|
||||||
throw err;
|
throw err;
|
||||||
} finally {
|
|
||||||
// Successful rename leaves rollback artifacts; we keep them until
|
|
||||||
// commit() (or rollback() explicitly deletes them).
|
|
||||||
void cleanupRollbackOnly;
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Restore the original live paths from their rollback snapshots. Safe
|
* Restore the original live paths from their rollback snapshots. Safe to call multiple times.
|
||||||
* to call multiple times.
|
|
||||||
*/
|
*/
|
||||||
async rollback(handle: SwapRollbackHandle): Promise<void> {
|
async rollback(handle: SwapRollbackHandle): Promise<void> {
|
||||||
if (!handle || handle.rolledBack) return;
|
if (!handle || handle.rolledBack) return;
|
||||||
@@ -146,10 +252,11 @@ export class FilesystemTransactionService {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Mark the swap committed; the rollback snapshots are deleted.
|
* Mark the swap committed; the rollback snapshots and (any) staged
|
||||||
|
* leftovers are deleted, and the manifest's terminal phase is recorded.
|
||||||
*/
|
*/
|
||||||
commit(handle: SwapRollbackHandle): void {
|
commit(handle: SwapRollbackHandle): void {
|
||||||
if (!handle || handle.rolledBack) return;
|
if (!handle || handle.rolledBack || handle.committed) return;
|
||||||
for (const r of handle.pairs) {
|
for (const r of handle.pairs) {
|
||||||
try {
|
try {
|
||||||
if (fs.existsSync(r.rollbackPath)) {
|
if (fs.existsSync(r.rollbackPath)) {
|
||||||
@@ -158,24 +265,403 @@ export class FilesystemTransactionService {
|
|||||||
if (fs.existsSync(r.stagedPath)) {
|
if (fs.existsSync(r.stagedPath)) {
|
||||||
fs.rmSync(r.stagedPath, { recursive: true, force: true });
|
fs.rmSync(r.stagedPath, { recursive: true, force: true });
|
||||||
}
|
}
|
||||||
|
// Sidecars: a successful commit means the new live has fresh
|
||||||
|
// sidecars, but any leftover from the staged tree should be gone.
|
||||||
|
for (const suffix of r.sidecarSuffixes) {
|
||||||
|
const stagedSidecar = `${r.stagedPath}${suffix}`;
|
||||||
|
if (fs.existsSync(stagedSidecar)) {
|
||||||
|
try { fs.rmSync(stagedSidecar, { recursive: true, force: true }); } catch { /* ignore */ }
|
||||||
|
}
|
||||||
|
}
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
this.logger.warn(
|
this.logger.warn(
|
||||||
`Failed to clean rollback artifact ${r.rollbackPath}: ${(err as Error)?.message}`,
|
`Failed to clean rollback artifact ${r.rollbackPath}: ${(err as Error)?.message}`,
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
if (handle.manifestDir) {
|
||||||
|
try {
|
||||||
|
FilesystemTransactionService.rmSafe(handle.manifestDir);
|
||||||
|
} catch {
|
||||||
|
// ignore
|
||||||
|
}
|
||||||
|
}
|
||||||
handle.rolledBack = true;
|
handle.rolledBack = true;
|
||||||
|
handle.committed = true;
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Reset the swap state after a successful commit (also clears
|
* Idempotent terminal step retained for backwards compatibility.
|
||||||
* rolledBack flag so commit() is safely idempotent).
|
|
||||||
*/
|
*/
|
||||||
finalize(handle: SwapRollbackHandle): void {
|
finalize(handle: SwapRollbackHandle): void {
|
||||||
if (!handle) return;
|
if (!handle) return;
|
||||||
this.commit(handle);
|
if (!handle.committed && !handle.rolledBack) this.commit(handle);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Scan a system-staging root for unfinished transactions and resolve
|
||||||
|
* each one to a coherent state. Intended to be called once during
|
||||||
|
* startup BEFORE the database / uploads are opened.
|
||||||
|
*
|
||||||
|
* Returns a summary that the caller can log. Throws if any transaction
|
||||||
|
* cannot be safely resolved (ambiguous manifest state). Throwing at
|
||||||
|
* startup is preferable to silently producing a hybrid state.
|
||||||
|
*/
|
||||||
|
recoverTransactions(stagingDir: string, opts: { now?: number; maxAgeMs?: number } = {}): RecoveryReport {
|
||||||
|
FilesystemTransactionService.ensureDir(stagingDir);
|
||||||
|
const report: RecoveryReport = { transactionsScanned: 0, resolved: 0, aborted: [], cleaned: [] };
|
||||||
|
|
||||||
|
const entries = fs.readdirSync(stagingDir, { withFileTypes: true });
|
||||||
|
const now = opts.now ?? Date.now();
|
||||||
|
const maxAgeMs = opts.maxAgeMs ?? 7 * 24 * 60 * 60_000;
|
||||||
|
|
||||||
|
for (const entry of entries) {
|
||||||
|
if (!entry.isDirectory()) continue;
|
||||||
|
const txDir = path.join(stagingDir, entry.name);
|
||||||
|
const manifestPath = path.join(txDir, FilesystemTransactionService.MANIFEST_FILENAME);
|
||||||
|
if (!fs.existsSync(manifestPath)) {
|
||||||
|
// Orphan tx dir without a manifest — only remove if stale (we never delete anything tied to a live manifest).
|
||||||
|
try {
|
||||||
|
const stat = fs.statSync(txDir);
|
||||||
|
if (now - stat.mtimeMs > maxAgeMs) {
|
||||||
|
FilesystemTransactionService.rmSafe(txDir);
|
||||||
|
report.cleaned.push(txDir);
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
// ignore
|
||||||
|
}
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
report.transactionsScanned += 1;
|
||||||
|
let manifest: TransactionManifest;
|
||||||
|
try {
|
||||||
|
manifest = JSON.parse(fs.readFileSync(manifestPath, 'utf8')) as TransactionManifest;
|
||||||
|
} catch (err) {
|
||||||
|
report.aborted.push(manifestPath);
|
||||||
|
throw new ApiError(
|
||||||
|
ERROR_CODES.SYSTEM_RESTORE_FAILED,
|
||||||
|
`Unreadable transaction manifest at ${manifestPath}: ${(err as Error).message}`,
|
||||||
|
500,
|
||||||
|
{ details: { phase: 'recovery' } },
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Terminal phases need no further action.
|
||||||
|
if (manifest.phase === 'committed' || manifest.phase === 'rolled-back') {
|
||||||
|
FilesystemTransactionService.rmSafe(txDir);
|
||||||
|
report.resolved += 1;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Mid-swap recovery decision.
|
||||||
|
try {
|
||||||
|
const decision = this.resolveInterruptedManifest(manifest);
|
||||||
|
if (decision === 'keep-old') {
|
||||||
|
this.restoreOldGeneration(manifest, txDir);
|
||||||
|
} else if (decision === 'keep-new') {
|
||||||
|
this.finalizeNewGeneration(manifest, txDir);
|
||||||
|
} else {
|
||||||
|
report.aborted.push(manifestPath);
|
||||||
|
throw new ApiError(
|
||||||
|
ERROR_CODES.SYSTEM_RESTORE_FAILED,
|
||||||
|
`Cannot resolve interrupted transaction ${manifest.transactionId} at phase=${manifest.phase}: neither old nor new generation is fully present`,
|
||||||
|
500,
|
||||||
|
{ details: { phase: 'recovery', transactionId: manifest.transactionId, transactionPhase: manifest.phase } },
|
||||||
|
);
|
||||||
|
}
|
||||||
|
FilesystemTransactionService.rmSafe(txDir);
|
||||||
|
report.resolved += 1;
|
||||||
|
} catch (err) {
|
||||||
|
if ((err as ApiError)?.code) throw err;
|
||||||
|
report.aborted.push(manifestPath);
|
||||||
|
throw new ApiError(
|
||||||
|
ERROR_CODES.SYSTEM_RESTORE_FAILED,
|
||||||
|
`Recovery of transaction ${manifest.transactionId} failed: ${(err as Error).message}`,
|
||||||
|
500,
|
||||||
|
{ details: { phase: 'recovery', transactionId: manifest.transactionId } },
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return report;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Sweep unowned rollback / restore-stage / wipe-stage fragments that
|
||||||
|
* sit NEXT to a live path (database or uploads) and are not currently
|
||||||
|
* tracked by any manifest. These can be left behind by an interrupted
|
||||||
|
* swap from before this durable manifest existed, OR by an interrupted
|
||||||
|
* wipe-challenges snapshot. Only artifacts older than `maxAgeMs` are
|
||||||
|
* removed so that a perfectly healthy in-flight operation is never
|
||||||
|
* disturbed.
|
||||||
|
*/
|
||||||
|
sweepUnownedArtifactsNearLivePaths(
|
||||||
|
paths: string[],
|
||||||
|
opts: { now?: number; maxAgeMs?: number; stagingDir?: string } = {},
|
||||||
|
): string[] {
|
||||||
|
const cleaned: string[] = [];
|
||||||
|
const now = opts.now ?? Date.now();
|
||||||
|
const maxAgeMs = opts.maxAgeMs ?? 24 * 60 * 60_000;
|
||||||
|
const stagingDir = opts.stagingDir;
|
||||||
|
|
||||||
|
const candidates = (livePath: string, suffixRegex: RegExp): string[] => {
|
||||||
|
const dir = path.dirname(livePath);
|
||||||
|
const base = path.basename(livePath);
|
||||||
|
if (!fs.existsSync(dir)) return [];
|
||||||
|
const result: string[] = [];
|
||||||
|
for (const entry of fs.readdirSync(dir)) {
|
||||||
|
if (!suffixRegex.test(entry)) continue;
|
||||||
|
// The artifact must be derived from this specific live path (its
|
||||||
|
// base name must appear at the start of the artifact).
|
||||||
|
if (!entry.startsWith(`${base}.`) && !entry.startsWith(`.${base}.`)) continue;
|
||||||
|
const full = path.join(dir, entry);
|
||||||
|
try {
|
||||||
|
const stat = fs.statSync(full);
|
||||||
|
if (now - stat.mtimeMs < maxAgeMs) continue;
|
||||||
|
} catch {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
// Skip artifacts actively tracked by a manifest.
|
||||||
|
if (stagingDir && FilesystemTransactionService.hasManifestTracking(stagingDir, full)) continue;
|
||||||
|
result.push(full);
|
||||||
|
}
|
||||||
|
return result;
|
||||||
|
};
|
||||||
|
|
||||||
|
const restoreStageRe = /\.restore-stage-[\w-]+$/;
|
||||||
|
// Either a hidden (`.<base>.rollback-…`) or visible (`<base>.rollback-…`)
|
||||||
|
// rollback artifact derived from the live path basename.
|
||||||
|
const rollbackRe = /^\.?[^/]+\.rollback-[\w-]+$/;
|
||||||
|
const wipeStageRe = /^\.challenges\.wipe-stage-[\w-]+$/;
|
||||||
|
|
||||||
|
for (const livePath of paths) {
|
||||||
|
const directDir = path.dirname(livePath);
|
||||||
|
if (!fs.existsSync(directDir)) continue;
|
||||||
|
// For the uploadDir live, also sweep the `challenges/` subdirectory
|
||||||
|
// (used by DangerZoneService.wipeChallenges()).
|
||||||
|
const extraDirs: string[] = [];
|
||||||
|
if (fs.existsSync(livePath)) {
|
||||||
|
const stat = fs.statSync(livePath);
|
||||||
|
if (stat.isDirectory()) {
|
||||||
|
extraDirs.push(path.join(livePath, 'challenges'));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
const allArtifacts = [
|
||||||
|
...candidates(livePath, restoreStageRe),
|
||||||
|
...candidates(livePath, rollbackRe),
|
||||||
|
...candidates(livePath, wipeStageRe),
|
||||||
|
];
|
||||||
|
for (const artifact of allArtifacts) {
|
||||||
|
FilesystemTransactionService.rmSafe(artifact);
|
||||||
|
cleaned.push(artifact);
|
||||||
|
}
|
||||||
|
for (const extra of extraDirs) {
|
||||||
|
if (!fs.existsSync(extra)) continue;
|
||||||
|
for (const entry of fs.readdirSync(extra)) {
|
||||||
|
if (!wipeStageRe.test(entry)) continue;
|
||||||
|
const full = path.join(extra, entry);
|
||||||
|
try {
|
||||||
|
const stat = fs.statSync(full);
|
||||||
|
if (now - stat.mtimeMs < maxAgeMs) continue;
|
||||||
|
} catch {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (stagingDir && FilesystemTransactionService.hasManifestTracking(stagingDir, full)) continue;
|
||||||
|
FilesystemTransactionService.rmSafe(full);
|
||||||
|
cleaned.push(full);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return cleaned;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Decide whether to restore the OLD generation (live+rollback both
|
||||||
|
* present) or keep the NEW generation (live already swapped + rollback
|
||||||
|
* present). Returns null when neither generation is complete.
|
||||||
|
*/
|
||||||
|
private resolveInterruptedManifest(manifest: TransactionManifest): 'keep-old' | 'keep-new' | null {
|
||||||
|
const generationPairs = manifest.pairs.filter((p) => p.contributesToGeneration);
|
||||||
|
if (generationPairs.length === 0) return 'keep-new';
|
||||||
|
|
||||||
|
// Old generation complete iff every livePath is currently at the
|
||||||
|
// rollbackPath (no swap happened for that pair). Once a swap has been
|
||||||
|
// recorded as `replacements-installed`, the live path is the new
|
||||||
|
// generation.
|
||||||
|
if (manifest.phase === 'prepared') {
|
||||||
|
// Nothing has happened yet — the live tree is still the old one and
|
||||||
|
// there is no rollback because the rename hasn't run. We treat this
|
||||||
|
// as a clean "keep-old" with no filesystem action.
|
||||||
|
return 'keep-old';
|
||||||
|
}
|
||||||
|
|
||||||
|
if (manifest.phase === 'live-snapshotted') {
|
||||||
|
const oldComplete = generationPairs.every((p) => fs.existsSync(p.rollbackPath) && !fs.existsSync(p.livePath));
|
||||||
|
return oldComplete ? 'keep-old' : null;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (manifest.phase === 'replacements-installed' || manifest.phase === 'verified' || manifest.phase === 'committed') {
|
||||||
|
const newComplete = generationPairs.every((p) => fs.existsSync(p.livePath) && fs.existsSync(p.rollbackPath));
|
||||||
|
return newComplete ? 'keep-new' : null;
|
||||||
|
}
|
||||||
|
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Restore the OLD generation: move each rollback path back to the live path
|
||||||
|
* (or, when the live path already holds the new generation due to an
|
||||||
|
* earlier partial swap, remove it first). The rollback is the snapshot of
|
||||||
|
* the pre-swap live tree.
|
||||||
|
*/
|
||||||
|
private restoreOldGeneration(manifest: TransactionManifest, txDir: string): void {
|
||||||
|
if (manifest.phase === 'prepared') {
|
||||||
|
// Nothing was moved; just record the terminal state.
|
||||||
|
this.updateManifest(manifest, txDir, 'rolled-back');
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
for (const p of manifest.pairs) {
|
||||||
|
// Remove any partial new-generation content that may already exist at livePath.
|
||||||
|
if (fs.existsSync(p.livePath)) {
|
||||||
|
FilesystemTransactionService.rmSafe(p.livePath);
|
||||||
|
}
|
||||||
|
if (fs.existsSync(p.rollbackPath)) {
|
||||||
|
FilesystemTransactionService.ensureDir(path.dirname(p.livePath));
|
||||||
|
try {
|
||||||
|
fs.renameSync(p.rollbackPath, p.livePath);
|
||||||
|
} catch {
|
||||||
|
FilesystemTransactionService.copyDirSync(p.rollbackPath, p.livePath);
|
||||||
|
try { fs.rmSync(p.rollbackPath, { recursive: true, force: true }); } catch { /* ignore */ }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Sidecars of the OLD generation are stored next to the rollback path
|
||||||
|
// (e.g. `rollbackPath-wal`); restore them too if present.
|
||||||
|
for (const suffix of p.sidecarSuffixes) {
|
||||||
|
const rollbackSidecar = `${p.rollbackPath}${suffix}`;
|
||||||
|
const liveSidecar = `${p.livePath}${suffix}`;
|
||||||
|
if (fs.existsSync(rollbackSidecar)) {
|
||||||
|
try { fs.rmSync(liveSidecar, { recursive: true, force: true }); } catch { /* ignore */ }
|
||||||
|
FilesystemTransactionService.ensureDir(path.dirname(liveSidecar));
|
||||||
|
try {
|
||||||
|
fs.renameSync(rollbackSidecar, liveSidecar);
|
||||||
|
} catch {
|
||||||
|
FilesystemTransactionService.copyDirSync(rollbackSidecar, liveSidecar);
|
||||||
|
try { fs.rmSync(rollbackSidecar, { recursive: true, force: true }); } catch { /* ignore */ }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (fs.existsSync(liveSidecar) && !fs.existsSync(rollbackSidecar)) {
|
||||||
|
// No source sidecar and the live sidecar remains — leave it removed.
|
||||||
|
try { fs.rmSync(liveSidecar, { recursive: true, force: true }); } catch { /* ignore */ }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
this.updateManifest(manifest, txDir, 'rolled-back');
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Finalize the NEW generation: keep the live replacement in place,
|
||||||
|
* remove the rollback snapshot and any stale sidecars.
|
||||||
|
*/
|
||||||
|
private finalizeNewGeneration(manifest: TransactionManifest, txDir: string): void {
|
||||||
|
for (const p of manifest.pairs) {
|
||||||
|
// Drop the rollback copy; the new live is now durable.
|
||||||
|
FilesystemTransactionService.rmSafe(p.rollbackPath);
|
||||||
|
// Remove any unused staged remnants and stale sidecars.
|
||||||
|
FilesystemTransactionService.rmSafe(p.stagedPath);
|
||||||
|
for (const suffix of p.sidecarSuffixes) {
|
||||||
|
const stagedSidecar = `${p.stagedPath}${suffix}`;
|
||||||
|
FilesystemTransactionService.rmSafe(stagedSidecar);
|
||||||
|
// Keep the live sidecar if it was written by the new replacement;
|
||||||
|
// it should be empty (just created) so nothing further to clean.
|
||||||
|
}
|
||||||
|
}
|
||||||
|
this.updateManifest(manifest, txDir, 'committed');
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Helper that callers (e.g. DatabaseInitService) can use to decide
|
||||||
|
* whether a given rollback/staged artifact is already tracked by a
|
||||||
|
* live manifest in the staging root.
|
||||||
|
*/
|
||||||
|
static hasManifestTracking(stagingDir: string, rollbackPath: string): boolean {
|
||||||
|
if (!fs.existsSync(stagingDir)) return false;
|
||||||
|
const entries = fs.readdirSync(stagingDir, { withFileTypes: true });
|
||||||
|
for (const entry of entries) {
|
||||||
|
if (!entry.isDirectory()) continue;
|
||||||
|
const manifestPath = path.join(stagingDir, entry.name, FilesystemTransactionService.MANIFEST_FILENAME);
|
||||||
|
if (!fs.existsSync(manifestPath)) continue;
|
||||||
|
try {
|
||||||
|
const manifest = JSON.parse(fs.readFileSync(manifestPath, 'utf8')) as TransactionManifest;
|
||||||
|
for (const p of manifest.pairs) {
|
||||||
|
if (p.rollbackPath === rollbackPath || p.stagedPath === rollbackPath || p.livePath === rollbackPath) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
// ignore unreadable manifests; recovery will surface them
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ───────── Manifest helpers ───────── */
|
||||||
|
|
||||||
|
private ensureTxDir(stagingDir: string): string {
|
||||||
|
const id = `${Date.now()}-${Math.random().toString(36).slice(2, 10)}`;
|
||||||
|
const full = path.join(stagingDir, id);
|
||||||
|
FilesystemTransactionService.ensureDir(full);
|
||||||
|
return full;
|
||||||
|
}
|
||||||
|
|
||||||
|
private makeInitialManifest(input: StageSwapInput, txDir: string): TransactionManifest {
|
||||||
|
const transactionId = path.basename(txDir);
|
||||||
|
const now = new Date().toISOString();
|
||||||
|
const manifest: TransactionManifest = {
|
||||||
|
schemaVersion: 1,
|
||||||
|
transactionId,
|
||||||
|
operation: input.name,
|
||||||
|
createdAt: now,
|
||||||
|
updatedAt: now,
|
||||||
|
phase: 'prepared',
|
||||||
|
pairs: [],
|
||||||
|
};
|
||||||
|
const stamp = `${Date.now()}-${Math.random().toString(36).slice(2, 10)}`;
|
||||||
|
for (const pair of input.pairs) {
|
||||||
|
const rollbackPath = FilesystemTransactionService.makeRollbackPath(pair.livePath, stamp);
|
||||||
|
manifest.pairs.push({
|
||||||
|
livePath: pair.livePath,
|
||||||
|
rollbackPath,
|
||||||
|
stagedPath: pair.stagedPath,
|
||||||
|
contributesToGeneration: pair.contributesToGeneration ?? true,
|
||||||
|
sidecarSuffixes: pair.sidecarSuffixes ?? [],
|
||||||
|
});
|
||||||
|
}
|
||||||
|
this.writeManifestAtomic(manifest, txDir);
|
||||||
|
return manifest;
|
||||||
|
}
|
||||||
|
|
||||||
|
private updateManifest(manifest: TransactionManifest, txDir: string, phase: TransactionPhase): void {
|
||||||
|
manifest.phase = phase;
|
||||||
|
manifest.updatedAt = new Date().toISOString();
|
||||||
|
this.writeManifestAtomic(manifest, txDir);
|
||||||
|
}
|
||||||
|
|
||||||
|
private writeManifestAtomic(manifest: TransactionManifest, txDir: string): void {
|
||||||
|
const target = path.join(txDir, FilesystemTransactionService.MANIFEST_FILENAME);
|
||||||
|
const tmp = `${target}.tmp-${Math.random().toString(36).slice(2, 8)}`;
|
||||||
|
FilesystemTransactionService.ensureDir(path.dirname(tmp));
|
||||||
|
fs.writeFileSync(tmp, JSON.stringify(manifest));
|
||||||
|
fs.renameSync(tmp, target);
|
||||||
|
}
|
||||||
|
|
||||||
|
private manifestPathFor(_transactionId: string): string {
|
||||||
|
// Used only by recovery paths that already know where the manifest lives.
|
||||||
|
void _transactionId;
|
||||||
|
return FilesystemTransactionService.MANIFEST_FILENAME;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ───────── Static filesystem helpers ───────── */
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Recursively remove a directory or file. Best-effort.
|
* Recursively remove a directory or file. Best-effort.
|
||||||
*/
|
*/
|
||||||
@@ -188,8 +674,7 @@ export class FilesystemTransactionService {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Copy a directory recursively, falling back gracefully if the source
|
* Copy a directory recursively; gracefully no-ops if the source doesn't exist.
|
||||||
* doesn't exist (returns immediately).
|
|
||||||
*/
|
*/
|
||||||
static copyDirSync(src: string, dest: string): void {
|
static copyDirSync(src: string, dest: string): void {
|
||||||
if (!fs.existsSync(src)) return;
|
if (!fs.existsSync(src)) return;
|
||||||
@@ -211,7 +696,7 @@ export class FilesystemTransactionService {
|
|||||||
if (!fs.existsSync(dir)) fs.mkdirSync(dir, { recursive: true });
|
if (!fs.existsSync(dir)) fs.mkdirSync(dir, { recursive: true });
|
||||||
}
|
}
|
||||||
|
|
||||||
private static makeRollbackPath(livePath: string, stamp: string): string {
|
static makeRollbackPath(livePath: string, stamp: string): string {
|
||||||
const dir = path.dirname(livePath);
|
const dir = path.dirname(livePath);
|
||||||
const base = path.basename(livePath);
|
const base = path.basename(livePath);
|
||||||
return path.join(dir, `.${base}.rollback-${stamp}`);
|
return path.join(dir, `.${base}.rollback-${stamp}`);
|
||||||
@@ -221,11 +706,9 @@ export class FilesystemTransactionService {
|
|||||||
if (!handle || handle.rolledBack) return;
|
if (!handle || handle.rolledBack) return;
|
||||||
// Walk pairs in reverse order to maximize recovery chances.
|
// Walk pairs in reverse order to maximize recovery chances.
|
||||||
for (const r of [...handle.pairs].reverse()) {
|
for (const r of [...handle.pairs].reverse()) {
|
||||||
const stageStillAtLive = fs.existsSync(r.livePath) && r.stagedPath
|
const stageStillAtLive = fs.existsSync(r.livePath)
|
||||||
? (() => {
|
? (() => {
|
||||||
try {
|
try {
|
||||||
const liveStat = fs.statSync(r.livePath);
|
|
||||||
// If we already swapped, stagedDir is gone.
|
|
||||||
return !fs.existsSync(r.stagedPath);
|
return !fs.existsSync(r.stagedPath);
|
||||||
} catch {
|
} catch {
|
||||||
return true;
|
return true;
|
||||||
@@ -235,11 +718,7 @@ export class FilesystemTransactionService {
|
|||||||
try {
|
try {
|
||||||
if (stageStillAtLive) {
|
if (stageStillAtLive) {
|
||||||
// swap happened — remove the now-swapped content before restoring.
|
// swap happened — remove the now-swapped content before restoring.
|
||||||
try {
|
FilesystemTransactionService.rmSafe(r.livePath);
|
||||||
fs.rmSync(r.livePath, { recursive: true, force: true });
|
|
||||||
} catch {
|
|
||||||
// ignore
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
if (fs.existsSync(r.rollbackPath)) {
|
if (fs.existsSync(r.rollbackPath)) {
|
||||||
FilesystemTransactionService.ensureDir(path.dirname(r.livePath));
|
FilesystemTransactionService.ensureDir(path.dirname(r.livePath));
|
||||||
@@ -247,19 +726,25 @@ export class FilesystemTransactionService {
|
|||||||
fs.renameSync(r.rollbackPath, r.livePath);
|
fs.renameSync(r.rollbackPath, r.livePath);
|
||||||
} catch {
|
} catch {
|
||||||
FilesystemTransactionService.copyDirSync(r.rollbackPath, r.livePath);
|
FilesystemTransactionService.copyDirSync(r.rollbackPath, r.livePath);
|
||||||
try {
|
try { fs.rmSync(r.rollbackPath, { recursive: true, force: true }); } catch { /* ignore */ }
|
||||||
fs.rmSync(r.rollbackPath, { recursive: true, force: true });
|
|
||||||
} catch {
|
|
||||||
// ignore
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
// If a staged dir remained because swap failed before move, clean it up.
|
|
||||||
if (fs.existsSync(r.stagedPath)) {
|
if (fs.existsSync(r.stagedPath)) {
|
||||||
try {
|
FilesystemTransactionService.rmSafe(r.stagedPath);
|
||||||
fs.rmSync(r.stagedPath, { recursive: true, force: true });
|
}
|
||||||
} catch {
|
// Restore sidecars if they were tracked.
|
||||||
// ignore
|
for (const suffix of r.sidecarSuffixes) {
|
||||||
|
const liveSidecar = `${r.livePath}${suffix}`;
|
||||||
|
const rollbackSidecar = `${r.rollbackPath}${suffix}`;
|
||||||
|
if (fs.existsSync(rollbackSidecar)) {
|
||||||
|
FilesystemTransactionService.rmSafe(liveSidecar);
|
||||||
|
FilesystemTransactionService.ensureDir(path.dirname(liveSidecar));
|
||||||
|
try {
|
||||||
|
fs.renameSync(rollbackSidecar, liveSidecar);
|
||||||
|
} catch {
|
||||||
|
FilesystemTransactionService.copyDirSync(rollbackSidecar, liveSidecar);
|
||||||
|
try { fs.rmSync(rollbackSidecar, { recursive: true, force: true }); } catch { /* ignore */ }
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
|
|||||||
@@ -65,6 +65,7 @@ export class RestoreService {
|
|||||||
private readonly logger = new Logger(RestoreService.name);
|
private readonly logger = new Logger(RestoreService.name);
|
||||||
private readonly uploadDir: string;
|
private readonly uploadDir: string;
|
||||||
private readonly databasePath: string;
|
private readonly databasePath: string;
|
||||||
|
private readonly systemStagingDir: string;
|
||||||
private readonly stageTtlMs: number;
|
private readonly stageTtlMs: number;
|
||||||
private readonly restoreUploadLimit: number;
|
private readonly restoreUploadLimit: number;
|
||||||
|
|
||||||
@@ -78,6 +79,7 @@ export class RestoreService {
|
|||||||
) {
|
) {
|
||||||
this.uploadDir = path.resolve(this.configService.get<string>('UPLOAD_DIR', './data/uploads'));
|
this.uploadDir = path.resolve(this.configService.get<string>('UPLOAD_DIR', './data/uploads'));
|
||||||
this.databasePath = path.resolve(this.configService.get<string>('DATABASE_PATH', './data/db.sqlite'));
|
this.databasePath = path.resolve(this.configService.get<string>('DATABASE_PATH', './data/db.sqlite'));
|
||||||
|
this.systemStagingDir = resolveSystemStagingDir(this.configService);
|
||||||
this.stageTtlMs = this.configService.get<number>('SYSTEM_OP_RESTORE_STAGE_TTL_MS', 15 * 60_000);
|
this.stageTtlMs = this.configService.get<number>('SYSTEM_OP_RESTORE_STAGE_TTL_MS', 15 * 60_000);
|
||||||
this.restoreUploadLimit = parseUploadSizeLimit(this.configService.get<string>('BODY_SIZE_LIMIT', '50mb'));
|
this.restoreUploadLimit = parseUploadSizeLimit(this.configService.get<string>('BODY_SIZE_LIMIT', '50mb'));
|
||||||
}
|
}
|
||||||
@@ -230,8 +232,6 @@ export class RestoreService {
|
|||||||
throw new ApiError(ERROR_CODES.SYSTEM_RESTORE_STAGE_EXPIRED, 'Restore stage has expired; please validate again', 400);
|
throw new ApiError(ERROR_CODES.SYSTEM_RESTORE_STAGE_EXPIRED, 'Restore stage has expired; please validate again', 400);
|
||||||
}
|
}
|
||||||
const stagingDbPath = path.join(staged.stagingRoot, 'rebuild.sqlite');
|
const stagingDbPath = path.join(staged.stagingRoot, 'rebuild.sqlite');
|
||||||
const liveBackupDb = `${this.databasePath}.rollback-${Date.now()}-${Math.random().toString(36).slice(2, 8)}`;
|
|
||||||
void liveBackupDb;
|
|
||||||
const stagedUploadsReplace = `${this.uploadDir}.restore-stage-${Date.now()}-${Math.random().toString(36).slice(2, 8)}`;
|
const stagedUploadsReplace = `${this.uploadDir}.restore-stage-${Date.now()}-${Math.random().toString(36).slice(2, 8)}`;
|
||||||
|
|
||||||
let handle: SwapRollbackHandle | null = null;
|
let handle: SwapRollbackHandle | null = null;
|
||||||
@@ -246,18 +246,30 @@ export class RestoreService {
|
|||||||
FilesystemTransactionService.ensureDir(this.uploadDir);
|
FilesystemTransactionService.ensureDir(this.uploadDir);
|
||||||
FilesystemTransactionService.copyDirSync(staged.uploadsRoot, stagedUploadsReplace);
|
FilesystemTransactionService.copyDirSync(staged.uploadsRoot, stagedUploadsReplace);
|
||||||
|
|
||||||
handle = await this.fsTx.stageSwap({
|
handle = await this.fsTx.stageSwap(
|
||||||
name: 'restore-backup',
|
{
|
||||||
pairs: [
|
name: 'restore-backup',
|
||||||
{ livePath: this.databasePath, stagedPath: stagingDbPath },
|
pairs: [
|
||||||
{ livePath: this.uploadDir, stagedPath: stagedUploadsReplace },
|
{
|
||||||
],
|
livePath: this.databasePath,
|
||||||
hooks: {
|
stagedPath: stagingDbPath,
|
||||||
afterSwap: async () => {
|
contributesToGeneration: true,
|
||||||
await this.verifySwappedDatabase();
|
sidecarSuffixes: ['-wal', '-shm'],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
livePath: this.uploadDir,
|
||||||
|
stagedPath: stagedUploadsReplace,
|
||||||
|
contributesToGeneration: true,
|
||||||
|
},
|
||||||
|
],
|
||||||
|
hooks: {
|
||||||
|
afterSwap: async () => {
|
||||||
|
await this.verifySwappedDatabase();
|
||||||
|
},
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
});
|
{ stagingDir: this.systemStagingDir },
|
||||||
|
);
|
||||||
|
|
||||||
// Re-establish the live DataSource against the new database. The
|
// Re-establish the live DataSource against the new database. The
|
||||||
// global TypeORM connection will pick up the swapped file on next
|
// global TypeORM connection will pick up the swapped file on next
|
||||||
|
|||||||
@@ -4,7 +4,7 @@ title: Admin System Endpoints
|
|||||||
description: Admin-only destructive System operations: backup download, restore validate/commit, re-authenticated confirmation tokens, reset-scores, and wipe-challenges.
|
description: Admin-only destructive System operations: backup download, restore validate/commit, re-authenticated confirmation tokens, reset-scores, and wipe-challenges.
|
||||||
resource: backend/src/modules/admin/system/admin-system.controller.ts
|
resource: backend/src/modules/admin/system/admin-system.controller.ts
|
||||||
tags: [api, admin, system, backup, restore, danger-zone, confirmation-tokens]
|
tags: [api, admin, system, backup, restore, danger-zone, confirmation-tokens]
|
||||||
timestamp: 2026-07-23T13:27:25Z
|
timestamp: 2026-07-23T15:46:55Z
|
||||||
---
|
---
|
||||||
|
|
||||||
# Overview
|
# Overview
|
||||||
@@ -127,9 +127,15 @@ Flow:
|
|||||||
by cloning the live DB, then clearing every application table and
|
by cloning the live DB, then clearing every application table and
|
||||||
inserting the archived rows in FK-safe order. Uploads are copied
|
inserting the archived rows in FK-safe order. Uploads are copied
|
||||||
aside as `<UPLOAD_DIR>.restore-stage-<stamp>` and the live
|
aside as `<UPLOAD_DIR>.restore-stage-<stamp>` and the live
|
||||||
database + uploads are swapped atomically via
|
database + uploads are swapped as one recoverable generation via
|
||||||
`FilesystemTransactionService.stageSwap()` (with a `restore-backup`
|
`FilesystemTransactionService.stageSwap(..., { stagingDir })` (with a
|
||||||
rollback handle).
|
`restore-backup` rollback handle). Before moving either live path, the
|
||||||
|
service writes `<SYSTEM_OP_STAGING_DIR>/<transactionId>/manifest.json`.
|
||||||
|
The manifest advances through `prepared`, `live-snapshotted`,
|
||||||
|
`replacements-installed`, and `verified`; commit removes rollback and
|
||||||
|
manifest artifacts. SQLite `-wal` and `-shm` sidecars travel with the
|
||||||
|
database rollback generation and stale sidecars are removed from the
|
||||||
|
replacement.
|
||||||
|
|
||||||
The offline clone rebuild begins by capturing the exact SQL of the
|
The offline clone rebuild begins by capturing the exact SQL of the
|
||||||
deployment-wide `trg_user_last_admin_update` and
|
deployment-wide `trg_user_last_admin_update` and
|
||||||
@@ -161,6 +167,25 @@ Flow:
|
|||||||
revoked via `AuthService.revokeAllRefreshSessions()` so the SPA is
|
revoked via `AuthService.revokeAllRefreshSessions()` so the SPA is
|
||||||
forced back to `/login`.
|
forced back to `/login`.
|
||||||
|
|
||||||
|
### Startup recovery contract
|
||||||
|
|
||||||
|
Before TypeORM opens SQLite, `DatabaseInitService.init()` calls
|
||||||
|
`FilesystemTransactionService.recoverTransactions()` using
|
||||||
|
`SYSTEM_OP_STAGING_DIR` (default `<DATA_DIR>/.system-staging`):
|
||||||
|
|
||||||
|
| Manifest phase | Required complete generation | Startup action |
|
||||||
|
|---|---|---|
|
||||||
|
| `prepared` | Existing live paths | Keep the old generation and remove the transaction manifest. |
|
||||||
|
| `live-snapshotted` | Every rollback path exists and corresponding live path is absent | Restore all rollback paths, including tracked SQLite sidecars. |
|
||||||
|
| `replacements-installed` or `verified` | Every replacement live path and rollback path exists | Keep the complete new generation and remove rollback/staged artifacts. |
|
||||||
|
| Any ambiguous or mixed state | Neither generation is complete | Throw `SYSTEM_RESTORE_FAILED`, preserve the manifest and artifacts, and abort startup. |
|
||||||
|
| `committed` or `rolled-back` | Terminal | Remove the transaction directory. |
|
||||||
|
|
||||||
|
After manifest recovery, startup best-effort removes stale unowned
|
||||||
|
`.rollback-*`, `.restore-stage-*`, and `.wipe-stage-*` artifacts near the
|
||||||
|
configured database/uploads paths. Fresh artifacts and paths referenced by a
|
||||||
|
manifest are retained.
|
||||||
|
|
||||||
Returns `{ ok: true, operation: 'restore-backup' }`.
|
Returns `{ ok: true, operation: 'restore-backup' }`.
|
||||||
|
|
||||||
## POST /confirmations
|
## POST /confirmations
|
||||||
@@ -301,7 +326,9 @@ curl -sS ... -d '{"operation":"reset-scores","token":"<raw>"}'
|
|||||||
| Restore | `backend/src/modules/admin/system/restore.service.ts` |
|
| Restore | `backend/src/modules/admin/system/restore.service.ts` |
|
||||||
| Confirmation tokens | `backend/src/modules/admin/system/confirmation-token.service.ts` |
|
| Confirmation tokens | `backend/src/modules/admin/system/confirmation-token.service.ts` |
|
||||||
| Danger zone | `backend/src/modules/admin/system/danger-zone.service.ts` |
|
| Danger zone | `backend/src/modules/admin/system/danger-zone.service.ts` |
|
||||||
| Filesystem swap | `backend/src/modules/admin/system/filesystem-transaction.service.ts` |
|
| Filesystem swap + recovery | `backend/src/modules/admin/system/filesystem-transaction.service.ts` |
|
||||||
|
| Shared filesystem module | `backend/src/modules/admin/system/filesystem-transaction.module.ts` |
|
||||||
|
| Startup recovery wiring | `backend/src/database/database-init.service.ts`, `backend/src/database/database.module.ts` |
|
||||||
| Staging dir helper | `backend/src/common/utils/upload.ts` (`resolveSystemStagingDir`) |
|
| Staging dir helper | `backend/src/common/utils/upload.ts` (`resolveSystemStagingDir`) |
|
||||||
| Password re-auth / revoke | `backend/src/modules/auth/auth.service.ts` (`reauthenticateAdmin`, `revokeAllRefreshSessions`) |
|
| Password re-auth / revoke | `backend/src/modules/auth/auth.service.ts` (`reauthenticateAdmin`, `revokeAllRefreshSessions`) |
|
||||||
| Error codes | `backend/src/common/errors/error-codes.ts` |
|
| Error codes | `backend/src/common/errors/error-codes.ts` |
|
||||||
@@ -313,4 +340,4 @@ curl -sS ... -d '{"operation":"reset-scores","token":"<raw>"}'
|
|||||||
| Front-end route | `frontend/src/app/app.routes.ts` (admin child `system`) |
|
| Front-end route | `frontend/src/app/app.routes.ts` (admin child `system`) |
|
||||||
| Cross-store invalidation | `frontend/src/app/core/services/system-data-change.service.ts` |
|
| Cross-store invalidation | `frontend/src/app/core/services/system-data-change.service.ts` |
|
||||||
| Forced logout helper | `frontend/src/app/core/services/auth.service.ts` (`forceServerInvalidation`) |
|
| Forced logout helper | `frontend/src/app/core/services/auth.service.ts` (`forceServerInvalidation`) |
|
||||||
| Tests | `tests/backend/admin-system-authorization.spec.ts`, `admin-system-backup.spec.ts`, `admin-system-confirmation-token.spec.ts`, `admin-system-danger.spec.ts`, `admin-system-restore-validation.spec.ts`, `admin-system-restore-commit.spec.ts`; `tests/frontend/admin-system.pure.spec.ts` |
|
| Tests | `tests/backend/admin-system-authorization.spec.ts`, `admin-system-backup.spec.ts`, `admin-system-confirmation-token.spec.ts`, `admin-system-danger.spec.ts`, `admin-system-restore-validation.spec.ts`, `admin-system-restore-commit.spec.ts`, `database-init-recovery.spec.ts`, `filesystem-transaction.spec.ts`; `tests/frontend/admin-system.pure.spec.ts` |
|
||||||
|
|||||||
@@ -3,7 +3,7 @@ type: architecture
|
|||||||
title: Backend Module Map
|
title: Backend Module Map
|
||||||
description: NestJS modules, controllers, services, and how they are wired together.
|
description: NestJS modules, controllers, services, and how they are wired together.
|
||||||
tags: [architecture, backend, nestjs, modules]
|
tags: [architecture, backend, nestjs, modules]
|
||||||
timestamp: 2026-07-23T10:12:24Z
|
timestamp: 2026-07-23T15:46:55Z
|
||||||
---
|
---
|
||||||
|
|
||||||
# Module Map
|
# Module Map
|
||||||
@@ -20,7 +20,8 @@ also registers two global providers:
|
|||||||
|
|
||||||
| Module | Path | Responsibility |
|
| Module | Path | Responsibility |
|
||||||
|------------------|-----------------------------------------------|-------------------------------------------------------------------------------------------------|
|
|------------------|-----------------------------------------------|-------------------------------------------------------------------------------------------------|
|
||||||
| `DatabaseModule` | `backend/src/database/database.module.ts` | Configures TypeORM with `better-sqlite3`, registers entities, runs migrations, enforces WAL. |
|
| `DatabaseModule` | `backend/src/database/database.module.ts` | Configures TypeORM with `better-sqlite3`, registers entities, runs migrations, enforces WAL, and imports `FilesystemTransactionModule` so startup recovery runs before the database is opened. |
|
||||||
|
| `FilesystemTransactionModule` | `backend/src/modules/admin/system/filesystem-transaction.module.ts` | Provides the shared durable filesystem swap service to both `DatabaseModule` startup recovery and `AdminSystemModule` restore operations without a circular dependency. |
|
||||||
| `CommonModule` | `backend/src/common/common.module.ts` | Provides shared services (CSRF middleware class, backoff, registration rate limit, SSE hub, theme loader, etc.). |
|
| `CommonModule` | `backend/src/common/common.module.ts` | Provides shared services (CSRF middleware class, backoff, registration rate limit, SSE hub, theme loader, etc.). |
|
||||||
| `SettingsModule` | `backend/src/modules/settings/settings.module.ts` | Exposes `SettingsService` (get/set/getAll over `setting` table). |
|
| `SettingsModule` | `backend/src/modules/settings/settings.module.ts` | Exposes `SettingsService` (get/set/getAll over `setting` table). |
|
||||||
| `AuthModule` | `backend/src/modules/auth/auth.module.ts` | `AuthController` (`/api/v1/auth/{register,login,refresh,logout,me,change-password,csrf}`) + `AuthService` (login/refresh/logout/register-first-admin + public `register` + `getMe` + `changePassword`). Imports `SettingsModule` so the public-register flow can read the `registrationsEnabled` flag, and `UsersModule` (forwardRef) for `UsersRankService`. |
|
| `AuthModule` | `backend/src/modules/auth/auth.module.ts` | `AuthController` (`/api/v1/auth/{register,login,refresh,logout,me,change-password,csrf}`) + `AuthService` (login/refresh/logout/register-first-admin + public `register` + `getMe` + `changePassword`). Imports `SettingsModule` so the public-register flow can read the `registrationsEnabled` flag, and `UsersModule` (forwardRef) for `UsersRankService`. |
|
||||||
@@ -28,7 +29,7 @@ also registers two global providers:
|
|||||||
| `SetupModule` | `backend/src/modules/setup/setup.module.ts` | `SetupController` (`POST /api/v1/setup/create-admin`) + `SetupService`. Returns 409 `SYSTEM_INITIALIZED` once any admin exists; serializes concurrent first-admin attempts via an in-process promise chain. |
|
| `SetupModule` | `backend/src/modules/setup/setup.module.ts` | `SetupController` (`POST /api/v1/setup/create-admin`) + `SetupService`. Returns 409 `SYSTEM_INITIALIZED` once any admin exists; serializes concurrent first-admin attempts via an in-process promise chain. |
|
||||||
| `SystemModule` | `backend/src/modules/system/system.module.ts` | `SystemController` (`/api/v1/bootstrap`, `/event/status`, SSE streams) + `SystemService`. |
|
| `SystemModule` | `backend/src/modules/system/system.module.ts` | `SystemController` (`/api/v1/bootstrap`, `/event/status`, SSE streams) + `SystemService`. |
|
||||||
| `AdminModule` | `backend/src/modules/admin/admin.module.ts` | Five controllers (`AdminController` for `/api/v1/admin/users*`, `AdminGeneralController` for `/api/v1/admin/general/*`, `AdminCategoriesController` for `/api/v1/admin/categories*`, `AdminChallengesController` for `/api/v1/admin/challenges*`, `AdminSystemController` for `/api/v1/admin/system/*`) plus their services (`AdminService`, `AdminGeneralService`, `AdminCategoriesService`, `AdminChallengesService`, `ChallengeFilesService`, `BackupService`, `RestoreService`, `DangerZoneService`, `FilesystemTransactionService`, `ConfirmationTokenService`). Gated by `AdminGuard` + `@Roles('admin')`. Imports `TypeOrmModule.forFeature([UserEntity, CategoryEntity, ChallengeEntity, ChallengeFileEntity, SolveEntity])`, `AuthModule`, `UsersModule`, `SettingsModule`, `CommonModule` (for `SseHubService`, `ThemeLoaderService`), and `AdminSystemModule` (which adds `AdminOperationTokenEntity` + `AuthModule`). |
|
| `AdminModule` | `backend/src/modules/admin/admin.module.ts` | Five controllers (`AdminController` for `/api/v1/admin/users*`, `AdminGeneralController` for `/api/v1/admin/general/*`, `AdminCategoriesController` for `/api/v1/admin/categories*`, `AdminChallengesController` for `/api/v1/admin/challenges*`, `AdminSystemController` for `/api/v1/admin/system/*`) plus their services (`AdminService`, `AdminGeneralService`, `AdminCategoriesService`, `AdminChallengesService`, `ChallengeFilesService`, `BackupService`, `RestoreService`, `DangerZoneService`, `FilesystemTransactionService`, `ConfirmationTokenService`). Gated by `AdminGuard` + `@Roles('admin')`. Imports `TypeOrmModule.forFeature([UserEntity, CategoryEntity, ChallengeEntity, ChallengeFileEntity, SolveEntity])`, `AuthModule`, `UsersModule`, `SettingsModule`, `CommonModule` (for `SseHubService`, `ThemeLoaderService`), and `AdminSystemModule` (which adds `AdminOperationTokenEntity` + `AuthModule`). |
|
||||||
| `AdminSystemModule` | `backend/src/modules/admin/system/admin-system.module.ts` | Hosts the destructive System operations: `AdminSystemController` + `BackupService` + `RestoreService` + `DangerZoneService` + `FilesystemTransactionService` + `ConfirmationTokenService`. Imports `TypeOrmModule.forFeature([AdminOperationTokenEntity])` and `AuthModule` (for `reauthenticateAdmin`/`revokeAllRefreshSessions`). |
|
| `AdminSystemModule` | `backend/src/modules/admin/system/admin-system.module.ts` | Hosts the destructive System operations: `AdminSystemController` + `BackupService` + `RestoreService` + `DangerZoneService` + `ConfirmationTokenService`. Imports shared `FilesystemTransactionModule`, `TypeOrmModule.forFeature([AdminOperationTokenEntity])`, and `AuthModule` (for `reauthenticateAdmin`/`revokeAllRefreshSessions`). |
|
||||||
| `UploadsModule` | `backend/src/modules/uploads/uploads.module.ts` | `UploadsController` (`/api/v1/uploads/*`). Admin-only multipart. |
|
| `UploadsModule` | `backend/src/modules/uploads/uploads.module.ts` | `UploadsController` (`/api/v1/uploads/*`). Admin-only multipart. |
|
||||||
| `BlogModule` | `backend/src/modules/blog/blog.module.ts` | `BlogController`/`BlogService` expose published posts at `GET /api/v1/blog/posts`; `AdminBlogController`/`AdminBlogService` provide guarded CRUD at `/api/v1/admin/blog/posts` using `BlogPostEntity`. |
|
| `BlogModule` | `backend/src/modules/blog/blog.module.ts` | `BlogController`/`BlogService` expose published posts at `GET /api/v1/blog/posts`; `AdminBlogController`/`AdminBlogService` provide guarded CRUD at `/api/v1/admin/blog/posts` using `BlogPostEntity`. |
|
||||||
| `ChallengesModule` | `backend/src/modules/challenges/challenges.module.ts` | `ChallengesController` (`/api/v1/challenges/{board,status,:id,:id/solves}`) + `ChallengesEventsController` (authenticated SSE `/api/v1/events`) + `ChallengesService` (board, detail, submit, scoring util). |
|
| `ChallengesModule` | `backend/src/modules/challenges/challenges.module.ts` | `ChallengesController` (`/api/v1/challenges/{board,status,:id,:id/solves}`) + `ChallengesEventsController` (authenticated SSE `/api/v1/events`) + `ChallengesService` (board, detail, submit, scoring util). |
|
||||||
@@ -82,7 +83,9 @@ also registers two global providers:
|
|||||||
# Startup chain
|
# Startup chain
|
||||||
|
|
||||||
`main.ts` → `AppModule` (loads `ConfigModule`, then all feature modules)
|
`main.ts` → `AppModule` (loads `ConfigModule`, then all feature modules)
|
||||||
→ `DatabaseInitService.init()` (runs migrations + WAL)
|
→ `DatabaseInitService.init()` → `FilesystemTransactionService.recoverTransactions()`
|
||||||
|
(resolves interrupted restore manifests before TypeORM opens the live SQLite file;
|
||||||
|
ambiguous generations abort startup) → stale unowned swap-artifact sweep → migrations + WAL
|
||||||
→ `app.use(...)` middlewares (helmet, parsers, CSRF, static)
|
→ `app.use(...)` middlewares (helmet, parsers, CSRF, static)
|
||||||
→ `SwaggerModule.setup(...)` (OpenAPI 3.1)
|
→ `SwaggerModule.setup(...)` (OpenAPI 3.1)
|
||||||
→ `SpaFallbackMiddleware` → `app.listen()`.
|
→ `SpaFallbackMiddleware` → `app.listen()`.
|
||||||
|
|||||||
@@ -3,7 +3,7 @@ type: architecture
|
|||||||
title: Key Files Index
|
title: Key Files Index
|
||||||
description: One-line responsibility for important source and contract-test files, including event-window, challenge, scoreboard, session, notification, and blog flows.
|
description: One-line responsibility for important source and contract-test files, including event-window, challenge, scoreboard, session, notification, and blog flows.
|
||||||
tags: [architecture, key-files, event-window, validation, default-challenge-ip, sse, bootstrap, migrations, challenges, import, board, notifications, per-user, session, blog]
|
tags: [architecture, key-files, event-window, validation, default-challenge-ip, sse, bootstrap, migrations, challenges, import, board, notifications, per-user, session, blog]
|
||||||
timestamp: 2026-07-23T13:27:25Z
|
timestamp: 2026-07-23T15:46:55Z
|
||||||
---
|
---
|
||||||
|
|
||||||
# Backend
|
# Backend
|
||||||
@@ -13,8 +13,9 @@ timestamp: 2026-07-23T13:27:25Z
|
|||||||
| `backend/src/main.ts` | Bootstraps Nest, middleware, OpenAPI, static assets, and SPA fallback. |
|
| `backend/src/main.ts` | Bootstraps Nest, middleware, OpenAPI, static assets, and SPA fallback. |
|
||||||
| `backend/src/app.module.ts` | Wires feature modules and global providers. |
|
| `backend/src/app.module.ts` | Wires feature modules and global providers. |
|
||||||
| `backend/src/config/env.schema.ts` | Zod-validated runtime config; canonical defaults for `DATABASE_PATH` (`./data/db.sqlite`) and `UPLOAD_DIR` (`./data/uploads`). |
|
| `backend/src/config/env.schema.ts` | Zod-validated runtime config; canonical defaults for `DATABASE_PATH` (`./data/db.sqlite`) and `UPLOAD_DIR` (`./data/uploads`). |
|
||||||
| `backend/src/database/database.module.ts` | Configures TypeORM with SQLite. |
|
| `backend/src/database/database.module.ts` | Configures TypeORM with SQLite and imports the shared `FilesystemTransactionModule` used by pre-TypeORM startup recovery. |
|
||||||
| `backend/src/database/database-init.service.ts` | Initializes the database and runs migrations; `verifySeed()` also asserts the canonical six system-category keys and reports missing/duplicate rows. |
|
| `backend/src/database/database-init.service.ts` | Before opening SQLite, resolves durable swap manifests to a coherent old or verified-new database/uploads generation, fails startup closed on ambiguity, best-effort sweeps stale unowned swap artifacts, then runs migrations, enforces WAL, and verifies seeds. |
|
||||||
|
| `backend/src/modules/admin/system/filesystem-transaction.module.ts` | Exports one shared `FilesystemTransactionService` provider to `DatabaseModule` and `AdminSystemModule`, avoiding a circular dependency. |
|
||||||
| `backend/src/database/migrations/1700000000400-RepairCategorySchemaAndSystemCategories.ts` | Forward-only repair migration: adds `created_at`/`updated_at` to legacy six-column `category` tables, deletes obsolete `FOR`/`OSI` system rows, rewrites canonical row metadata, inserts any missing canonical key, and re-asserts unique indexes. Exports `CANONICAL_SYSTEM_CATEGORIES`. |
|
| `backend/src/database/migrations/1700000000400-RepairCategorySchemaAndSystemCategories.ts` | Forward-only repair migration: adds `created_at`/`updated_at` to legacy six-column `category` tables, deletes obsolete `FOR`/`OSI` system rows, rewrites canonical row metadata, inserts any missing canonical key, and re-asserts unique indexes. Exports `CANONICAL_SYSTEM_CATEGORIES`. |
|
||||||
| `backend/src/common/guards/jwt-auth.guard.ts` | Global JWT authorization guard. |
|
| `backend/src/common/guards/jwt-auth.guard.ts` | Global JWT authorization guard. |
|
||||||
| `backend/src/common/middleware/csrf.middleware.ts` | Double-submit CSRF protection. |
|
| `backend/src/common/middleware/csrf.middleware.ts` | Double-submit CSRF protection. |
|
||||||
@@ -124,11 +125,11 @@ timestamp: 2026-07-23T13:27:25Z
|
|||||||
| `tests/frontend/scoreboard.pure.spec.ts` | Pure helpers: `applyRankingSort` (competition-rank numbering), `parseSolveEventIntoRanking` (add-or-increment + sort), `dedupEventLogBySolveId`, `applySolveToGraph` (anchors + plateau + top-10 trim), `mutateMatrixFromSolve`, `stablePlayerColorIndex`, `formatSolveDateTime`. |
|
| `tests/frontend/scoreboard.pure.spec.ts` | Pure helpers: `applyRankingSort` (competition-rank numbering), `parseSolveEventIntoRanking` (add-or-increment + sort), `dedupEventLogBySolveId`, `applySolveToGraph` (anchors + plateau + top-10 trim), `mutateMatrixFromSolve`, `stablePlayerColorIndex`, `formatSolveDateTime`. |
|
||||||
| `tests/frontend/scoreboard.store.spec.ts` | Store lifecycle + live merge: `loadAll` aggregation + idempotency, SSE `solve` frame applying to all four tabs, `stop()` teardown, exponential reconnect. |
|
| `tests/frontend/scoreboard.store.spec.ts` | Store lifecycle + live merge: `loadAll` aggregation + idempotency, SSE `solve` frame applying to all four tabs, `stop()` teardown, exponential reconnect. |
|
||||||
| `backend/src/modules/admin/system/admin-system.controller.ts` | Mounts `/api/v1/admin/system/*` (backup download, restore validate/commit, confirmations, scores/reset, challenges/wipe); all handlers require `AdminGuard` + `@Roles('admin')` and the destructive ones also consume a confirmation token. |
|
| `backend/src/modules/admin/system/admin-system.controller.ts` | Mounts `/api/v1/admin/system/*` (backup download, restore validate/commit, confirmations, scores/reset, challenges/wipe); all handlers require `AdminGuard` + `@Roles('admin')` and the destructive ones also consume a confirmation token. |
|
||||||
| `backend/src/modules/admin/system/admin-system.module.ts` | Wires `AdminSystemController`, `BackupService`, `RestoreService`, `DangerZoneService`, `FilesystemTransactionService`, `ConfirmationTokenService`; imports `TypeOrmModule.forFeature([AdminOperationTokenEntity])` and `AuthModule`. |
|
| `backend/src/modules/admin/system/admin-system.module.ts` | Wires `AdminSystemController`, `BackupService`, `RestoreService`, `DangerZoneService`, and `ConfirmationTokenService`; imports the shared `FilesystemTransactionModule`, `TypeOrmModule.forFeature([AdminOperationTokenEntity])`, and `AuthModule`. |
|
||||||
| `backend/src/modules/admin/system/backup.service.ts` | Builds the full backup JSON: every application table (`sqlite_master` discovery, excluding `admin_operation_token`/`migrations`/`sqlite_*`), with base64-encoded uploads walked recursively from `UPLOAD_DIR` (skipping `.staging`). Exposes `discoverTables()`, `getUploadDir()`, and `BackupService.stringify(doc)`. |
|
| `backend/src/modules/admin/system/backup.service.ts` | Builds the full backup JSON: every application table (`sqlite_master` discovery, excluding `admin_operation_token`/`migrations`/`sqlite_*`), with base64-encoded uploads walked recursively from `UPLOAD_DIR` (skipping `.staging`). Exposes `discoverTables()`, `getUploadDir()`, and `BackupService.stringify(doc)`. |
|
||||||
| `backend/src/modules/admin/system/restore.service.ts` | Two-phase restore: `stageArchive(rawText, userId)` validates the archive (zod), decodes uploads into `<DATA_DIR>/.system-staging/restore-<id>/`, and returns a summary; `commitRestore(stagingId)` clones the live DB, captures and temporarily drops the `trg_user_last_admin_update` / `trg_user_last_admin_delete` triggers in the offline candidate, clears every application table, re-inserts archived rows in FK-safe order, asserts the archive still contains at least one admin, re-creates the captured triggers, then atomically swaps the live DB + uploads via `FilesystemTransactionService.stageSwap({ name: 'restore-backup', ... })`. On any failure the swap is rolled back and `SYSTEM_RESTORE_ROLLED_BACK` is returned; the rebuild also best-effort re-creates any missing trigger before rethrowing so a rolled-back candidate never leaks missing the `LAST_ADMIN` safety net. |
|
| `backend/src/modules/admin/system/restore.service.ts` | Two-phase restore: validates/stages the archive, rebuilds an offline SQLite candidate while preserving the last-admin triggers, then swaps database + uploads as one durable manifest-backed generation; tracks SQLite `-wal`/`-shm` sidecars and reconnects TypeORM only after verification. |
|
||||||
| `backend/src/modules/admin/system/danger-zone.service.ts` | `resetScores()` deletes every `solve` row transactionally; `wipeChallenges()` snapshots `<UPLOAD_DIR>/challenges` to a side directory, deletes every challenge inside a transaction, physically removes the live `-challenges` directory post-commit, and either deletes the snapshot or restores from it on failure. |
|
| `backend/src/modules/admin/system/danger-zone.service.ts` | `resetScores()` deletes every `solve` row transactionally; `wipeChallenges()` snapshots `<UPLOAD_DIR>/challenges` to a side directory, deletes every challenge inside a transaction, physically removes the live `-challenges` directory post-commit, and either deletes the snapshot or restores from it on failure. |
|
||||||
| `backend/src/modules/admin/system/filesystem-transaction.service.ts` | Generic file/directory swap primitive: `stageSwap({ pairs, hooks })` renames each live path to a rollback location, then renames the staged path into place (with copy+unlink cross-device fallback); `commit(handle)` removes rollback artifacts; `rollback(handle)` restores the original live paths. Provides static helpers `rmSafe`, `copyDirSync`, `ensureDir`. |
|
| `backend/src/modules/admin/system/filesystem-transaction.service.ts` | Crash-safe multi-path swap primitive. Persists per-transaction phase/path manifests under `SYSTEM_OP_STAGING_DIR`, moves configured sidecars with their generation, commits or rolls back idempotently, restores a complete old generation after an interrupted snapshot, finalizes a complete verified new generation, fails closed on mixed/ambiguous state, and sweeps stale unowned rollback/restore/wipe artifacts. |
|
||||||
| `backend/src/modules/admin/system/confirmation-token.service.ts` | Issues SHA-256-hashed single-use tokens with `SYSTEM_OP_CONFIRM_TOKEN_TTL_MS` TTL; `consume()` runs in a transaction with conditional `WHERE consumedAt IS NULL` updates so only the first concurrent caller succeeds. `purgeExpired()` deletes expired and >24h-old consumed rows. |
|
| `backend/src/modules/admin/system/confirmation-token.service.ts` | Issues SHA-256-hashed single-use tokens with `SYSTEM_OP_CONFIRM_TOKEN_TTL_MS` TTL; `consume()` runs in a transaction with conditional `WHERE consumedAt IS NULL` updates so only the first concurrent caller succeeds. `purgeExpired()` deletes expired and >24h-old consumed rows. |
|
||||||
| `backend/src/modules/admin/system/dto/admin-system.dto.ts` | Re-exports `ADMIN_OPERATION_KINDS` and zod contracts for the re-auth body, danger confirm body, and restore-commit body; declares the backup format identifier + version constants. |
|
| `backend/src/modules/admin/system/dto/admin-system.dto.ts` | Re-exports `ADMIN_OPERATION_KINDS` and zod contracts for the re-auth body, danger confirm body, and restore-commit body; declares the backup format identifier + version constants. |
|
||||||
| `backend/src/database/entities/admin-operation-token.entity.ts` | TypeORM entity and the canonical `ADMIN_OPERATION_KINDS` literal union (`restore-backup`, `reset-scores`, `wipe-challenges`). |
|
| `backend/src/database/entities/admin-operation-token.entity.ts` | TypeORM entity and the canonical `ADMIN_OPERATION_KINDS` literal union (`restore-backup`, `reset-scores`, `wipe-challenges`). |
|
||||||
@@ -142,7 +143,9 @@ timestamp: 2026-07-23T13:27:25Z
|
|||||||
| `tests/backend/admin-system-confirmation-token.spec.ts` | Issue/consume/expire/reuse/mismatch flow and TTL config. |
|
| `tests/backend/admin-system-confirmation-token.spec.ts` | Issue/consume/expire/reuse/mismatch flow and TTL config. |
|
||||||
| `tests/backend/admin-system-danger.spec.ts` | Reset-scores and wipe-challenges happy path + DB/file rollback. |
|
| `tests/backend/admin-system-danger.spec.ts` | Reset-scores and wipe-challenges happy path + DB/file rollback. |
|
||||||
| `tests/backend/admin-system-restore-validation.spec.ts` | Archive schema validation, base64 size mismatch, sha256 mismatch, safe-path rejection, and duplicate-path rejection. |
|
| `tests/backend/admin-system-restore-validation.spec.ts` | Archive schema validation, base64 size mismatch, sha256 mismatch, safe-path rejection, and duplicate-path rejection. |
|
||||||
| `tests/backend/admin-system-restore-commit.spec.ts` | Controller forwards `stagingId` to `RestoreService.commitRestore`, plus a real-file service-level regression that restores a backup containing the same admin as the live system, asserts swapped table settings + uploads + trigger preservation, and verifies an admin-less archive is rejected with `SYSTEM_RESTORE_ROLLED_BACK`. |
|
| `tests/backend/admin-system-restore-commit.spec.ts` | Controller forwarding, real-file restore, trigger preservation, admin-less archive rejection, and kill-during-swap regression proving restart recovers one coherent database/uploads generation. |
|
||||||
|
| `tests/backend/database-init-recovery.spec.ts` | Startup recovery contracts: restores old generations from interrupted manifests, fails closed without mutation on hybrid state, and removes stale unowned artifacts. |
|
||||||
|
| `tests/backend/filesystem-transaction.spec.ts` | Durable manifest phase, commit/rollback, old/new generation recovery, ambiguous fail-closed, and stale artifact sweep contracts. |
|
||||||
| `frontend/src/app/features/admin/system/system.component.ts` | `/admin/system` smart page: two panels (Database + Danger Zone), backup download, restore pick + validate, confirm modal trigger, and per-operation success/error surfacing + cross-store invalidation. |
|
| `frontend/src/app/features/admin/system/system.component.ts` | `/admin/system` smart page: two panels (Database + Danger Zone), backup download, restore pick + validate, confirm modal trigger, and per-operation success/error surfacing + cross-store invalidation. |
|
||||||
| `frontend/src/app/features/admin/system/system-confirm-modal.component.ts` | Re-authentication + confirmation-phrase modal with operation-specific copy from `system.pure.ts`. |
|
| `frontend/src/app/features/admin/system/system-confirm-modal.component.ts` | Re-authentication + confirmation-phrase modal with operation-specific copy from `system.pure.ts`. |
|
||||||
| `frontend/src/app/features/admin/system/system.service.ts` | HTTP client for the six `/api/v1/admin/system/*` endpoints. |
|
| `frontend/src/app/features/admin/system/system.service.ts` | HTTP client for the six `/api/v1/admin/system/*` endpoints. |
|
||||||
|
|||||||
@@ -3,7 +3,7 @@ type: guide
|
|||||||
title: Admin — System (Backup, Restore, Danger Zone)
|
title: Admin — System (Backup, Restore, Danger Zone)
|
||||||
description: How an admin navigates to /admin/system, creates a backup, stages and commits a restore, resets all scores, and wipes challenges — including the re-authentication confirmation, progress, and side effects.
|
description: How an admin navigates to /admin/system, creates a backup, stages and commits a restore, resets all scores, and wipes challenges — including the re-authentication confirmation, progress, and side effects.
|
||||||
tags: [guide, admin, system, backup, restore, danger-zone, confirmation]
|
tags: [guide, admin, system, backup, restore, danger-zone, confirmation]
|
||||||
timestamp: 2026-07-23T12:04:50Z
|
timestamp: 2026-07-23T15:46:55Z
|
||||||
---
|
---
|
||||||
|
|
||||||
# Navigation
|
# Navigation
|
||||||
@@ -157,7 +157,13 @@ messages. Notable mappings:
|
|||||||
password, type `RESTORE BACKUP`, click **Confirm**.
|
password, type `RESTORE BACKUP`, click **Confirm**.
|
||||||
6. The modal closes, the page shows a brief *Restore complete.
|
6. The modal closes, the page shows a brief *Restore complete.
|
||||||
Logging out…* toast, and the SPA navigates to `/login`.
|
Logging out…* toast, and the SPA navigates to `/login`.
|
||||||
7. Sign in again with the same admin credentials. The data on the
|
7. If the server process is terminated during the database/uploads swap,
|
||||||
|
restart the server. Before accepting requests, startup recovery uses the
|
||||||
|
durable restore manifest to restore the complete old generation or finalize
|
||||||
|
the complete verified new generation; it never intentionally boots with a
|
||||||
|
mixed database/uploads generation. An ambiguous incomplete generation fails
|
||||||
|
startup closed and preserves the manifest and artifacts for diagnosis.
|
||||||
|
8. Sign in again with the same admin credentials. The data on the
|
||||||
target environment now matches the source backup.
|
target environment now matches the source backup.
|
||||||
|
|
||||||
## Manual reset happy path
|
## Manual reset happy path
|
||||||
|
|||||||
+1
-1
@@ -12,7 +12,7 @@ controls, administrator-managed Markdown blog publishing, and public and
|
|||||||
signed-in blog views.
|
signed-in blog views.
|
||||||
|
|
||||||
The docs below are organized by purpose so agents can pull just the slice
|
The docs below are organized by purpose so agents can pull just the slice
|
||||||
they need. Last regenerated 2026-07-23T13:27:25Z.
|
they need. Last regenerated 2026-07-23T15:46:55Z.
|
||||||
|
|
||||||
# Architecture
|
# Architecture
|
||||||
|
|
||||||
|
|||||||
@@ -356,3 +356,113 @@ describe('Job 918: Restore commit succeeds end-to-end with a real backup file',
|
|||||||
fs.rmSync(work, { recursive: true, force: true });
|
fs.rmSync(work, { recursive: true, force: true });
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
describe('Job 919: Restore commit survives a kill-9 mid-swap without leaving a hybrid on-disk state', () => {
|
||||||
|
// The reported Job 919 incident: a process kill during commit produced
|
||||||
|
// a hybrid where the live DB reverted to the old generation but the live
|
||||||
|
// uploads contained the new generation's file (and a rollback directory
|
||||||
|
// remained on disk). After the fix, a subsequent startup must clean
|
||||||
|
// the orphan rollback directory and bring the uploads back in line with
|
||||||
|
// the database.
|
||||||
|
|
||||||
|
beforeEach(() => {
|
||||||
|
jest.resetModules();
|
||||||
|
delete process.env.SYSTEM_OP_RESTORE_STAGE_TTL_MS;
|
||||||
|
delete process.env.SYSTEM_OP_CONFIRM_TOKEN_TTL_MS;
|
||||||
|
delete process.env.SYSTEM_OP_STAGING_DIR;
|
||||||
|
delete process.env.UPLOAD_DIR;
|
||||||
|
delete process.env.DATABASE_PATH;
|
||||||
|
});
|
||||||
|
|
||||||
|
it('simulated kill after the first pair swap leaves a manifest that startup recovery restores to a single coherent generation', async () => {
|
||||||
|
const fs = require('fs');
|
||||||
|
const os = require('os');
|
||||||
|
const path = require('path');
|
||||||
|
const Database = require('better-sqlite3');
|
||||||
|
|
||||||
|
const work = fs.mkdtempSync(path.join(os.tmpdir(), 'hipctf-job919-'));
|
||||||
|
const dataDir = path.join(work, 'data');
|
||||||
|
const uploadDir = path.join(dataDir, 'uploads');
|
||||||
|
fs.mkdirSync(uploadDir, { recursive: true });
|
||||||
|
const dbPath = path.join(dataDir, 'db.sqlite');
|
||||||
|
|
||||||
|
// Live state: OLD DB, OLD upload (forensics icon), with a stale rollback
|
||||||
|
// directory left behind by the prior broken swap attempt.
|
||||||
|
const DatabaseLib = require('better-sqlite3');
|
||||||
|
const live = new DatabaseLib(dbPath);
|
||||||
|
live.exec(`
|
||||||
|
CREATE TABLE "user" (id TEXT PRIMARY KEY, username TEXT, role TEXT);
|
||||||
|
CREATE TABLE "setting" (key TEXT PRIMARY KEY, value TEXT);
|
||||||
|
INSERT INTO "user" (id, username, role) VALUES ('admin-live', 'admin', 'admin');
|
||||||
|
INSERT INTO "setting" (key, value) VALUES ('site_title', 'OLD');
|
||||||
|
`);
|
||||||
|
live.close();
|
||||||
|
fs.writeFileSync(path.join(uploadDir, 'old-icon.png'), Buffer.from('OLD-ICON'));
|
||||||
|
const oldIconBytes = Buffer.from('OLD-ICON');
|
||||||
|
|
||||||
|
// Simulate the in-progress snapshot has already moved the live trees
|
||||||
|
// into the rollback locations, but the staged replacements never reached
|
||||||
|
// the live path. Place the OLD uploads content in the rollback dir.
|
||||||
|
const stamp = `${Date.now()}-sim`;
|
||||||
|
const oldDbRollback = path.join(dataDir, `.db.sqlite.rollback-${stamp}`);
|
||||||
|
const oldUpRollback = `${uploadDir}.rollback-${stamp}`;
|
||||||
|
fs.copyFileSync(dbPath, oldDbRollback);
|
||||||
|
fs.rmSync(dbPath, { recursive: true, force: true });
|
||||||
|
fs.mkdirSync(oldUpRollback, { recursive: true });
|
||||||
|
fs.copyFileSync(path.join(uploadDir, 'old-icon.png'), path.join(oldUpRollback, 'old-icon.png'));
|
||||||
|
fs.rmSync(uploadDir, { recursive: true, force: true });
|
||||||
|
|
||||||
|
const id = `${Date.now()}-kr`;
|
||||||
|
const stagingDir = path.join(dataDir, '.system-staging');
|
||||||
|
fs.mkdirSync(stagingDir, { recursive: true });
|
||||||
|
const txDir = path.join(stagingDir, id);
|
||||||
|
fs.mkdirSync(txDir, { recursive: true });
|
||||||
|
const manifest = {
|
||||||
|
schemaVersion: 1,
|
||||||
|
transactionId: id,
|
||||||
|
operation: 'restore-backup',
|
||||||
|
createdAt: new Date().toISOString(),
|
||||||
|
updatedAt: new Date().toISOString(),
|
||||||
|
phase: 'live-snapshotted',
|
||||||
|
pairs: [
|
||||||
|
{ livePath: dbPath, rollbackPath: oldDbRollback, stagedPath: path.join(stagingDir, 'unused.db'), contributesToGeneration: true, sidecarSuffixes: ['-wal', '-shm'] },
|
||||||
|
{ livePath: uploadDir, rollbackPath: oldUpRollback, stagedPath: path.join(stagingDir, 'unused.up'), contributesToGeneration: true, sidecarSuffixes: [] },
|
||||||
|
],
|
||||||
|
};
|
||||||
|
fs.writeFileSync(path.join(txDir, 'manifest.json'), JSON.stringify(manifest));
|
||||||
|
|
||||||
|
process.env.DATABASE_PATH = dbPath;
|
||||||
|
process.env.UPLOAD_DIR = uploadDir;
|
||||||
|
process.env.SYSTEM_OP_STAGING_DIR = stagingDir;
|
||||||
|
|
||||||
|
const { ConfigService } = require('@nestjs/config');
|
||||||
|
const config = new ConfigService({
|
||||||
|
DATABASE_PATH: dbPath,
|
||||||
|
UPLOAD_DIR: uploadDir,
|
||||||
|
SYSTEM_OP_STAGING_DIR: stagingDir,
|
||||||
|
SYSTEM_OP_RESTORE_STAGE_TTL_MS: 900000,
|
||||||
|
SYSTEM_OP_CONFIRM_TOKEN_TTL_MS: 300000,
|
||||||
|
BODY_SIZE_LIMIT: '50mb',
|
||||||
|
} as any);
|
||||||
|
const { DatabaseInitService } = require('../../backend/src/database/database-init.service');
|
||||||
|
const { FilesystemTransactionService } = require('../../backend/src/modules/admin/system/filesystem-transaction.service');
|
||||||
|
const fakeDataSource = { isInitialized: true, query: async () => [] };
|
||||||
|
const init = new DatabaseInitService(fakeDataSource, config, new FilesystemTransactionService());
|
||||||
|
|
||||||
|
await (init as any).recoverInterruptedSystemTransactions();
|
||||||
|
|
||||||
|
// Coherent pre-restore state on disk: DB and uploads agree on the OLD
|
||||||
|
// generation; the rollback artifacts and the manifest are gone.
|
||||||
|
expect(fs.existsSync(dbPath)).toBe(true);
|
||||||
|
const reopened = new Database(dbPath, { readonly: true });
|
||||||
|
expect(reopened.prepare('SELECT value FROM "setting" WHERE key=?').get('site_title').value).toBe('OLD');
|
||||||
|
expect(reopened.prepare('SELECT COUNT(*) AS c FROM "user"').get().c).toBe(1);
|
||||||
|
reopened.close();
|
||||||
|
expect(fs.readFileSync(path.join(uploadDir, 'old-icon.png'))).toEqual(oldIconBytes);
|
||||||
|
expect(fs.existsSync(oldDbRollback)).toBe(false);
|
||||||
|
expect(fs.existsSync(oldUpRollback)).toBe(false);
|
||||||
|
expect(fs.existsSync(txDir)).toBe(false);
|
||||||
|
|
||||||
|
try { fs.rmSync(work, { recursive: true, force: true }); } catch { /* ignore */ }
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|||||||
@@ -0,0 +1,169 @@
|
|||||||
|
process.env.DATABASE_PATH = ':memory:';
|
||||||
|
process.env.THEMES_DIR = './themes';
|
||||||
|
process.env.FRONTEND_DIST = './frontend/dist';
|
||||||
|
process.env.UPLOAD_SIZE_LIMIT = '5mb';
|
||||||
|
process.env.NODE_ENV = 'test';
|
||||||
|
|
||||||
|
import { ConfigService } from '@nestjs/config';
|
||||||
|
|
||||||
|
// ConfigService reads process.env with precedence over its internal
|
||||||
|
// overrides, so unset the env-driven defaults before constructing it
|
||||||
|
// for these tests so the per-test data roots are honored.
|
||||||
|
delete process.env.DATABASE_PATH;
|
||||||
|
delete process.env.UPLOAD_DIR;
|
||||||
|
delete process.env.SYSTEM_OP_STAGING_DIR;
|
||||||
|
import * as fs from 'fs';
|
||||||
|
import * as os from 'os';
|
||||||
|
import * as path from 'path';
|
||||||
|
import { DatabaseInitService } from '../../backend/src/database/database-init.service';
|
||||||
|
import {
|
||||||
|
FilesystemTransactionService,
|
||||||
|
TransactionManifest,
|
||||||
|
} from '../../backend/src/modules/admin/system/filesystem-transaction.service';
|
||||||
|
import { ERROR_CODES } from '../../backend/src/common/errors/error-codes';
|
||||||
|
|
||||||
|
function makeRoot(prefix: string): string {
|
||||||
|
return fs.mkdtempSync(path.join(os.tmpdir(), prefix));
|
||||||
|
}
|
||||||
|
|
||||||
|
describe('Job 919: DatabaseInitService recovery on startup', () => {
|
||||||
|
let work: string;
|
||||||
|
let dbPath: string;
|
||||||
|
let uploadDir: string;
|
||||||
|
let stagingDir: string;
|
||||||
|
|
||||||
|
beforeEach(() => {
|
||||||
|
work = makeRoot('hipctf-job919-init-');
|
||||||
|
dbPath = path.join(work, 'db.sqlite');
|
||||||
|
uploadDir = path.join(work, 'uploads');
|
||||||
|
stagingDir = path.join(work, '.system-staging');
|
||||||
|
// Note: `uploadDir` is intentionally NOT created here. Individual tests
|
||||||
|
// arrange the filesystem in the exact on-disk shape they want to
|
||||||
|
// exercise (e.g. an interrupted-swap manifest expects the live uploads
|
||||||
|
// dir to NOT exist on disk because the in-process snapshot has moved it).
|
||||||
|
});
|
||||||
|
|
||||||
|
afterEach(() => {
|
||||||
|
if (work && fs.existsSync(work)) {
|
||||||
|
try { fs.rmSync(work, { recursive: true, force: true }); } catch { /* ignore */ }
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
function buildInit(): DatabaseInitService {
|
||||||
|
const config = new ConfigService({
|
||||||
|
DATABASE_PATH: dbPath,
|
||||||
|
UPLOAD_DIR: uploadDir,
|
||||||
|
SYSTEM_OP_STAGING_DIR: stagingDir,
|
||||||
|
} as any);
|
||||||
|
const fsTx = new FilesystemTransactionService();
|
||||||
|
// The DataSource is not used by the recovery code path, so a stub suffices.
|
||||||
|
const fakeDataSource: any = { isInitialized: true, query: async () => [] };
|
||||||
|
const init = new DatabaseInitService(fakeDataSource, config, fsTx);
|
||||||
|
return init;
|
||||||
|
}
|
||||||
|
|
||||||
|
it('restores the old database + uploads generation from an interrupted restore manifest', async () => {
|
||||||
|
const stamp = `${Date.now()}-11111`;
|
||||||
|
const liveDb = dbPath;
|
||||||
|
const liveUploads = uploadDir;
|
||||||
|
const oldDbRollback = `${liveDb}.rollback-${stamp}`;
|
||||||
|
const oldUpRollback = `${liveUploads}.rollback-${stamp}`;
|
||||||
|
const stagedDb = path.join(stagingDir, 'unused.db');
|
||||||
|
const stagedUploads = path.join(stagingDir, 'unused.up');
|
||||||
|
|
||||||
|
// Simulate the in-process state AFTER the live-snapshot pair rename has
|
||||||
|
// moved the live tree to the rollback location but BEFORE the staged
|
||||||
|
// replacement has been moved into place.
|
||||||
|
fs.writeFileSync(oldDbRollback, 'OLD-DB-SNAPSHOT');
|
||||||
|
fs.mkdirSync(oldUpRollback, { recursive: true });
|
||||||
|
fs.writeFileSync(path.join(oldUpRollback, 'keep.txt'), 'OLD-UPLOAD-SNAPSHOT');
|
||||||
|
|
||||||
|
const id = `${Date.now()}-tyy`;
|
||||||
|
const txDir = path.join(stagingDir, id);
|
||||||
|
fs.mkdirSync(txDir, { recursive: true });
|
||||||
|
const manifest: TransactionManifest = {
|
||||||
|
schemaVersion: 1,
|
||||||
|
transactionId: id,
|
||||||
|
operation: 'restore-backup',
|
||||||
|
createdAt: new Date().toISOString(),
|
||||||
|
updatedAt: new Date().toISOString(),
|
||||||
|
phase: 'live-snapshotted',
|
||||||
|
pairs: [
|
||||||
|
{ livePath: liveDb, rollbackPath: oldDbRollback, stagedPath: stagedDb, contributesToGeneration: true, sidecarSuffixes: ['-wal', '-shm'] },
|
||||||
|
{ livePath: liveUploads, rollbackPath: oldUpRollback, stagedPath: stagedUploads, contributesToGeneration: true, sidecarSuffixes: [] },
|
||||||
|
],
|
||||||
|
};
|
||||||
|
fs.writeFileSync(path.join(txDir, FilesystemTransactionService.MANIFEST_FILENAME), JSON.stringify(manifest));
|
||||||
|
|
||||||
|
const init = buildInit();
|
||||||
|
await (init as any).recoverInterruptedSystemTransactions();
|
||||||
|
|
||||||
|
expect(fs.readFileSync(liveDb, 'utf8')).toBe('OLD-DB-SNAPSHOT');
|
||||||
|
expect(fs.readFileSync(path.join(liveUploads, 'keep.txt'), 'utf8')).toBe('OLD-UPLOAD-SNAPSHOT');
|
||||||
|
expect(fs.existsSync(path.join(liveUploads, 'replaced.txt'))).toBe(false);
|
||||||
|
expect(fs.existsSync(oldDbRollback)).toBe(false);
|
||||||
|
expect(fs.existsSync(oldUpRollback)).toBe(false);
|
||||||
|
expect(fs.existsSync(txDir)).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('fails closed (throws) when an interrupted manifest has a hybrid state (no full old or new generation)', async () => {
|
||||||
|
const stamp = `${Date.now()}-22222`;
|
||||||
|
const oldDbRollback = `${dbPath}.rollback-${stamp}`;
|
||||||
|
fs.writeFileSync(oldDbRollback, 'OLD-DB-SNAPSHOT');
|
||||||
|
fs.mkdirSync(uploadDir, { recursive: true });
|
||||||
|
fs.writeFileSync(path.join(uploadDir, 'replaced.txt'), 'NEW-ORPHAN');
|
||||||
|
const oldUpRollback = `${uploadDir}.rollback-${stamp}`;
|
||||||
|
fs.mkdirSync(oldUpRollback, { recursive: true });
|
||||||
|
fs.writeFileSync(path.join(oldUpRollback, 'keep.txt'), 'OLD-UPLOAD-SNAPSHOT');
|
||||||
|
|
||||||
|
const id = `${Date.now()}-hyb`;
|
||||||
|
const txDir = path.join(stagingDir, id);
|
||||||
|
fs.mkdirSync(txDir, { recursive: true });
|
||||||
|
const manifest: TransactionManifest = {
|
||||||
|
schemaVersion: 1,
|
||||||
|
transactionId: id,
|
||||||
|
operation: 'restore-backup',
|
||||||
|
createdAt: new Date().toISOString(),
|
||||||
|
updatedAt: new Date().toISOString(),
|
||||||
|
phase: 'verified',
|
||||||
|
pairs: [
|
||||||
|
{ livePath: dbPath, rollbackPath: oldDbRollback, stagedPath: path.join(stagingDir, 'unused.db'), contributesToGeneration: true, sidecarSuffixes: [] },
|
||||||
|
{ livePath: uploadDir, rollbackPath: oldUpRollback, stagedPath: path.join(stagingDir, 'unused.up'), contributesToGeneration: true, sidecarSuffixes: [] },
|
||||||
|
],
|
||||||
|
};
|
||||||
|
fs.writeFileSync(path.join(txDir, FilesystemTransactionService.MANIFEST_FILENAME), JSON.stringify(manifest));
|
||||||
|
|
||||||
|
const init = buildInit();
|
||||||
|
await expect((init as any).recoverInterruptedSystemTransactions()).rejects.toMatchObject({
|
||||||
|
code: ERROR_CODES.SYSTEM_RESTORE_FAILED,
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(fs.readFileSync(oldDbRollback, 'utf8')).toBe('OLD-DB-SNAPSHOT');
|
||||||
|
expect(fs.readFileSync(path.join(uploadDir, 'replaced.txt'), 'utf8')).toBe('NEW-ORPHAN');
|
||||||
|
expect(fs.readFileSync(path.join(oldUpRollback, 'keep.txt'), 'utf8')).toBe('OLD-UPLOAD-SNAPSHOT');
|
||||||
|
expect(fs.existsSync(txDir)).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('removes stale unowned rollback/restore-stage/wipe-stage artifacts during startup sweep', async () => {
|
||||||
|
const now = Date.now();
|
||||||
|
const oldStamp = now - 2 * 24 * 60 * 60_000; // 2 days old
|
||||||
|
|
||||||
|
const staleDbRollback = path.join(work, `.db.sqlite.rollback-${oldStamp}-abc`);
|
||||||
|
const staleUploadsRollback = `${uploadDir}.rollback-${oldStamp}-def`;
|
||||||
|
const staleUploadsStage = `${uploadDir}.restore-stage-${oldStamp}-ghi`;
|
||||||
|
fs.writeFileSync(staleDbRollback, 'orphan');
|
||||||
|
fs.mkdirSync(staleUploadsRollback, { recursive: true });
|
||||||
|
fs.mkdirSync(staleUploadsStage, { recursive: true });
|
||||||
|
const oldTime = new Date(oldStamp);
|
||||||
|
fs.utimesSync(staleDbRollback, oldTime, oldTime);
|
||||||
|
fs.utimesSync(staleUploadsRollback, oldTime, oldTime);
|
||||||
|
fs.utimesSync(staleUploadsStage, oldTime, oldTime);
|
||||||
|
|
||||||
|
const init = buildInit();
|
||||||
|
await (init as any).recoverInterruptedSystemTransactions();
|
||||||
|
|
||||||
|
expect(fs.existsSync(staleDbRollback)).toBe(false);
|
||||||
|
expect(fs.existsSync(staleUploadsRollback)).toBe(false);
|
||||||
|
expect(fs.existsSync(staleUploadsStage)).toBe(false);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,300 @@
|
|||||||
|
process.env.DATABASE_PATH = ':memory:';
|
||||||
|
process.env.THEMES_DIR = './themes';
|
||||||
|
process.env.FRONTEND_DIST = './frontend/dist';
|
||||||
|
process.env.UPLOAD_SIZE_LIMIT = '5mb';
|
||||||
|
process.env.NODE_ENV = 'test';
|
||||||
|
|
||||||
|
import * as fs from 'fs';
|
||||||
|
import * as os from 'os';
|
||||||
|
import * as path from 'path';
|
||||||
|
import { FilesystemTransactionService } from '../../backend/src/modules/admin/system/filesystem-transaction.service';
|
||||||
|
import { ERROR_CODES } from '../../backend/src/common/errors/error-codes';
|
||||||
|
import { TransactionManifest } from '../../backend/src/modules/admin/system/filesystem-transaction.service';
|
||||||
|
|
||||||
|
function makeRoot(prefix: string): { root: string; cleanup: () => void } {
|
||||||
|
const root = fs.mkdtempSync(path.join(os.tmpdir(), prefix));
|
||||||
|
return {
|
||||||
|
root,
|
||||||
|
cleanup: () => {
|
||||||
|
try { fs.rmSync(root, { recursive: true, force: true }); } catch { /* ignore */ }
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function readManifest(stagingDir: string, id: string): TransactionManifest {
|
||||||
|
const full = path.join(stagingDir, id, FilesystemTransactionService.MANIFEST_FILENAME);
|
||||||
|
return JSON.parse(fs.readFileSync(full, 'utf8')) as TransactionManifest;
|
||||||
|
}
|
||||||
|
|
||||||
|
describe('Job 919: FilesystemTransactionService durable manifest', () => {
|
||||||
|
let svc: FilesystemTransactionService;
|
||||||
|
let dataRoot: { root: string; cleanup: () => void };
|
||||||
|
|
||||||
|
beforeEach(() => {
|
||||||
|
svc = new FilesystemTransactionService();
|
||||||
|
dataRoot = makeRoot('hipctf-fs-tx-');
|
||||||
|
});
|
||||||
|
|
||||||
|
afterEach(() => {
|
||||||
|
dataRoot.cleanup();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('stageSwap + commit leaves the new generation in place and removes rollback/staged artifacts', async () => {
|
||||||
|
const live = path.join(dataRoot.root, 'live.txt');
|
||||||
|
const staged = path.join(dataRoot.root, 'staged.txt');
|
||||||
|
fs.writeFileSync(live, 'OLD');
|
||||||
|
fs.writeFileSync(staged, 'NEW');
|
||||||
|
|
||||||
|
const handle = await svc.stageSwap(
|
||||||
|
{
|
||||||
|
name: 'unit-swap',
|
||||||
|
pairs: [{ livePath: live, stagedPath: staged, contributesToGeneration: true }],
|
||||||
|
},
|
||||||
|
{ stagingDir: dataRoot.root },
|
||||||
|
);
|
||||||
|
expect(handle.transactionId).toBeTruthy();
|
||||||
|
expect(fs.readFileSync(live, 'utf8')).toBe('NEW');
|
||||||
|
// Manifest exists and is at the `verified` phase before commit
|
||||||
|
const manifest = readManifest(dataRoot.root, handle.transactionId!);
|
||||||
|
expect(manifest.phase).toBe('verified');
|
||||||
|
|
||||||
|
svc.commit(handle);
|
||||||
|
expect(fs.existsSync(handle.manifestDir!)).toBe(false);
|
||||||
|
expect(fs.readFileSync(live, 'utf8')).toBe('NEW');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('stageSwap failure rolls the live path back to its prior content and marks manifest rolled-back', async () => {
|
||||||
|
const dataDir = path.join(dataRoot.root, 'data');
|
||||||
|
fs.mkdirSync(dataDir, { recursive: true });
|
||||||
|
const stagingRoot = path.join(dataRoot.root, '.system-staging');
|
||||||
|
fs.mkdirSync(stagingRoot, { recursive: true });
|
||||||
|
|
||||||
|
const live = path.join(dataDir, 'live.bin');
|
||||||
|
const staged = path.join(dataDir, 'staged.bin');
|
||||||
|
fs.writeFileSync(live, 'OLD-CONTENT');
|
||||||
|
fs.writeFileSync(staged, 'NEW-CONTENT');
|
||||||
|
|
||||||
|
await expect(
|
||||||
|
svc.stageSwap(
|
||||||
|
{
|
||||||
|
name: 'unit-swap-fail',
|
||||||
|
pairs: [{ livePath: live, stagedPath: staged, contributesToGeneration: true }],
|
||||||
|
hooks: {
|
||||||
|
beforeSwap: async () => {
|
||||||
|
throw new Error('boom');
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{ stagingDir: stagingRoot },
|
||||||
|
),
|
||||||
|
).rejects.toMatchObject({ message: 'boom' });
|
||||||
|
|
||||||
|
expect(fs.readFileSync(live, 'utf8')).toBe('OLD-CONTENT');
|
||||||
|
const manifestDirs = fs.readdirSync(stagingRoot);
|
||||||
|
expect(manifestDirs.length).toBe(1);
|
||||||
|
const manifest = readManifest(stagingRoot, manifestDirs[0]);
|
||||||
|
expect(manifest.phase).toBe('rolled-back');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('recoverTransactions restores the OLD generation when the manifest is at live-snapshotted and both rollback dirs are intact', () => {
|
||||||
|
const liveDb = path.join(dataRoot.root, 'db.sqlite');
|
||||||
|
const stagedDb = path.join(dataRoot.root, 'db.staged.sqlite');
|
||||||
|
fs.writeFileSync(liveDb, 'OLD-DB');
|
||||||
|
fs.writeFileSync(stagedDb, 'NEW-DB');
|
||||||
|
|
||||||
|
const liveUploads = path.join(dataRoot.root, 'uploads');
|
||||||
|
const stagedUploads = path.join(dataRoot.root, 'uploads.stage');
|
||||||
|
fs.mkdirSync(liveUploads, { recursive: true });
|
||||||
|
fs.mkdirSync(stagedUploads, { recursive: true });
|
||||||
|
fs.writeFileSync(path.join(liveUploads, 'keep.txt'), 'OLD-UPLOAD');
|
||||||
|
fs.writeFileSync(path.join(stagedUploads, 'replaced.txt'), 'NEW-UPLOAD');
|
||||||
|
|
||||||
|
const id = `${Date.now()}-abcdef`;
|
||||||
|
const txDir = path.join(dataRoot.root, id);
|
||||||
|
fs.mkdirSync(txDir, { recursive: true });
|
||||||
|
const stamp = `${Date.now()}-12345`;
|
||||||
|
const manifest: TransactionManifest = {
|
||||||
|
schemaVersion: 1,
|
||||||
|
transactionId: id,
|
||||||
|
operation: 'restore-backup',
|
||||||
|
createdAt: new Date().toISOString(),
|
||||||
|
updatedAt: new Date().toISOString(),
|
||||||
|
phase: 'live-snapshotted',
|
||||||
|
pairs: [
|
||||||
|
{
|
||||||
|
livePath: liveDb,
|
||||||
|
rollbackPath: `${liveDb}.rollback-${stamp}`,
|
||||||
|
stagedPath: stagedDb,
|
||||||
|
contributesToGeneration: true,
|
||||||
|
sidecarSuffixes: ['-wal', '-shm'],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
livePath: liveUploads,
|
||||||
|
rollbackPath: `${liveUploads}.rollback-${stamp}`,
|
||||||
|
stagedPath: stagedUploads,
|
||||||
|
contributesToGeneration: true,
|
||||||
|
sidecarSuffixes: [],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
};
|
||||||
|
fs.writeFileSync(path.join(txDir, FilesystemTransactionService.MANIFEST_FILENAME), JSON.stringify(manifest));
|
||||||
|
// Simulate the in-progress state where the snapshot-to-rollback has already happened.
|
||||||
|
fs.renameSync(liveDb, manifest.pairs[0].rollbackPath);
|
||||||
|
fs.renameSync(liveUploads, manifest.pairs[1].rollbackPath);
|
||||||
|
|
||||||
|
const report = svc.recoverTransactions(dataRoot.root);
|
||||||
|
expect(report.transactionsScanned).toBe(1);
|
||||||
|
expect(report.resolved).toBe(1);
|
||||||
|
expect(fs.readFileSync(liveDb, 'utf8')).toBe('OLD-DB');
|
||||||
|
expect(fs.readFileSync(path.join(liveUploads, 'keep.txt'), 'utf8')).toBe('OLD-UPLOAD');
|
||||||
|
// Staged trees should not have ended up at the live paths.
|
||||||
|
expect(fs.existsSync(stagedDb)).toBe(true);
|
||||||
|
expect(fs.existsSync(stagedUploads)).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('recoverTransactions keeps the NEW generation when both pairs swapped but manifest is at verified', () => {
|
||||||
|
const liveDb = path.join(dataRoot.root, 'db.sqlite');
|
||||||
|
const liveUploads = path.join(dataRoot.root, 'uploads');
|
||||||
|
fs.writeFileSync(liveDb, 'NEW-DB');
|
||||||
|
fs.mkdirSync(liveUploads, { recursive: true });
|
||||||
|
fs.writeFileSync(path.join(liveUploads, 'replaced.txt'), 'NEW-UPLOAD');
|
||||||
|
|
||||||
|
const stamp = `${Date.now()}-99999`;
|
||||||
|
const oldDbPath = `${liveDb}.rollback-${stamp}`;
|
||||||
|
const oldUpPath = `${liveUploads}.rollback-${stamp}`;
|
||||||
|
fs.writeFileSync(oldDbPath, 'OLD-DB');
|
||||||
|
fs.mkdirSync(oldUpPath, { recursive: true });
|
||||||
|
fs.writeFileSync(path.join(oldUpPath, 'keep.txt'), 'OLD-UPLOAD');
|
||||||
|
|
||||||
|
const id = `${Date.now()}-x9`;
|
||||||
|
const txDir = path.join(dataRoot.root, id);
|
||||||
|
fs.mkdirSync(txDir, { recursive: true });
|
||||||
|
const manifest: TransactionManifest = {
|
||||||
|
schemaVersion: 1,
|
||||||
|
transactionId: id,
|
||||||
|
operation: 'restore-backup',
|
||||||
|
createdAt: new Date().toISOString(),
|
||||||
|
updatedAt: new Date().toISOString(),
|
||||||
|
phase: 'verified',
|
||||||
|
pairs: [
|
||||||
|
{ livePath: liveDb, rollbackPath: oldDbPath, stagedPath: path.join(dataRoot.root, 'unused.db'), contributesToGeneration: true, sidecarSuffixes: [] },
|
||||||
|
{ livePath: liveUploads, rollbackPath: oldUpPath, stagedPath: path.join(dataRoot.root, 'unused.up'), contributesToGeneration: true, sidecarSuffixes: [] },
|
||||||
|
],
|
||||||
|
};
|
||||||
|
fs.writeFileSync(path.join(txDir, FilesystemTransactionService.MANIFEST_FILENAME), JSON.stringify(manifest));
|
||||||
|
|
||||||
|
const report = svc.recoverTransactions(dataRoot.root);
|
||||||
|
expect(report.resolved).toBe(1);
|
||||||
|
expect(fs.readFileSync(liveDb, 'utf8')).toBe('NEW-DB');
|
||||||
|
expect(fs.readFileSync(path.join(liveUploads, 'replaced.txt'), 'utf8')).toBe('NEW-UPLOAD');
|
||||||
|
// Rollback trees for the (now finalized) new generation must be gone.
|
||||||
|
expect(fs.existsSync(oldDbPath)).toBe(false);
|
||||||
|
expect(fs.existsSync(oldUpPath)).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('recoverTransactions fails closed when neither generation is fully present (the Job 919 hybrid state)', () => {
|
||||||
|
// Reproduce the actual reported hybrid: the swap committed the new
|
||||||
|
// uploads tree to the live path and wrote a rollback of the OLD
|
||||||
|
// uploads tree, but the DB live→rollback snapshot happened while the
|
||||||
|
// staged DB never reached the live path. From the resolver's point
|
||||||
|
// of view the DB pair looks "old-complete" (livePath missing,
|
||||||
|
// rollbackPath present), the uploads pair looks "new-complete"
|
||||||
|
// (livePath present, rollbackPath present). Both pairs are not
|
||||||
|
// consistent; recovery must fail closed without modifying either
|
||||||
|
// generation.
|
||||||
|
const dataDir = path.join(dataRoot.root, 'data');
|
||||||
|
fs.mkdirSync(dataDir, { recursive: true });
|
||||||
|
const liveDb = path.join(dataDir, 'db.sqlite');
|
||||||
|
const liveUploads = path.join(dataDir, 'uploads');
|
||||||
|
fs.mkdirSync(liveUploads, { recursive: true });
|
||||||
|
fs.writeFileSync(path.join(liveUploads, 'replaced.txt'), 'NEW-ORPHAN'); // stale from a failed swap
|
||||||
|
|
||||||
|
const stamp = `${Date.now()}-hybrid`;
|
||||||
|
const oldDbRollback = `${liveDb}.rollback-${stamp}`;
|
||||||
|
const oldUpRollback = `${liveUploads}.rollback-${stamp}`;
|
||||||
|
fs.writeFileSync(oldDbRollback, 'OLD-DB-SNAPSHOT');
|
||||||
|
fs.mkdirSync(oldUpRollback, { recursive: true });
|
||||||
|
fs.writeFileSync(path.join(oldUpRollback, 'keep.txt'), 'OLD-UPLOAD-SNAPSHOT');
|
||||||
|
|
||||||
|
const id = `${Date.now()}-hybrid`;
|
||||||
|
const stagingRoot = path.join(dataRoot.root, '.system-staging');
|
||||||
|
fs.mkdirSync(stagingRoot, { recursive: true });
|
||||||
|
const txDir = path.join(stagingRoot, id);
|
||||||
|
fs.mkdirSync(txDir, { recursive: true });
|
||||||
|
const manifest: TransactionManifest = {
|
||||||
|
schemaVersion: 1,
|
||||||
|
transactionId: id,
|
||||||
|
operation: 'restore-backup',
|
||||||
|
createdAt: new Date().toISOString(),
|
||||||
|
updatedAt: new Date().toISOString(),
|
||||||
|
phase: 'verified',
|
||||||
|
pairs: [
|
||||||
|
{ livePath: liveDb, rollbackPath: oldDbRollback, stagedPath: path.join(dataDir, 'unused.db'), contributesToGeneration: true, sidecarSuffixes: [] },
|
||||||
|
{ livePath: liveUploads, rollbackPath: oldUpRollback, stagedPath: path.join(dataDir, 'unused.up'), contributesToGeneration: true, sidecarSuffixes: [] },
|
||||||
|
],
|
||||||
|
};
|
||||||
|
fs.writeFileSync(path.join(txDir, FilesystemTransactionService.MANIFEST_FILENAME), JSON.stringify(manifest));
|
||||||
|
|
||||||
|
expect(() => svc.recoverTransactions(stagingRoot)).toThrow(
|
||||||
|
expect.objectContaining({ code: ERROR_CODES.SYSTEM_RESTORE_FAILED }),
|
||||||
|
);
|
||||||
|
|
||||||
|
// Fail-closed must NOT delete or move anything in either generation.
|
||||||
|
expect(fs.readFileSync(oldDbRollback, 'utf8')).toBe('OLD-DB-SNAPSHOT');
|
||||||
|
expect(fs.readFileSync(path.join(liveUploads, 'replaced.txt'), 'utf8')).toBe('NEW-ORPHAN');
|
||||||
|
expect(fs.readFileSync(path.join(oldUpRollback, 'keep.txt'), 'utf8')).toBe('OLD-UPLOAD-SNAPSHOT');
|
||||||
|
// Manifest stays on disk for the operator.
|
||||||
|
expect(fs.existsSync(txDir)).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('sweepUnownedArtifactsNearLivePaths cleans up stale rollback/restore-stage/wipe-stage directories', () => {
|
||||||
|
const dataDir = path.join(dataRoot.root, 'data');
|
||||||
|
fs.mkdirSync(dataDir, { recursive: true });
|
||||||
|
const liveDb = path.join(dataDir, 'db.sqlite');
|
||||||
|
fs.writeFileSync(liveDb, 'OLD');
|
||||||
|
const liveUploads = path.join(dataDir, 'uploads');
|
||||||
|
fs.mkdirSync(liveUploads, { recursive: true });
|
||||||
|
const challenges = path.join(liveUploads, 'challenges');
|
||||||
|
fs.mkdirSync(challenges, { recursive: true });
|
||||||
|
|
||||||
|
const now = Date.now();
|
||||||
|
const oldStamp = now - 2 * 24 * 60 * 60_000; // 2 days old
|
||||||
|
const freshStamp = now - 60_000; // 1 minute old
|
||||||
|
|
||||||
|
const rollbackUploads = `${liveUploads}.rollback-${oldStamp}-abc`;
|
||||||
|
const rollbackDb = path.join(dataDir, `.db.sqlite.rollback-${oldStamp}-def`);
|
||||||
|
const stageUploads = `${liveUploads}.restore-stage-${oldStamp}-ghi`;
|
||||||
|
const wipeChallenges = path.join(challenges, `.challenges.wipe-stage-${oldStamp}-jkl`);
|
||||||
|
fs.mkdirSync(rollbackUploads, { recursive: true });
|
||||||
|
fs.writeFileSync(rollbackDb, 'orphan');
|
||||||
|
fs.mkdirSync(stageUploads, { recursive: true });
|
||||||
|
fs.mkdirSync(wipeChallenges, { recursive: true });
|
||||||
|
|
||||||
|
// A fresh rollback pattern must NOT be removed.
|
||||||
|
const freshRollbackUploads = `${liveUploads}.rollback-${freshStamp}-mno`;
|
||||||
|
fs.mkdirSync(freshRollbackUploads, { recursive: true });
|
||||||
|
|
||||||
|
// Backdate the old artifacts.
|
||||||
|
const oldTime = new Date(oldStamp);
|
||||||
|
fs.utimesSync(rollbackUploads, oldTime, oldTime);
|
||||||
|
fs.utimesSync(rollbackDb, oldTime, oldTime);
|
||||||
|
fs.utimesSync(stageUploads, oldTime, oldTime);
|
||||||
|
fs.utimesSync(wipeChallenges, oldTime, oldTime);
|
||||||
|
|
||||||
|
const cleaned = svc.sweepUnownedArtifactsNearLivePaths([liveDb, liveUploads], {
|
||||||
|
now,
|
||||||
|
maxAgeMs: 60 * 60_000,
|
||||||
|
stagingDir: dataRoot.root,
|
||||||
|
});
|
||||||
|
expect(cleaned).toEqual(expect.arrayContaining([rollbackUploads, rollbackDb, stageUploads, wipeChallenges]));
|
||||||
|
expect(fs.existsSync(rollbackUploads)).toBe(false);
|
||||||
|
expect(fs.existsSync(rollbackDb)).toBe(false);
|
||||||
|
expect(fs.existsSync(stageUploads)).toBe(false);
|
||||||
|
expect(fs.existsSync(wipeChallenges)).toBe(false);
|
||||||
|
// Fresh rollback path remains.
|
||||||
|
expect(fs.existsSync(freshRollbackUploads)).toBe(true);
|
||||||
|
|
||||||
|
// Cleanup helper for the remaining fresh artifact.
|
||||||
|
try { fs.rmSync(freshRollbackUploads, { recursive: true, force: true }); } catch { /* ignore */ }
|
||||||
|
});
|
||||||
|
});
|
||||||
Reference in New Issue
Block a user