From b02369dd4117f1bbacd1d9ea54b78423787effb3 Mon Sep 17 00:00:00 2001 From: OpenVelo Agent Date: Wed, 22 Jul 2026 16:45:58 +0000 Subject: [PATCH] docs: update documentation to OKF v0.1 format --- docs/architecture/key-files.md | 7 +- docs/database/category-repair-migration.md | 107 +++++++++++++++++++++ docs/database/challenges.md | 13 ++- docs/database/schema.md | 15 ++- docs/index.md | 6 +- 5 files changed, 135 insertions(+), 13 deletions(-) create mode 100644 docs/database/category-repair-migration.md diff --git a/docs/architecture/key-files.md b/docs/architecture/key-files.md index 4c9816e..e52dc6e 100644 --- a/docs/architecture/key-files.md +++ b/docs/architecture/key-files.md @@ -2,8 +2,8 @@ type: architecture title: Key Files Index description: One-line responsibility for important source and contract-test files, including strict event-window validation and the public bootstrap SSE listener. -tags: [architecture, key-files, event-window, validation, default-challenge-ip, sse, bootstrap] -timestamp: 2026-07-22T16:15:00Z +tags: [architecture, key-files, event-window, validation, default-challenge-ip, sse, bootstrap, migrations] +timestamp: 2026-07-22T16:44:54Z --- # Backend @@ -13,7 +13,8 @@ timestamp: 2026-07-22T16:15:00Z | `backend/src/main.ts` | Bootstraps Nest, middleware, OpenAPI, static assets, and SPA fallback. | | `backend/src/app.module.ts` | Wires feature modules and global providers. | | `backend/src/database/database.module.ts` | Configures TypeORM with SQLite. | -| `backend/src/database/database-init.service.ts` | Initializes the database and runs migrations. | +| `backend/src/database/database-init.service.ts` | Initializes the database and runs migrations; `verifySeed()` also asserts the canonical six system-category keys and reports missing/duplicate rows. | +| `backend/src/database/migrations/1700000000400-RepairCategorySchemaAndSystemCategories.ts` | Forward-only repair migration: adds `created_at`/`updated_at` to legacy six-column `category` tables, deletes obsolete `FOR`/`OSI` system rows, rewrites canonical row metadata, inserts any missing canonical key, and re-asserts unique indexes. Exports `CANONICAL_SYSTEM_CATEGORIES`. | | `backend/src/common/guards/jwt-auth.guard.ts` | Global JWT authorization guard. | | `backend/src/common/middleware/csrf.middleware.ts` | Double-submit CSRF protection. | | `backend/src/common/services/event-status.service.ts` | Computes the event-window state machine. | diff --git a/docs/database/category-repair-migration.md b/docs/database/category-repair-migration.md new file mode 100644 index 0000000..809a910 --- /dev/null +++ b/docs/database/category-repair-migration.md @@ -0,0 +1,107 @@ +--- +type: database +title: Category Repair Migration (1700000000400) +description: Forward-only migration that repairs legacy `category` schemas (adds timestamps) and reconciles system rows to the canonical CRY/HW/MSC/PWN/REV/WEB set. +tags: [database, migration, category, repair, system-categories] +timestamp: 2026-07-22T16:44:54Z +--- + +# Purpose + +Some persistent `/data/hipctf/db.sqlite` databases were created before +the canonical six system categories existed. They contain a legacy +six-column `category` table (no `created_at` / `updated_at`) and +system rows whose `system_key` / `abbreviation` no longer match the +canonical set (e.g. `FOR` for Forensics, `OSI` for OSINT, missing +`HW` / `REV`). + +When the backend starts against such a database, `CategoryEntity` +fails to `SELECT c.*` because the timestamp columns are absent, and +`GET /api/v1/admin/categories` returns 500. Migrations that may +already be recorded in those databases cannot be edited or relied on +to re-run, so a new forward-only migration is required. + +# File + +`backend/src/database/migrations/1700000000400-RepairCategorySchemaAndSystemCategories.ts` + +Registered last in `backend/src/database/database.module.ts` so it +runs after `UpdateSystemCategoryKeys1700000000300`. + +# Canonical system categories + +Exported as `CANONICAL_SYSTEM_CATEGORIES` (typed +`ReadonlyArray`) so the migration and any +future seed consumer share one source of truth: + +| `system_key` | `name` | `abbreviation` | `description` | `icon_path` | +|--------------|-------------------------|----------------|------------------------------|------------------------------| +| `CRY` | Cryptography | `CRY` | Cryptographic challenges | `/uploads/icons/CRY.png` | +| `HW` | Hardware | `HW` | Hardware challenges | `/uploads/icons/HW.png` | +| `MSC` | Misc | `MSC` | Miscellaneous challenges | `/uploads/icons/MSC.png` | +| `PWN` | Pwn | `PWN` | Binary exploitation | `/uploads/icons/PWN.png` | +| `REV` | Reverse Engineering | `REV` | Reverse engineering challenges | `/uploads/icons/REV.png` | +| `WEB` | Web | `WEB` | Web exploitation challenges | `/uploads/icons/WEB.png` | + +# What the `up()` does + +1. **`ensureCategoryTimestamps`** — reads `PRAGMA table_info("category")`. + * If `created_at` is absent → `ALTER TABLE "category" ADD COLUMN "created_at" TEXT NOT NULL DEFAULT ''`. + * If `updated_at` is absent → `ALTER TABLE "category" ADD COLUMN "updated_at" TEXT NOT NULL DEFAULT ''`. + * `UPDATE "category" SET "created_at" = strftime(...)` and same for `updated_at` for any row still blank. +2. **`reconcileSystemCategories`** — inside the migration's + transaction: + * Loads all rows with `system_key IS NOT NULL`. + * Deletes every system row whose `system_key` is **not** in the + canonical set (e.g. legacy `FOR`, `OSI`). + * For each canonical key: + * If a row already exists with that `system_key` → updates + `name`, `abbreviation`, `description`, `icon_path`, bumps + timestamps. + * Else if a row exists with the matching `abbreviation` (no + `system_key` or wrong key) → adopts it as the canonical system + row by setting `system_key` and metadata. + * Else → `INSERT` a fresh row with a new UUID v4. +3. **`ensureCategoryIndexes`** — + `CREATE UNIQUE INDEX IF NOT EXISTS "uq_category_system_key" ON "category"("system_key") WHERE "system_key" IS NOT NULL;` + and `CREATE UNIQUE INDEX IF NOT EXISTS "uq_category_abbreviation" ON "category"("abbreviation");`. + +# Idempotency + +Running the migration twice (or restarting after it has been recorded +plus re-running it in tests) leaves exactly six system rows — one per +canonical key — with no duplicates. The reconciliation step only +*deletes* non-canonical system rows and only *updates / inserts* +canonical keys, so user-created categories (`system_key IS NULL`) and +challenges attached to them are never touched. + +# `down()` + +Conservative: only re-asserts the unique indexes. It does **not** +remove canonical rows, delete user categories, or drop the new +timestamp columns, so a rollback cannot break challenge references. + +# Startup verification + +`DatabaseInitService.verifySeed()` was tightened (no schema change) +so that, after every startup, it now: + +* Counts total categories and total settings (informational). +* Selects `system_key, abbreviation FROM "category" WHERE system_key IS NOT NULL ORDER BY abbreviation`. +* Compares the set to the canonical `['CRY','HW','MSC','PWN','REV','WEB']` + and logs a `WARN Canonical system categories mismatch: missing=[...] duplicateSystemRows=N` + if any canonical key is missing or duplicated. + +# Tests + +| File | What it asserts | +|---|---| +| `tests/backend/category-repair-migration.spec.ts` | Builds the exact legacy six-column `category` schema with CRY/FOR/MSC/OSI/PWN/WEB rows, runs the migration on an isolated in-memory SQLite DB, then asserts: timestamps exist and are populated; a TypeORM repository list query no longer throws; `GET /admin/categories` returns exactly `[CRY, HW, MSC, PWN, REV, WEB]` in API sort order with icon + description metadata; running the repair twice still leaves six unique system rows. | +| `tests/backend/migrations.spec.ts` | Registers the new migration in the fresh-schema migration list and strengthens canonical system-category assertions to require exactly CRY/HW/MSC/PWN/REV/WEB once each with populated metadata. | +| `tests/backend/database-init.spec.ts` | Asserts that startup initialization yields the exact canonical set and remains duplicate-free across repeated initialization/startup behavior. | + +# See also + +* [Database Schema Overview](/database/schema.md) — full migration list and bootstrap behavior. +* [Challenge Tables](/database/challenges.md) — `category` schema, indexes, and provenance of `created_at` / `updated_at`. +* [Admin — Categories](/guides/admin-categories.md) — user-visible behavior that depends on the canonical six keys. \ No newline at end of file diff --git a/docs/database/challenges.md b/docs/database/challenges.md index 8164b55..d273204 100644 --- a/docs/database/challenges.md +++ b/docs/database/challenges.md @@ -3,7 +3,7 @@ type: database title: Challenge Tables description: category, challenge, challenge_file, and solve tables — how CTF challenges and scoring are stored. tags: [database, challenge, category, solve] -timestamp: 2026-07-22T14:50:25Z +timestamp: 2026-07-22T16:44:54Z --- # Tables @@ -25,14 +25,19 @@ System rows are seeded by the `UpdateSystemCategoryKeys1700000000300` migration so the canonical keys are `CRY` (Cryptography), `MSC` (Misc), `PWN` (Binary exploitation), `REV` (Reverse Engineering), `WEB` (Web), and `HW` -(Hardware). The migration drops any legacy seeded rows that no -longer match the canonical set before inserting the missing entries. +(Hardware). A forward-only repair migration, +`RepairCategorySchemaAndSystemCategories1700000000400`, runs after +that one on existing databases that pre-date the canonical six keys +(it adds `created_at`/`updated_at` if missing, deletes obsolete +`FOR`/`OSI` system rows, rewrites name/abbreviation/description/icon +metadata on canonical rows, inserts any missing canonical row, and +guarantees unique indexes on `system_key` and `abbreviation`). User-created categories leave `system_key` as `NULL`. The uniqueness on `abbreviation` is enforced both by application validation (uppercased on save, duplicates rejected) and by the `uq_category_abbreviation` index created in migration -`1700000000200`. +`1700000000200` (re-asserted by `1700000000400`). ## `challenge` diff --git a/docs/database/schema.md b/docs/database/schema.md index a2f2f07..d39fa50 100644 --- a/docs/database/schema.md +++ b/docs/database/schema.md @@ -3,7 +3,7 @@ type: database title: Database Schema Overview description: SQLite (better-sqlite3) schema for HIPCTF: tables, relationships, indexes, and WAL journal mode. tags: [database, sqlite, typeorm, schema] -timestamp: 2026-07-22T14:50:25Z +timestamp: 2026-07-22T16:44:54Z --- # Overview @@ -19,10 +19,15 @@ by a second migration (`backend/src/database/migrations/1700000000100-SeedSystemData.ts`). The initial migration declares every column used by the entities (including `category.created_at` and `category.updated_at`), so the -later `AddCategoryTimestampsAndUniqueAbbrev1700000000200` and -`UpdateSystemCategoryKeys1700000000300` migrations only patch legacy -databases that pre-date those columns and reseed the canonical system -rows. +later `AddCategoryTimestampsAndUniqueAbbrev1700000000200`, +`UpdateSystemCategoryKeys1700000000300`, and +`RepairCategorySchemaAndSystemCategories1700000000400` migrations only +patch legacy databases that pre-date those columns and reseed the +canonical system rows. The `...0400` migration is idempotent and +forward-only: it adds `created_at`/`updated_at` to legacy six-column +`category` tables, deletes obsolete `FOR`/`OSI` system rows, rewrites +metadata on canonical rows, inserts any missing canonical row, and +re-asserts the unique indexes on `system_key` and `abbreviation`. # Tables diff --git a/docs/index.md b/docs/index.md index dad838f..d8ddee9 100644 --- a/docs/index.md +++ b/docs/index.md @@ -11,7 +11,7 @@ scoreboard, an event window with a public countdown, theming, and admin controls. The docs below are organized by purpose so agents can pull just the slice -they need. Last regenerated 2026-07-22T16:15:00Z. +they need. Last regenerated 2026-07-22T16:44:54Z. # Architecture @@ -31,6 +31,10 @@ they need. Last regenerated 2026-07-22T16:15:00Z. * [User Table](/database/users.md) - `user` table schema and seed behavior. * [Challenge Tables](/database/challenges.md) - `category`, `challenge`, `challenge_file`, `solve` tables. +* [Category Repair Migration](/database/category-repair-migration.md) - + Forward-only migration that adds `created_at`/`updated_at` to legacy + `category` tables and reconciles system rows to the canonical + CRY/HW/MSC/PWN/REV/WEB set. * [Auth and Settings Tables](/database/auth-settings.md) - `refresh_token` and `setting` tables. * [Blog Post Table](/database/blog-posts.md) - `blog_post` table.