HIPCTF2/docs/database/category-repair-migration.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
database	Category Repair Migration (1700000000400)	Forward-only migration that repairs legacy `category` schemas (adds timestamps) and reconciles system rows to the canonical CRY/HW/MSC/PWN/REV/WEB set.	database migration category repair system-categories	2026-07-22T16:44:54Z

Purpose

Some persistent /data/hipctf/db.sqlite databases were created before the canonical six system categories existed. They contain a legacy six-column category table (no created_at / updated_at) and system rows whose system_key / abbreviation no longer match the canonical set (e.g. FOR for Forensics, OSI for OSINT, missing HW / REV).

When the backend starts against such a database, CategoryEntity fails to SELECT c.* because the timestamp columns are absent, and GET /api/v1/admin/categories returns 500. Migrations that may already be recorded in those databases cannot be edited or relied on to re-run, so a new forward-only migration is required.

File

backend/src/database/migrations/1700000000400-RepairCategorySchemaAndSystemCategories.ts

Registered last in backend/src/database/database.module.ts so it runs after UpdateSystemCategoryKeys1700000000300.

Canonical system categories

Exported as CANONICAL_SYSTEM_CATEGORIES (typed ReadonlyArray<CanonicalSystemCategory>) so the migration and any future seed consumer share one source of truth:

system_key	name	abbreviation	description	icon_path
CRY	Cryptography	CRY	Cryptographic challenges	/uploads/icons/CRY.png
HW	Hardware	HW	Hardware challenges	/uploads/icons/HW.png
MSC	Misc	MSC	Miscellaneous challenges	/uploads/icons/MSC.png
PWN	Pwn	PWN	Binary exploitation	/uploads/icons/PWN.png
REV	Reverse Engineering	REV	Reverse engineering challenges	/uploads/icons/REV.png
WEB	Web	WEB	Web exploitation challenges	/uploads/icons/WEB.png

What the up() does

1. ensureCategoryTimestamps — reads PRAGMA table_info("category").
   • If created_at is absent → ALTER TABLE "category" ADD COLUMN "created_at" TEXT NOT NULL DEFAULT ''.
   • If updated_at is absent → ALTER TABLE "category" ADD COLUMN "updated_at" TEXT NOT NULL DEFAULT ''.
   • UPDATE "category" SET "created_at" = strftime(...) and same for updated_at for any row still blank.
2. reconcileSystemCategories — inside the migration's transaction:
   • Loads all rows with system_key IS NOT NULL.
   • Deletes every system row whose system_key is not in the canonical set (e.g. legacy FOR, OSI).
   • For each canonical key:
     • If a row already exists with that system_key → updates name, abbreviation, description, icon_path, bumps timestamps.
     • Else if a row exists with the matching abbreviation (no system_key or wrong key) → adopts it as the canonical system row by setting system_key and metadata.
     • Else → INSERT a fresh row with a new UUID v4.
3. ensureCategoryIndexes — CREATE UNIQUE INDEX IF NOT EXISTS "uq_category_system_key" ON "category"("system_key") WHERE "system_key" IS NOT NULL; and CREATE UNIQUE INDEX IF NOT EXISTS "uq_category_abbreviation" ON "category"("abbreviation");.

Idempotency

Running the migration twice (or restarting after it has been recorded plus re-running it in tests) leaves exactly six system rows — one per canonical key — with no duplicates. The reconciliation step only deletes non-canonical system rows and only updates / inserts canonical keys, so user-created categories (system_key IS NULL) and challenges attached to them are never touched.

down()

Conservative: only re-asserts the unique indexes. It does not remove canonical rows, delete user categories, or drop the new timestamp columns, so a rollback cannot break challenge references.

Startup verification

DatabaseInitService.verifySeed() was tightened (no schema change) so that, after every startup, it now:

• Counts total categories and total settings (informational).
• Selects system_key, abbreviation FROM "category" WHERE system_key IS NOT NULL ORDER BY abbreviation.
• Compares the set to the canonical ['CRY','HW','MSC','PWN','REV','WEB'] and logs a WARN Canonical system categories mismatch: missing=[...] duplicateSystemRows=N if any canonical key is missing or duplicated.

Tests

File	What it asserts
tests/backend/category-repair-migration.spec.ts	Builds the exact legacy six-column category schema with CRY/FOR/MSC/OSI/PWN/WEB rows, runs the migration on an isolated in-memory SQLite DB, then asserts: timestamps exist and are populated; a TypeORM repository list query no longer throws; GET /admin/categories returns exactly [CRY, HW, MSC, PWN, REV, WEB] in API sort order with icon + description metadata; running the repair twice still leaves six unique system rows.
tests/backend/migrations.spec.ts	Registers the new migration in the fresh-schema migration list and strengthens canonical system-category assertions to require exactly CRY/HW/MSC/PWN/REV/WEB once each with populated metadata.
tests/backend/database-init.spec.ts	Asserts that startup initialization yields the exact canonical set and remains duplicate-free across repeated initialization/startup behavior.

See also

• Database Schema Overview — full migration list and bootstrap behavior.
• Challenge Tables — category schema, indexes, and provenance of created_at / updated_at.
• Admin — Categories — user-visible behavior that depends on the canonical six keys.