OpenVelo Agent
107 lines
6.1 KiB
Markdown
107 lines
6.1 KiB
Markdown
---
|
|
type: database
|
|
title: Category Repair Migration (1700000000400)
|
|
description: Forward-only migration that repairs legacy `category` schemas (adds timestamps) and reconciles system rows to the canonical CRY/HW/MSC/PWN/REV/WEB set.
|
|
tags: [database, migration, category, repair, system-categories]
|
|
timestamp: 2026-07-22T16:44:54Z
|
|
---
|
|
|
|
# Purpose
|
|
|
|
Some persistent `/data/hipctf/db.sqlite` databases were created before
|
|
the canonical six system categories existed. They contain a legacy
|
|
six-column `category` table (no `created_at` / `updated_at`) and
|
|
system rows whose `system_key` / `abbreviation` no longer match the
|
|
canonical set (e.g. `FOR` for Forensics, `OSI` for OSINT, missing
|
|
`HW` / `REV`).
|
|
|
|
When the backend starts against such a database, `CategoryEntity`
|
|
fails to `SELECT c.*` because the timestamp columns are absent, and
|
|
`GET /api/v1/admin/categories` returns 500. Migrations that may
|
|
already be recorded in those databases cannot be edited or relied on
|
|
to re-run, so a new forward-only migration is required.
|
|
|
|
# File
|
|
|
|
`backend/src/database/migrations/1700000000400-RepairCategorySchemaAndSystemCategories.ts`
|
|
|
|
Registered last in `backend/src/database/database.module.ts` so it
|
|
runs after `UpdateSystemCategoryKeys1700000000300`.
|
|
|
|
# Canonical system categories
|
|
|
|
Exported as `CANONICAL_SYSTEM_CATEGORIES` (typed
|
|
`ReadonlyArray<CanonicalSystemCategory>`) so the migration and any
|
|
future seed consumer share one source of truth:
|
|
|
|
| `system_key` | `name` | `abbreviation` | `description` | `icon_path` |
|
|
|--------------|-------------------------|----------------|------------------------------|------------------------------|
|
|
| `CRY` | Cryptography | `CRY` | Cryptographic challenges | `/uploads/icons/CRY.png` |
|
|
| `HW` | Hardware | `HW` | Hardware challenges | `/uploads/icons/HW.png` |
|
|
| `MSC` | Misc | `MSC` | Miscellaneous challenges | `/uploads/icons/MSC.png` |
|
|
| `PWN` | Pwn | `PWN` | Binary exploitation | `/uploads/icons/PWN.png` |
|
|
| `REV` | Reverse Engineering | `REV` | Reverse engineering challenges | `/uploads/icons/REV.png` |
|
|
| `WEB` | Web | `WEB` | Web exploitation challenges | `/uploads/icons/WEB.png` |
|
|
|
|
# What the `up()` does
|
|
|
|
1. **`ensureCategoryTimestamps`** — reads `PRAGMA table_info("category")`.
|
|
* If `created_at` is absent → `ALTER TABLE "category" ADD COLUMN "created_at" TEXT NOT NULL DEFAULT ''`.
|
|
* If `updated_at` is absent → `ALTER TABLE "category" ADD COLUMN "updated_at" TEXT NOT NULL DEFAULT ''`.
|
|
* `UPDATE "category" SET "created_at" = strftime(...)` and same for `updated_at` for any row still blank.
|
|
2. **`reconcileSystemCategories`** — inside the migration's
|
|
transaction:
|
|
* Loads all rows with `system_key IS NOT NULL`.
|
|
* Deletes every system row whose `system_key` is **not** in the
|
|
canonical set (e.g. legacy `FOR`, `OSI`).
|
|
* For each canonical key:
|
|
* If a row already exists with that `system_key` → updates
|
|
`name`, `abbreviation`, `description`, `icon_path`, bumps
|
|
timestamps.
|
|
* Else if a row exists with the matching `abbreviation` (no
|
|
`system_key` or wrong key) → adopts it as the canonical system
|
|
row by setting `system_key` and metadata.
|
|
* Else → `INSERT` a fresh row with a new UUID v4.
|
|
3. **`ensureCategoryIndexes`** —
|
|
`CREATE UNIQUE INDEX IF NOT EXISTS "uq_category_system_key" ON "category"("system_key") WHERE "system_key" IS NOT NULL;`
|
|
and `CREATE UNIQUE INDEX IF NOT EXISTS "uq_category_abbreviation" ON "category"("abbreviation");`.
|
|
|
|
# Idempotency
|
|
|
|
Running the migration twice (or restarting after it has been recorded
|
|
plus re-running it in tests) leaves exactly six system rows — one per
|
|
canonical key — with no duplicates. The reconciliation step only
|
|
*deletes* non-canonical system rows and only *updates / inserts*
|
|
canonical keys, so user-created categories (`system_key IS NULL`) and
|
|
challenges attached to them are never touched.
|
|
|
|
# `down()`
|
|
|
|
Conservative: only re-asserts the unique indexes. It does **not**
|
|
remove canonical rows, delete user categories, or drop the new
|
|
timestamp columns, so a rollback cannot break challenge references.
|
|
|
|
# Startup verification
|
|
|
|
`DatabaseInitService.verifySeed()` was tightened (no schema change)
|
|
so that, after every startup, it now:
|
|
|
|
* Counts total categories and total settings (informational).
|
|
* Selects `system_key, abbreviation FROM "category" WHERE system_key IS NOT NULL ORDER BY abbreviation`.
|
|
* Compares the set to the canonical `['CRY','HW','MSC','PWN','REV','WEB']`
|
|
and logs a `WARN Canonical system categories mismatch: missing=[...] duplicateSystemRows=N`
|
|
if any canonical key is missing or duplicated.
|
|
|
|
# Tests
|
|
|
|
| File | What it asserts |
|
|
|---|---|
|
|
| `tests/backend/category-repair-migration.spec.ts` | Builds the exact legacy six-column `category` schema with CRY/FOR/MSC/OSI/PWN/WEB rows, runs the migration on an isolated in-memory SQLite DB, then asserts: timestamps exist and are populated; a TypeORM repository list query no longer throws; `GET /admin/categories` returns exactly `[CRY, HW, MSC, PWN, REV, WEB]` in API sort order with icon + description metadata; running the repair twice still leaves six unique system rows. |
|
|
| `tests/backend/migrations.spec.ts` | Registers the new migration in the fresh-schema migration list and strengthens canonical system-category assertions to require exactly CRY/HW/MSC/PWN/REV/WEB once each with populated metadata. |
|
|
| `tests/backend/database-init.spec.ts` | Asserts that startup initialization yields the exact canonical set and remains duplicate-free across repeated initialization/startup behavior. |
|
|
|
|
# See also
|
|
|
|
* [Database Schema Overview](/database/schema.md) — full migration list and bootstrap behavior.
|
|
* [Challenge Tables](/database/challenges.md) — `category` schema, indexes, and provenance of `created_at` / `updated_at`.
|
|
* [Admin — Categories](/guides/admin-categories.md) — user-visible behavior that depends on the canonical six keys. |