From abbfad76dcf6d9a2041b1bcdf77ff69a2013d0dd Mon Sep 17 00:00:00 2001 From: OpenVelo Agent Date: Tue, 4 Aug 2026 20:48:13 +0000 Subject: [PATCH] docs: update documentation to OKF v0.1 format --- docs/architecture/config-schema.md | 58 ++++++++----------- docs/architecture/mastodon-publishing.md | 12 ++-- docs/architecture/pipeline-runner.md | 15 ++--- docs/architecture/runtime-changes.md | 66 +++++++++++++++++++++ docs/guides/daily-run.md | 14 +---- docs/guides/runtime-verification.md | 74 ++++++++++++++++++++++++ docs/index.md | 4 +- docs/operations/environment-setup.md | 37 +++++------- 8 files changed, 193 insertions(+), 87 deletions(-) create mode 100644 docs/architecture/runtime-changes.md create mode 100644 docs/guides/runtime-verification.md diff --git a/docs/architecture/config-schema.md b/docs/architecture/config-schema.md index 591db8e..dc4babd 100644 --- a/docs/architecture/config-schema.md +++ b/docs/architecture/config-schema.md @@ -16,45 +16,33 @@ dataclass. Job 1083 changes every variable that ships in # Required Environment Variables -All eight of these must be present and non-empty in the process -environment for the container to boot. +The following keys must be present and non-empty in the merged dotenv/process environment before application defaults are applied. -| Key | Purpose | Validation | -|----------------------|--------------------------------------------------------|------------------------------------------------------------------| -| `MASTODON_BASE_URL` | Mastodon instance to post against. | Must parse with `http` or `https` scheme and a non-empty netloc. | -| `MASTODON_ACCESS_TOKEN` | OAuth token used by the future Mastodon.py client. | Required string. | -| `VISIBILITY` | Default post visibility. | Must be `public` or `unlisted`. | -| `SITE_URL` | Source blog URL (used by the future clone step). | Must parse with `http` or `https` scheme and a non-empty netloc. | -| `HASHTAGS` | Comma-separated hashtags appended to every throwback. | Required string (formatting handled by the future poster). | -| `THROWNBACK_PREFIX` | Per-post title prefix (default: `Throwback:`). | Required string. | -| `MAX_RETRIES` | Number of additional retries after the first attempt. | Non-negative integer (parsed by `_parse_max_retries`). | -| `RUN_AT` | Daily fire time for the cron entry. | `HH:MM` 24-hour format. | -| `TZ` | IANA timezone used by cron + the container clock. | Must resolve via `zoneinfo.ZoneInfo`. | +| Key | Purpose | Validation | +|---|---|---| +| `MASTODON_BASE_URL` | Mastodon instance to post against. | Valid `http` or `https` URL. | +| `MASTODON_ACCESS_TOKEN` | OAuth token used by the Mastodon client. | Required string; redacted in logs. | +| `MASTODON_VISIBILITY` | Post visibility. | `public` or `unlisted`. | +| `SITE_URL` | Source blog URL. | Valid `http` or `https` URL. | +| `HASHTAGS` | Comma-separated hashtags. | Required string. | +| `THROWBACK_PREFIX` | Prefix prepended to each status. | Required string. | +| `MAX_RETRIES` | Maximum number of pipeline attempts. | Non-negative integer; defaults to `5` when absent. | +| `RUN_AT` | Daily cron fire time. | `HH:MM` 24-hour format. | +| `TZ` | Cron and application timezone. | Valid IANA timezone. | -> **Note** — All nine keys are required since Job 1083. The previously -> optional `MASTODON_VISIBILITY`, `THROWBACK_PREFIX`, and `RETRY_COUNT` -> were promoted to first-class citizens and renamed to `VISIBILITY`, -> `THROWNBACK_PREFIX`, and `MAX_RETRIES`. +`BLOG_REPO_URL` and `BLOG_DIR` are optional and receive defaults after this required-key check. `DATA_DIR` is read separately from the process environment and defaults to `/app/data`. # Defaults -`DEFAULTS` in `config.py` provides fallback strings that `apply_defaults` -fills into the merged map **before** validation runs: +Defaults are applied only after required-key validation. The current defaults are: -| Default key | Default value | -|-----------------------|-------------------------------------------| -| `VISIBILITY` | `public` | -| `THROWNBACK_PREFIX` | `Heute vor 10 Jahren:` | -| `MAX_RETRIES` | `3` | -| `TZ` | `Europe/Berlin` | -| `RUN_AT` | `09:00` | -| `BLOG_REPO_URL` | `https://git.chaospott.de/Chaospott/site` | -| `BLOG_DIR` | `blog` (resolved relative to `data_dir`) | +| Default key | Default value | +|---|---| +| `MAX_RETRIES` | `5` | +| `BLOG_REPO_URL` | `https://git.chaospott.de/Chaospott/site` | +| `BLOG_DIR` | `blog` (resolved relative to `data_dir`) | -Because every required key carries a default, a freshly-initialised -container can still boot to validate the scaffold, but the operator -must supply `MASTODON_BASE_URL` and `MASTODON_ACCESS_TOKEN` (and -ideally `SITE_URL`) to make a real run. +Application-facing required values are not silently supplied by `config.py`; the container environment may still provide `TZ=Europe/Berlin` and Compose may provide development fallback values. # `Config` Dataclass @@ -62,14 +50,14 @@ ideally `SITE_URL`) to make a real run. |-------------------------|-----------|----------------------------------------------| | `mastodon_base_url` | `str` | `MASTODON_BASE_URL` | | `mastodon_access_token` | `str` | `MASTODON_ACCESS_TOKEN` | -| `visibility` | `str` | `VISIBILITY` | +| `mastodon_visibility` | `str` | `MASTODON_VISIBILITY` | | `site_url` | `str` | `SITE_URL` | | `hashtags` | `str` | `HASHTAGS` | -| `throwback_prefix` | `str` | `THROWNBACK_PREFIX` | +| `throwback_prefix` | `str` | `THROWBACK_PREFIX` | | `max_retries` | `int` | `MAX_RETRIES` (parsed as non-negative int) | | `run_at` | `str` | `RUN_AT` | | `tz` | `str` | `TZ` | -| `data_dir` | `Path` | Constant: `Path("/app/data")` | +| `data_dir` | `Path` | `DATA_DIR`, default `/app/data` | | `blog_repo_url` | `str` | `BLOG_REPO_URL` | | `blog_dir` | `Path` | `BLOG_DIR` resolved against `data_dir` | diff --git a/docs/architecture/mastodon-publishing.md b/docs/architecture/mastodon-publishing.md index f7e20e8..54e3537 100644 --- a/docs/architecture/mastodon-publishing.md +++ b/docs/architecture/mastodon-publishing.md @@ -45,8 +45,7 @@ of matching posts, in this shape: {hashtags} ``` -* The prefix comes from `Config.throwback_prefix` (default - `Heute vor 10 Jahren:`, overridable via `THROWNBACK_PREFIX`). +* The prefix comes from `Config.throwback_prefix` (configured via `THROWBACK_PREFIX`). * Posts are sorted by `(date, path)` so output is deterministic regardless of upstream order. * `hashtags` is a comma-separated string; commas are collapsed to a @@ -106,10 +105,7 @@ publish failure. # Mastodon API Call -`_post_status_via_mastodon_py()` constructs a `mastodon.Mastodon` -client with `(access_token=config.mastodon_access_token, -api_base_url=config.mastodon_base_url)` and calls -`status_post(status, visibility=config.visibility)`. +`_post_status_via_mastodon_py()` constructs a `mastodon.Mastodon` client with `(access_token=config.mastodon_access_token, api_base_url=config.mastodon_base_url)` and calls `status_post(status, visibility=config.mastodon_visibility)`. The `client_factory` keyword argument on `publish_mastodon` lets tests inject a fake client without monkey-patching. Production @@ -150,8 +146,8 @@ re-attempt on the next iteration. |---|---|---| | `MASTODON_BASE_URL` | `Config.mastodon_base_url` | Mastodon instance URL. | | `MASTODON_ACCESS_TOKEN` | `Config.mastodon_access_token` | OAuth token passed to the `Mastodon` client. | -| `VISIBILITY` | `Config.visibility` | Passed as `visibility=` to `status_post`. | -| `THROWNBACK_PREFIX` | `Config.throwback_prefix` | First line of every published status. | +| `MASTODON_VISIBILITY` | `Config.mastodon_visibility` | Passed as `visibility=` to `status_post`. | +| `THROWBACK_PREFIX` | `Config.throwback_prefix` | First line of every published status. | | `HASHTAGS` | `Config.hashtags` | Trailing hashtag line; commas become spaces. | See [Config Schema](/architecture/config-schema.md) for full validation diff --git a/docs/architecture/pipeline-runner.md b/docs/architecture/pipeline-runner.md index 1aa782e..73a9be6 100644 --- a/docs/architecture/pipeline-runner.md +++ b/docs/architecture/pipeline-runner.md @@ -97,15 +97,12 @@ characters); the runner fails safely (raises) rather than truncating. # Retry Behaviour (`_run_with_retry`) -* `attempts = max(1, config.max_retries + 1)` — at least one attempt - even if `MAX_RETRIES=0`. -* Any exception inside `_run_once` is caught (`noqa: BLE001` — - intentional, the whole pass is opaque to the runner). -* Each failed attempt is recorded via - `log_error("pipeline_error", exc=exc, attempt=attempt, max_attempts=attempts)`. -* When the budget is exhausted, the runner emits - `log_error("pipeline_failed", exc=last_exc, attempts=attempts)` and - returns `None` so `main` can translate it to `exit 1`. +* `attempts = max(1, config.max_retries)` — at least one attempt even if `MAX_RETRIES=0`. +* The runner owns the unified retry budget; `_run_once` calls `ensure_repo` with `max_retries=0`. +* Transient Git and Mastodon failures are retried with delays of `2 ** (attempt - 1)` seconds between attempts. +* Each failed attempt emits `retry_attempt` with `operation`, `attempt`, and the exception `error` text. +* Fatal configuration, local repository, and publishing validation errors emit `retry_exhausted` immediately without retrying. +* When the budget is exhausted, `retry_exhausted` includes the final operation, total attempts, and error text, and `main` returns exit code `1`. # Silent-on-No-Matches Contract diff --git a/docs/architecture/runtime-changes.md b/docs/architecture/runtime-changes.md new file mode 100644 index 0000000..b5fb0fa --- /dev/null +++ b/docs/architecture/runtime-changes.md @@ -0,0 +1,66 @@ +--- +type: architecture +title: Job 1089 Runtime Changes +description: Container privilege separation, unified retry logging, configuration validation, and persistent data wiring introduced by the current implementation. +tags: [architecture, docker, retry, logging, configuration] +timestamp: 2026-08-04T20:44:00Z +--- + +# Overview + +The current runtime keeps the container entrypoint and cron daemon as root while executing the Python bot as the unprivileged `bot` user. The pipeline owns one retry budget for Git synchronization and Mastodon publishing, and structured retry events include the operation and exception text. Configuration now validates required values before optional defaults are applied. + +# Wiring + +```text +Docker ENTRYPOINT /app/entrypoint.sh (root) + -> validates environment and renders /etc/cron.d/tenbackward + -> cron -f + -> /usr/local/bin/run-bot.sh + -> setpriv/su to bot + -> python -m tenbackward + -> load_config() + -> _run_with_retry() + -> _run_once() + -> ensure_repo(..., max_retries=0, logger=...) + -> match posts + -> PostedStore dedupe + -> publish_mastodon() + -> PostedStore.mark_posted_many() +``` + +The single `./data:/app/data` Compose bind mount contains both the blog clone and `posted.json`; no file-level bind mount is used. + +# Retry and Logging Contract + +`MAX_RETRIES` is interpreted as the maximum number of attempts, with at least one attempt when configured as zero. Backoff delays are `1, 2, 4, ...` seconds between attempts. Git clone/pull retries are surfaced to the runner with operation labels (`git clone` or `git pull`), while publishing failures use `mastodon post`. Fatal configuration, local-repository, and publishing validation errors are not retried. + +Retry events include an `error` field containing the exception message. `log_event()` sends structured fields through the same redaction path as other log helpers. + +# Configuration Validation + +`load_config()` merges dotenv values and process environment values, validates required keys against that raw merged map, then applies defaults. `MAX_RETRIES`, `BLOG_REPO_URL`, and `BLOG_DIR` may use defaults; the required Mastodon, site, hashtag, prefix, schedule, and timezone values must be explicitly non-empty. + +# Key Files + +| Path | Responsibility | +|---|---| +| `/repo/Dockerfile` | Installs cron, Git, timezone data, and `setpriv`; leaves the final process root-capable for cron and owns `/app/data` by `bot`. | +| `/repo/entrypoint.sh` | Validates environment, renders the daily cron entry, installs it as root, and starts foreground cron. | +| `/repo/run-bot.sh` | Drops from root to `bot` using `setpriv` or `su`, then invokes the Python module. | +| `/repo/docker-compose.yml` | Supplies environment values and mounts `./data` at `/app/data`. | +| `/repo/src/tenbackward/main.py` | Owns the unified retry loop and passes logger/retry controls into repository synchronization. | +| `/repo/src/tenbackward/blog.py` | Clones or fast-forwards the blog and emits operation-specific retry events. | +| `/repo/src/tenbackward/config.py` | Performs raw required-key validation and builds the typed configuration. | +| `/repo/src/tenbackward/logging_setup.py` | Provides structured events, redaction, and JSON-per-line formatting. | +| `/repo/tests/test_docker_artifacts.py` | Verifies container privilege and artifact wiring. | +| `/repo/tests/test_config.py` | Verifies required-key validation and defaults. | +| `/repo/tests/test_run_logging.py` | Verifies retry fields, attempt behavior, and state safety. | + +# Related + +* [System Architecture](/architecture/system-overview.md) +* [Config Schema](/architecture/config-schema.md) +* [Pipeline Runner](/architecture/pipeline-runner.md) +* [Environment Variable Setup](/operations/environment-setup.md) +* [Daily Run Guide](/guides/daily-run.md) diff --git a/docs/guides/daily-run.md b/docs/guides/daily-run.md index d170cd6..baad7fb 100644 --- a/docs/guides/daily-run.md +++ b/docs/guides/daily-run.md @@ -26,15 +26,7 @@ scheduled tick. # What the Run Does -Before scanning, the process ensures the configured blog repository exists -under `/app/data/blog` and is fast-forwarded from `BLOG_REPO_URL`. It then -walks `_posts/blog/**/*.md`, finds posts whose date is exactly ten years -before today, skips IDs already present in `posted.json`, composes a -single German-language Mastodon status (default prefix -`Heute vor 10 Jahren:`) listing each new match's title and canonical URL -followed by the configured hashtags, and — only after the Mastodon API -call succeeds — records the relative paths of the published posts in -`posted.json`. +Before scanning, the process ensures the configured blog repository exists under `/app/data/blog` and is fast-forwarded from `BLOG_REPO_URL`. It then walks `_posts/blog/**/*.md`, finds posts whose date is exactly ten years before today, skips IDs already present in `posted.json`, composes a single Mastodon status listing each new match's title and canonical URL followed by the configured hashtags, and — only after the Mastodon API call succeeds — records the relative paths of the published posts in `posted.json`. The scheduled wrapper executes this Python process as the unprivileged `bot` user. # What You Should See @@ -61,8 +53,8 @@ this is the **only** line you should see — the runner emits no | Blog clone or pull fails | 1× `blog_repo_error` | `1` | | Config valid, matching IDs are new | 1× `startup`, then 1× `run_complete` with `posted=N` | `0` | | Config valid, all matches already posted | 1× `startup`, then 1× `run_complete` with `posted=0` | `0` | -| Pipeline raises once, recovers | 1× `pipeline_error`, then 1× `run_complete` | `0` | -| Pipeline keeps raising (budget exhausted) | `MAX_RETRIES + 1` × `pipeline_error`, then 1× `pipeline_failed` | `1` | +| Pipeline raises once, recovers | `retry_attempt`, then `run_complete` | `0` | +| Pipeline keeps raising (budget exhausted) | `retry_attempt` events, then `retry_exhausted` with operation and error | `1` | | Config invalid | 1× `configuration_error` (extras describe what failed) | `2` | > **Tester tip** — `docker compose ps` should report the container as diff --git a/docs/guides/runtime-verification.md b/docs/guides/runtime-verification.md new file mode 100644 index 0000000..303067a --- /dev/null +++ b/docs/guides/runtime-verification.md @@ -0,0 +1,74 @@ +--- +type: guide +title: Runtime Verification Guide +description: Tester workflow for validating root cron startup, unprivileged bot execution, retry events, and persistent data behavior. +tags: [guide, tester, docker, runtime] +timestamp: 2026-08-04T20:44:00Z +--- + +# Setup + +1. Copy `.env.example` to `.env` and provide valid required values. +2. Build and start the service: + + ```bash + docker compose up --build -d + ``` + +3. Follow container output: + + ```bash + docker compose logs -f bot + ``` + +# Expected Runtime Behavior + +The container remains running with `cron -f` as its foreground process. Startup output includes the configured `RUN_AT` and `TZ`. The scheduled cron command invokes `/usr/local/bin/run-bot.sh`, which runs the Python bot as user `bot`, not root. + +The host `./data` directory should contain the synchronized `blog/` clone and, after the state store is loaded, `posted.json`. Both are under the one `/app/data` mount. + +# Manual Run + +Run the same application command without waiting for cron: + +```bash +docker compose exec bot /usr/local/bin/python -m tenbackward +``` + +A valid run exits `0`. A configuration failure exits `2`; an exhausted pipeline retry budget exits `1`. + +# Retry Checks + +To exercise transient failure handling, use a test repository or injected test double that fails once and then succeeds. Verify JSON events contain: + +| Event | Expected fields | +|---|---| +| `retry_attempt` | `operation`, `attempt`, and the original `error` text. | +| `retry_exhausted` | Final `operation`, total `attempts`, and `error` text when all attempts fail. | +| `run_complete` | Counters and `posted_ids` only after a successful pass. | + +With `MAX_RETRIES=0`, one attempt is still made and no sleep occurs. With a larger value, delays occur only between attempts and follow powers of two beginning at one second. + +# Configuration Checks + +Remove a required variable from `.env` and restart the service. The entrypoint should report the missing variable before cron starts. If the shell check is bypassed and the Python module runs directly, `configuration_error` should identify the missing key and return exit code `2` rather than silently substituting an application default. + +# Persistence and Failure Safety + +After a successful publication, inspect `/app/data/posted.json` and confirm new relative post paths appear under the `posted` list. If Mastodon publishing fails, the path must not be recorded; a later retry should be able to publish it again. + +# Key Visual/Observable Elements + +This service has no graphical UI. Test-facing outputs are: + +* Container status and `docker compose logs` output. +* JSON-per-line startup, retry, failure, and completion events. +* Exit codes from manual runs. +* `/app/data/blog` and `/app/data/posted.json` on the mounted host directory. + +# Related + +* [Daily Run Guide](/guides/daily-run.md) +* [Job 1089 Runtime Changes](/architecture/runtime-changes.md) +* [Environment Variable Setup](/operations/environment-setup.md) +* [Cron Lifecycle](/operations/cron-lifecycle.md) diff --git a/docs/index.md b/docs/index.md index a87fcb5..619c14c 100644 --- a/docs/index.md +++ b/docs/index.md @@ -4,6 +4,7 @@ okf_version: "0.1" # Architecture * [System Architecture](/architecture/system-overview.md) — Component map of 10Backward: entrypoint, config, logging, runner, state, publishing boundary, and cron wiring. +* [Runtime Changes](/architecture/runtime-changes.md) — Root cron/unprivileged bot separation, unified retries, raw configuration validation, and data-volume wiring. * [Config Schema](/architecture/config-schema.md) — Required/optional env vars, validation rules, and the typed Config dataclass. * [Logging & Run Summary](/architecture/logging.md) — JSON formatter, secret redaction, and structured event helpers. * [Pipeline Runner](/architecture/pipeline-runner.md) — How a cron tick synchronizes the blog, matches anniversaries, applies deduplication, publishes to Mastodon, retries failures, and emits run events. @@ -15,4 +16,5 @@ okf_version: "0.1" * [Cron Lifecycle](/operations/cron-lifecycle.md) — How `entrypoint.sh` renders `/etc/cron.d/tenbackward` and hands off to `cron -f`. # User Guides -* [Daily Run Guide](/guides/daily-run.md) — Tester/operator walkthrough of repository sync, anniversary matching, deduplication, Mastodon publishing, and expected log/output behaviour. +* [Daily Run Guide](/guides/daily-run.md) — Operator walkthrough of repository sync, anniversary matching, deduplication, publishing, and expected logs. +* [Runtime Verification Guide](/guides/runtime-verification.md) — Tester steps for privilege separation, retry events, configuration failures, and persistent state. diff --git a/docs/operations/environment-setup.md b/docs/operations/environment-setup.md index 1f1385f..614bfb1 100644 --- a/docs/operations/environment-setup.md +++ b/docs/operations/environment-setup.md @@ -18,26 +18,21 @@ full set before the container will boot. |--------------------------|--------------------------------|---------------------------------------------------| | `MASTODON_BASE_URL` | `https://mastodon.social` | Mastodon instance to post against. | | `MASTODON_ACCESS_TOKEN` | _(from your Mastodon account)_ | OAuth access token. | -| `VISIBILITY` | `public` | Post visibility (`public` or `unlisted`). | -| `SITE_URL` | `https://blog.example.com` | Source blog URL (used by the future clone step). | -| `HASHTAGS` | `#throwback,#10backward` | Hashtags appended to every throwback post. | -| `THROWNBACK_PREFIX` | `Heute vor 10 Jahren:` | Prefix prepended to every Mastodon status. | -| `MAX_RETRIES` | `3` | Non-negative retry count for the pipeline. | -| `RUN_AT` | `09:00` | Daily fire time (HH:MM, 24-hour). | -| `TZ` | `Europe/Berlin` | IANA timezone for cron + container clock. | +| `MASTODON_VISIBILITY` | `public` | Post visibility (`public` or `unlisted`). | +| `SITE_URL` | `https://blog.example.com` | Source blog URL. | +| `HASHTAGS` | `#throwback,#10backward` | Hashtags appended to every throwback post. | +| `THROWBACK_PREFIX` | `Heute vor 10 Jahren:` | Prefix prepended to every Mastodon status. | +| `MAX_RETRIES` | `5` | Maximum number of pipeline attempts. | +| `RUN_AT` | `09:00` | Daily fire time (`HH:MM`, 24-hour). | +| `TZ` | `Europe/Berlin` | IANA timezone for cron and application. | +| `BLOG_REPO_URL` | project default | Optional source repository URL. | +| `BLOG_DIR` | `blog` | Optional clone directory, relative to `DATA_DIR` unless absolute. | # Where Each Name Is Enforced -* `entrypoint.sh` — exports `TZ`, validates all nine are non-empty, - and validates `RUN_AT` matches `^([01][0-9]|2[0-3]):[0-5][0-9]$` - before writing the cron file. A missing var aborts the container - with `ERROR: missing required environment variable(s): ...` and - exits `1`. -* `tenbackward.config.validate_config` — same set, plus URL/visibility/ - TZ/MAX_RETRIES validation. Failures surface as a single - `ConfigError` listing every problem. -* The `kilo.json` config file is **not** an env-var file — it is the - agent runtime configuration. +* `entrypoint.sh` — validates all nine required keys, validates `RUN_AT`, and installs the cron file as root. The cron-launched wrapper then drops privileges to `bot` before Python runs. +* `tenbackward.config.validate_config` — validates the merged raw required keys plus URL, visibility, timezone, and retry syntax. Defaults are applied only after required-key validation. +* `docker-compose.yml` — supplies development environment values and mounts `./data` at `/app/data`. # Setup Steps @@ -80,13 +75,9 @@ If you have an existing `.env` from the scaffold, rename these manually. The renamed variables are **not** backwards compatible — the container will refuse to start with both versions set. -# Defaults That Can Be Removed +# Defaults -`DEFAULTS` provides fallbacks for `VISIBILITY`, `THROWNBACK_PREFIX`, -`MAX_RETRIES`, `TZ`, and `RUN_AT`. You may leave them out of your -`.env`, but the operator contract is "every required key is set" — -prod deployments should set them explicitly so a missing key is -caught at boot instead of silently used as a default. +`MAX_RETRIES` defaults to `5`; `BLOG_REPO_URL` and `BLOG_DIR` also have application defaults. Other application-facing required keys must be present in the raw merged dotenv/process environment. The Docker image supplies `TZ=Europe/Berlin`, and Compose supplies development fallbacks, but production deployments should set every required value explicitly. # Related