m0rph3us1987/10Backward
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

AI Implementation feature(1083): Environment Configuration and Logging Baseline (#2)

Browse Source
This commit was merged in pull request #2.
This commit is contained in:
m0rph3us1987
2026-08-04 17:53:19 +00:00
parent 33c6c76ce9
commit 207e2e6bbb
21 changed files with 1429 additions and 77 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture/config-schema.md
+103
View File
@@ -0,0 +1,103 @@
---
type: api
title: Config Schema
description: Required/optional environment variables, validation rules, and the typed Config dataclass returned by tenbackward.config.load_config.
tags: [config, env, schema]
timestamp: 2026-08-04T17:51:00Z
---
# Purpose
`tenbackward.config.load_config()` is the single boundary that turns
process environment + optional `.env` file into a typed `Config`
dataclass. Job 1083 changes every variable that ships in
`.env.example` so that it is **explicitly required** at every layer
(`entrypoint.sh`, `config.validate_config`, and the rendered cron).
# Required Environment Variables
All eight of these must be present and non-empty in the process
environment for the container to boot.
| Key | Purpose | Validation |
|----------------------|--------------------------------------------------------|------------------------------------------------------------------|
| `MASTODON_BASE_URL` | Mastodon instance to post against. | Must parse with `http` or `https` scheme and a non-empty netloc. |
| `MASTODON_ACCESS_TOKEN` | OAuth token used by the future Mastodon.py client. | Required string. |
| `VISIBILITY` | Default post visibility. | Must be `public` or `unlisted`. |
| `SITE_URL` | Source blog URL (used by the future clone step). | Must parse with `http` or `https` scheme and a non-empty netloc. |
| `HASHTAGS` | Comma-separated hashtags appended to every throwback. | Required string (formatting handled by the future poster). |
| `THROWNBACK_PREFIX` | Per-post title prefix (default: `Throwback:`). | Required string. |
| `MAX_RETRIES` | Number of additional retries after the first attempt. | Non-negative integer (parsed by `_parse_max_retries`). |
| `RUN_AT` | Daily fire time for the cron entry. | `HH:MM` 24-hour format. |
| `TZ` | IANA timezone used by cron + the container clock. | Must resolve via `zoneinfo.ZoneInfo`. |
> **Note** — All nine keys are required since Job 1083. The previously
> optional `MASTODON_VISIBILITY`, `THROWBACK_PREFIX`, and `RETRY_COUNT`
> were promoted to first-class citizens and renamed to `VISIBILITY`,
> `THROWNBACK_PREFIX`, and `MAX_RETRIES`.
# Defaults
`DEFAULTS` in `config.py` provides fallback strings that `apply_defaults`
fills into the merged map **before** validation runs:
| Default key | Default value |
|-----------------------|-------------------|
| `VISIBILITY` | `public` |
| `THROWNBACK_PREFIX` | `Throwback:` |
| `MAX_RETRIES` | `3` |
| `TZ` | `Europe/Berlin` |
| `RUN_AT` | `09:00` |
Because every required key carries a default, a freshly-initialised
container can still boot to validate the scaffold, but the operator
must supply `MASTODON_BASE_URL` and `MASTODON_ACCESS_TOKEN` (and
ideally `SITE_URL`) to make a real run.
# `Config` Dataclass
| Field | Type | Source |
|-------------------------|-----------|----------------------------------------------|
| `mastodon_base_url` | `str` | `MASTODON_BASE_URL` |
| `mastodon_access_token` | `str` | `MASTODON_ACCESS_TOKEN` |
| `visibility` | `str` | `VISIBILITY` |
| `site_url` | `str` | `SITE_URL` |
| `hashtags` | `str` | `HASHTAGS` |
| `throwback_prefix` | `str` | `THROWNBACK_PREFIX` |
| `max_retries` | `int` | `MAX_RETRIES` (parsed as non-negative int) |
| `run_at` | `str` | `RUN_AT` |
| `tz` | `str` | `TZ` |
| `data_dir` | `Path` | Constant: `Path("/app/data")` |
# Validation Behaviour
`validate_config(values)` collects all failures before raising so the
operator sees every problem at once. A single `ConfigError` lists each
problem joined with `"; "`. Validation covers:
* Required keys are non-empty.
* `RUN_AT` parses as `HH:MM`.
* `MASTODON_BASE_URL` and `SITE_URL` parse via `urllib.parse.urlparse`
with an `http`/`https` scheme and a non-empty `netloc`.
* `VISIBILITY` is one of `{"public", "unlisted"}` (`ALLOWED_VISIBILITY`).
* `TZ` resolves via `zoneinfo.ZoneInfo`.
* `MAX_RETRIES` parses as a non-negative integer (rejects bools).
# Exit Codes
| Code | Source | Meaning |
|------|-----------------------------------------|----------------------------------------------------------------------|
| `1` | `main.main()` (retry budget exhausted) | Pipeline failed on every attempt after `MAX_RETRIES + 1` tries. |
| `2` | `main.main()` (`ConfigError`) | Configuration was present but invalid; details are logged. |
# Citations
* [1] `src/tenbackward/config.py` — `REQUIRED_KEYS`, `DEFAULTS`, `ALLOWED_VISIBILITY`, `Config`, `load_config`, `validate_config`.
* [2] `.env.example` — canonical env-var list.
* [3] `entrypoint.sh` — shell-side mirror of the required keys.
# Related
* [System Architecture](/architecture/system-overview.md)
* [Logging & Run Summary](/architecture/logging.md)
* [Environment Variable Setup](/operations/environment-setup.md)
docs/architecture/logging.md
+125
View File
@@ -0,0 +1,125 @@
---
type: architecture
title: Logging & Run Summary
description: Structured JSON logging in 10Backward — JsonFormatter, secret redaction, and the startup / error / run_complete event helpers.
tags: [logging, observability, json]
timestamp: 2026-08-04T17:51:00Z
---
# Purpose
`tenbackward.logging_setup` replaces the scaffold-era `basicConfig`
text logger with a **single-line JSON formatter** that downstream
log shippers can index. It also owns the three helpers the runner
uses to record structured events: `log_startup`, `log_error`,
`log_run_summary`.
# JsonFormatter
`JsonFormatter.format(record)` returns one JSON object per log record:
| Key | Source |
|------------|---------------------------------------------------------------------|
| `ts` | UTC `datetime.fromtimestamp(record.created).isoformat()`. |
| `level` | `record.levelname` (e.g. `INFO`, `ERROR`). |
| `logger` | `record.name` (always `tenbackward`). |
| `message` | `record.getMessage()`. |
| _extras_ | Any keyword passed via `logging.info(..., extra={...})` that is not a reserved `LogRecord` attribute and whose key is **not** in `_SECRET_KEYS`. |
| `exc_type` | When `record.exc_info` is set, the exception class name. |
Reserved `LogRecord` attributes and any key starting with `_` are
dropped before serialization, so the payload contains only the values
that callers intentionally attached.
## Secret Redaction
`_SECRET_KEYS` is the canonical denylist applied in two places:
1. **At format time** for keys passed via `extra={...}`.
2. **At helper time** by `_redact_dict(...)` so values reaching
`log_startup` / `log_error` never expose known-sensitive fields
even before they hit the formatter.
Deny-list contents (case-insensitive):
```
token, access_token, mastodon_access_token, password, secret,
authorization, api_key
```
Any matching key is replaced with the string `[REDACTED]`.
# Helper Functions
| Helper | When it fires | Logger / Level | `event` field |
|---------------------------------------|-----------------------------------------------------------------------|----------------|--------------------|
| `configure_json_logging(level=INFO)` | First line of `main.main()`. Idempotent — clears existing handlers. | n/a | n/a |
| `log_startup(**fields)` | After config load + `data_dir.mkdir`, before the pipeline pass. | `INFO` | `startup` |
| `log_error(event, *, exc=None, **fields)` | When a pipeline attempt raises, or when the retry budget is exhausted. | `ERROR` | the supplied event name; `exc_type` added when `exc` is provided. |
| `log_run_summary(scanned, matched, posted, skipped, posted_ids)` | Once per successful pass. **Silently skipped** when all counters are zero and no ids were posted. | `INFO` | `run_complete` |
## Startup payload
`log_startup` is invoked with:
```
version, site_url, run_at, tz, hashtags, throwback_prefix
```
Each is added to the `startup` event payload under its own key.
## Run-summary payload
`log_run_summary` is the canonical "what did the bot do today?" record:
```json
{
"ts": "2026-08-04T09:00:00+00:00",
"level": "INFO",
"logger": "tenbackward",
"message": "run complete",
"event": "run_complete",
"scanned": 0,
"matched": 0,
"posted": 0,
"skipped": 0,
"posted_ids": []
}
```
A pass with no candidates emits **no line at all**, matching the
"silent on no matches" contract.
# Examples
## Successful run with one new post
```json
{"ts":"2026-08-04T09:00:00+00:00","level":"INFO","logger":"tenbackward","message":"startup","event":"startup","version":"0.1.0","site_url":"https://blog.example.com","run_at":"09:00","tz":"Europe/Berlin","hashtags":"#throwback,#10backward","throwback_prefix":"Throwback:"}
{"ts":"2026-08-04T09:00:01+00:00","level":"INFO","logger":"tenbackward","message":"run complete","event":"run_complete","scanned":1,"matched":1,"posted":1,"skipped":0,"posted_ids":["2025-08-04-post-slug"]}
```
## Retry budget exhausted
```json
{"ts":"...","level":"ERROR","logger":"tenbackward","message":"pipeline_error","event":"pipeline_error","attempt":1,"max_attempts":4,"exc_type":"ConnectionError"}
{"ts":"...","level":"ERROR","logger":"tenbackward","message":"pipeline_error","event":"pipeline_error","attempt":2,"max_attempts":4,"exc_type":"ConnectionError"}
{"ts":"...","level":"ERROR","logger":"tenbackward","message":"pipeline_failed","event":"pipeline_failed","attempts":4,"exc_type":"ConnectionError"}
```
The container then exits with status `1`.
## Token accidentally logged
Even if a future call wrote `extra={"access_token": "..."}`, the
formatter replaces it:
```json
{ "...": "...", "access_token": "[REDACTED]" }
```
# Related
* [System Architecture](/architecture/system-overview.md)
* [Pipeline Runner](/architecture/pipeline-runner.md)
* [Config Schema](/architecture/config-schema.md)
docs/architecture/pipeline-runner.md
+111
View File
@@ -0,0 +1,111 @@
---
type: architecture
title: Pipeline Runner
description: How tenbackward.main turns one cron tick into a startup log line, a retry-wrapped pipeline pass, and a single run_complete summary line.
tags: [pipeline, runner, retry]
timestamp: 2026-08-04T17:51:00Z
---
# Purpose
`tenbackward.main` is the entry point executed by cron. Job 1083
rewrote it to: (a) install JSON logging, (b) execute a retry-wrapped
pipeline pass that is **silent when there are no candidates**, and
(c) emit a single structured `run_complete` summary on success.
# Call Flow
```
main.main()
├── configure_json_logging()
├── load_config() ── raises ConfigError → exit 2
├── config.data_dir.mkdir(parents=True, exist_ok=True)
├── log_startup(version, site_url, run_at, tz, hashtags, throwback_prefix)
├── result = _run_with_retry(config)
│ ├── for attempt in 1 .. max_retries+1:
│ │ try: return _run_once(config)
│ │ except Exception as exc:
│ │ log_error("pipeline_error", exc=exc, attempt=attempt, max_attempts=attempts)
│ ├── log_error("pipeline_failed", exc=last_exc, attempts=attempts)
│ └── return None
├── if result is None: return 1
└── log_run_summary(scanned, matched, posted, skipped, posted_ids)
```
# Extension Point: `_iter_candidates`
```python
def _iter_candidates(config: Config) -> Iterable[str]: ...
```
Job 1083 ships `_iter_candidates` as an **empty iterator** so the
silent-on-no-matches contract is the default behaviour. The future
blog-clone + matching pipeline plugs into this function without
touching `_run_once` or `_run_with_retry`.
# `_run_once` — Summary Counters
`_run_once` returns a 5-tuple:
```python
(scanned, matched, posted, skipped, posted_ids)
```
| Counter | Meaning |
|-------------|----------------------------------------------------------------------|
| `scanned` | Items yielded by `_iter_candidates`. |
| `matched` | Items that survived the dedupe-vs-`already_posted` check. |
| `posted` | Items newly recorded in `posted.json` during this run. |
| `skipped` | Items that matched but were already in `posted.json` (deduped). |
| `posted_ids`| The list of post identifiers written to state. |
`state.save_posted` is called **only** when `posted_ids` is non-empty,
so a no-op run does not touch `posted.json` on disk.
# Retry Behaviour (`_run_with_retry`)
* `attempts = max(1, config.max_retries + 1)` — at least one attempt
even if `MAX_RETRIES=0`.
* Any exception inside `_run_once` is caught (`noqa: BLE001` —
intentional, the whole pass is opaque to the runner).
* Each failed attempt is recorded via
`log_error("pipeline_error", exc=exc, attempt=attempt, max_attempts=attempts)`.
* When the budget is exhausted, the runner emits
`log_error("pipeline_failed", exc=last_exc, attempts=attempts)` and
returns `None` so `main` can translate it to `exit 1`.
* `time.sleep(0)` between attempts is the placeholder seam where the
next job can introduce real backoff.
# Silent-on-No-Matches Contract
A run with zero candidates produces exactly one line:
```json
{"ts":"...","level":"INFO","message":"startup","event":"startup",...}
```
No `run_complete` line is emitted. Tests assert this by counting log
records (`tests/test_run_logging.py::test_run_summary_silent_when_no_candidates`).
# Example Sequence (Healthy Run)
1. `startup` line at INFO.
2. `_run_with_retry` returns `(0, 0, 0, 0, [])` (no candidates yet
because `_iter_candidates` is empty) → `main` exits `0`
**without** a `run_complete` line.
3. Next-day tick repeats the cycle.
# Example Sequence (Future Blog Step in Place)
1. `startup` line at INFO.
2. `_iter_candidates` yields `[id-1, id-2]`. `id-1` is already in
`posted.json`, so `skipped += 1`. `id-2` is new and appended to
`posted_ids` and persisted.
3. `run_complete` line at INFO with
`scanned=2, matched=2, posted=1, skipped=1, posted_ids=["id-2"]`.
# Related
* [Logging & Run Summary](/architecture/logging.md)
* [Config Schema](/architecture/config-schema.md)
* [Cron Lifecycle](/operations/cron-lifecycle.md)
docs/architecture/system-overview.md
+98
View File
@@ -0,0 +1,98 @@
---
type: architecture
title: System Architecture
description: Component map of 10Backward — how config, logging, the pipeline runner, state persistence, and container entrypoint are wired together.
tags: [architecture, overview]
timestamp: 2026-08-04T17:51:00Z
---
# Overview
`10Backward` is a Mastodon daily-throwback bot that runs as a single
containerised cron job. Job 1083 adds a structured logging layer and a
retry-capable pipeline runner on top of the scaffold shipped by Job 1082.
The container boots, validates environment configuration, renders a
`/etc/cron.d/tenbackward` entry that fires once per day at the
configured `RUN_AT`, and then runs `python -m tenbackward` in the
foreground of the `cron` process. Each scheduled invocation calls
`tenbackward.main.main()`, which:
1. Installs the JSON logging formatter on the root logger.
2. Loads and validates the `Config`.
3. Ensures the data directory exists.
4. Emits a `startup` log line.
5. Executes one pipeline pass wrapped in a retry loop.
6. Emits a `run_complete` summary (skipped silently if there were no candidates).
# Components
| Component | Responsibility |
|--------------------------|------------------------------------------------------------------------------------------------------------|
| `entrypoint.sh` | Validates required env vars, renders the cron file from `RUN_AT`/`TZ`, then `exec cron -f`. |
| `tenbackward.main` | CLI entry point; orchestrates startup logging, retry-wrapped pipeline pass, and run summary. |
| `tenbackward.config` | Loads `.env` + process env, applies defaults, validates schema, produces a typed `Config` dataclass. |
| `tenbackward.logging_setup` | JSON formatter (one log record per line), secret redaction, and `log_startup` / `log_error` / `log_run_summary` helpers. |
| `tenbackward.state` | Read/write of `posted.json` (atomic temp-file replace). |
| `tenbackward.blog` | Stubbed blog clone/update helper (lands in a follow-up job). |
| `/etc/cron.d/tenbackward`| Rendered cron file. One daily line that `cd /app` and runs `python -m tenbackward`. |
# Communication & Wiring
```
+-------------------+ env vars +-------------------------+
| entrypoint | -------------------> | config.load_config |
| (.sh) | | (Config dataclass) |
+---------+---------+ +-----------+-------------+
| |
| installs cron file | feeds
v v
+-------------------+ +-------------------------+
| /etc/cron.d/ | -- daily fires --> | tenbackward.main.main |
| tenbackward | | - configure logging |
+-------------------+ | - retry-wrapped pass |
+-----------+-------------+
|
v
+-------------------------+
| _run_with_retry(...) |
| -> _run_once(...) |
| -> state.{load,save}|
+-------------------------+
```
* **Env → Config.** `load_config()` merges a `.env` file (when
present) under the process environment via `dotenv_values` + `load_dotenv`,
applies defaults, then validates the merged map before producing the
`Config` dataclass.
* **Cron → main.** Each scheduled tick re-runs `python -m tenbackward`,
so every run is a fresh interpreter invocation.
* **Pipeline state.** `_run_once` reads `posted.json` via
`state.load_posted()` for dedupe, then writes it back via
`state.save_posted()` only when at least one new post was recorded.
# Key Files
| Path | Responsibility |
|-------------------------------------|---------------------------------------------------------------------------------|
| `/repo/entrypoint.sh` | Bootstraps cron; validates required env vars; renders `/etc/cron.d/tenbackward`. |
| `/repo/src/tenbackward/main.py` | CLI entry, retry wrapper, pipeline counters. |
| `/repo/src/tenbackward/config.py` | Env merging, defaults, validation, `Config` dataclass, `ConfigError`. |
| `/repo/src/tenbackward/logging_setup.py` | JSON formatter, secret redaction, structured event helpers. |
| `/repo/src/tenbackward/state.py` | `posted.json` read/write with atomic temp-file replace. |
| `/repo/src/tenbackward/blog.py` | Stubbed blog clone helper (not yet wired into the pipeline). |
| `/repo/.env.example` | Canonical list of environment variables. |
| `/repo/docker-compose.yml` | Service definition; binds env vars from the host `.env`. |
| `/repo/Dockerfile` | Builds the runtime image. |
| `/repo/crontab/tenbackward.cron` | Cron template shipped in the image. |
| `/repo/crontab/install-cron.sh` | Installs the rendered cron file with mode 0644; refuses placeholder leftovers. |
| `/repo/tests/` | Pytest suite covering config, logging, runner, cron, and entrypoint validation. |
# Related
* [Config Schema](/architecture/config-schema.md)
* [Logging & Run Summary](/architecture/logging.md)
* [Pipeline Runner](/architecture/pipeline-runner.md)
* [Daily Run Guide](/guides/daily-run.md)
* [Environment Variable Setup](/operations/environment-setup.md)
* [Cron Lifecycle](/operations/cron-lifecycle.md)