10Backward/docs/architecture/pipeline-runner.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
architecture	Pipeline Runner	How tenbackward.main turns one cron tick into a startup log line, a retry-wrapped pipeline pass, and a single run_complete summary line.	pipeline runner retry	2026-08-04T17:51:00Z

Purpose

tenbackward.main is the entry point executed by cron. Job 1083 rewrote it to: (a) install JSON logging, (b) execute a retry-wrapped pipeline pass that is silent when there are no candidates, and (c) emit a single structured run_complete summary on success.

Call Flow

main.main()
  ├── configure_json_logging()
  ├── load_config()  ── raises ConfigError → exit 2
  ├── config.data_dir.mkdir(parents=True, exist_ok=True)
  ├── log_startup(version, site_url, run_at, tz, hashtags, throwback_prefix)
  ├── result = _run_with_retry(config)
  │       ├── for attempt in 1 .. max_retries+1:
  │       │     try: return _run_once(config)
  │       │     except Exception as exc:
  │       │           log_error("pipeline_error", exc=exc, attempt=attempt, max_attempts=attempts)
  │       ├── log_error("pipeline_failed", exc=last_exc, attempts=attempts)
  │       └── return None
  ├── if result is None: return 1
  └── log_run_summary(scanned, matched, posted, skipped, posted_ids)

Extension Point: _iter_candidates

def _iter_candidates(config: Config) -> Iterable[str]: ...

Job 1083 ships _iter_candidates as an empty iterator so the silent-on-no-matches contract is the default behaviour. The future blog-clone + matching pipeline plugs into this function without touching _run_once or _run_with_retry.

_run_once — Summary Counters

_run_once returns a 5-tuple:

(scanned, matched, posted, skipped, posted_ids)

Counter	Meaning
scanned	Items yielded by _iter_candidates.
matched	Items that survived the dedupe-vs-already_posted check.
posted	Items newly recorded in posted.json during this run.
skipped	Items that matched but were already in posted.json (deduped).
posted_ids	The list of post identifiers written to state.

state.save_posted is called only when posted_ids is non-empty, so a no-op run does not touch posted.json on disk.

Retry Behaviour (_run_with_retry)

• attempts = max(1, config.max_retries + 1) — at least one attempt even if MAX_RETRIES=0.
• Any exception inside _run_once is caught (noqa: BLE001 — intentional, the whole pass is opaque to the runner).
• Each failed attempt is recorded via log_error("pipeline_error", exc=exc, attempt=attempt, max_attempts=attempts).
• When the budget is exhausted, the runner emits log_error("pipeline_failed", exc=last_exc, attempts=attempts) and returns None so main can translate it to exit 1.
• time.sleep(0) between attempts is the placeholder seam where the next job can introduce real backoff.

Silent-on-No-Matches Contract

A run with zero candidates produces exactly one line:

{"ts":"...","level":"INFO","message":"startup","event":"startup",...}

No run_complete line is emitted. Tests assert this by counting log records (tests/test_run_logging.py::test_run_summary_silent_when_no_candidates).

Example Sequence (Healthy Run)

1. startup line at INFO.
2. _run_with_retry returns (0, 0, 0, 0, []) (no candidates yet because _iter_candidates is empty) → main exits 0 without a run_complete line.
3. Next-day tick repeats the cycle.

Example Sequence (Future Blog Step in Place)

1. startup line at INFO.
2. _iter_candidates yields [id-1, id-2]. id-1 is already in posted.json, so skipped += 1. id-2 is new and appended to posted_ids and persisted.
3. run_complete line at INFO with scanned=2, matched=2, posted=1, skipped=1, posted_ids=["id-2"].

Related

• Logging & Run Summary
• Config Schema
• Cron Lifecycle