10Backward/docs/architecture/pipeline-runner.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
architecture	Pipeline Runner	How tenbackward.main synchronizes the blog, discovers anniversary candidates, applies deduplication, retries failures, and emits a run summary.	pipeline runner retry	2026-08-04T17:51:00Z

Purpose

tenbackward.main is the entry point executed by cron. The pipeline first ensures that the configured blog repository is available and current, then scans Jekyll posts for today's ten-year anniversary, filters already-recorded paths, publishes a single combined Mastodon status for the new matches, and finally persists the freshly published identifiers.

Call Flow

main.main()
  ├── configure_json_logging()
  ├── load_config()  ── raises ConfigError → exit 2
  ├── config.data_dir.mkdir(parents=True, exist_ok=True)
  ├── ensure_repo(config.data_dir, blog_repo_url, blog_dir, max_retries)
  │       └── BlogRepoError → log blog_repo_error → exit 1
  ├── log_startup(version, site_url, run_at, tz, hashtags, throwback_prefix)
  ├── result = _run_with_retry(config)
  │       ├── for attempt in 1 .. max_retries+1:
  │       │     try: return _run_once(config)
  │       │     └── _run_once → ensure_repo → find_anniversary_matches
  │       │         → PostedStore deduplication → publish_mastodon
  │       │         → mark_posted_many   # state only after publish OK
  │       ├── log_error("pipeline_error", exc=exc, attempt=attempt, max_attempts=attempts)
  │       ├── log_error("pipeline_failed", exc=last_exc, attempts=attempts)
  │       └── return None
  ├── if result is None: return 1
  └── log_run_summary(scanned, matched, posted, skipped, posted_ids)

Extension Point: _iter_candidates

def _iter_candidates(config: Config) -> Iterable[MatchedPost]: ...

_iter_candidates delegates to find_anniversary_matches() with config.blog_dir / "_posts" / "blog" and config.site_url. The matcher returns full MatchedPost values (relative path, title, date, canonical URL); the runner uses MatchedPost.path as a stable deduplication ID and passes the full objects to publish_mastodon() for status composition. See Anniversary Matching for the file and front matter rules.

Blog Synchronization

ensure_repo() runs before candidate discovery. An empty target is cloned from BLOG_REPO_URL; an existing repository is fetched and fast-forwarded. Transient Git failures use exponential backoff, while local modifications fail immediately with BlogRepoError. The same synchronization occurs once in main() before startup and again inside _run_once() as the retryable pipeline boundary.

_run_once — Summary Counters

_run_once returns a 5-tuple:

(scanned, matched, posted, skipped, posted_ids)

Counter	Meaning
scanned	Items yielded by _iter_candidates.
matched	Items that survived the dedupe-vs-already_posted check.
posted	Items newly recorded in posted.json during this run.
skipped	Items that matched but were already in posted.json (deduped).
posted_ids	The list of post identifiers written to state.

PostedStore.mark_posted_many is called only when posted_ids is non-empty, so a no-op run does not touch posted.json on disk. The store persists the list under a "posted" key ({"posted": ["2014/2014-08-04-foo.md", ...]}) and serialises concurrent runs with an fcntl.flock.

Publish Boundary

publish_mastodon() is the single success boundary. The runner calls it after dedupe and before mark_posted_many. If the call raises a PublishError, the runner treats it like any other pipeline exception — the retry loop in _run_with_retry re-attempts, and no identifier is written to posted.json. Identical-day matches are published as one combined status: prefix line + one title\nurl block per post + hashtags line, sorted by (date, path) for deterministic ordering. The default prefix is Heute vor 10 Jahren: (overridable via THROWNBACK_PREFIX). Generated statuses are validated against MASTODON_STATUS_LIMIT (500 characters); the runner fails safely (raises) rather than truncating.

Retry Behaviour (_run_with_retry)

• attempts = max(1, config.max_retries) — at least one attempt even if MAX_RETRIES=0.
• The runner owns the unified retry budget; _run_once calls ensure_repo with max_retries=0.
• Transient Git and Mastodon failures are retried with delays of 2 ** (attempt - 1) seconds between attempts.
• Each failed attempt emits retry_attempt with operation, attempt, and the exception error text.
• Fatal configuration, local repository, and publishing validation errors emit retry_exhausted immediately without retrying.
• When the budget is exhausted, retry_exhausted includes the final operation, total attempts, and error text, and main returns exit code 1.

Silent-on-No-Matches Contract

• run_complete is emitted for a successful pass, including when all discovered candidates were already posted.
• A run with zero candidates emits only startup; no run_complete line is emitted.

Example Sequence (Healthy Run)

1. startup line at INFO.
2. _run_with_retry returns (0, 0, 0, 0, []) (no candidates yet because _iter_candidates is empty) → main exits 0 without a run_complete line.
3. Next-day tick repeats the cycle.

Example Sequence (Future Blog Step in Place)

1. startup line at INFO.
2. _iter_candidates yields [id-1, id-2]. id-1 is already in posted.json, so skipped += 1. id-2 is new and appended to posted_ids and persisted.
3. run_complete line at INFO with scanned=2, matched=2, posted=1, skipped=1, posted_ids=["id-2"].

Related

• Anniversary Matching
• Logging & Run Summary
• Config Schema
• Cron Lifecycle