---
type: architecture
title: Pipeline Runner
description: How tenbackward.main synchronizes the blog, discovers anniversary candidates, applies deduplication, retries failures, and emits a run summary.
tags: [pipeline, runner, retry]
timestamp: 2026-08-04T17:51:00Z
---

# Purpose
`tenbackward.main` is the entry point executed by cron. The pipeline first
ensures that the configured blog repository is available and current, then
scans Jekyll posts for today's ten-year anniversary, filters already-recorded
paths, publishes a single combined Mastodon status for the new matches, and
finally persists the freshly published identifiers.


# Call Flow

```
main.main()
  ├── configure_json_logging()
  ├── load_config()  ── raises ConfigError → exit 2
  ├── config.data_dir.mkdir(parents=True, exist_ok=True)
  ├── ensure_repo(config.data_dir, blog_repo_url, blog_dir, max_retries)
  │       └── BlogRepoError → log blog_repo_error → exit 1
  ├── log_startup(version, site_url, run_at, tz, hashtags, throwback_prefix)
  ├── result = _run_with_retry(config)
  │       ├── for attempt in 1 .. max_retries+1:
  │       │     try: return _run_once(config)
  │       │     └── _run_once → ensure_repo → find_anniversary_matches
  │       │         → PostedStore deduplication → publish_mastodon
  │       │         → mark_posted_many   # state only after publish OK
  │       ├── log_error("pipeline_error", exc=exc, attempt=attempt, max_attempts=attempts)
  │       ├── log_error("pipeline_failed", exc=last_exc, attempts=attempts)
  │       └── return None
  ├── if result is None: return 1
  └── log_run_summary(scanned, matched, posted, skipped, posted_ids)
```

# Extension Point: `_iter_candidates`

```python
def _iter_candidates(config: Config) -> Iterable[MatchedPost]: ...
```

`_iter_candidates` delegates to `find_anniversary_matches()` with
`config.blog_dir / "_posts" / "blog"` and `config.site_url`. The matcher
returns full `MatchedPost` values (relative path, title, date, canonical
URL); the runner uses `MatchedPost.path` as a stable deduplication ID and
passes the full objects to `publish_mastodon()` for status composition.
See [Anniversary Matching](/architecture/anniversary-matching.md) for the
file and front matter rules.

## Blog Synchronization

`ensure_repo()` runs before candidate discovery. An empty target is cloned
from `BLOG_REPO_URL`; an existing repository is fetched and fast-forwarded.
Transient Git failures use exponential backoff, while local modifications
fail immediately with `BlogRepoError`. The same synchronization occurs once
in `main()` before `startup` and again inside `_run_once()` as the retryable
pipeline boundary.

# `_run_once` — Summary Counters

`_run_once` returns a 5-tuple:

```python
(scanned, matched, posted, skipped, posted_ids)
```

| Counter     | Meaning                                                              |
|-------------|----------------------------------------------------------------------|
| `scanned`   | Items yielded by `_iter_candidates`.                                 |
| `matched`   | Items that survived the dedupe-vs-`already_posted` check.            |
| `posted`    | Items newly recorded in `posted.json` during this run.               |
| `skipped`   | Items that matched but were already in `posted.json` (deduped).      |
| `posted_ids`| The list of post identifiers written to state.                       |

`PostedStore.mark_posted_many` is called **only** when `posted_ids`
is non-empty, so a no-op run does not touch `posted.json` on disk.
The store persists the list under a `"posted"` key
(`{"posted": ["2014/2014-08-04-foo.md", ...]}`) and serialises
concurrent runs with an `fcntl.flock`.

# Publish Boundary

`publish_mastodon()` is the single success boundary. The runner calls it
**after** dedupe and **before** `mark_posted_many`. If the call raises a
`PublishError`, the runner treats it like any other pipeline exception —
the retry loop in `_run_with_retry` re-attempts, and no identifier is
written to `posted.json`. Identical-day matches are published as **one**
combined status: prefix line + one `title\nurl` block per post + hashtags
line, sorted by `(date, path)` for deterministic ordering. The default
prefix is `Heute vor 10 Jahren:` (overridable via `THROWNBACK_PREFIX`).
Generated statuses are validated against `MASTODON_STATUS_LIMIT` (500
characters); the runner fails safely (raises) rather than truncating.

# Retry Behaviour (`_run_with_retry`)

* `attempts = max(1, config.max_retries)` — at least one attempt even if `MAX_RETRIES=0`.
* The runner owns the unified retry budget; `_run_once` calls `ensure_repo` with `max_retries=0`.
* Transient Git and Mastodon failures are retried with delays of `2 ** (attempt - 1)` seconds between attempts.
* Each failed attempt emits `retry_attempt` with `operation`, `attempt`, and the exception `error` text.
* Fatal configuration, local repository, and publishing validation errors emit `retry_exhausted` immediately without retrying.
* When the budget is exhausted, `retry_exhausted` includes the final operation, total attempts, and error text, and `main` returns exit code `1`.

# Silent-on-No-Matches Contract

* `run_complete` is emitted for a successful pass, including when all discovered candidates were already posted.
* A run with zero candidates emits only `startup`; no `run_complete` line is emitted.

# Example Sequence (Healthy Run)

1.  `startup` line at INFO.
2.  `_run_with_retry` returns `(0, 0, 0, 0, [])` (no candidates yet
    because `_iter_candidates` is empty) → `main` exits `0`
    **without** a `run_complete` line.
3.  Next-day tick repeats the cycle.

# Example Sequence (Future Blog Step in Place)

1.  `startup` line at INFO.
2.  `_iter_candidates` yields `[id-1, id-2]`. `id-1` is already in
    `posted.json`, so `skipped += 1`. `id-2` is new and appended to
    `posted_ids` and persisted.
3.  `run_complete` line at INFO with
    `scanned=2, matched=2, posted=1, skipped=1, posted_ids=["id-2"]`.

# Related

* [Anniversary Matching](/architecture/anniversary-matching.md)
* [Logging & Run Summary](/architecture/logging.md)
* [Config Schema](/architecture/config-schema.md)
* [Cron Lifecycle](/operations/cron-lifecycle.md)