OpenVelo Agent
112 lines
4.2 KiB
Markdown
112 lines
4.2 KiB
Markdown
---
|
|
type: architecture
|
|
title: Pipeline Runner
|
|
description: How tenbackward.main turns one cron tick into a startup log line, a retry-wrapped pipeline pass, and a single run_complete summary line.
|
|
tags: [pipeline, runner, retry]
|
|
timestamp: 2026-08-04T17:51:00Z
|
|
---
|
|
|
|
# Purpose
|
|
|
|
`tenbackward.main` is the entry point executed by cron. Job 1083
|
|
rewrote it to: (a) install JSON logging, (b) execute a retry-wrapped
|
|
pipeline pass that is **silent when there are no candidates**, and
|
|
(c) emit a single structured `run_complete` summary on success.
|
|
|
|
# Call Flow
|
|
|
|
```
|
|
main.main()
|
|
├── configure_json_logging()
|
|
├── load_config() ── raises ConfigError → exit 2
|
|
├── config.data_dir.mkdir(parents=True, exist_ok=True)
|
|
├── log_startup(version, site_url, run_at, tz, hashtags, throwback_prefix)
|
|
├── result = _run_with_retry(config)
|
|
│ ├── for attempt in 1 .. max_retries+1:
|
|
│ │ try: return _run_once(config)
|
|
│ │ except Exception as exc:
|
|
│ │ log_error("pipeline_error", exc=exc, attempt=attempt, max_attempts=attempts)
|
|
│ ├── log_error("pipeline_failed", exc=last_exc, attempts=attempts)
|
|
│ └── return None
|
|
├── if result is None: return 1
|
|
└── log_run_summary(scanned, matched, posted, skipped, posted_ids)
|
|
```
|
|
|
|
# Extension Point: `_iter_candidates`
|
|
|
|
```python
|
|
def _iter_candidates(config: Config) -> Iterable[str]: ...
|
|
```
|
|
|
|
Job 1083 ships `_iter_candidates` as an **empty iterator** so the
|
|
silent-on-no-matches contract is the default behaviour. The future
|
|
blog-clone + matching pipeline plugs into this function without
|
|
touching `_run_once` or `_run_with_retry`.
|
|
|
|
# `_run_once` — Summary Counters
|
|
|
|
`_run_once` returns a 5-tuple:
|
|
|
|
```python
|
|
(scanned, matched, posted, skipped, posted_ids)
|
|
```
|
|
|
|
| Counter | Meaning |
|
|
|-------------|----------------------------------------------------------------------|
|
|
| `scanned` | Items yielded by `_iter_candidates`. |
|
|
| `matched` | Items that survived the dedupe-vs-`already_posted` check. |
|
|
| `posted` | Items newly recorded in `posted.json` during this run. |
|
|
| `skipped` | Items that matched but were already in `posted.json` (deduped). |
|
|
| `posted_ids`| The list of post identifiers written to state. |
|
|
|
|
`state.save_posted` is called **only** when `posted_ids` is non-empty,
|
|
so a no-op run does not touch `posted.json` on disk.
|
|
|
|
# Retry Behaviour (`_run_with_retry`)
|
|
|
|
* `attempts = max(1, config.max_retries + 1)` — at least one attempt
|
|
even if `MAX_RETRIES=0`.
|
|
* Any exception inside `_run_once` is caught (`noqa: BLE001` —
|
|
intentional, the whole pass is opaque to the runner).
|
|
* Each failed attempt is recorded via
|
|
`log_error("pipeline_error", exc=exc, attempt=attempt, max_attempts=attempts)`.
|
|
* When the budget is exhausted, the runner emits
|
|
`log_error("pipeline_failed", exc=last_exc, attempts=attempts)` and
|
|
returns `None` so `main` can translate it to `exit 1`.
|
|
* `time.sleep(0)` between attempts is the placeholder seam where the
|
|
next job can introduce real backoff.
|
|
|
|
# Silent-on-No-Matches Contract
|
|
|
|
A run with zero candidates produces exactly one line:
|
|
|
|
```json
|
|
{"ts":"...","level":"INFO","message":"startup","event":"startup",...}
|
|
```
|
|
|
|
No `run_complete` line is emitted. Tests assert this by counting log
|
|
records (`tests/test_run_logging.py::test_run_summary_silent_when_no_candidates`).
|
|
|
|
# Example Sequence (Healthy Run)
|
|
|
|
1. `startup` line at INFO.
|
|
2. `_run_with_retry` returns `(0, 0, 0, 0, [])` (no candidates yet
|
|
because `_iter_candidates` is empty) → `main` exits `0`
|
|
**without** a `run_complete` line.
|
|
3. Next-day tick repeats the cycle.
|
|
|
|
# Example Sequence (Future Blog Step in Place)
|
|
|
|
1. `startup` line at INFO.
|
|
2. `_iter_candidates` yields `[id-1, id-2]`. `id-1` is already in
|
|
`posted.json`, so `skipped += 1`. `id-2` is new and appended to
|
|
`posted_ids` and persisted.
|
|
3. `run_complete` line at INFO with
|
|
`scanned=2, matched=2, posted=1, skipped=1, posted_ids=["id-2"]`.
|
|
|
|
# Related
|
|
|
|
* [Logging & Run Summary](/architecture/logging.md)
|
|
* [Config Schema](/architecture/config-schema.md)
|
|
* [Cron Lifecycle](/operations/cron-lifecycle.md)
|