m0rph3us1987/10Backward
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

AI Implementation feature(1086): Posted-Anniversary Dedup Store (#5)

Browse Source
This commit was merged in pull request #5.
This commit is contained in:
m0rph3us1987
2026-08-04 18:35:28 +00:00
parent 72c2369160
commit 7d39f3a339
10 changed files with 516 additions and 86 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture/anniversary-matching.md
+76
View File
@@ -0,0 +1,76 @@
---
type: architecture
title: Anniversary Matching
description: Rules and file boundaries used to discover Jekyll posts whose publication anniversary is exactly ten years before the current date.
tags: [matching, jekyll, anniversary, posts]
timestamp: 2026-08-04T18:35:00Z
---
# Purpose
`tenbackward.matching` scans the synchronized blog at
`<blog_dir>/_posts/blog` and returns posts whose date is exactly ten years
before the current date. The current date is evaluated in the configured
IANA timezone, defaulting to `Europe/Berlin` in the matcher.
# Discovery Rules
1. Walk every Markdown file below the configured post root recursively.
2. Parse YAML front matter when present.
3. Require a filename stem shaped as `YYYY-MM-DD-slug` with a valid calendar
date and non-empty slug.
4. Prefer a valid front matter `date`; otherwise use the date in the filename.
5. Match the post year to `today.year - 10` and the month/day to today.
6. Skip posts with `published: false`.
7. Return a `MatchedPost` containing the relative path, title, date, and
canonical site URL. If front matter has no title, use the filename slug.
# Leap-Day Behavior
A February 29 post matches February 29 when the current year is a leap year.
In a non-leap year it matches March 1, preserving one anniversary event for
leap-day posts.
# Output and Integration
`find_anniversary_matches()` returns `MatchedPost` values for callers that
need metadata. `iter_anniversary_paths()` yields only relative paths; the
pipeline uses those paths as deduplication identifiers in the
[system state store](/architecture/system-overview.md) and persists new IDs in
`posted.json`.
Unreadable files, malformed front matter, invalid filenames, unpublished
posts, and unexpected per-file errors are skipped with structured warning
events rather than aborting the full scan.
# Key Files
| Path | Responsibility |
|---|---|
| `/repo/src/tenbackward/matching.py` | Date parsing, Jekyll front matter handling, anniversary rules, URL construction, and recursive scanning. |
| `/repo/src/tenbackward/main.py` | Supplies the blog post root and site URL, then applies state deduplication. |
| `/repo/tests/test_matching.py` | Covers filename/front matter parsing, date matching, unpublished posts, malformed files, URL output, and leap-day behavior. |
| `/repo/src/tenbackward/blog.py` | Ensures the source repository is cloned or fast-forwarded before matching. |
# Examples
## Matching post
On 2026-08-04, a file named
`_posts/blog/2016/2016-08-04-release.md` is eligible. Its identifier is
`2016/2016-08-04-release.md` and its generated URL is:
```text
https://example.com/2016/08/04/release/
```
## Skipped post
A file with `published: false`, an invalid date, or a non-matching
month/day is not yielded and does not enter `posted.json`.
# Related
* [Pipeline Runner](/architecture/pipeline-runner.md)
* [System Architecture](/architecture/system-overview.md)
* [Daily Run Guide](/guides/daily-run.md)
docs/architecture/pipeline-runner.md
+40 -23
View File
@@ -1,17 +1,19 @@
---
type: architecture
title: Pipeline Runner
description: How tenbackward.main turns one cron tick into a startup log line, a retry-wrapped pipeline pass, and a single run_complete summary line.
description: How tenbackward.main synchronizes the blog, discovers anniversary candidates, applies deduplication, retries failures, and emits a run summary.
tags: [pipeline, runner, retry]
timestamp: 2026-08-04T17:51:00Z
---
# Purpose
`tenbackward.main` is the entry point executed by cron. The pipeline first
ensures that the configured blog repository is available and current, then
scans Jekyll posts for today's ten-year anniversary, filters already-recorded
paths, and persists newly discovered identifiers. Job 1086 wires the
anniversary matcher into the runner; Mastodon publishing remains a future
step.
`tenbackward.main` is the entry point executed by cron. Job 1083
rewrote it to: (a) install JSON logging, (b) execute a retry-wrapped
pipeline pass that is **silent when there are no candidates**, and
(c) emit a single structured `run_complete` summary on success.
# Call Flow
@@ -20,12 +22,15 @@ main.main()
├── configure_json_logging()
├── load_config() ── raises ConfigError → exit 2
├── config.data_dir.mkdir(parents=True, exist_ok=True)
├── ensure_repo(config.data_dir, blog_repo_url, blog_dir, max_retries)
│ └── BlogRepoError → log blog_repo_error → exit 1
├── log_startup(version, site_url, run_at, tz, hashtags, throwback_prefix)
├── result = _run_with_retry(config)
│ ├── for attempt in 1 .. max_retries+1:
│ │ try: return _run_once(config)
│ │ except Exception as exc:
│ │ log_error("pipeline_error", exc=exc, attempt=attempt, max_attempts=attempts)
│ │ └── _run_once → ensure_repo → iter_anniversary_paths
│ │ → PostedStore deduplication → mark_posted_many
│ ├── log_error("pipeline_error", exc=exc, attempt=attempt, max_attempts=attempts)
│ ├── log_error("pipeline_failed", exc=last_exc, attempts=attempts)
│ └── return None
├── if result is None: return 1
@@ -38,10 +43,26 @@ main.main()
def _iter_candidates(config: Config) -> Iterable[str]: ...
```
Job 1083 ships `_iter_candidates` as an **empty iterator** so the
silent-on-no-matches contract is the default behaviour. The future
blog-clone + matching pipeline plugs into this function without
touching `_run_once` or `_run_with_retry`.
Job 1086 wires `_iter_candidates` to the anniversary matcher. The matcher
is now active; it discovers matching Jekyll posts after the repository is
synchronized.
## Blog Synchronization
`ensure_repo()` runs before candidate discovery. An empty target is cloned
from `BLOG_REPO_URL`; an existing repository is fetched and fast-forwarded.
Transient Git failures use exponential backoff, while local modifications
fail immediately with `BlogRepoError`. The same synchronization occurs once
in `main()` before `startup` and again inside `_run_once()` as the retryable
pipeline boundary.
`_iter_candidates(config)` delegates to `iter_anniversary_paths()` with
`config.blog_dir / "_posts" / "blog"` and `config.site_url`. The matcher
returns relative Markdown paths, such as
`2016/2016-08-04-example.md`, which are used as stable deduplication IDs.
See [Anniversary Matching](/architecture/anniversary-matching.md) for the
file and front matter rules.
# `_run_once` — Summary Counters
@@ -59,8 +80,11 @@ touching `_run_once` or `_run_with_retry`.
| `skipped` | Items that matched but were already in `posted.json` (deduped). |
| `posted_ids`| The list of post identifiers written to state. |
`state.save_posted` is called **only** when `posted_ids` is non-empty,
so a no-op run does not touch `posted.json` on disk.
`PostedStore.mark_posted_many` is called **only** when `posted_ids`
is non-empty, so a no-op run does not touch `posted.json` on disk.
The store persists the list under a `"posted"` key
(`{"posted": ["2014/2014-08-04-foo.md", ...]}`) and serialises
concurrent runs with an `fcntl.flock`.
# Retry Behaviour (`_run_with_retry`)
@@ -73,19 +97,11 @@ so a no-op run does not touch `posted.json` on disk.
* When the budget is exhausted, the runner emits
`log_error("pipeline_failed", exc=last_exc, attempts=attempts)` and
returns `None` so `main` can translate it to `exit 1`.
* `time.sleep(0)` between attempts is the placeholder seam where the
next job can introduce real backoff.
# Silent-on-No-Matches Contract
A run with zero candidates produces exactly one line:
```json
{"ts":"...","level":"INFO","message":"startup","event":"startup",...}
```
No `run_complete` line is emitted. Tests assert this by counting log
records (`tests/test_run_logging.py::test_run_summary_silent_when_no_candidates`).
* `run_complete` is emitted for a successful pass, including when all discovered candidates were already posted.
* A run with zero candidates emits only `startup`; no `run_complete` line is emitted.
# Example Sequence (Healthy Run)
@@ -106,6 +122,7 @@ records (`tests/test_run_logging.py::test_run_summary_silent_when_no_candidates`
# Related
* [Anniversary Matching](/architecture/anniversary-matching.md)
* [Logging & Run Summary](/architecture/logging.md)
* [Config Schema](/architecture/config-schema.md)
* [Cron Lifecycle](/operations/cron-lifecycle.md)
docs/architecture/system-overview.md
+24 -13
View File
@@ -1,7 +1,7 @@
---
type: architecture
title: System Architecture
description: Component map of 10Backward — how config, logging, the pipeline runner, state persistence, and container entrypoint are wired together.
description: Component map of 10Backward — how configuration, blog synchronization, anniversary matching, logging, the pipeline runner, state persistence, and container entrypoint are wired together.
tags: [architecture, overview]
timestamp: 2026-08-04T17:51:00Z
---
@@ -9,8 +9,11 @@ timestamp: 2026-08-04T17:51:00Z
# Overview
`10Backward` is a Mastodon daily-throwback bot that runs as a single
containerised cron job. Job 1083 adds a structured logging layer and a
retry-capable pipeline runner on top of the scaffold shipped by Job 1082.
containerised cron job. Job 1086 adds anniversary matching across Jekyll
posts, backed by the blog clone/pull workflow from Job 1084 and the
structured logging, retry-capable runner, and deduplication state from
Job 1083. The current pipeline identifies matching post paths and records
those identifiers; Mastodon publishing is still a future integration.
The container boots, validates environment configuration, renders a
`/etc/cron.d/tenbackward` entry that fires once per day at the
@@ -33,8 +36,9 @@ foreground of the `cron` process. Each scheduled invocation calls
| `tenbackward.main` | CLI entry point; orchestrates startup logging, retry-wrapped pipeline pass, and run summary. |
| `tenbackward.config` | Loads `.env` + process env, applies defaults, validates schema, produces a typed `Config` dataclass. |
| `tenbackward.logging_setup` | JSON formatter (one log record per line), secret redaction, and `log_startup` / `log_error` / `log_run_summary` helpers. |
| `tenbackward.state` | Read/write of `posted.json` (atomic temp-file replace). |
| `tenbackward.state` | `PostedStore` — self-contained dedup store. `posted.json` holds `{"posted": [str, ...]}`; mutations are serialised by an `fcntl.flock` on a sibling lock file, writes go through a temp-file replace, and `load()` auto-creates an empty list when the file is missing. |
| `tenbackward.blog` | GitPython `ensure_repo()` — clones the configured blog repo on first run, fast-forwards it via `pull --ff-only` thereafter, with exponential-backoff retries on transient network errors. |
| `tenbackward.matching` | Walks `_posts/blog/**/*.md`, parses Jekyll front matter and filenames, and yields posts exactly ten years before the current date using Berlin-time and leap-day rules. |
| `/etc/cron.d/tenbackward`| Rendered cron file. One daily line that `cd /app` and runs `python -m tenbackward`. |
# Communication & Wiring
@@ -54,11 +58,13 @@ foreground of the `cron` process. Each scheduled invocation calls
+-----------+-------------+
|
v
+-------------------------+
| _run_with_retry(...) |
| -> _run_once(...) |
| -> state.{load,save}|
+-------------------------+
+-------------------------+
| _run_with_retry(...) |
| -> _run_once(...) |
| -> PostedStore. |
| {is_posted, |
| mark_posted_many}|
+-------------------------+
```
* **Env → Config.** `load_config()` merges a `.env` file (when
@@ -68,8 +74,11 @@ foreground of the `cron` process. Each scheduled invocation calls
* **Cron → main.** Each scheduled tick re-runs `python -m tenbackward`,
so every run is a fresh interpreter invocation.
* **Pipeline state.** `_run_once` reads `posted.json` via
`state.load_posted()` for dedupe, then writes it back via
`state.save_posted()` only when at least one new post was recorded.
`PostedStore.is_posted()` for dedupe, then writes it back via
`PostedStore.mark_posted_many()` only when at least one new post
was recorded. The store serialises concurrent runs with an
`fcntl.flock` on a sibling lock file and writes via temp-file
rename.
# Key Files
@@ -79,8 +88,10 @@ foreground of the `cron` process. Each scheduled invocation calls
| `/repo/src/tenbackward/main.py` | CLI entry, retry wrapper, pipeline counters. |
| `/repo/src/tenbackward/config.py` | Env merging, defaults, validation, `Config` dataclass, `ConfigError`. |
| `/repo/src/tenbackward/logging_setup.py` | JSON formatter, secret redaction, structured event helpers. |
| `/repo/src/tenbackward/state.py` | `posted.json` read/write with atomic temp-file replace. |
| `/repo/src/tenbackward/state.py` | `PostedStore` + module helpers (`load_posted`, `is_posted`, `mark_posted`, `mark_posted_many`); list-shaped JSON, `fcntl.flock`, atomic temp-file replace, env-var data dir. |
| `/repo/src/tenbackward/blog.py` | `ensure_repo()` — GitPython clone + fast-forward pull with `2^n` retry/backoff; raises `BlogRepoError` on local modifications or exhausted retries. |
| `/repo/src/tenbackward/matching.py` | Jekyll post discovery, front matter parsing, anniversary and leap-day matching, and canonical URL construction. |
| `/repo/src/tenbackward/__main__.py` | Module entrypoint that invokes `tenbackward.main.main()`. |
| `/repo/.env.example` | Canonical list of environment variables. |
| `/repo/docker-compose.yml` | Service definition; binds env vars from the host `.env`. |
| `/repo/Dockerfile` | Builds the runtime image. |
@@ -93,6 +104,6 @@ foreground of the `cron` process. Each scheduled invocation calls
* [Config Schema](/architecture/config-schema.md)
* [Logging & Run Summary](/architecture/logging.md)
* [Pipeline Runner](/architecture/pipeline-runner.md)
* [Daily Run Guide](/guides/daily-run.md)
* [Anniversary Matching](/architecture/anniversary-matching.md)
* [Environment Variable Setup](/operations/environment-setup.md)
* [Cron Lifecycle](/operations/cron-lifecycle.md)