how to run a daily content pipeline with cron jobs

TL;DR

Run the daily content pipeline as cron jobs with three disciplines: each job is idempotent (reruns are safe), every run logs to a durable file, and failures alert rather than silently stopping. Schedule generation, review, and publish as separate steps so a failure in one does not corrupt the others.

how to run a daily content pipeline with cron jobs

Use this when

• Publishing content on a daily or regular cadence
• The pipeline has multiple stages (draft, review, publish)
• Runs happen unattended

Not for this skill when

• Publishing is irregular (run manually)
• Every step needs human judgment (automate less, not more)
• You have no alerting (silent failures will eat you)

Steps

1. Split the pipeline into staged jobs. Generate, review-queue, and publish as separate cron entries with clear handoffs via files. Expected: a failure in generation never publishes a half-done piece.

1. Make every job idempotent. Deterministic output keys and "skip if already done" checks mean reruns are safe. Expected: running the job twice produces one result.

pipeline stages:
  06:00 generate -> drafts/[date].md (skip if exists)
  07:00 review   -> moves approved to queue/
  08:00 publish  -> publishes queue/, logs URLs

1. Log every run durably. Append-only log with timestamps, outside /tmp, one line per significant event. Expected: you can reconstruct any day's run from the log.

1. Alert on failure, not on success. A failed stage pages or messages you; successful runs stay quiet. Expected: silence means healthy, noise means broken.

1. Keep a manual override. A flag file or command that pauses the pipeline for launch holds or incidents. Expected: you can stop the machine in seconds.

Variant phrasings

cron-based publishing pipeline

Staged jobs, idempotent reruns, durable logs, failure alerts, manual override.

automating a daily content run

Generate, review, publish as separate scheduled steps with file handoffs.

scheduled content pipeline best practices

Idempotency first, logging second, alerting third; never let stages blur.

Why it happens

Daily cadence plus unattended execution means failures compound silently: a stuck job publishes nothing for a week, or worse, publishes broken content daily. Staged, idempotent, logged jobs make the pipeline boring, which is the goal.

Edge cases / pitfalls

• Cron environments are minimal; use absolute paths and explicit environment setup.
• Overlapping runs corrupt state; use a lock file per job.
• Daylight saving shifts cron times; schedule in UTC or accept the shift.
• Test the failure alert path; an alert that never fires is decoration.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_vT7-GoReb71wPw1R3erv-w