Pipeline Runs, Schedules & Artifacts
Overview
Once a Custom Pipeline exists, everything you do with it happens on its detail page: run it, put it on a schedule, watch what each run did, and download what it produced. This page covers that lifecycle.
Running a pipeline
There are two ways a pipeline runs:
- On demand — you click Run. Use this for the first test after the Agent builds it, for one-off reruns, and for anything you'd rather trigger by hand.
- On a schedule — the pipeline runs itself on a recurring cadence.
A pipeline does not need a schedule to be useful. Plenty of pipelines exist purely as an on-demand button — a backfill you run twice a year, a QA export you pull before a client call.
Before the first full run, ask the Agent to limit the pipeline — one day, one account, a row cap — and review the output. It is much cheaper than debugging an hour-long run, and it is the only way to catch a business-logic mistake before it reaches your warehouse or your ad accounts.
Schedules
A schedule is defined as a cron expression and always runs in UTC. If you want a report to land at 9am local time, do the conversion — the Agent will do it for you if you say which timezone you mean.
Two behaviours are worth deciding deliberately when you set one up:
What happens if a run is still going when the next one is due
| Behaviour | What it does | Good default for |
|---|---|---|
| Skip | Drops the new trigger while a run is still going | Most recurring ETL, where re-running the same window is harmless |
| Queue one | Holds exactly one missed trigger and runs it after | Cases where missing a run is worse than delaying it |
| Cancel the running one | Stops the in-flight run and starts the new one | Refreshes where the latest data wins — audience syncs, pacing checks |
Under Skip, a run that never finishes will silently swallow every future trigger — the pipeline looks scheduled but nothing fires. If a scheduled pipeline goes quiet, check Run History for a run stuck in a running state before looking anywhere else.
Catch-up
Triggers missed while the platform was unavailable are replayed within a catch-up window. For hourly-or-slower jobs this is what you want — one missed tick replays. For very frequent jobs, keep the window short so a brief outage doesn't produce a burst of runs.
Enable the schedule only after a manual run has produced output you're happy with. Once cron is live, a bug fires on every tick.
Run history
Every execution is recorded. For each run you get:
- Status and live progress while it's going
- Duration and start/finish times
- A step timeline — which step is running, which ones completed, which one failed
- Output — whatever the pipeline returned as its result
- Logs and metrics
- Errors, attached to the step that raised them
Steps are isolated: one failing step does not cascade through the rest, and a step that fails on a transient problem — a timeout, a rate limit, a platform 5xx — is retried automatically. An error that won't get better on retry (a revoked token, a policy rejection, a permission problem) is surfaced to you instead of consuming the retry budget.
Artifacts
Any file a run writes — a report, an export, a debug dump — is listed under that run as an artifact. There is no separate publish step: if the pipeline wrote it, it's there. Download links are short-lived and generated on demand, so use the run page rather than saving a link.
Artifacts are per run, which makes them the natural place to keep evidence: the list of rows a data-quality run rejected, the before/after of a governance rename, the exact file a client was sent last Tuesday.
Incremental runs and pipeline state
Pipelines can keep durable state between runs — a cursor, a bookmark, a set of already-processed ids. This is what makes incremental processing and resumable backfills work: the run picks up where the last one stopped rather than reprocessing everything.
The rule that matters when you're reviewing a pipeline the Agent built: the cursor should advance only after the target has accepted the data, never before. If it advances first, a failure halfway through loses that window permanently.
Editing, pausing and archiving
- Edit with AI — describe the change (different filter, wider window, new destination, different schedule) and the Agent patches the existing pipeline. It edits what's there rather than rebuilding from scratch, so your run history and state survive.
- Pause / Resume — stops and restarts scheduled execution without deleting anything. On-demand runs still work while paused.
- Archive — removes the pipeline from the active list and deletes its schedule. A run already in flight finishes.
Editing a pipeline while a run is in flight does not change that run. The next run picks up the new version, a short moment after the edit is saved.
Troubleshooting
| Symptom | Usual cause |
|---|---|
| Scheduled pipeline stopped producing runs | A previous run is stuck in a running state and the schedule is set to Skip |
| Run fails with an authorization error | The underlying connection's credentials expired — reconnect the source, then rerun |
| Run completes but writes nothing | The cursor is already past the window, or the filter matched no rows — check the run output, which reports counts, not just success |
| Run fails only at full volume | The run exceeded its resource budget — see Limits & Availability |
| Audience uploaded, size shows 0 | Expected — population is asynchronous on every ad platform. Check again later; do not re-upload |
If a pipeline is failing and the cause isn't obvious from the run timeline, ask the Agent in chat — it can read the failing run's logs and errors directly.
Related articles
Was this article helpful?
Thanks for the feedback!