Custom Pipelines — Limits & Availability

Updated on Aug 17, 2026

Availability

Custom Pipelines are enabled per workspace. If you don't see the Custom Pipelines area, or the AI Agent tells you it can't create one, the capability isn't switched on for your workspace — contact your Improvado representative.

Some features are built on top of Custom Pipelines and inherit the same requirement — most notably Discovery API governance rules, which create and maintain a pipeline for you behind the scenes.

How a pipeline reaches your data

A pipeline never holds credentials. The connections it needs — data sources, destinations, Improvado storage — are declared on the pipeline itself, and the credentials are resolved securely at run time and scoped to that run.

Two consequences worth knowing:

A pipeline can only reach what you gave it. Adding a new source to an existing pipeline means adding the connection, not pasting a key.
When a connection breaks, the pipeline breaks with it. Expired or revoked credentials surface as an authorization error on the run. Reconnect the source and rerun — there is nothing to change in the pipeline itself.

Pipeline code runs in an isolated per-workspace environment with no access to the wider system, and identifiers are hashed before they leave for a third-party platform. Logs and run output carry counts and request ids, not personal data.

Scale and volume

Each run gets a fixed slice of compute and a time limit, so a pipeline has to work through a large dataset rather than hold all of it at once. The AI Agent designs for this automatically — but it can only do so if it knows the size up front, which is why volume belongs in your prompt.

What this means in practice:

Say how much data you're moving. “This table is about 40 million rows”, “that's roughly 18 months of history” — the pipeline is then built to process it in batches from the start, instead of being rewritten after the first full run fails.
Test on a slice. A pipeline that succeeds on one day and fails on a year is almost always a volume problem, not a logic problem.
Long jobs belong in batches, not in one long run. Each run has a time limit set on the pipeline. Resumable batch processing is the supported way to move a lot of history — raising the limit is not.

If a run fails at full volume after working on a smaller slice, tell the Agent — it can read the failing run and restructure the pipeline to batch or stream the work.

What Custom Pipelines are not for

Bulk historical extraction for reporting. Use Extract & Load — it is built for volume, it has schema handling, and it is cheaper. A pipeline is the right tool when the workflow is custom, not when the volume is large.
One-off exploration. If you just want to see what a platform returns, do it in chat with the Discovery API. Build a pipeline once you want to repeat it.
Threshold alerting on a metric. Discovery API rules give you that through Marketing Data Governance, including the digest email, and build the pipeline for you.

Credits and cost

Building and editing a pipeline is AI Agent work and consumes credits like any other Agent session — see Agent Usage & Credits. Once a pipeline exists, its scheduled runs do not use the AI Agent and therefore don't consume Agent credits: the pipeline executes on its own. Ordinary platform costs still apply to whatever the pipeline reads and writes.

Runtime versions

Pipelines you create today run on Improvado's current workflow runtime, which gives them durable execution, automatic retries, step timelines and scheduling. Older pipelines created before that runtime shipped continue to run as they are; they can be maintained in place, and are only moved across when you explicitly ask for it.