Pipelines — Limits & Availability

Availability

Pipelines are enabled per workspace. If Automations has no Pipelines tab, or the AI Agent tells you it can't create a pipeline, the capability isn't switched on for your workspace — contact your Improvado representative.

Some features are built on top of Pipelines and inherit the same requirement — most notably Discovery API governance rules, which create and maintain a pipeline for you behind the scenes.

How a pipeline reaches your data

Pipeline code never contains credentials. A pipeline reaches other systems in one of two ways, and in both the credential is resolved securely at run time:

• Connections — data sources, destinations and Improvado storage already connected in your workspace. They are declared on the pipeline itself and scoped to each run.
• Secrets — an API key or password for a service that has no Improvado connection. You add the value on the pipeline's Secrets tab or as a workspace secret; the pipeline reads it by name. See Pipeline secrets.

Two consequences worth knowing:

• A pipeline can only reach what you gave it. Adding a connected source to an existing pipeline means adding the connection; any other service needs a secret, never a key pasted into chat or code.
• When a connection or key breaks, the pipeline breaks with it. Expired or revoked credentials surface as an authorization error on the run. Reconnect the source or update the secret and rerun — there is nothing to change in the pipeline itself.

Pipeline code runs in an isolated per-workspace environment with no access to the wider system, and identifiers are hashed before they leave for a third-party platform. Logs and run output carry counts and request ids, not personal data.

Scale and volume

Each run gets a fixed slice of compute and a time limit, so a pipeline has to work through a large dataset rather than hold all of it at once. The AI Agent designs for this automatically — but it can only do so if it knows the size up front, which is why volume belongs in your prompt.

What this means in practice:

• Say how much data you're moving. “This table is about 40 million rows”, “that's roughly 18 months of history” — the pipeline is then built to process it in batches from the start, instead of being rewritten after the first full run fails.
• Test on a slice. A pipeline that succeeds on one day and fails on a year is almost always a volume problem, not a logic problem.
• Long jobs belong in batches, not in one long run. Each run has a time limit set on the pipeline. Resumable batch processing is the supported way to move a lot of history — raising the limit is not.

If a run fails at full volume after working on a smaller slice, tell the Agent — it can read the failing run and restructure the pipeline to batch or stream the work.

What Pipelines are not for

• Bulk historical extraction for reporting. Use Extract & Load — it is built for volume, it has schema handling, and it is cheaper. A pipeline is the right tool when the workflow is custom, not when the volume is large.
• One-off exploration. If you just want to see what a platform returns, do it in chat with the Discovery API. Build a pipeline once you want to repeat it.
• Threshold alerting on a metric. Discovery API rules give you that through Marketing Data Governance, including the digest email, and build the pipeline for you.

Credits and cost

Building and editing a pipeline is AI Agent work and consumes credits like any other Agent session — see Agent Usage & Credits. Once a pipeline exists, its scheduled runs do not use the AI Agent and therefore don't consume Agent credits: the pipeline executes on its own. Ordinary platform costs still apply to whatever the pipeline reads and writes.

Runtime versions

Pipelines you create today run on Improvado's current workflow runtime, which gives them durable execution, automatic retries, step timelines and scheduling. Older pipelines created before that runtime shipped continue to run as they are; they can be maintained in place, and are only moved across when you explicitly ask for it.

Related articles