Documentation

Automated load

Automated load allows for data push of all new data tables created after data extraction into your current destination automatically. When you extract a new data table, we will create a load order for it as well.

{%docs-informer info%}

New data tables created through extracts are automatically loaded into the destination but with varying delays. Generally, it takes about 5 minutes for the table to appear in the Data Explorer. Data loading typically begins an additional 15–20 minutes afterward. Learn more about transformation tables here.

{%docs-informer-end%}

To enable automated load for your connection:

1. Select a specific destination from the Connected destinations list.
2. Go to Automation tab.
3. Mark the Load all new data tables after data extraction automatically checkbox.

By default, this checkbox is unchecked. If you mark the checkbox for two or more destinations, we will create a separate load order for all of them.

How to set up data load

You can set up data load for active connected destinations in two ways:

• Click Setup load for the required destination in the Connected destinations list.

• Click the Setup load button on the top right corner for the necessary destination page.

Step 1. Choose your data

Here you need to select the required data tables for upload. You can use the search bar for easy navigation.

To continue working with the data table in Step 2, mark a checkbox near it. Click it again to remove the selection.

Also, you can select a checkbox near the table header to choose all tables only on this page! To choose data tables on other pages, move between them. Marked checkboxes are saved when moving.

When all the required tables are selected, click the Continue button. This button is non-clickable until at least one data table is selected.

Step 2. Setup load

Here you need to define the following settings:

• Load time
• Load entire historical data
• Name of the data object on the destination

Load time

Load time defines how often Improvado loads your data to the Improvado database. By default, this happens every time once the data extraction is finished.

Load entire historical data

This parameter allows you to enable/disable historical data loading.

Possible options:

• Yes
• No

Name of the data object on the destination

The type of data object that will be created on the destination depends on the current destination.

We use the File name on the destination for:

• FTP
• SFTP
• Databricks
• Amazon S3
• Azure Blob Storage
• Google Cloud Storage

In all other cases, you will see the Table name on the destination.

By default, a name compiles according to the corresponding data table name. All words are written with lowercase letters and underscores replace all the spaces.

Of course, you can change it to any other name. Please note that this name must be unique!

When you configure all the necessary settings, click the Create button.

Step 3. Correct the name of the Data object on the destination (optional)

Suppose another destination table with the same name already exists. In that case, you will see the following error message “Destination table is already used in another load”.

At this step, you can do the following:

• Click the Finish button to save only data objects with a unique name
• Сhange duplicate names and click the Create button again to check them

Also, you can click the Back button to return to Step 1 and select any other tables. If the table has not been deselected, the settings configured at Step 2 will be saved.

Results

• You see the specific destination page for which you set up the data load.
• There you will find new load orders.
• The name of each load order corresponds to the data object name specified in the previous step.

How to set up data load in a Recipe

{%docs-informer info%}

Now the Automated load is not enabled by default for transformation tables.

{%docs-informer-end%}

Step 1. Add a Load Step

Click the "+" button in the transformation wizard flow and select Load as the step type.

Step 2. Choose a Destination

Pick your target from the Destination dropdown where the transformed data will be loaded.

Step 3. Select Load Mode

Decide between:

• Auto-load after data extraction
• Scheduled load (you’ll need to set the schedule parameters)

Step 4. Save and Enable

Click Save and Load to apply your settings and activate the load step.