Documentation

Improvado can ingest your data from an Amazon S3 bucket - whether is it your storage or ours.

Requirements and preconditions

If your data provider fits the following conditions, we can ingest this data via S3 easily:

1. Data should be in one of the following formats:

• tsv, csv, xls(x) files
• compressed versions (zip, gzip, tar.gz archives)

2. Data set should be presented as a flat table:

• one table per file
• column names in the first row
• no tabulation
• no data hierarchy
• no images

3. Data structure (field names, field order and data types) should NOT be changed without warning

4. Your data provider or your own software should be able to send reports to Amazon S3 buckets automatically and regularly

How to setup Amazon S3 data ingestion

Step 1. Contact our technical support and tell us what you want:

• which data source do you need to connect?
• an example of your data frame is very helpful (for uncommon report types or data formats it is necessary)
• how often should this data be refreshed (typically - on a daily basis)?
• update policy:
  - should we append, update or replace data?
  - in case of an update or replacement, you should specify the key column
• explain S3 folder structure

Step 2. We will confirm whether we can ingest your data or what needs to be changed in order to make it possible.

Step 3. Setup access:

• ‍in case you want to use the Improavado S3 bucket
  - we can grant you access to our S3 bucket in the us-east-1 region, create a bucket for your company and provide access keys.
• in case you want us to load data from your S3 bucket
  - we will need you to provide us with your access keys.

Step 4. Send test data to a pre-determined destination

Step 5. We will confirm whether we have successfully loaded your data

Step 6. Schedule reports from your data platform

That’s all! Your data platform sends us your data at the selected frequency and we do the rest of the job on our side.

Special options

In addition, we can offer the following options:

• one-time historical data download or reference data download if required for custom reports or data transformations
• client data analysis (for unstructured or atypical data formats) to select the best data loading method
  - in this case, examples of data set needed
• data update frequency changes (if necessary)
• report format or data structure update (if data source changes the set of fields available)