IMPORTANT: This article covers setup of warehouse for load data from Improvado, not customer data warehouse from which data is being extracted. This article doesn't cover setup of customer data warehouse for Data Prep as well.
Required information
- Title
- Host
- Port
- User Name
- Password
- Folder
- ~a forward slash ( / ) means that you need to enter the root path
- File format
- File name
- Separator (optional)
- ~the maximum length of the separator is 2 characters
- Partition by (the way of splitting data for uploading to the file)
- Encryption
- Use static IP
Folder
Possible parameters:
- /{{ data_source }}/{{ data_table_title }}/{{report_type}}/{{ YYYY }}/{{ MM }}/{{ DD }}
- ~{ data_source } is a data provider, integration, connector
- ~{ data_table } is an object that contains all extraction orders with the same granularity (dimensional schema)
- ~{report_type} is a set of such fields as metrics, properties, dimensions, etc.
If you use /{YYYY}/{MM}/{DD} settings, the data will be added to folders daily. Each new record will not delete the previous one, even for data that contains no date.
File format
Possible formats:
- csv
- csv+gzip
- json
- json+gzip
- parquet
File name
Possible parameters:
- {{filename}}-{{YYYY}}-{{MM}}-{{DD}}
- ~{ filename } is the same as destination table name
IMPORTANT: you cannot use {{ DD }} for partition by month
- ~{{filename}}-{{YYYY}}-{{MM}}-{{DD}} – for partition by day
- ~{{filename}}-{{YYYY}}-{{MM}} – for partition by month
Also, you can use “_” instead of “-” or do not use any symbols at all, for example:
- {{filename}}_{{YYYY}}-{{MM}}-{{DD}}
- {{filename}}{{YYYY}}{{MM}}{{DD}}
Partition by
Possible ways of splitting data:
Encryption
Possible parameters:
- Require explicit FTP over TLS (FTPS)
- Explicit FTP over TLS if available (FTPS)
- Plain FTP (insecure)
Use static IP
Select Yes for Use static IP option if you allow Improvado to connect your database by the following static IPs only:
- 34.226.37.150
- 18.213.72.135
- 54.146.15.122
- 3.86.170.178
- 23.21.191.65
Select No if you have permitted access to your database from any IP. In this case, Improvado will connect your database using dynamic IPs not listed above.