Note: Azure Data Lake is currently supported only as a Destination. This guide doesn’t cover the DataPrep setup for Azure Data Lake.
Description
Azure Data Lake is a scalable cloud storage solution that enables the efficient storage, processing, and analysis of large amounts of structured and unstructured data. It supports big data analytics by integrating with Azure's analytics services, offering seamless data ingestion, high-throughput processing, and advanced security features.
Setup guide
Follow our setup guide to connect Azure Data Lake to Improvado.
File System Name length must be between 3 and 63 characters and must satisfy the following regular expression: r'^(?!.*--.*)[a-z0-9][a-z0-9]*[a-z0-9]$’
Select the necessary Encryption type option from the dropdown.
Possible options:
No encryption (default cloud storage encryption is still enabled)
Customer-provided key
(Customer-provided keys only) Enter the Encryption key.
If you have selected the Default Cloud Storage encryption type, you will not be able to edit this field.
Otherwise, you should enter your AES-256 key, encoded in standard Base64 or resource name of Cloud KMS key used to encrypt the blob’s contents. For more info, see Azure Data Lake encryption docs.
data_source is a data provider, integration, connector
data_table_title is an object that contains all extraction orders with the same granularity (dimensional schema)
report_type is a set of such fields as metrics, properties, dimensions, etc.
timestamp is the date and time when data load started
If you use /YYYY/MM/DD settings, the data will be added to folders daily. Each new record will not delete the previous one, even for data that contains no date.
The maximum length is 254 characters.
Select the necessary File format option from the dropdown.
Possible formats:
csv
csv+gzip
json
json+gzip
parquet
avro
Enter the Filename.
Possible parameters:
filename-YYYY-MM-DD
filename is the same as destination table name
Note: you cannot use DD for partition by month.
filename-YYYY-MM-DD – for partition by day
filename-YYYY-MM – for partition by month
Also, you can use “_” instead of “-” or do not use any symbols at all, for example:
filenameYYYY-MM-DD
filenameYYYYMMDD
Select the necessary Separator option from the dropdown.
Possible delimiters that can separate data in your file:
comma
semicolon
tab
Select the necessary Partition by option from the dropdown.
Possible ways of splitting data:
Day
Month
Select the necessary Use binary strings option from the dropdown.
The "Use binary strings" determines whether string fields are encoded in binary format when loading data.
