Note: Databricks is currently supported only as a Destination. This guide doesn’t cover the DataPrep setup for Databricks.
Description
Databricks is a unified data analytics platform that accelerates innovation by enabling data science, engineering, and business teams to collaborate seamlessly.
Setup guide
Follow our setup guide to connect Databricks to Improvado.
filename is the same as the destination table name
timestamp is the date and time when the data load started
Note: you cannot use {{DD}} for partition by month
filename-YYYY-MM-DD – for partition by day
filename-YYYY-MM – for partition by month
Important
{{YYYY}}/{{MM}}/{{DD}} partitioning is based on a date field in the exported data. If the dataset does not contain an actual date field (for example, no “date” column/field), these partition values cannot be derived and date-based partitioning will not work as expected. In that case, use {{YYYY_today}}/{{MM_today}}/{{DD_today}} to partition by the export run date instead.
Note
{{YYYY}}/{{MM}}/{{DD}} refer to the record’s date-based partition values (if applicable), while {{YYYY_today}}/{{MM_today}}/{{DD_today}} always use today’s date.
Also, you can use “_” instead of “-” or do not use any symbols at all, for example:
filenameYYYY-MM-DD-timestamp
filenameYYYYMMDDtimestamp
Select the necessary Partition by option from the dropdown.
Possible ways of splitting data:
Day
Month
Select the necessary File format option from the dropdown.
Possible formats:
csv
csv+gzip
json
json+gzip
parquet
avro
Select the necessary Separator option from the dropdown.
Possible delimiters that can separate data in your file:
comma
semicolon
tab
Select the necessary Use static IP option from the dropdown.
Select Yes for Use static IP option if you allow Improvado to connect your database by the static IPs mentioned on the Destination connection page.
Select No if you have permitted access to your database from any IP. In this case, Improvado will connect your database using dynamic IPs not listed on the Destination connection page.
Select the necessary Use binary strings option from the dropdown.
The "Use binary strings" determines whether string fields are encoded in binary format when loading data.
Recommended setting: "No" (default).
(For Unity Catalog) Enter the HTTP Path. Learn more here.
