Note: Databricks is currently supported only as a Destination. This guide doesn’t cover the DataPrep setup for Databricks.
Description
Databricks is a unified data analytics platform that accelerates innovation by enabling data science, engineering, and business teams to collaborate seamlessly.
Setup guide
Follow our setup guide to connect Databricks to Improvado.
filename is the same as the destination table name
timestamp is the date and time when the data load started
Note: you cannot use {{DD}} for partition by month
filename-YYYY-MM-DD – for partition by day
filename-YYYY-MM – for partition by month
Important
{{YYYY}}/{{MM}}/{{DD}} partitioning is based on a date field in the exported data. If the dataset does not contain an actual date field (for example, no “date” column/field), these partition values cannot be derived and date-based partitioning will not work as expected. In that case, use {{YYYY_today}}/{{MM_today}}/{{DD_today}} to partition by the export run date instead.
Note
{{YYYY}}/{{MM}}/{{DD}} refer to the record’s date-based partition values (if applicable), while {{YYYY_today}}/{{MM_today}}/{{DD_today}} always use today’s date.
Also, you can use “_” instead of “-” or do not use any symbols at all, for example:
filenameYYYY-MM-DD-timestamp
filenameYYYYMMDDtimestamp
Select the necessary Partition by option from the dropdown.
Possible ways of splitting data:
Day
Month
Select the necessary File format option from the dropdown.
Possible formats:
csv
csv+gzip
json
json+gzip
parquet
avro
Select the necessary Separator option from the dropdown.
Possible delimiters that can separate data in your file:
comma
semicolon
tab
Select the necessary Use static IP option from the dropdown.
Select Yes for Use static IP option if you allow Improvado to connect your database by the static IPs mentioned on the Destination connection page.
Select No if you have permitted access to your database from any IP. In this case, Improvado will connect your database using dynamic IPs not listed on the Destination connection page.
Select the necessary Use binary strings option from the dropdown.
The "Use binary strings" determines whether string fields are encoded in binary format when loading data.
Recommended setting: "No" (default).
(For Unity Catalog) Enter the HTTP Path. Learn more here.
(For Unity Catalog) Enter the Catalog name.
(For Unity Catalog) Enter the Schema name.
Was this article helpful?
Thanks for the feedback!
Your privacy choices
We use necessary cookies to run this site and, with your consent, cookies to measure traffic and support our marketing. Read our Privacy policy and Cookie policy.
Privacy preferences
Choose which cookies this site may use. Necessary cookies are always on. See the Cookie policy for the full list.
Necessary
Required for core functionality and security. Always active.
Analytics
Helps us understand how the site is used so we can improve it.
Marketing
Lets us measure campaigns and show relevant messaging across channels.
