Databricks Destination

Updated on

Oct 10, 2024

Note: Databricks is currently supported only as a Destination. This guide doesn’t cover the DataPrep setup for Databricks.

Description

‍Databricks is a unified data analytics platform that accelerates innovation by enabling data science, engineering, and business teams to collaborate seamlessly.

Schema information

Setup guide

Follow our setup guide to connect Databricks to Improvado.

Complete configuration

On the Databricks connection page, fill in the following fields:

Enter a name for your Destination connection in the Title.
Enter the Server hostname. {%dropdown-button name="server-hostname"%}

{%dropdown-body name="server-hostname"%}

Learn more here.

Azure Databricks - ```https://adb-ХХХХ.ХХ.azuredatabricks.net```

AWS Databricks - ```https://dbc-ХХХХ.cloud.databricks.com```

Google Cloud Databricks - ```https://XXXX.X.gcs.databricks.com```

{%dropdown-end%}

Enter the Databricks Access Token. Learn more here.
You can enable Write data to Unity Catalog option to write data directly into your Databricks Unity Catalog. Learn more here.
Enter the Filepath. {%dropdown-button name="filepath"%}

{%dropdown-body name="filepath"%}

Possible parameters:

```/FileStore/{{filename}}-{{YYYY}}-{{MM}}-{{DD}}-{{timestamp}}```

```filename``` is the same as the destination table name
```timestamp``` is the date and time when the data load started

Note: you cannot use ```{{DD}}``` for partition by month

```filename-YYYY-MM-DD``` – for partition by day
```filename-YYYY-MM``` – for partition by month

Also, you can use “_” instead of “-” or do not use any symbols at all, for example:

```filenameYYYY-MM-DD-timestamp```
```filenameYYYYMMDDtimestamp```

{%dropdown-end%}

Select the necessary Partition by option from the dropdown. {%dropdown-button name="partition-by"%}

{%dropdown-body name="partition-by"%}

Possible ways of splitting data:

Day
Month

{%dropdown-end%}

Select the necessary File format option from the dropdown. {%dropdown-button name="file-format"%}

{%dropdown-body name="file-format"%}

Possible formats:

csv
csv+gzip
json
json+gzip
parquet
avro

{%dropdown-end%}

Select the necessary Separator option from the dropdown. {%dropdown-button name="separator"%}

{%dropdown-body name="separator"%}

Possible delimiters that can separate data in your file:

comma
semicolon
tab

{%dropdown-end%}

Select the necessary Use static IP option from the dropdown. {%dropdown-button name="use-static-ip"%}

{%dropdown-body name="use-static-ip"%}

Select Yes for Use static IP option if you allow Improvado to connect your database by the static IPs mentioned on the Destination connection page.

Select No if you have permitted access to your database from any IP. In this case, Improvado will connect your database using dynamic IPs not listed on the Destination connection page.

{%dropdown-end%}