Documentation

Note: Databricks is currently supported only as a Destination. This guide doesn’t cover the DataPrep setup for Databricks.

Description

‍Databricks is a unified data analytics platform that accelerates innovation by enabling data science, engineering, and business teams to collaborate seamlessly.

Follow our setup guide to connect Databricks to Improvado.

Complete configuration

On the Databricks connection page, fill in the following fields:

1. Enter a name for your Destination connection in the Title.
2. Enter the Server hostname. {%dropdown-button name="server-hostname"%}

{%dropdown-body name="server-hostname"%}

Learn more here.

Azure Databricks - ```https://adb-ХХХХ.ХХ.azuredatabricks.net```

AWS Databricks - ```https://dbc-ХХХХ.cloud.databricks.com```

Google Cloud Databricks - ```https://XXXX.X.gcs.databricks.com```

{%dropdown-end%}

3. Enter the Databricks Access Token. Learn more here.
4. Enter the Filepath. {%dropdown-button name="filepath"%}

{%dropdown-body name="filepath"%}

Possible parameters:

```/FileStore/{{filename}}-{{YYYY}}-{{MM}}-{{DD}}-{{timestamp}}```

• ```filename``` is the same as the destination table name
• ```timestamp``` is the date and time when the data load started

Note: you cannot use ```{{DD}}``` for partition by month

• ```filename-YYYY-MM-DD``` – for partition by day
• ```filename-YYYY-MM``` – for partition by month

Also, you can use “_” instead of “-” or do not use any symbols at all, for example:

• ```filenameYYYY-MM-DD-timestamp```
• ```filenameYYYYMMDDtimestamp```

{%dropdown-end%}

5. Select the necessary Partition by option from the dropdown. {%dropdown-button name="partition-by"%}

{%dropdown-body name="partition-by"%}

Possible ways of splitting data:

• Day
• Month

{%dropdown-end%}

6. Select the necessary File format option from the dropdown. {%dropdown-button name="file-format"%}

{%dropdown-body name="file-format"%}

Possible formats:

• csv
• csv+gzip
• json
• json+gzip
• parquet
• avro

{%dropdown-end%}

7. Select the necessary Separator option from the dropdown. {%dropdown-button name="separator"%}

{%dropdown-body name="separator"%}

Possible delimiters that can separate data in your file:

• comma
• semicolon
• tab

{%dropdown-end%}

8. You can enable Write data to Unity Catalog option to write data directly into your Databricks Unity Catalog. Learn more here.
9. (For Unity Catalog) Enter the HTTP Path. Learn more here.
10. (For Unity Catalog) Enter the Catalog name.
11. (For Unity Catalog) Enter the Schema name.