Documentation

Databricks Destination

Updated on

Sep 5, 2024

Note: Databricks is currently supported only as a Destination. This guide doesn’t cover the DataPrep setup for Databricks.

Description

Databricks is a unified data analytics platform that accelerates innovation by enabling data science, engineering, and business teams to collaborate seamlessly.

Schema information

Setup guide

Follow our setup guide to connect Databricks to Improvado.

Complete configuration

On the Databricks connection page, fill in the following fields:

  1. Enter a name for your Destination connection in the Title.
  2. Enter the Server hostname. {%dropdown-button name="server-hostname"%}

{%dropdown-body name="server-hostname"%}

Learn more here.

Azure Databricks - ```https://adb-ХХХХ.ХХ.azuredatabricks.net```

AWS Databricks - ```https://dbc-ХХХХ.cloud.databricks.com```

Google Cloud Databricks - ```https://XXXX.X.gcs.databricks.com```

{%dropdown-end%}

  1. Enter the Databricks Access Token. Learn more here.
  2. Enter the Filepath. {%dropdown-button name="filepath"%}

{%dropdown-body name="filepath"%}

Possible parameters:

```/FileStore/{{filename}}-{{YYYY}}-{{MM}}-{{DD}}-{{timestamp}}```

  • ```filename``` is the same as the destination table name
  • ```timestamp``` is the date and time when the data load started

Note: you cannot use ```{{DD}}``` for partition by month

  • ```filename-YYYY-MM-DD``` – for partition by day
  • ```filename-YYYY-MM``` – for partition by month

Also, you can use “_” instead of “-” or do not use any symbols at all, for example:

  • ```filenameYYYY-MM-DD-timestamp```
  • ```filenameYYYYMMDDtimestamp```

{%dropdown-end%}

  1. Select the necessary Partition by option from the dropdown. {%dropdown-button name="partition-by"%}

{%dropdown-body name="partition-by"%}

Possible ways of splitting data:

  • Day
  • Month

{%dropdown-end%}

  1. Select the necessary File format option from the dropdown. {%dropdown-button name="file-format"%}

{%dropdown-body name="file-format"%}

Possible formats:

  • csv
  • csv+gzip
  • json
  • json+gzip
  • parquet
  • avro

{%dropdown-end%}

  1. Select the necessary Separator option from the dropdown. {%dropdown-button name="separator"%}

{%dropdown-body name="separator"%}

Possible delimiters that can separate data in your file:

  • comma
  • semicolon
  • tab

{%dropdown-end%}

  1. You can enable Write data to Unity Catalog option to write data directly into your Databricks Unity Catalog. Learn more here.
  2. (For Unity Catalog) Enter the HTTP Path. Learn more here.
  3. (For Unity Catalog) Enter the Catalog name.
  4. (For Unity Catalog) Enter the Schema name.

Settings

No items found.

Troubleshooting

Troubleshooting guides

Check out troubleshooting guides for
Databricks Destination
here:

Limits

Frequently asked questions

No items found.
☶ On this page
Description
Related articles
No items found.
No items found.

Questions?

Improvado team is always happy to help with any other questions you might have! Send us an email.

Contact your Customer Success Manager or raise a request in Improvado Service Desk.