Databricks Data source

Connect a Databricks SQL warehouse to read tables, views, or the results of a custom SQL query into Improvado. This guide covers Databricks as a data source. To load data into Databricks instead, see the Databricks destination guide.

Before you begin

• Have a Databricks SQL warehouse in a commercial AWS, Azure, or Google Cloud workspace. This integration uses SQL warehouse connection details; all-purpose clusters, jobs compute, GovCloud, and Azure China are not supported.
• Have a personal access token (PAT) for the same Databricks workspace. The connection form supports PAT authentication; OAuth is not available for this source.
• Grant the token's user or service principal Can use on the SQL warehouse. For Unity Catalog data, also grant USE CATALOG on the catalog, USE SCHEMA on the schema, and SELECT on the tables or views to read. See Databricks documentation on SQL warehouse permissions and Unity Catalog privileges.
• If your workspace restricts network access, allow Improvado to reach the workspace and the cloud storage endpoints used to download query results over HTTPS. Contact your Improvado account team for the network requirements for your region.

Ask your Databricks administrator to prepare access if you cannot manage warehouse or data permissions. Write permissions on the source tables are not required.

Step 1. Get your connection details

1. 1 In your Databricks workspace, open SQL Warehouses and select the warehouse Improvado will use.
2. 2 Open Connection details and copy Server hostname and HTTP Path. See Databricks connection details.
3. 3 In Catalog (Catalog Explorer), find the catalog and schema containing the tables or views you want to extract. Copy their names separately. For example, main.marketing.daily_spend uses catalog main and schema marketing. For the legacy Hive metastore, use hive_metastore as the catalog and the name of your database as the schema.
4. 4 Obtain a PAT for the identity that has access to this warehouse and data. To create a user token, open your Databricks user settings, select Developer, then Access tokens > Manage > Generate new token. Save the token securely and reconnect Improvado when you rotate it. For administrator restrictions and service principal tokens, see Databricks personal access token authentication.

Step 2. Connect Databricks in Improvado

1. 1 Open Connections, choose to create a connection, and select Databricks. You can also open the Databricks authorization page.
2. 2Complete all five fields using the values from your Databricks workspace.

• Server hostname — the warehouse's Server hostname, without https:// or a path.
• HTTP Path — the full warehouse path from Connection details: /sql/1.0/warehouses/ followed by your warehouse ID.
• Catalog name — the catalog to read, for example main or hive_metastore.
• Schema name — the schema inside that catalog, for example marketing. Enter only the schema name.
• Access token — the PAT value only, without quotes, JSON, or a Bearer prefix.

3. 3Click Authorize to validate and save the connection.

Start with a small table or view to verify the connection and field mapping before setting up a larger historical extraction.

Step 3. Select data for extraction

Select a table

1. 1Choose Select a table (recommended).
2. 2Select a table or view from the catalog and schema configured in your connection.
3. 3Click Continue.

Enter a custom SQL query

1. 1Choose Enter SQL query.
2. 2 Enter one read-only SELECT query. A query beginning with WITH is also supported. Use fully qualified table names when referring to another schema or catalog, and give calculated columns unique aliases.
3. 3Validate the query, review the returned fields, and click Continue.

For example, replace the catalog, schema, table, and column names in this query with your own:

SELECT
  report_date,
  campaign_id,
  spend
FROM main.marketing.daily_spend
WHERE campaign_id IS NOT NULL

To filter by date in the extraction settings, include the date column in the query output. For an aggregated query, keep the required date column in the output and grouping. Queries that modify data and multiple SQL statements are not supported.

Step 4. Configure dates, scheduling, and fields

1. 1 On Setup Scheduling, choose the initial historical period and how often Improvado should refresh the data.
2. 2 On Match, review the fields and their types, select the fields to import, and choose the date field if you want extraction by date range. For custom SQL, select a date field returned by your query.
3. 3 Set the data table name and review the write strategy, including which date range a refresh replaces. See Dates and write strategies in Flat Data.
4. 4 Continue to Preview, check the sample and field mapping, and confirm the extraction.

When a date field is selected, the configured date range filters both table extracts and custom SQL results. The end date includes the full day. Native timestamp filtering uses UTC; if you need a local business date, expose that date as a separate DATE column in your table, view, or query and select it as the date field.

Limits and large historical extractions

Each extraction result is limited to 1,000,000 rows and 1 GiB. The byte limit also applies to the CSV representation of the result, so a wide result can reach the limit before one million rows. The limit applies to the result for the requested range, not to the total number of rows in the source table. Results that exceed the limit fail explicitly; Improvado does not load a truncated result as a successful extraction.

Historical ranges are not automatically split into smaller windows. If a result is too large, reduce the historical period or use a view or custom SQL to return fewer columns and rows, then retry. Selecting fewer fields only on the Match page does not reduce the result returned by Databricks.

An interrupted download can resume from a retained query result while that result remains available. If it has expired, the query may run again.

Keep warehouse usage efficient

• Return only the columns and rows you need in your view or custom SQL.
• Use selective date filters and a refresh schedule that matches how often your source data changes. The result-size limit does not limit how much source data a query scans.
• Connection checks, previews, and extractions use your Databricks warehouse and may incur compute charges. Review warehouse usage and Auto Stop settings with your Databricks administrator.

Troubleshooting

Authorization fails

Recheck the hostname and HTTP Path, and confirm that the PAT belongs to the same workspace and has not expired or been revoked. Paste only the token value in Access token. Ask your administrator to verify warehouse and data permissions and any network restrictions.

The table list is empty or a table is missing

Verify the catalog and schema names, including spelling and capitalization. The selector lists only objects in that configured scope that the token's identity can access. Check that the same identity can query the table in Databricks. After permissions or tables change, allow up to 15 minutes for cached metadata to refresh. If the issue persists, contact Improvado Support with the connection name and error message; do not send your token.

A query fails or the result is too large

Run the read-only query on the same warehouse and check its object names, aliases, and permissions. Confirm that the selected date field is returned by the query. For a result-size error, reduce the date range, columns, or rows as described above before retrying.