Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 11 additions & 5 deletions site/guide/integrations/configure-connections.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -116,14 +116,20 @@ A unified analytics platform for data engineering, data science, and machine lea

Required configuration details:

**Databricks Host**
**Authentication Method**
: **OAuth (user login)** (recommended) or **Personal Access Token**. OAuth redirects you to Databricks to authorize the connection; PAT uses an organization secret.

**Workspace URL**
: The URL of your Databricks workspace, such as `https://yourcompany.cloud.databricks.com`.

**Client ID**
: The ID of your SQL Warehouse (found in Databricks SQL Warehouses settings).
**SQL Warehouse ID** (optional)
: The ID of your SQL Warehouse (found in Databricks SQL Warehouses settings). Required only when querying table row data via SQL.

**OAuth Client ID** / **OAuth Client Secret** (optional, OAuth only)
: Leave blank to use the default ValidMind partner OAuth app, or supply credentials from your own Databricks OAuth application. For confidential apps, store the client secret as an organization secret.

**Client Secret**
: The secret that stores your Databricks personal access token (PAT).
**Personal Access Token** (PAT only)
: Select a secret containing your Databricks Personal Access Token.

### Custom

Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
Expand Up @@ -62,14 +62,14 @@ flowchart LR

- [x] {{< var link.login >}}
- [x] You are a [{{< fa hand >}} Customer Admin]{.bubble} or assigned another role with sufficient permissions to configure connections.[^1]
- [x] A secret is configured for your Databricks Personal Access Token.[^2]
- [x] You have admin access to your Databricks workspace.
- [x] You have access to your Databricks workspace and can authorize the connection (OAuth) or create a personal access token (PAT).
- [x] For **Personal Access Token** authentication only: a secret is configured for your Databricks PAT.[^2]

:::

## Sync {{< var vm.product >}} records with Databricks
## Sync {{< var vm.product >}} records with Databricks {#sync-records-with-databricks}

::: {.panel-tabset}
:::: {.panel-tabset}

### 1. Configure Databricks connection

Expand All @@ -81,16 +81,33 @@ c. Click **{{< fa plus >}} Add Connection**.

d. In the modal that opens, select **Databricks**.[^3]

e. Complete:
e. Complete the shared fields:

- **Integration Name** — How other admins can identify the connection.
- **Description** (optional) — The intended usage or additional details.
- **Authentication Method** — Choose **OAuth (user login)** (recommended) or **Personal Access Token**.
- **Workspace URL** — Your Databricks workspace URL, found in the browser address bar, such as `https://adb-1234567890.azuredatabricks.net`.
- **SQL Warehouse ID** — The ID of your SQL Warehouse, found in SQL Warehouses settings.
- **Personal Access Token** — Select a secret containing your Databricks Personal Access Token.
- **SQL Warehouse ID** (optional) — The ID of your SQL Warehouse, found in SQL Warehouses settings. Required only when querying table row data via SQL.

f. Complete the fields for your authentication method:

::: {.panel-tabset}

### OAuth (user login)

- **OAuth Client ID** (optional) — Leave blank to use the default ValidMind partner OAuth app, or enter a client ID from your own Databricks OAuth application.
- **OAuth Client Secret** (optional) — Select a secret only for confidential OAuth apps. Leave blank for the ValidMind partner app or public PKCE apps.

Click **Connect with Databricks**. You are redirected to Databricks to sign in and authorize access. After you authorize, you return to Connections with the new connection created. Access and refresh tokens are stored securely and are never shown in the form.

### Personal Access Token

- **Personal Access Token** — Select a secret containing your Databricks Personal Access Token.[^2]
- **Initial Status** — Set to `Operational` to enable immediately or `Disabled` if you plan to finish setup later.

f. Click **Save Integration**.
Click **Save Integration**.

:::

g. Test the connection:

Expand All @@ -99,6 +116,12 @@ g. Test the connection:

If the test is successful, the message **{{< fa check-circle >}} Connection successful** is displayed.

::: {.callout-note}
## Reconnect OAuth connections

To refresh OAuth tokens later, edit the Databricks connection and click **Reconnect with Databricks**. Complete the Databricks sign-in flow again. Switching an existing connection from Personal Access Token to OAuth also requires reconnecting.
:::

### 2. Link records to Databricks

Once the connection is configured, you can link {{< var vm.product >}} inventory records to your Databricks Unity Catalog resources:
Expand All @@ -122,7 +145,7 @@ g. Click **Link Model** to complete the link.

After linking, metadata from the Unity Catalog resource syncs to {{< var vm.product >}}. You can use linked fields in custom calculated fields to surface Databricks metadata directly in your inventory views.

:::
::::

## What's next

Expand All @@ -138,6 +161,6 @@ After linking, metadata from the Unity Catalog resource syncs to {{< var vm.prod

[^3]:

![Configure the Databricks connection](configure-databricks.png){width=80% fig-alt="Screenshot of the Configure Databricks connection dialog showing the required fields described in step 5." .screenshot}
![Configure the Databricks connection](configure-databricks.png){width=80% fig-alt="Screenshot of the Configure Databricks connection dialog with Authentication Method set to OAuth (user login), showing Workspace URL, optional SQL Warehouse ID, optional OAuth Client ID and Client Secret, and the Connect with Databricks button." .screenshot}

[^4]: [Working with the inventory](/guide/inventory/working-with-the-inventory.qmd#search-filter-and-sort-records)
18 changes: 10 additions & 8 deletions site/notebooks/databricks/validmind_databricks_quickstart.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -21,10 +21,11 @@
"1. A running Databricks workspace with Unity Catalog enabled\n",
"2. A ValidMind account with a registered model\n",
"3. Your ValidMind API credentials (API key, API secret, model identifier)\n",
"4. A Databricks connection in ValidMind (OAuth user login recommended, or a personal access token)\n",
"\n",
"To get your credentials: log in to ValidMind → **Model Inventory** → select your model → **Getting Started** → **Copy snippet to clipboard**.\n",
"\n",
"For step-by-step instructions on setting up the Databricks integration and linking a Unity Catalog table to your model, refer to [Synchronize with Databricks](https://docs.validmind.ai/guide/integrations/integrations-examples/synchronize-with-databricks.html).\n",
"For step-by-step instructions on connecting with **OAuth (user login)** or a personal access token, and linking a Unity Catalog table to your model, refer to [Synchronize with Databricks](https://docs.validmind.ai/guide/integrations/integrations-examples/synchronize-with-databricks.html).\n",
"\n",
"> **Note:** If you don't have a Unity Catalog table linked to your model yet, this notebook includes a synthetic-data fallback so you can still run through the full workflow."
]
Expand Down Expand Up @@ -140,7 +141,7 @@
"\n",
"Before running the next cell, make sure you have:\n",
"\n",
"1. A Databricks integration configured in **Settings → Integrations → Databricks**\n",
"1. A Databricks connection configured in **Settings → Integrations → Connections** (OAuth user login recommended, or a personal access token). Include a **SQL Warehouse ID** if you need table row data via SQL.\n",
"2. A `table` binding created for your model that links a Unity Catalog table to it\n",
"3. At least one successful sync (the initial sync runs automatically when you create the binding)\n",
"\n",
Expand Down Expand Up @@ -205,10 +206,11 @@
" raise RuntimeError(\n",
" \"No active Databricks table binding found for this model.\\n\\n\"\n",
" \"To fix:\\n\"\n",
" \" 1. Go to ValidMind → Settings → Integrations → Databricks\\n\"\n",
" \" 2. Open the model binding browser and select a Unity Catalog table\\n\"\n",
" \" 3. Wait ~30 seconds for the initial sync to complete\\n\"\n",
" \" 4. Re-run this cell\\n\\n\"\n",
" \" 1. Go to ValidMind → Settings → Integrations → Connections\\n\"\n",
" \" 2. Connect Databricks with OAuth (user login) or a personal access token\\n\"\n",
" \" 3. Open the model binding browser and select a Unity Catalog table\\n\"\n",
" \" 4. Wait ~30 seconds for the initial sync to complete\\n\"\n",
" \" 5. Re-run this cell\\n\\n\"\n",
" \"Or set USE_SYNTHETIC_FALLBACK = True above to continue with generated data.\"\n",
" )\n",
" else:\n",
Expand Down Expand Up @@ -506,8 +508,8 @@
"| `ConnectionError` on `vm.init()` | Your workspace may block outbound traffic. Check your network policy, or use a cluster with internet access. |\n",
"| `401 Unauthorized` on `vm.init()` | The API key or secret is incorrect. Copy your credentials again from the ValidMind Platform. |\n",
"| `numpy` version conflict | Pin a compatible version with `%pip install -q validmind \"numpy>=1.23,<2.0.0\"`. |\n",
"| `404` on dataset load | No Databricks table binding was found. Create one in **Settings → Integrations → Databricks**, then wait for the initial sync to complete. |\n",
"| `row_data is empty` after binding created | The initial sync is still running. Wait about 30 seconds and re-run Step 4. |\n",
"| `404` on dataset load | No Databricks table binding was found. Create a Databricks connection in **Settings → Integrations → Connections** (OAuth recommended), link a Unity Catalog table, then wait for the initial sync to complete. |\n",
"| `row_data is empty` after binding created | The initial sync is still running. Wait about 30 seconds and re-run Step 4. Confirm the connection includes a **SQL Warehouse ID** if you need row data via SQL. |\n",
"| Wrong columns or target not found | Update `TARGET_COLUMN` in Step 4 to match the target column in your Unity Catalog table. |\n",
"| Want to try the notebook without a binding | Set `USE_SYNTHETIC_FALLBACK = True` in Step 4 to use generated data. |"
]
Expand Down
Loading