diff --git a/site/guide/integrations/configure-connections.qmd b/site/guide/integrations/configure-connections.qmd index f6f2966a35..b3a2d5861e 100644 --- a/site/guide/integrations/configure-connections.qmd +++ b/site/guide/integrations/configure-connections.qmd @@ -116,14 +116,20 @@ A unified analytics platform for data engineering, data science, and machine lea Required configuration details: -**Databricks Host** +**Authentication Method** +: **OAuth (user login)** (recommended) or **Personal Access Token**. OAuth redirects you to Databricks to authorize the connection; PAT uses an organization secret. + +**Workspace URL** : The URL of your Databricks workspace, such as `https://yourcompany.cloud.databricks.com`. -**Client ID** -: The ID of your SQL Warehouse (found in Databricks SQL Warehouses settings). +**SQL Warehouse ID** (optional) +: The ID of your SQL Warehouse (found in Databricks SQL Warehouses settings). Required only when querying table row data via SQL. + +**OAuth Client ID** / **OAuth Client Secret** (optional, OAuth only) +: Leave blank to use the default ValidMind partner OAuth app, or supply credentials from your own Databricks OAuth application. For confidential apps, store the client secret as an organization secret. -**Client Secret** -: The secret that stores your Databricks personal access token (PAT). +**Personal Access Token** (PAT only) +: Select a secret containing your Databricks Personal Access Token. ### Custom diff --git a/site/guide/integrations/integrations-examples/configure-databricks.png b/site/guide/integrations/integrations-examples/configure-databricks.png index d524781af9..d5fc251914 100644 Binary files a/site/guide/integrations/integrations-examples/configure-databricks.png and b/site/guide/integrations/integrations-examples/configure-databricks.png differ diff --git a/site/guide/integrations/integrations-examples/synchronize-with-databricks.qmd b/site/guide/integrations/integrations-examples/synchronize-with-databricks.qmd index 61e93d5b33..5806f3103c 100644 --- a/site/guide/integrations/integrations-examples/synchronize-with-databricks.qmd +++ b/site/guide/integrations/integrations-examples/synchronize-with-databricks.qmd @@ -62,14 +62,14 @@ flowchart LR - [x] {{< var link.login >}} - [x] You are a [{{< fa hand >}} Customer Admin]{.bubble} or assigned another role with sufficient permissions to configure connections.[^1] -- [x] A secret is configured for your Databricks Personal Access Token.[^2] -- [x] You have admin access to your Databricks workspace. +- [x] You have access to your Databricks workspace and can authorize the connection (OAuth) or create a personal access token (PAT). +- [x] For **Personal Access Token** authentication only: a secret is configured for your Databricks PAT.[^2] ::: -## Sync {{< var vm.product >}} records with Databricks +## Sync {{< var vm.product >}} records with Databricks {#sync-records-with-databricks} -::: {.panel-tabset} +:::: {.panel-tabset} ### 1. Configure Databricks connection @@ -81,16 +81,33 @@ c. Click **{{< fa plus >}} Add Connection**. d. In the modal that opens, select **Databricks**.[^3] -e. Complete: +e. Complete the shared fields: - **Integration Name** — How other admins can identify the connection. - **Description** (optional) — The intended usage or additional details. + - **Authentication Method** — Choose **OAuth (user login)** (recommended) or **Personal Access Token**. - **Workspace URL** — Your Databricks workspace URL, found in the browser address bar, such as `https://adb-1234567890.azuredatabricks.net`. - - **SQL Warehouse ID** — The ID of your SQL Warehouse, found in SQL Warehouses settings. - - **Personal Access Token** — Select a secret containing your Databricks Personal Access Token. + - **SQL Warehouse ID** (optional) — The ID of your SQL Warehouse, found in SQL Warehouses settings. Required only when querying table row data via SQL. + +f. Complete the fields for your authentication method: + + ::: {.panel-tabset} + + ### OAuth (user login) + + - **OAuth Client ID** (optional) — Leave blank to use the default ValidMind partner OAuth app, or enter a client ID from your own Databricks OAuth application. + - **OAuth Client Secret** (optional) — Select a secret only for confidential OAuth apps. Leave blank for the ValidMind partner app or public PKCE apps. + + Click **Connect with Databricks**. You are redirected to Databricks to sign in and authorize access. After you authorize, you return to Connections with the new connection created. Access and refresh tokens are stored securely and are never shown in the form. + + ### Personal Access Token + + - **Personal Access Token** — Select a secret containing your Databricks Personal Access Token.[^2] - **Initial Status** — Set to `Operational` to enable immediately or `Disabled` if you plan to finish setup later. -f. Click **Save Integration**. + Click **Save Integration**. + + ::: g. Test the connection: @@ -99,6 +116,12 @@ g. Test the connection: If the test is successful, the message **{{< fa check-circle >}} Connection successful** is displayed. +::: {.callout-note} +## Reconnect OAuth connections + +To refresh OAuth tokens later, edit the Databricks connection and click **Reconnect with Databricks**. Complete the Databricks sign-in flow again. Switching an existing connection from Personal Access Token to OAuth also requires reconnecting. +::: + ### 2. Link records to Databricks Once the connection is configured, you can link {{< var vm.product >}} inventory records to your Databricks Unity Catalog resources: @@ -122,7 +145,7 @@ g. Click **Link Model** to complete the link. After linking, metadata from the Unity Catalog resource syncs to {{< var vm.product >}}. You can use linked fields in custom calculated fields to surface Databricks metadata directly in your inventory views. -::: +:::: ## What's next @@ -138,6 +161,6 @@ After linking, metadata from the Unity Catalog resource syncs to {{< var vm.prod [^3]: - ![Configure the Databricks connection](configure-databricks.png){width=80% fig-alt="Screenshot of the Configure Databricks connection dialog showing the required fields described in step 5." .screenshot} + ![Configure the Databricks connection](configure-databricks.png){width=80% fig-alt="Screenshot of the Configure Databricks connection dialog with Authentication Method set to OAuth (user login), showing Workspace URL, optional SQL Warehouse ID, optional OAuth Client ID and Client Secret, and the Connect with Databricks button." .screenshot} [^4]: [Working with the inventory](/guide/inventory/working-with-the-inventory.qmd#search-filter-and-sort-records) diff --git a/site/notebooks/databricks/validmind_databricks_quickstart.ipynb b/site/notebooks/databricks/validmind_databricks_quickstart.ipynb index c54c52ae8a..42bb04f099 100644 --- a/site/notebooks/databricks/validmind_databricks_quickstart.ipynb +++ b/site/notebooks/databricks/validmind_databricks_quickstart.ipynb @@ -21,10 +21,11 @@ "1. A running Databricks workspace with Unity Catalog enabled\n", "2. A ValidMind account with a registered model\n", "3. Your ValidMind API credentials (API key, API secret, model identifier)\n", + "4. A Databricks connection in ValidMind (OAuth user login recommended, or a personal access token)\n", "\n", "To get your credentials: log in to ValidMind → **Model Inventory** → select your model → **Getting Started** → **Copy snippet to clipboard**.\n", "\n", - "For step-by-step instructions on setting up the Databricks integration and linking a Unity Catalog table to your model, refer to [Synchronize with Databricks](https://docs.validmind.ai/guide/integrations/integrations-examples/synchronize-with-databricks.html).\n", + "For step-by-step instructions on connecting with **OAuth (user login)** or a personal access token, and linking a Unity Catalog table to your model, refer to [Synchronize with Databricks](https://docs.validmind.ai/guide/integrations/integrations-examples/synchronize-with-databricks.html).\n", "\n", "> **Note:** If you don't have a Unity Catalog table linked to your model yet, this notebook includes a synthetic-data fallback so you can still run through the full workflow." ] @@ -140,7 +141,7 @@ "\n", "Before running the next cell, make sure you have:\n", "\n", - "1. A Databricks integration configured in **Settings → Integrations → Databricks**\n", + "1. A Databricks connection configured in **Settings → Integrations → Connections** (OAuth user login recommended, or a personal access token). Include a **SQL Warehouse ID** if you need table row data via SQL.\n", "2. A `table` binding created for your model that links a Unity Catalog table to it\n", "3. At least one successful sync (the initial sync runs automatically when you create the binding)\n", "\n", @@ -205,10 +206,11 @@ " raise RuntimeError(\n", " \"No active Databricks table binding found for this model.\\n\\n\"\n", " \"To fix:\\n\"\n", - " \" 1. Go to ValidMind → Settings → Integrations → Databricks\\n\"\n", - " \" 2. Open the model binding browser and select a Unity Catalog table\\n\"\n", - " \" 3. Wait ~30 seconds for the initial sync to complete\\n\"\n", - " \" 4. Re-run this cell\\n\\n\"\n", + " \" 1. Go to ValidMind → Settings → Integrations → Connections\\n\"\n", + " \" 2. Connect Databricks with OAuth (user login) or a personal access token\\n\"\n", + " \" 3. Open the model binding browser and select a Unity Catalog table\\n\"\n", + " \" 4. Wait ~30 seconds for the initial sync to complete\\n\"\n", + " \" 5. Re-run this cell\\n\\n\"\n", " \"Or set USE_SYNTHETIC_FALLBACK = True above to continue with generated data.\"\n", " )\n", " else:\n", @@ -506,8 +508,8 @@ "| `ConnectionError` on `vm.init()` | Your workspace may block outbound traffic. Check your network policy, or use a cluster with internet access. |\n", "| `401 Unauthorized` on `vm.init()` | The API key or secret is incorrect. Copy your credentials again from the ValidMind Platform. |\n", "| `numpy` version conflict | Pin a compatible version with `%pip install -q validmind \"numpy>=1.23,<2.0.0\"`. |\n", - "| `404` on dataset load | No Databricks table binding was found. Create one in **Settings → Integrations → Databricks**, then wait for the initial sync to complete. |\n", - "| `row_data is empty` after binding created | The initial sync is still running. Wait about 30 seconds and re-run Step 4. |\n", + "| `404` on dataset load | No Databricks table binding was found. Create a Databricks connection in **Settings → Integrations → Connections** (OAuth recommended), link a Unity Catalog table, then wait for the initial sync to complete. |\n", + "| `row_data is empty` after binding created | The initial sync is still running. Wait about 30 seconds and re-run Step 4. Confirm the connection includes a **SQL Warehouse ID** if you need row data via SQL. |\n", "| Wrong columns or target not found | Update `TARGET_COLUMN` in Step 4 to match the target column in your Unity Catalog table. |\n", "| Want to try the notebook without a binding | Set `USE_SYNTHETIC_FALLBACK = True` in Step 4 to use generated data. |" ]