Register completed project runs as CKAN datasets on ckan.tacc.utexas.edu

[wmobley]

Feature Request

Add a Publish to CKAN button to the task detail page that registers a completed WebODM run as a CKAN dataset on ckan.tacc.utexas.edu.

Motivation

ODM outputs (orthophotos, point clouds, DEMs, reports) are research artifacts that should be discoverable and citable. Registering each run in CKAN provides persistent metadata, enables cross-project search, and connects field data to the broader DSO data infrastructure.

Proposed Behavior

• A Publish to CKAN button is prominently visible on the task detail page whenever a task status is completed — it should not be hidden, collapsed, or require a menu to find
• Clicking the button collects run metadata (project name, task name, parameters, date, output file list) and creates a CKAN dataset on ckan.tacc.utexas.edu
• Output files (orthophoto, point cloud, DSM/DTM, report) are registered as CKAN resources on the dataset
• The resulting CKAN dataset URL is stored back in the WebODM task record and displayed on the task page

Open Questions

• Auth: how should WebODM authenticate to CKAN (API key stored in settings, or per-user)?
• Output scope: all outputs, or user-selectable before publishing?
• Organization/group mapping in CKAN — derived from project, user-selected, or configured globally?

[wmobley]

Proposed Solution — Design Spec v1.3

Design spec: WebODM/docs/design/2026-07-08-publish-to-ckan.md

---

Architecture

A WebODM coreplugin (coreplugins/ckan/) that adds a "Publish to CKAN" button to the task detail panel. Clicking it opens a chat panel where the user works with the dso-agent-api service to review and confirm metadata before anything is written to CKAN.

Publishing is mediated entirely through the existing DSO agent stack — WebODM never calls CKAN directly.

---

Interaction flow

User clicks "Publish to CKAN"
  → chat panel slides open
  → agent analyzes task outputs, proposes dataset title/description/tags/resources
  → user reviews, types corrections or asks questions
  → user clicks "Confirm & Publish"
  → agent creates the CKAN package + registers all outputs as URL resources
  → chat shows dataset link; task stores ckan_url

The agent's analyze → [human review] → apply pipeline maps directly to the chat. No dry-run step in MVP.

---

Auth

All publishing goes through Tapis JWT — no static CKAN API tokens. WebODM holds a service-account WO_TAPIS_USERNAME / WO_TAPIS_PASSWORD in env vars; a fresh JWT is obtained per publish job via dso-agent-api /v1/auth/login.

Tapis credentials never reach the browser — all agent calls are proxied through Django.

---

Plugin API (4 endpoints, all scoped to the task)

Endpoint	Purpose
POST /api/plugins/ckan/task/{pk}/chat/start	Opens agent run (action: "analyze"), seeds with task metadata + asset URLs
POST /api/plugins/ckan/task/{pk}/chat/message	Proxies user message to runs/{thread_id}/resume
POST /api/plugins/ckan/task/{pk}/chat/confirm	Sends action: "apply", approval: "REGISTER" → queues Celery apply job
GET /api/plugins/ckan/task/{pk}/publish-status	Returns status / ckan_url once done

---

Assets registered

All files in task.available_assets are registered as remote URL resources pointing back to WebODM's download endpoints — no file upload to CKAN. This avoids the 100 MB CKAN upload limit entirely. The agent receives these via the remote_resources field (CkanRegistrationBaseInput).

---

Schema change

One new nullable field on Task:

ckan_url = models.URLField(null=True, blank=True)

---

Config (new env vars)

WO_DSO_AGENT_URL   # e.g. https://dso-agent-api.pods.portals.tapis.io
WO_TAPIS_USERNAME  # service account
WO_TAPIS_PASSWORD  # service account
WO_URL             # externally reachable base URL for resource links

Plugin is hidden unless WO_DSO_AGENT_URL is set.

---

Dependency

dso-agent-api pod must be live (GHCR image visibility → Public + GitHub Secrets set in In-For-Disaster-Analytics/agents).

[wmobley]

Status Update — Implementation Substantially Complete

The core feature described in this issue is implemented. Here's the current state:

Done

• All 4 API endpoints implemented and mounted (chat/start, chat/message, chat/confirm, publish-status)
• React chat panel (CKANPublishPanel.jsx) — opens on "Publish to CKAN" button click, renders agent messages with markdown, polls publish status, supports confirm/retry flow, and shows a "View on CKAN →" link on success
• Celery publish task (publisher.py) with rich metadata builders: 3-tier title fallback, structured abstract from stats.json/cameras.json, tag inference (multispectral/RGB, product type, georeferencing method), spatial bbox, and temporal coverage
• Task.ckan_url nullable URLField + migration (0047_task_ckan_url.py)
• Auth: uses the logged-in user's TapisOAuth2Token — no static CKAN API keys or service-account credentials
• Task auto-set to public=True before publishing so CKAN resource URLs are externally accessible from the agent pod
• Re-publish support: owner_org is persisted in GlobalDataStore and re-seeded on subsequent publish sessions, skipping the org-selection interrupt
• Plugin is hidden unless WO_DSO_AGENT_URL is set in env

Still Missing

• Tests — no tests/plugins/ckan/ directory exists; unit and integration tests for views, publisher helpers, and the Celery task are unwritten
• Plugin README — env var reference, token scope requirements, and network prerequisites are undocumented
• Minor: publish-status response does not include a timestamp field (spec called for one)
• Minor: ckan_url is not in TaskSerializer.Meta.read_only_fields, so it could be PATCHed directly via the task API

Notable Implementation Details (not in original spec)

• has_pending_interrupt routing: after the agent's propose step reaches END, follow-up messages call POST /runs with session_id rather than POST /resume — calling /resume on a completed graph is a no-op
• Web Map Viewer and 3D Model Viewer links are registered as HTML resources in CKAN alongside the downloadable assets
• TAS email lookup (_tas_email) includes Redis caching (1-hour TTL on hits, 5-minute TTL on misses) and a direct HTTP fallback if pytas is unavailable