[FEATURE] Add OpenSearch datasource and log query plugin with PPL support

[Oliver-ke]

Description

Adds an OpenSearch datasource + log query plugin

These changes includes:

• Plugin scaffold — OpenSearchDatasource + OpenSearchLogQuery plugins (TypeScript, React, Module Federation via rsbuild).
• PPL client (opensearch-client.ts) - POSTs to /_plugins/_ppl, handles relative datasource URLs, throws a typed OpenSearchPPLError(status, body) on non-200.
• Time bounding + row mapping (get-opensearch-log-data.ts) - wraps user PPL with a where clause on the configured timestamp field; maps rows to LogEntry and surfaces
  remaining columns (e.g. traceId) as labels.
• Field-name overrides - timestampField / messageField on the query spec for indexes that don't use @timestamp / message (e.g. OpenTelemetry's body).
• Editor (OpenSearchLogQueryEditor.tsx) - datasource picker, index pattern, timestamp/message field overrides, and a PPL textarea.
• Trace-to-logs pivot - relies on the existing replaceVariables path, so a $traceId dashboard variable substitutes into the PPL before send. Example dashboard at
  opensearch/docs/examples/trace-to-logs.json (Tempo + OpenSearch; Jaeger swap documented).
• CUE schema (schemas/queries/opensearch-log-query/query.cue) - mirrors the TS spec including the new optional fields.
• Go SDK (sdk/go/query/log) - PluginSpec + TimestampField() / MessageField() options for dashboards-as-code.
• Tests - 26 TS tests (PPL bounder, row mapper with overrides + fallbacks, $traceId substitution + dependsOn, PPL client URL/headers/error, editor inputs)
  and 2 Go tests (builder JSON shape + omitempty for optional fields).

Screenshots

DataSource

Checklist

• Pull request has a descriptive title and context useful to a reviewer.
• Pull request title follows the [<catalog_entry>] <commit message> naming convention using one of the
  following catalog_entry values: FEATURE, ENHANCEMENT, BUGFIX, BREAKINGCHANGE, DOC,IGNORE.
• All commits have DCO signoffs.

UI Changes

• Changes that impact the UI include screenshots and/or screencasts of the relevant changes.
• Code follows the UI guidelines.

[ibakshay]

Hi @Oliver-ke,
Great work! 👏
Could you please write down on how one can test this plugin locally?

[Oliver-ke]

Hi @Oliver-ke, Great work! 👏 Could you please write down on how one can test this plugin locally?

Sure, I'd share my testing.md file shortly

[Oliver-ke]

@ibakshay

TESTING.md

Testing the OpenSearch plugin

Manual QA steps. Verifies datasource, log query, and trace → logs pivot.

Prerequisites

• Node 18.17+ (Jest needs os.availableParallelism)
• npm
• Go 1.21+ (for the SDK tests)
• Docker (for local OpenSearch)
• Perses dev server running locally (see top-level README.md)
• percli on $PATH (built from the perses repo: make build produces bin/percli)

1. Run unit tests

Install once at the workspace root so react/react-dom hoist correctly. Running
npm install inside the plugin directory creates a duplicate React install and
breaks the component tests with Cannot read properties of null (reading 'useState').

cd plugins
npm install
cd opensearch
npm test
npm run type-check
npm run lint

Validate the plugin schemas with percli (built from the perses repo):

cd plugins/opensearch
percli plugin lint
percli plugin test-schemas

Go SDK:

cd plugins/opensearch
go test ./sdk/...

Expect all green. No skipped suites.

2. Start a local OpenSearch

Use the bundled compose file (also boots OpenSearch Dashboards on :5601).
The compose file enables CORS so the Perses UI (running on :3000) can hit
the cluster directly when the datasource uses directUrl. Without these
headers the browser preflight is blocked and queries fail silently with no
status code in the network tab.

cd docs/examples
docker compose up -d
docker compose logs -f opensearch    # watch boot, Ctrl+C once "started" appears

Wait until ready:

curl -s http://localhost:9200/_cluster/health | jq .status

Expect "green" or "yellow".

Teardown when done:

docker compose down -v

3. Seed sample log data

Index two log documents with a shared traceId:

curl -X POST "http://localhost:9200/otel-logs-2026.04.29/_doc" \
  -H 'Content-Type: application/json' -d '{
    "@timestamp": "2026-04-29T10:00:00Z",
    "body": "request received",
    "traceId": "abc123",
    "severity": "INFO"
  }'

curl -X POST "http://localhost:9200/otel-logs-2026.04.29/_doc" \
  -H 'Content-Type: application/json' -d '{
    "@timestamp": "2026-04-29T10:00:01Z",
    "body": "request failed",
    "traceId": "abc123",
    "severity": "ERROR"
  }'

curl -X POST "http://localhost:9200/otel-logs-2026.04.29/_refresh"

Sanity-check PPL endpoint directly:

curl -X POST "http://localhost:9200/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query":"source=otel-logs-* | where traceId='\''abc123'\''"}'

Expect 2 datarows in response.

4. Build and serve the plugin

cd plugins/opensearch
npm run dev

Dev server listens on http://localhost:3135 (configured in rsbuild.config.ts).
The module-federation manifest is served at http://localhost:3135/mf-manifest.json;
point Perses at that URL when registering the plugin. Keep the dev server running.

5. Register the datasource in Perses

The current Perses UI does not accept raw YAML for datasource creation, so create
the resource one of two ways.

Option A — via the UI form

Navigate to Admin → Global Datasources → Add Datasource. Fill in:

• Name: opensearch
• Plugin Options → Source: OpenSearch Datasource
• Direct URL: http://localhost:9200

Save.

Option B — via percli (declarative)

Drop this YAML in a file opensearch-ds.yaml:

kind: GlobalDatasource
metadata:
  name: opensearch
spec:
  default: false
  plugin:
    kind: OpenSearchDatasource
    spec:
      directUrl: http://localhost:9200

Apply it:

percli login http://localhost:8080
percli apply -f opensearch-ds.yaml

Verify

• Datasource list shows opensearch with kind OpenSearchDatasource.
• Edit form renders without console errors. (Open browser devtools → Console
  while clicking through the form. Any red errors here block step 6.)

6. Smoke test the log query editor

Create dashboard, add LogsTable panel, pick OpenSearchLogQuery.

• Datasource picker lists opensearch. Select it.
• Type query: source=otel-logs-* | where traceId='abc123'.
• Hit Apply. Table shows 2 rows. Timestamp + message columns populated.

Edge cases:

Case	Input	Expected
Empty query	``	No request, no error toast
Bad PPL	source=foo | invalid	Error banner with PPL error body (SyntaxCheckException)
Wrong index	source=does-not-exist	Error banner with IndexNotFoundException from OpenSearch (the PPL endpoint returns 404; the plugin surfaces that as an OpenSearchPPLError)
Unreachable DS	stop docker, re-run query	Error banner, retry works after restart
Aggregation (stats)	source=otel-logs-* | stats count() by service	Returns one row per service, no error
Field projection	source=otel-logs-* | fields service, body | head 10	Returns 10 rows with only those two columns

7. Field overrides

Index a doc with non-standard field names:

curl -X POST "http://localhost:9200/legacy-logs/_doc" \
  -H 'Content-Type: application/json' -d '{
    "time": "2026-04-29T10:05:00Z",
    "message": "legacy log line",
    "trace_id": "abc123"
  }'
curl -X POST "http://localhost:9200/legacy-logs/_refresh"

In the query spec, set:

timestampField: time
messageField: message
query: source=legacy-logs | where trace_id='abc123'

Expect row to render with time as timestamp and message as body. Without
the overrides, the row still renders without crashing: missing timestamp falls
back to epoch 0, and missing message falls back to a JSON dump of the row.

8. Trace → logs pivot

Load docs/examples/trace-to-logs.json as a dashboard. Requires a Tempo (or Jaeger) datasource also configured.

• Set traceId variable to abc123.
• Logs panel re-runs and shows the 2 seeded rows.
• Change variable to xyz999. Logs panel goes empty without error.
• Swap Tempo panel for Jaeger per docs/examples/README.md. Logs panel unchanged.

9. Cleanup

cd plugins/opensearch/docs/examples
docker compose down -v

[horovits]

@ibakshay

TESTING.md

Testing the OpenSearch plugin

Manual QA steps. Verifies datasource, log query, and trace → logs pivot.

Prerequisites

• Node 18.17+ (Jest needs os.availableParallelism)
• npm
• Go 1.21+ (for the SDK tests)
• Docker (for local OpenSearch)
• Perses dev server running locally (see top-level README.md)
• percli on $PATH (built from the perses repo: make build produces bin/percli)

1. Run unit tests

Install once at the workspace root so react/react-dom hoist correctly. Running npm install inside the plugin directory creates a duplicate React install and breaks the component tests with Cannot read properties of null (reading 'useState').

cd plugins
npm install
cd opensearch
npm test
npm run type-check
npm run lint

Validate the plugin schemas with percli (built from the perses repo):

cd plugins/opensearch
percli plugin lint
percli plugin test-schemas

Go SDK:

cd plugins/opensearch
go test ./sdk/...

Expect all green. No skipped suites.

2. Start a local OpenSearch

Use the bundled compose file (also boots OpenSearch Dashboards on :5601). The compose file enables CORS so the Perses UI (running on :3000) can hit the cluster directly when the datasource uses directUrl. Without these headers the browser preflight is blocked and queries fail silently with no status code in the network tab.

cd docs/examples
docker compose up -d
docker compose logs -f opensearch    # watch boot, Ctrl+C once "started" appears

Wait until ready:

curl -s http://localhost:9200/_cluster/health | jq .status

Expect "green" or "yellow".

Teardown when done:

docker compose down -v

3. Seed sample log data

Index two log documents with a shared traceId:

curl -X POST "http://localhost:9200/otel-logs-2026.04.29/_doc" \
  -H 'Content-Type: application/json' -d '{
    "@timestamp": "2026-04-29T10:00:00Z",
    "body": "request received",
    "traceId": "abc123",
    "severity": "INFO"
  }'

curl -X POST "http://localhost:9200/otel-logs-2026.04.29/_doc" \
  -H 'Content-Type: application/json' -d '{
    "@timestamp": "2026-04-29T10:00:01Z",
    "body": "request failed",
    "traceId": "abc123",
    "severity": "ERROR"
  }'

curl -X POST "http://localhost:9200/otel-logs-2026.04.29/_refresh"

Sanity-check PPL endpoint directly:

curl -X POST "http://localhost:9200/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query":"source=otel-logs-* | where traceId='\''abc123'\''"}'

Expect 2 datarows in response.

4. Build and serve the plugin

cd plugins/opensearch
npm run dev

Dev server listens on http://localhost:3135 (configured in rsbuild.config.ts). The module-federation manifest is served at http://localhost:3135/mf-manifest.json; point Perses at that URL when registering the plugin. Keep the dev server running.

5. Register the datasource in Perses

The current Perses UI does not accept raw YAML for datasource creation, so create the resource one of two ways.

Option A — via the UI form

Navigate to Admin → Global Datasources → Add Datasource. Fill in:

• Name: opensearch
• Plugin Options → Source: OpenSearch Datasource
• Direct URL: http://localhost:9200

Save.

Option B — via percli (declarative)

Drop this YAML in a file opensearch-ds.yaml:

kind: GlobalDatasource
metadata:
  name: opensearch
spec:
  default: false
  plugin:
    kind: OpenSearchDatasource
    spec:
      directUrl: http://localhost:9200

Apply it:

percli login http://localhost:8080
percli apply -f opensearch-ds.yaml

Verify

• Datasource list shows opensearch with kind OpenSearchDatasource.
• Edit form renders without console errors. (Open browser devtools → Console
  while clicking through the form. Any red errors here block step 6.)

6. Smoke test the log query editor

Create dashboard, add LogsTable panel, pick OpenSearchLogQuery.

• Datasource picker lists opensearch. Select it.
• Type query: source=otel-logs-* | where traceId='abc123'.
• Hit Apply. Table shows 2 rows. Timestamp + message columns populated.

Edge cases:

Case Input Expected
Empty query `` No request, no error toast
Bad PPL source=foo | invalid Error banner with PPL error body (`SyntaxCheckException`)
Wrong index `source=does-not-exist` Error banner with `IndexNotFoundException` from OpenSearch (the PPL endpoint returns 404; the plugin surfaces that as an `OpenSearchPPLError`)
Unreachable DS stop docker, re-run query Error banner, retry works after restart
Aggregation (stats) `source=otel-logs-* | stats count() by service` Returns one row per service, no error
Field projection `source=otel-logs-* | fields service, body | head 10` Returns 10 rows with only those two columns

7. Field overrides

Index a doc with non-standard field names:

curl -X POST "http://localhost:9200/legacy-logs/_doc" \
  -H 'Content-Type: application/json' -d '{
    "time": "2026-04-29T10:05:00Z",
    "message": "legacy log line",
    "trace_id": "abc123"
  }'
curl -X POST "http://localhost:9200/legacy-logs/_refresh"

In the query spec, set:

timestampField: time
messageField: message
query: source=legacy-logs | where trace_id='abc123'

Expect row to render with time as timestamp and message as body. Without the overrides, the row still renders without crashing: missing timestamp falls back to epoch 0, and missing message falls back to a JSON dump of the row.

8. Trace → logs pivot

Load docs/examples/trace-to-logs.json as a dashboard. Requires a Tempo (or Jaeger) datasource also configured.

• Set traceId variable to abc123.
• Logs panel re-runs and shows the 2 seeded rows.
• Change variable to xyz999. Logs panel goes empty without error.
• Swap Tempo panel for Jaeger per docs/examples/README.md. Logs panel unchanged.

9. Cleanup

cd plugins/opensearch/docs/examples
docker compose down -v

looks like a comprehensive test plan

[ibakshay]

Hi @Oliver-ke, first of all great work!

I have tested against a local OpenSearch with the opensearch_dashboards_sample_data_logs sample. I did not review the source-code. I have only tested the functionality and below are my findings.

Bugs/Improvements

1. With Timestamp field blank, querying opensearch_dashboards_sample_data_logs returns 400: Invalid Query

Screenshot 1

Screenshot 2

Working after I add timestamp in the @timestamp input field

Timestamp field (optional) placeholder @timestamp implies the plugin will fall back gracefully if left blank. it doesn't

2. Index pattern is ignored when PPL Query already contains source=....

3. The plugin's auto-generated where @timestamp` >= ...` clause is always prepended, even when the user writes their own time bound. No escape hatch.

4. Run Query with an empty PPL Query is a silent no-op — no request fires, panel just shows No logs to display.
   UX

5. Run Query button is always enabled, even when the form has nothing runnable.

6.. PPL Query field is a plain textarea. It has no syntax highlighting, no autocomplete, no Query Examples disclosure (Please compare ClickHouse query editor in the same panel type).

7. No way to discover indices or field names from the editor. User must know them in advance.

http://localhost:9200/_cat/indices?v
health status index                                       uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   otel-v1-apm-service-map-sample              Nio8tHEHTWK3uNI4PHmyXA   1   0         49            0     17.7kb         17.7kb
yellow open   otel-logs-2026.04.29                        _3tNZKIVTU2eseJhMHb_cw   1   1          4            0       17kb           17kb
green  open   otel-v1-apm-span-sample                     14q5nFcqQq-OwewznwFAZQ   1   0      13061            0      6.1mb          6.1mb
green  open   .ql-datasources                             KdnzV0EoT96Y1XpY9NfQ7A   1   0          0            0       208b           208b
green  open   top_queries-2026.05.22-84957                hi43N36cRxqNUGtFImwVSg   1   0         16            2    104.1kb        104.1kb
green  open   ss4o_metrics-otel-sample                    2Z4CkH5JTGWQXW6NZgHc7A   1   0      39923            0        4mb            4mb
green  open   .kibana_1                                   uKgkLwTLThyWFoSS5ztn2Q   1   0        246            1    155.6kb        155.6kb
green  open   ss4o_logs-otel-sample                       EkQueqJiQJmMWszbrNRe-Q   1   0      16286            0      5.5mb          5.5mb
green  open   .plugins-ml-config                          QmvGWldgQ_-5TnMuA35KeQ   1   0          1            0        4kb            4kb
green  open   .opensearch-observability                   BqmQfOVnSNaeth8GcjBEkw   1   0          0            0       208b           208b
green  open   opensearch_dashboards_sample_data_logs      OgFiObqIR4GAuT_GPovfFA   1   0      14074            0      7.4mb          7.4mb
green  open   opensearch_dashboards_sample_data_flights   1jp6tGfmSuqBK1YUjKYmNA   1   0      13059            0      5.5mb          5.5mb
green  open   opensearch_dashboards_sample_data_ecommerce dxMt1PfqSP-9-4G5Cd-WbQ   1   0       4675            0      3.9mb          3.9mb

8 Error banner reads OpenSearch PPL request failed (400): Invalid Query — the actual details field from the response (e.g. can't resolve Symbol(name=@timestamp)) is dropped, removing the important info.
9. No logs to display is shown for at least four distinct states (empty query, zero hits, missing index, time-range miss). Each wants its own message.
Docs / discoverability

10. PPL is not introduced anywhere in the editor, no link to OpenSearch PPL syntax docs, no mention that the OpenSearch PPL plugin must be enabled cluster-side for any of this to work.

11. keyboard shortcut should be using the @tanstack/react-hotkeys like the other plugins.

[Oliver-ke]

Hi @Oliver-ke, first of all great work!

I have tested against a local OpenSearch with the opensearch_dashboards_sample_data_logs sample. I did not review the source-code. I have only tested the functionality and below are my findings.

Bugs/Improvements

1. With Timestamp field blank, querying opensearch_dashboards_sample_data_logs returns 400: Invalid Query

Screenshot 1

#### Screenshot 2 Working after I add `timestamp` in the `@timestamp` input field

Timestamp field (optional) placeholder @timestamp implies the plugin will fall back gracefully if left blank. it doesn't

2. Index pattern is silently ignored when PPL Query already contains source=.... Two visible inputs, only one wins, no signal to the user.
3. The plugin's auto-generated where @timestamp >= ... clause is always prepended, even when the user writes their own time bound. No escape hatch.
4. Run Query with an empty PPL Query is a silent no-op — no request fires, panel just shows No logs to display.
   UX
5. Run Query button is always enabled, even when the form has nothing runnable.

6.. PPL Query field is a plain textarea. It has no syntax highlighting, no autocomplete, no Query Examples disclosure (Please compare ClickHouse query editor in the same panel type).

7. No way to discover indices or field names from the editor. User must know them in advance.

http://localhost:9200/_cat/indices?v
health status index                                       uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   otel-v1-apm-service-map-sample              Nio8tHEHTWK3uNI4PHmyXA   1   0         49            0     17.7kb         17.7kb
yellow open   otel-logs-2026.04.29                        _3tNZKIVTU2eseJhMHb_cw   1   1          4            0       17kb           17kb
green  open   otel-v1-apm-span-sample                     14q5nFcqQq-OwewznwFAZQ   1   0      13061            0      6.1mb          6.1mb
green  open   .ql-datasources                             KdnzV0EoT96Y1XpY9NfQ7A   1   0          0            0       208b           208b
green  open   top_queries-2026.05.22-84957                hi43N36cRxqNUGtFImwVSg   1   0         16            2    104.1kb        104.1kb
green  open   ss4o_metrics-otel-sample                    2Z4CkH5JTGWQXW6NZgHc7A   1   0      39923            0        4mb            4mb
green  open   .kibana_1                                   uKgkLwTLThyWFoSS5ztn2Q   1   0        246            1    155.6kb        155.6kb
green  open   ss4o_logs-otel-sample                       EkQueqJiQJmMWszbrNRe-Q   1   0      16286            0      5.5mb          5.5mb
green  open   .plugins-ml-config                          QmvGWldgQ_-5TnMuA35KeQ   1   0          1            0        4kb            4kb
green  open   .opensearch-observability                   BqmQfOVnSNaeth8GcjBEkw   1   0          0            0       208b           208b
green  open   opensearch_dashboards_sample_data_logs      OgFiObqIR4GAuT_GPovfFA   1   0      14074            0      7.4mb          7.4mb
green  open   opensearch_dashboards_sample_data_flights   1jp6tGfmSuqBK1YUjKYmNA   1   0      13059            0      5.5mb          5.5mb
green  open   opensearch_dashboards_sample_data_ecommerce dxMt1PfqSP-9-4G5Cd-WbQ   1   0       4675            0      3.9mb          3.9mb

8. Add second Query on an OpenSearch panel defaults to ClickHouse Log Query

9. Error banner reads OpenSearch PPL request failed (400): Invalid Query — the actual details field from the response (e.g. can't resolve Symbol(name=@timestamp)) is dropped, removing the important info. 10. No logs to display is shown for at least four distinct states (empty query, zero hits, missing index, time-range miss). Each wants its own message. Docs / discoverability 11. PPL is not introduced anywhere in the editor — no link to OpenSearch PPL syntax docs, no mention that the OpenSearch PPL plugin must be enabled cluster-side for any of this to work. 12. keyboard shortcut should be using the `@tanstack/react-hotkeys` like the other plugins.

@ibakshay Thanks for the feedback.
I'd address those and update the PR.

[Oliver-ke]

@ibakshay Had to go through your feedbacks again.

The following feedback (3, 8, 10, 11) has been clearly addressed on this PR. While 1, 2, and 6 have been partially addressed due to the increasing scope. Issues 4 and 7 also deferred as they result in a larger scope, while 5 and 9 are outside the plugin repo (Perses hosted UI).

In summary, all feedback in scope for this PR, at least for the initial release, has been addressed. Further enhancements can be added in the coming PR's.

[ibakshay]

@ibakshay Had to go through your feedbacks again.

The following feedback (3, 8, 10, 11) has been clearly addressed on this PR. While 1, 2, and 6 have been partially addressed due to the increasing scope. Issues 4 and 7 also deferred as they result in a larger scope, while 5 and 9 are outside the plugin repo (Perses hosted UI).

In summary, all feedback in scope for this PR, at least for the initial release, has been addressed. Further enhancements can be added in the coming PR's.

I think I did not build properly before. Everything works as it is supposed to be now. Thank you for the clarification.

[Copilot]

Pull request overview

Copilot reviewed 52 out of 55 changed files in this pull request and generated 3 comments.

[Oliver-ke]

Resolved all outstanding comments. Bump perses version

[Oliver-ke]

Any new feedback on this PR?

[Oliver-ke]

PR has been updated based on feedbacks

[Oliver-ke]

Please kindly review

[jgbernalp]

we might need to trim this first. A query with might generate invalid PPL

[jgbernalp]

empty string reason will not include the details:

Suggested change

	const suffix = reason && details && reason !== details ? `${reason} — ${details}` : (reason ?? details);
	const suffix = reason && details && reason !== details ? `${reason} — ${details}` : (reason || details);

[jgbernalp]

should index be escaped with escapeIdentifier?

[jgbernalp]

We should avoid using harcoded colors, this won't work for dark mode cases and won't match the general style, prefer using theme colors.

[jgbernalp]

We don't need to duplicate the HTTPProxy fields, we can import them from the existing https://github.com/perses/spec/blob/main/go/datasource/datasource.go#L39

[jgbernalp]

rest or other variables might be empty. I suggest using join with a filter, something like

parts.filter(notEmpty).join(' | ')

[jgbernalp]

The heuristic lacks coverage for nano or microseconds, as this values need a different division factor. Specially for logs which might have nano second precision, the current heuristic will parse these values way far in the future. I suggest we add some test for this function.