From 72c1953f6f809c6927426487b9fe8370a41f0311 Mon Sep 17 00:00:00 2001 From: Kelechi Oliver Azorji Date: Mon, 27 Jul 2026 13:43:52 +0000 Subject: [PATCH] docs: add documentation for opensearch plugin Signed-off-by: Kelechi Oliver Azorji --- docs/opensearch/README.md | 59 +++++++++++ docs/opensearch/go-sdk/datasource.md | 58 +++++++++++ docs/opensearch/go-sdk/log-query.md | 107 ++++++++++++++++++++ docs/opensearch/model.md | 146 +++++++++++++++++++++++++++ 4 files changed, 370 insertions(+) create mode 100644 docs/opensearch/README.md create mode 100644 docs/opensearch/go-sdk/datasource.md create mode 100644 docs/opensearch/go-sdk/log-query.md create mode 100644 docs/opensearch/model.md diff --git a/docs/opensearch/README.md b/docs/opensearch/README.md new file mode 100644 index 000000000..898286dda --- /dev/null +++ b/docs/opensearch/README.md @@ -0,0 +1,59 @@ +--- +tags: + - datasource +--- +# OpenSearch plugins + +The OpenSearch plugin package provides support for [OpenSearch](https://opensearch.org/) in Perses dashboards. Logs are queried with [PPL (Piped Processing Language)](https://opensearch.org/docs/latest/search-plugins/sql/ppl/index/), which requires the SQL/PPL plugin on your cluster — it ships with OpenSearch by default. + +## Datasource (`OpenSearchDatasource`) + +The OpenSearch datasource enables connection between Perses and your OpenSearch cluster for log aggregation and querying. It works with log-related panels and queries. + +It supports the [proxy](https://perses.dev/perses/docs/concepts/proxy/) feature of Perses that allows to restrict the access to your data source. All queries are sent to a single endpoint, `POST /_plugins/_ppl`, so that is the only endpoint you need to allow. + +See also technical docs related to this plugin: + +- [Data model](./model.md#opensearchdatasource) +- [Dashboard-as-Code Go lib](./go-sdk/datasource.md) + +## Log Query (`OpenSearchLogQuery`) + +The OpenSearch log query plugin enables querying log data from your OpenSearch cluster using PPL syntax. It supports filtering, projection and aggregation through PPL pipes. + +See also technical docs related to this plugin: + +- [Data model](./model.md#opensearchlogquery) +- [Dashboard-as-Code Go lib](./go-sdk/log-query.md) + +### Index pattern + +The `index` field of the query is prepended to your expression as a `source=` clause. It is ignored when your expression already starts with `source=` (or `search source=`), so you can either set the index once on the query or write the source clause yourself. + +### Automatic time filtering + +The panel time range is injected into the query as a `where` clause on the timestamp field, placed immediately after the source clause so it applies before any pipe that drops the timestamp column, such as `stats` or `fields`. A query written as: + +```ppl +source=otel-logs-demo | where severityText='ERROR' +``` + +is sent to OpenSearch as: + +```ppl +source=otel-logs-demo | where `@timestamp` >= '2026-07-14T03:53:33.000Z' and `@timestamp` <= '2026-07-14T04:53:33.000Z' | where severityText='ERROR' +``` + +Set `disableTimeFilter` to `true` to turn this off and manage the time bounds yourself. + +### Field mapping + +OpenSearch does not enforce a schema, and field names vary by exporter. Each row of the PPL response is mapped to a log entry by looking up a timestamp column and a message column, trying these names in order: + +- **timestamp**: `@timestamp`, `timestamp`, `time` +- **message**: `message`, `log`, `body` + +Set `timestampField` or `messageField` on the query to prefer a different name; the defaults above are still used as fallbacks. Every remaining column of the response becomes a label on the log entry. If no message column matches, the whole row is serialized to JSON as the log line so no data is lost. + +!!! note + The `timestampField` override also determines the column used by the automatic time filter, which defaults to `@timestamp` alone. If your index stores its timestamp as `time` or `timestamp`, set `timestampField` explicitly, otherwise the injected `where` clause references a column that does not exist and OpenSearch rejects the query. diff --git a/docs/opensearch/go-sdk/datasource.md b/docs/opensearch/go-sdk/datasource.md new file mode 100644 index 000000000..7ae2e0f1c --- /dev/null +++ b/docs/opensearch/go-sdk/datasource.md @@ -0,0 +1,58 @@ +# OpenSearch Datasource Go SDK + +## Constructor + +```golang +import "github.com/perses/plugins/opensearch/sdk/go/datasource" + +var options []datasource.Option +datasource.OpenSearch(options...) +``` + +Need a list of options. Exactly one of direct URL or proxy must be provided, in order to work. + +## Default options + +- None + +## Available options + +#### Direct URL + +```golang +import "github.com/perses/plugins/opensearch/sdk/go/datasource" + +datasource.DirectURL("http://opensearch.example.com:9200") +``` + +Configure the access to the OpenSearch datasource with a direct URL. The browser then calls OpenSearch directly, which requires CORS to be enabled on the cluster. + +#### Proxy + +```golang +import "github.com/perses/plugins/opensearch/sdk/go/datasource" + +datasource.HTTPProxy("http://opensearch.example.com:9200", httpProxyOptions...) +``` + +Configure the access to the OpenSearch datasource with a proxy URL. More info at [HTTP Proxy](https://perses.dev/perses/docs/dac/go/helper/http-proxy). + +This is the recommended setup: requests go through the Perses backend, so the cluster needs no CORS configuration and any credentials stay server-side. + +## Example + +```golang +package main + +import ( + "github.com/perses/perses/go-sdk/dashboard" + + osDs "github.com/perses/plugins/opensearch/sdk/go/datasource" +) + +func main() { + dashboard.New("OpenSearch Dashboard", + dashboard.AddDatasource("openSearchMain", osDs.OpenSearch(osDs.DirectURL("http://opensearch.example.com:9200"))), + ) +} +``` diff --git a/docs/opensearch/go-sdk/log-query.md b/docs/opensearch/go-sdk/log-query.md new file mode 100644 index 000000000..2bbdecd2c --- /dev/null +++ b/docs/opensearch/go-sdk/log-query.md @@ -0,0 +1,107 @@ +# OpenSearch Log Query Go SDK + +## Constructor + +```golang +import "github.com/perses/plugins/opensearch/sdk/go/query/log" + +var options []log.Option +log.OpenSearchLogQuery(`source=otel-logs-* | where severityText='ERROR'`, options...) +``` + +Need to provide the PPL expression and a list of options. The expression cannot be empty. + +## Default options + +- [Query()](#query): with the expression provided in the constructor. + +## Available options + +#### Query + +```golang +import "github.com/perses/plugins/opensearch/sdk/go/query/log" + +log.Query(`source=otel-logs-* | where severityText='ERROR' | head 20`) +``` + +Define the PPL query expression for log data. + +#### Datasource + +```golang +import "github.com/perses/plugins/opensearch/sdk/go/query/log" + +log.Datasource("MyOpenSearchDatasource") +``` + +Define the datasource the query will use. + +#### Index + +```golang +import "github.com/perses/plugins/opensearch/sdk/go/query/log" + +log.Index("otel-logs-*") +``` + +Define the index or index pattern to read from. It is prepended to the query as a `source=` clause, and ignored when the query already starts with `source=`. + +#### TimestampField + +```golang +import "github.com/perses/plugins/opensearch/sdk/go/query/log" + +log.TimestampField("time") +``` + +Override which column carries the log timestamp, for indices that do not use `@timestamp`, `timestamp` or `time`. This is also the column used by the automatic time filter, which defaults to `@timestamp`. + +#### MessageField + +```golang +import "github.com/perses/plugins/opensearch/sdk/go/query/log" + +log.MessageField("msg") +``` + +Override which column becomes the log line, for indices that do not use `message`, `log` or `body`. + +#### DisableTimeFilter + +```golang +import "github.com/perses/plugins/opensearch/sdk/go/query/log" + +log.DisableTimeFilter(true) +``` + +Stop the panel time range from being injected as a `where` clause on the timestamp field, so the query manages its own time bounds. + +## Example + +```golang +package main + +import ( + "github.com/perses/perses/go-sdk/dashboard" + "github.com/perses/perses/go-sdk/panel" + panelgroup "github.com/perses/perses/go-sdk/panel-group" + logstable "github.com/perses/plugins/logstable/sdk/go" + osLog "github.com/perses/plugins/opensearch/sdk/go/query/log" +) + +func main() { + dashboard.New("OpenSearch Dashboard", + dashboard.AddPanelGroup("Application Logs", + panelgroup.AddPanel("Error Logs", + logstable.LogsTable(), + panel.AddQuery( + osLog.OpenSearchLogQuery(`where severityText='ERROR' | head 100`, + osLog.Index("otel-logs-*"), + ), + ), + ), + ), + ) +} +``` diff --git a/docs/opensearch/model.md b/docs/opensearch/model.md new file mode 100644 index 000000000..bdcdb2552 --- /dev/null +++ b/docs/opensearch/model.md @@ -0,0 +1,146 @@ +# OpenSearch plugin models + +This documentation provides the definition of the different plugins related to OpenSearch. + +## OpenSearchDatasource + +OpenSearch as a datasource is basically an HTTP server. So we need to define an HTTP config. + +```yaml +kind: "OpenSearchDatasource" +spec: + # It is the url of the datasource. + # Leave it empty if you don't want to access the datasource directly from the UI. + # You should define a proxy if you want to access the datasource through the Perses' server. + directUrl: # Optional + + # It is the http configuration that will be used by the Perses' server to redirect to the datasource any query sent by the UI. + proxy: # Optional +``` + +!!! warning + Exactly one of `directUrl` or `proxy` must be set. Providing neither, or both, is rejected. + +### HTTP Proxy specification + +See [common plugin definitions](https://perses.dev/perses/docs/plugins/common/#http-proxy-specification). + +The plugin sends every query to `POST /_plugins/_ppl`, so that single endpoint is all you need to allow. + +### Example + +A simple OpenSearch datasource would be + +```yaml +kind: "Datasource" +metadata: + name: "OpenSearchMain" + project: "logging" +spec: + default: true + plugin: + kind: "OpenSearchDatasource" + spec: + directUrl: "http://opensearch.example.com:9200" +``` + +A more complex one: + +```yaml +kind: "Datasource" +metadata: + name: "OpenSearchMain" + project: "logging" +spec: + default: true + plugin: + kind: "OpenSearchDatasource" + spec: + proxy: + kind: "HTTPProxy" + spec: + url: "http://opensearch.example.com:9200" + allowedEndpoints: + - endpointPattern: "/_plugins/_ppl" + method: "POST" + secret: "opensearch_secret_config" +``` + +## OpenSearchLogQuery + +Perses supports log queries for OpenSearch: `OpenSearchLogQuery`. + +```yaml +kind: "OpenSearchLogQuery" +spec: + # `query` is the PPL expression for log data. It cannot be empty. + query: + + # `datasource` is a datasource selector. If not provided, the default OpenSearchDatasource is used. + # See the documentation about the datasources to understand how it is selected. + datasource: # Optional + + # `index` is the index or index pattern to read from, e.g. `logs-*`. + # It is prepended to the query as a `source=` clause, and ignored when the + # query already starts with `source=`. + index: # Optional + + # `timestampField` overrides which column carries the log timestamp. + # It is also the column used by the automatic time filter. + # Defaults to `@timestamp`, then `timestamp`, then `time`. + timestampField: # Optional + + # `messageField` overrides which column becomes the log line. + # Defaults to `message`, then `log`, then `body`. + messageField: # Optional + + # When `disableTimeFilter` is true, the panel time range is NOT injected as a + # `where` clause on the timestamp field. + disableTimeFilter: # Optional +``` + +- See [OpenSearch Datasource selector](#opensearch-datasource-selector) + +### Example + +A simple log query: + +```yaml +kind: "LogQuery" +spec: + plugin: + kind: "OpenSearchLogQuery" + spec: + index: "otel-logs-*" + query: "where severityText='ERROR'" +``` + +A more complex one, reading from an index whose fields do not match the defaults, and managing its own time bounds: + +```yaml +kind: "LogQuery" +spec: + plugin: + kind: "OpenSearchLogQuery" + spec: + datasource: + kind: "OpenSearchDatasource" + name: "OpenSearchMain" + query: "source=app-logs-* | where `service.name`='payment-svc' | sort - time | head 100" + timestampField: "time" + messageField: "msg" + disableTimeFilter: true +``` + +## Shared definitions + +### OpenSearch Datasource selector + +!!! note + See [Selecting / Referencing a Datasource](https://github.com/perses/perses/blob/main/docs/api/datasource.md#selecting--referencing-a-datasource) + +```yaml +kind: "OpenSearchDatasource" +# The name of the datasource regardless its level +name: # Optional +```