Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 59 additions & 0 deletions docs/opensearch/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
---
tags:
- datasource
---
# OpenSearch plugins

The OpenSearch plugin package provides support for [OpenSearch](https://opensearch.org/) in Perses dashboards. Logs are queried with [PPL (Piped Processing Language)](https://opensearch.org/docs/latest/search-plugins/sql/ppl/index/), which requires the SQL/PPL plugin on your cluster — it ships with OpenSearch by default.

## Datasource (`OpenSearchDatasource`)

The OpenSearch datasource enables connection between Perses and your OpenSearch cluster for log aggregation and querying. It works with log-related panels and queries.

It supports the [proxy](https://perses.dev/perses/docs/concepts/proxy/) feature of Perses that allows to restrict the access to your data source. All queries are sent to a single endpoint, `POST /_plugins/_ppl`, so that is the only endpoint you need to allow.

See also technical docs related to this plugin:

- [Data model](./model.md#opensearchdatasource)
- [Dashboard-as-Code Go lib](./go-sdk/datasource.md)

## Log Query (`OpenSearchLogQuery`)

The OpenSearch log query plugin enables querying log data from your OpenSearch cluster using PPL syntax. It supports filtering, projection and aggregation through PPL pipes.

See also technical docs related to this plugin:

- [Data model](./model.md#opensearchlogquery)
- [Dashboard-as-Code Go lib](./go-sdk/log-query.md)

### Index pattern

The `index` field of the query is prepended to your expression as a `source=<index>` clause. It is ignored when your expression already starts with `source=` (or `search source=`), so you can either set the index once on the query or write the source clause yourself.

### Automatic time filtering

The panel time range is injected into the query as a `where` clause on the timestamp field, placed immediately after the source clause so it applies before any pipe that drops the timestamp column, such as `stats` or `fields`. A query written as:

```ppl
source=otel-logs-demo | where severityText='ERROR'
```

is sent to OpenSearch as:

```ppl
source=otel-logs-demo | where `@timestamp` >= '2026-07-14T03:53:33.000Z' and `@timestamp` <= '2026-07-14T04:53:33.000Z' | where severityText='ERROR'
```

Set `disableTimeFilter` to `true` to turn this off and manage the time bounds yourself.

### Field mapping

OpenSearch does not enforce a schema, and field names vary by exporter. Each row of the PPL response is mapped to a log entry by looking up a timestamp column and a message column, trying these names in order:

- **timestamp**: `@timestamp`, `timestamp`, `time`
- **message**: `message`, `log`, `body`

Set `timestampField` or `messageField` on the query to prefer a different name; the defaults above are still used as fallbacks. Every remaining column of the response becomes a label on the log entry. If no message column matches, the whole row is serialized to JSON as the log line so no data is lost.

!!! note
The `timestampField` override also determines the column used by the automatic time filter, which defaults to `@timestamp` alone. If your index stores its timestamp as `time` or `timestamp`, set `timestampField` explicitly, otherwise the injected `where` clause references a column that does not exist and OpenSearch rejects the query.
58 changes: 58 additions & 0 deletions docs/opensearch/go-sdk/datasource.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# OpenSearch Datasource Go SDK

## Constructor

```golang
import "github.com/perses/plugins/opensearch/sdk/go/datasource"

var options []datasource.Option
datasource.OpenSearch(options...)
```

Need a list of options. Exactly one of direct URL or proxy must be provided, in order to work.

## Default options

- None

## Available options

#### Direct URL

```golang
import "github.com/perses/plugins/opensearch/sdk/go/datasource"

datasource.DirectURL("http://opensearch.example.com:9200")
```

Configure the access to the OpenSearch datasource with a direct URL. The browser then calls OpenSearch directly, which requires CORS to be enabled on the cluster.

#### Proxy

```golang
import "github.com/perses/plugins/opensearch/sdk/go/datasource"

datasource.HTTPProxy("http://opensearch.example.com:9200", httpProxyOptions...)
```

Configure the access to the OpenSearch datasource with a proxy URL. More info at [HTTP Proxy](https://perses.dev/perses/docs/dac/go/helper/http-proxy).

This is the recommended setup: requests go through the Perses backend, so the cluster needs no CORS configuration and any credentials stay server-side.

## Example

```golang
package main

import (
"github.com/perses/perses/go-sdk/dashboard"

osDs "github.com/perses/plugins/opensearch/sdk/go/datasource"
)

func main() {
dashboard.New("OpenSearch Dashboard",
dashboard.AddDatasource("openSearchMain", osDs.OpenSearch(osDs.DirectURL("http://opensearch.example.com:9200"))),
)
}
```
107 changes: 107 additions & 0 deletions docs/opensearch/go-sdk/log-query.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
# OpenSearch Log Query Go SDK

## Constructor

```golang
import "github.com/perses/plugins/opensearch/sdk/go/query/log"

var options []log.Option
log.OpenSearchLogQuery(`source=otel-logs-* | where severityText='ERROR'`, options...)
```

Need to provide the PPL expression and a list of options. The expression cannot be empty.

## Default options

- [Query()](#query): with the expression provided in the constructor.

## Available options

#### Query

```golang
import "github.com/perses/plugins/opensearch/sdk/go/query/log"

log.Query(`source=otel-logs-* | where severityText='ERROR' | head 20`)
```

Define the PPL query expression for log data.

#### Datasource

```golang
import "github.com/perses/plugins/opensearch/sdk/go/query/log"

log.Datasource("MyOpenSearchDatasource")
```

Define the datasource the query will use.

#### Index

```golang
import "github.com/perses/plugins/opensearch/sdk/go/query/log"

log.Index("otel-logs-*")
```

Define the index or index pattern to read from. It is prepended to the query as a `source=<index>` clause, and ignored when the query already starts with `source=`.

#### TimestampField

```golang
import "github.com/perses/plugins/opensearch/sdk/go/query/log"

log.TimestampField("time")
```

Override which column carries the log timestamp, for indices that do not use `@timestamp`, `timestamp` or `time`. This is also the column used by the automatic time filter, which defaults to `@timestamp`.

#### MessageField

```golang
import "github.com/perses/plugins/opensearch/sdk/go/query/log"

log.MessageField("msg")
```

Override which column becomes the log line, for indices that do not use `message`, `log` or `body`.

#### DisableTimeFilter

```golang
import "github.com/perses/plugins/opensearch/sdk/go/query/log"

log.DisableTimeFilter(true)
```

Stop the panel time range from being injected as a `where` clause on the timestamp field, so the query manages its own time bounds.

## Example

```golang
package main

import (
"github.com/perses/perses/go-sdk/dashboard"
"github.com/perses/perses/go-sdk/panel"
panelgroup "github.com/perses/perses/go-sdk/panel-group"
logstable "github.com/perses/plugins/logstable/sdk/go"
osLog "github.com/perses/plugins/opensearch/sdk/go/query/log"
)

func main() {
dashboard.New("OpenSearch Dashboard",
dashboard.AddPanelGroup("Application Logs",
panelgroup.AddPanel("Error Logs",
logstable.LogsTable(),
panel.AddQuery(
osLog.OpenSearchLogQuery(`where severityText='ERROR' | head 100`,
osLog.Index("otel-logs-*"),
),
),
),
),
)
}
```
146 changes: 146 additions & 0 deletions docs/opensearch/model.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,146 @@
# OpenSearch plugin models

This documentation provides the definition of the different plugins related to OpenSearch.

## OpenSearchDatasource

OpenSearch as a datasource is basically an HTTP server. So we need to define an HTTP config.

```yaml
kind: "OpenSearchDatasource"
spec:
# It is the url of the datasource.
# Leave it empty if you don't want to access the datasource directly from the UI.
# You should define a proxy if you want to access the datasource through the Perses' server.
directUrl: <url> # Optional

# It is the http configuration that will be used by the Perses' server to redirect to the datasource any query sent by the UI.
proxy: <HTTP Proxy specification> # Optional
```

!!! warning
Exactly one of `directUrl` or `proxy` must be set. Providing neither, or both, is rejected.

### HTTP Proxy specification

See [common plugin definitions](https://perses.dev/perses/docs/plugins/common/#http-proxy-specification).

The plugin sends every query to `POST /_plugins/_ppl`, so that single endpoint is all you need to allow.

### Example

A simple OpenSearch datasource would be

```yaml
kind: "Datasource"
metadata:
name: "OpenSearchMain"
project: "logging"
spec:
default: true
plugin:
kind: "OpenSearchDatasource"
spec:
directUrl: "http://opensearch.example.com:9200"
```

A more complex one:

```yaml
kind: "Datasource"
metadata:
name: "OpenSearchMain"
project: "logging"
spec:
default: true
plugin:
kind: "OpenSearchDatasource"
spec:
proxy:
kind: "HTTPProxy"
spec:
url: "http://opensearch.example.com:9200"
allowedEndpoints:
- endpointPattern: "/_plugins/_ppl"
method: "POST"
secret: "opensearch_secret_config"
```

## OpenSearchLogQuery

Perses supports log queries for OpenSearch: `OpenSearchLogQuery`.

```yaml
kind: "OpenSearchLogQuery"
spec:
# `query` is the PPL expression for log data. It cannot be empty.
query: <string>

# `datasource` is a datasource selector. If not provided, the default OpenSearchDatasource is used.
# See the documentation about the datasources to understand how it is selected.
datasource: <OpenSearch Datasource selector> # Optional

# `index` is the index or index pattern to read from, e.g. `logs-*`.
# It is prepended to the query as a `source=<index>` clause, and ignored when the
# query already starts with `source=`.
index: <string> # Optional

# `timestampField` overrides which column carries the log timestamp.
# It is also the column used by the automatic time filter.
# Defaults to `@timestamp`, then `timestamp`, then `time`.
timestampField: <string> # Optional

# `messageField` overrides which column becomes the log line.
# Defaults to `message`, then `log`, then `body`.
messageField: <string> # Optional

# When `disableTimeFilter` is true, the panel time range is NOT injected as a
# `where` clause on the timestamp field.
disableTimeFilter: <boolean> # Optional
```

- See [OpenSearch Datasource selector](#opensearch-datasource-selector)

### Example

A simple log query:

```yaml
kind: "LogQuery"
spec:
plugin:
kind: "OpenSearchLogQuery"
spec:
index: "otel-logs-*"
query: "where severityText='ERROR'"
```

A more complex one, reading from an index whose fields do not match the defaults, and managing its own time bounds:

```yaml
kind: "LogQuery"
spec:
plugin:
kind: "OpenSearchLogQuery"
spec:
datasource:
kind: "OpenSearchDatasource"
name: "OpenSearchMain"
query: "source=app-logs-* | where `service.name`='payment-svc' | sort - time | head 100"
timestampField: "time"
messageField: "msg"
disableTimeFilter: true
```

## Shared definitions

### OpenSearch Datasource selector

!!! note
See [Selecting / Referencing a Datasource](https://github.com/perses/perses/blob/main/docs/api/datasource.md#selecting--referencing-a-datasource)

```yaml
kind: "OpenSearchDatasource"
# The name of the datasource regardless its level
name: <string> # Optional
```
Loading