README

                   -----------------------------------------
                      The HAProxy OpenTelemetry filter (OTel)
                                  Version 1.0
                          ( Last update: 2026-07-02 )
                   -----------------------------------------
                           Author : Miroslav Zagorac
                     Contact : mzagorac at haproxy dot com


SUMMARY
--------

  0.    Terms
  1.    Introduction
  2.    Build instructions
  3.    Basic concepts in OpenTelemetry
  4.    OTel configuration
  4.1.    OTel scope
  4.2.    "otel-instrumentation" section
  4.3.    "otel-scope" section
  4.4.    "otel-group" section
  5.    Examples
  5.1.    Benchmarking results
  6.    OTel CLI
  7.    Known bugs and limitations


0. Terms
---------

* OTel: The HAProxy OpenTelemetry filter

OTel is the HAProxy filter that allows you to send telemetry data (traces,
metrics and logs) to observability backends via the OpenTelemetry protocol.


1. Introduction
----------------

Nowadays there is a growing need to divide a process into microservices and
there is a problem of monitoring the work of the same process.  One way to solve
this problem is to use a distributed tracing service in a central location.

The OTel filter is the successor to the OpenTracing (OT) filter and is built on
the OpenTelemetry standard, which unifies distributed tracing, metrics and
logging into a single observability framework.  Unlike the older OpenTracing
filter which relied on vendor-specific tracer plugins, the OTel filter uses the
OpenTelemetry protocol (OTLP) to export data directly to any compatible backend.

The OTel filter is a standard HAProxy filter, so what applies to others also
applies to this one (of course, by that I mean what is described in the
documentation, more precisely in the doc/internals/filters.txt file).

The OTel filter activation is done explicitly by specifying it in the HAProxy
configuration.  If this is not done, the OTel filter in no way participates in
the work of HAProxy.

As for the impact on HAProxy speed, this is documented with test results located
in the test directory (see section 5.1).  The speed of operation depends on the
way the filter is used and the complexity of the configuration.  In typical
production use with a rate limit of 10% or less, the performance impact should
be negligible (see the 'rate-limit' keyword).

The OTel filter allows intensive use of ACLs, which can be defined anywhere in
the configuration.  Thus, it is possible to use the filter only for those
connections that are of interest to us.


2. Build instructions
----------------------

OTel is the HAProxy filter and as such is compiled together with HAProxy.  It
supports all HAProxy versions from 3.4 onward; the sources are available at
https://github.com/haproxy/haproxy/ .

To communicate with an OpenTelemetry compatible backend, the OTel filter uses
the OpenTelemetry C Wrapper library version 2.2.0 or later (which again uses
the OpenTelemetry C++ SDK).  This means that we must have the library installed
on the system on which we want to compile or use HAProxy.

Note that the build does not enforce this minimum version: pkg-config is asked
only whether the library exists.  At startup the filter verifies instead that
the linked library and the header files it was compiled against carry the same
version string, and refuses to initialize on a mismatch.

Instructions for compiling and installing the required library can be found at
https://github.com/haproxytech/opentelemetry-c-wrapper .

The OTel filter is built as a standalone addon outside the HAProxy source tree
and is plugged into the HAProxy build via the EXTRA_MAKE variable, which lists
directories whose Makefile.mk fragments are included by the HAProxy top-level
Makefile.  The OTel addon's Makefile.mk sits at the top of this source tree, so
EXTRA_MAKE must point to that directory.

The examples below assume that the OTel addon checkout sits next to the HAProxy
source tree and that make is run from the HAProxy directory, referencing the
addon as "../haproxy-opentelemetry".  Adjust the path if your layout differs.

The OTel filter can be more easily compiled using the pkg-config tool, if we
have the OpenTelemetry C Wrapper library installed so that it contains
pkg-config files (which have the .pc extension).  If the pkg-config tool cannot
be used, then the path to the directory where the include files and libraries
are located can be explicitly specified.

Below are examples of the two ways to compile HAProxy with the OTel filter, the
first using the pkg-config tool and the second explicitly specifying the path to
the OpenTelemetry C Wrapper include and library.

Note: prompt '%' indicates that the command is executed under an unprivileged
      user, while prompt '#' indicates that the command is executed under the
      root user.

Example of compiling HAProxy using the pkg-config tool (assuming the
OpenTelemetry C Wrapper library is installed in the /opt directory):

  % PKG_CONFIG_PATH=/opt/lib/pkgconfig make -j8 TARGET=linux-glibc EXTRA_MAKE="../haproxy-opentelemetry"

The OTel filter can also be compiled in debug mode as follows:

  % PKG_CONFIG_PATH=/opt/lib/pkgconfig make -j8 TARGET=linux-glibc EXTRA_MAKE="../haproxy-opentelemetry" OTEL_DEBUG=1

HAProxy compilation example explicitly specifying path to the OpenTelemetry C
Wrapper include and library:

  % make -j8 TARGET=linux-glibc EXTRA_MAKE="../haproxy-opentelemetry" OTEL_INC=/opt/include OTEL_LIB=/opt/lib

In case we want to use debug mode, then it looks like this:

  % make -j8 TARGET=linux-glibc EXTRA_MAKE="../haproxy-opentelemetry" OTEL_INC=/opt/include OTEL_LIB=/opt/lib OTEL_DEBUG=1

To enable OpenTelemetry context propagation via HAProxy variables (in addition
to HTTP headers), add the OTEL_USE_VARS=1 option:

  % PKG_CONFIG_PATH=/opt/lib/pkgconfig make -j8 TARGET=linux-glibc EXTRA_MAKE="../haproxy-opentelemetry" OTEL_USE_VARS=1

If the library we want to use is not installed on a unix system, then a locally
installed library can be used (say, which is compiled and installed in the user
home directory).  In this case instead of /opt/include and /opt/lib the
equivalent paths to the local installation should be specified.  Of course, in
that case the pkg-config tool can also be used if we have a complete
installation (with .pc files).

Last but not least, if the pkg-config tool is not used when compiling, then the
HAProxy executable may not be able to find the OpenTelemetry C Wrapper library
at startup.  This can be solved in several ways, for example using the
LD_LIBRARY_PATH environment variable which should be set to the path where the
library is located before starting the HAProxy.

  % LD_LIBRARY_PATH=/opt/lib /path-to/haproxy ...

Another way is to add RUNPATH to HAProxy executable that contains the path to
the library in question.

  % make -j8 TARGET=linux-glibc EXTRA_MAKE="../haproxy-opentelemetry" OTEL_INC=/opt/include OTEL_LIB=/opt/lib OTEL_RUNPATH=1

After HAProxy is compiled, we can check if the OTel filter is enabled:

  % ./haproxy -vv | grep -i opentelemetry
  --- command output ----------
  Built with OpenTelemetry support (C++ version 1.26.0, C Wrapper version 2.2.0-868).
          [OTEL] opentelemetry
  --- command output ----------

A summary of all OTel build options:

  EXTRA_MAKE    - path to the OTel addon directory (enables the filter)
  OTEL_DEBUG    - compile the filter in debug mode
  OTEL_INC      - force path to opentelemetry-c-wrapper include files
  OTEL_LIB      - force path to opentelemetry-c-wrapper library
  OTEL_RUNPATH  - add opentelemetry-c-wrapper RUNPATH (needs OTEL_LIB)
  OTEL_STATIC   - pass --static to pkg-config for static linking
  OTEL_USE_VARS - enable context propagation via HAProxy variables


3. Basic concepts in OpenTelemetry
-----------------------------------

Basic concepts of OpenTelemetry can be read on the OpenTelemetry documentation
website https://opentelemetry.io/docs/concepts/ .

Here we will list only the most important elements of distributed tracing.

A 'trace' is a description of the complete transaction we want to record in the
tracing system.  A 'span' is an operation that represents a unit of work that is
recorded in a tracing system.  A 'span context' is a group of information
related to a particular span that is passed on to the system (from service to
service).  Using this context, we can add new spans to already open trace (or
supplement data in already open spans).

An individual span may contain one or more attributes, events, links and baggage
items.

An 'attribute' is a key-value element that is valid for the entire span.
Attributes describe properties of the span such as HTTP method, URL, status
code, and so on.

A span 'event' is a named key-value element that allows you to write some data
at a certain time within the span's lifetime.  It can be used for debugging or
recording notable occurrences.

A 'link' is a reference to another span (possibly in a different trace) that is
causally related to the current span.  Unlike the parent-child relationship,
links represent non-hierarchical associations between spans.

A 'baggage' item is a key-value data pair that can be used for the duration of
an entire trace, from the moment it is added to the span.

A span 'status' indicates the outcome of the operation: unset (default), ok
(successful) or error (failed).  An optional description string can accompany
the error status.

Note: telemetry delivery is best-effort.  The OpenTelemetry specification does
not mandate reliable delivery, so there is no guarantee that all telemetry
reaches its destination.  Data may be dropped at any stage -- collection,
processing or export -- through sampling, buffer or queue limits, network
failures or backend unavailability.  Treat occasional loss as a normal
operational condition rather than a fault, and do not rely on OTel data where
completeness is critical (for example billing or auditing).


4. OTel configuration
----------------------

The OTel filter must also be included in the HAProxy configuration, in the
proxy section (frontend / listen / backend):

   frontend otel-test
     ...
     filter opentelemetry [id <id>] config <file>
     ...

If no filter id is specified, 'otel-filter' is used as default.  The 'config'
parameter must be specified and it contains the path of the OTel filter
configuration file.  This file defines the OTel scopes, groups and
instrumentation sections (see section 4.1).  The YAML configuration for the
OpenTelemetry SDK is a separate file, referenced by the 'config' keyword inside
the "otel-instrumentation" section (see section 4.2).

Note: the parser does not limit the number of OTel filters; it only requires
each filter id to be unique.  The OTel C wrapper, however, supports a single
active configuration, so only one filter instance is operational per HAProxy
process.  Declaring several distinct filters is thus unsupported for now: the
parser accepts them, but the extra instances will not work.  Once the wrapper
supports simultaneous operation with different configurations, this limitation
will be lifted.


4.1. OTel scope
----------------

If the filter id is defined for the OTel filter, then the OTel scope with the
same name should be defined in the configuration file.  In the same
configuration file we can have several defined OTel scopes.

Each OTel scope must have a defined (only one) "otel-instrumentation" section
that is used to configure the operation of the OTel filter and define the used
groups and scopes.

OTel scope starts with the id of the filter specified in square brackets and
ends with the end of the file or when a new OTel scope is defined.

For example, this defines two OTel scopes in the same configuration file:
  [my-first-otel-filter]
    otel-instrumentation instrumentation1
    ...
    otel-group group1
    ...
    otel-scope scope1
    ...

  [my-second-otel-filter]
    ...

Although several OTel scopes may coexist in a single configuration file, only
one is operational at a time.  The parser does not prevent declaring multiple
filters -- it only requires their ids to be unique -- but the OTel C wrapper
supports a single active configuration, so just one filter instance runs per
HAProxy process.  Once the OTel C wrapper supports simultaneous operation with
different configurations, this restriction will be lifted.


4.2. "otel-instrumentation" section
-------------------------------------

Only one "otel-instrumentation" section must be defined for each OTel scope.

The mandatory 'config' keyword defines the YAML configuration file for the
OpenTelemetry SDK.  This file specifies the telemetry pipeline: exporters,
processors, samplers, readers, providers and signals.

Through optional keywords can be defined ACLs, logging, rate limit, and groups
and scopes that define the tracing model.


otel-instrumentation <name>
  A new OTel instrumentation with the name <name> is created.

  Arguments :
    name - the name of the OpenTelemetry instrumentation section


  The following keywords are supported in this section:
    - mandatory keywords:
      - config

    - optional keywords:
      - acl
      - debug-level
      - groups
      - [no] log
      - [no] option disabled
      - [no] option dontlog-normal
      - [no] option hard-errors
      - rate-limit
      - scopes


acl <aclname> <criterion> [flags] [operator] <value> ...
  Declare or complete an access list.

  To configure and use the ACL, see section 7 of the HAProxy Configuration
  Manual.


config <file>
  The mandatory keyword associated with the OTel instrumentation configuration.
  This keyword sets the path of the YAML configuration file for the
  OpenTelemetry SDK.  The YAML file defines the complete telemetry pipeline
  including the exporters, samplers, processors, readers, providers and signal
  routing.

  The YAML configuration file supports the following top-level sections:

  'exporters' - defines telemetry data destinations.  Supported exporter types
  are:
    - otlp_grpc     : export via OTLP over gRPC
    - otlp_http     : export via OTLP over HTTP (JSON or Protobuf)
    - otlp_file     : export to local files in OTLP format
    - zipkin        : export to Zipkin-compatible backends
    - elasticsearch : export to Elasticsearch
    - ostream       : write to a file (text output, useful for debugging)
    - memory        : in-memory buffer (useful for testing)

  'samplers' - defines trace sampling strategies.  Supported types:
    - always_on            : sample every trace
    - always_off           : sample no traces
    - trace_id_ratio_based : sample a fraction of traces (set by ratio)
    - parent_based         : sampling decision based on parent span

  'processors' - defines how telemetry data is processed before export:
    - batch  : batch spans before exporting (configurable queue size, export
               interval and batch size)
    - single : export each span individually

  'readers' - defines metric readers with configurable export interval and
  timeout.

  'providers' - defines resource attributes (service name, version, instance ID,
  namespace, etc.) that are attached to all telemetry data.

  'signals' - binds the above components together for each signal type (traces,
  metrics, logs), specifying which exporter, sampler, processor, reader and
  provider to use.

  Arguments :
    file - the path of the YAML configuration file


debug-level <value>
  This keyword sets the value of the debug level related to the display of debug
  messages in the OTel filter.  The 'debug-level' value is a bitmask, ie a
  single value bit enables or disables the display of the corresponding debug
  message that uses that bit.  The default value is set via the
  FLT_OTEL_DEBUG_LEVEL macro in the include/config.h file.  Debug level value is
  used only if the OTel filter is compiled with the debug mode enabled,
  otherwise it is ignored.

  Arguments :
    value - bitmask value (hexadecimal notation, e.g. 0x77f)


groups <name> ...
  A list of "otel-group" groups used for the currently defined instrumentation
  is declared.  Several groups can be specified in one line, and the keyword may
  be repeated on multiple lines.

  Arguments :
    name - the name of the OTel group


log global
log <addr> [len <len>] [format <fmt>] <facility> [<level> [<minlevel>]]
no log
  Enable per-instance logging of events and traffic.

  To configure and use the logging system, see section 4.2 of the HAProxy
  Configuration Manual.


option disabled
no option disabled
  Keyword which turns the operation of the OTel filter on or off.  By default
  the filter is on.


option dontlog-normal
no option dontlog-normal
  Drop the below-error log lines (warnings and informational) while keeping the
  error-level messages.  By default, this option is disabled and all levels are
  logged.  For this option to be considered, logging must be turned on.

  See also: 'log' keyword description.


option hard-errors
no option hard-errors
  During the operation of the filter, some errors may occur, caused by incorrect
  configuration of the instrumentation or some error related to the operation of
  HAProxy.  By default, such an error will not interrupt the filter operation
  for the stream in which the error occurred.  If the 'hard-errors' option is
  enabled, the operation error prohibits all further processing of events and
  groups in the stream in which the error occurred.


rate-limit <value>
  This option allows limiting the use of the OTel filter, ie it can be
  influenced whether the OTel filter is activated for a stream or not.
  Determining whether or not a filter is activated depends on the value of this
  option that is compared to a randomly selected value when attaching the filter
  to the stream.  By default, the value of this option is set to 100.0, ie the
  OTel filter is activated for each stream.

  Arguments :
    value - floating point value ranging from 0.0 to 100.0


scopes <name> ...
  This keyword lists the "otel-scope" definitions used by the currently defined
  instrumentation.  Multiple scopes may be given on one line, and the keyword
  may be repeated on multiple lines.

  Arguments :
    name - the name of the OTel scope


4.3. "otel-scope" section
--------------------------

Stream processing begins with filter attachment, then continues with the
processing of a number of defined events and groups, and ends with filter
detachment.  The "otel-scope" section is used to define actions related to
individual events.  However, this section may be part of a group, so the event
does not have to be part of the definition.


otel-scope <name>
  Creates a new OTel scope definition named <name>.

  Arguments :
    name - the name of the OTel scope


  The following keywords are supported in this section:
    - acl
    - attribute
    - baggage
    - event
    - exception
    - extract
    - finish
    - idle-timeout
    - inject
    - instrument
    - link
    - log-record
    - otel-event
    - otel-stop
    - set-var
    - set-var-ctx
    - span
    - status
    - unset-var


acl <aclname> <criterion> [flags] [operator] <value> ...
  Declare or complete an access list.

  To configure and use the ACL, see section 7 of the HAProxy Configuration
  Manual.


attribute <key> <sample> ... [{ if | unless } <condition>]
  This keyword allows setting an attribute for the currently active span.  The
  first argument is the name of the attribute (key) and the rest are its value.
  A value can consist of one or more sample expressions.  If the value is only
  one sample, then the type of that data depends on the type of the HAProxy
  sample.  If the value contains more samples, then the data type is string.
  The data conversion table is below:

   HAProxy sample data type | the OpenTelemetry data type
  --------------------------+----------------------------
            BOOL            |        BOOL
            SINT            |        INT64
            IPV4            |        STRING
            IPV6            |        STRING
            STRING          |        STRING
            METH            |        STRING
            BINARY          |        UNSUPPORTED
  --------------------------+----------------------------

  An optional trailing 'if'/'unless' condition (same syntax as 'otel-event')
  gates the attribute; it is set only when the condition holds at request time.
  The keyword may be repeated to set several attributes on one span, and every
  line whose condition holds is applied.

  Arguments :
    key       - key part of a data pair (attribute name)
    sample    - sample expression (value part of a data pair), at least
                one sample must be present
    condition - an optional ACL-based 'if' or 'unless' condition


baggage <key> <sample> ... [{ if | unless } <condition>]
  Baggage items allow the propagation of data between spans, ie allow the
  assignment of metadata that is propagated to future children spans.  This data
  is formatted in the style of key-value pairs and is part of the context that
  can be transferred between processes that are part of a server architecture.

  This keyword allows setting the baggage for the currently active span.  The
  data type is always a string, ie any sample type is converted to a string.
  The exception is a binary value that is not supported by the OTel filter.

  See the 'attribute' keyword description for the data type conversion table.

  An optional trailing 'if'/'unless' condition (same syntax as 'otel-event')
  gates the baggage.  The keyword may be repeated to set several baggage items
  on one span, and every line whose condition holds is applied.

  Arguments :
    key       - key part of a data pair
    sample    - sample expression (value part of a data pair), at least one
                sample must be present
    condition - an optional ACL-based 'if' or 'unless' condition


event <name> [time [<unit>] <sample>] <key> <sample> ... [{ if | unless } <condition>]
  This keyword allows adding a span event to the currently active span.  A span
  event is a named, timestamped annotation with optional attributes.  The data
  type is always a string, ie any sample type is converted to a string.

  See the 'attribute' keyword description for the data type conversion table.

  An optional 'time [<unit>] <sample>' clause, placed between the event name and
  the first key, overrides the event timestamp.  The unit is 's' (default),
  'ms', 'us' or 'ns', and the sample must evaluate to a number; without it the
  event inherits the span timestamp.  A 'time' token directly after the name is
  read as this clause, so it cannot also serve as an attribute key there.

  An optional trailing 'if'/'unless' condition (same syntax as 'otel-event')
  gates the event.  The keyword may be repeated to add several events to one
  span, and every line whose condition holds is applied.

  Arguments :
    name      - name of the span event
    time      - optional explicit timestamp (with optional unit selector)
    key       - key part of a data pair (attribute name within the event)
    sample    - sample expression (value part of a data pair), at least one
                sample must be present
    condition - an optional ACL-based 'if' or 'unless' condition


extract <name-prefix> [use-vars | use-headers]
  For a more detailed description of the propagation process of the span
  context, see the description of the keyword 'inject'.  Only the process of
  extracting data from the carrier is described here.

  The default carrier is HTTP headers.  If OTEL_USE_VARS is enabled at compile
  time, the 'use-vars' option can be used instead to extract context from
  HAProxy variables.

  When the OpenTelemetry context is transferred via the HTTP header and the
  first character of the context name is '-', the context name is excluded
  from the header.  This behavior enables the use of OpenTelemetry contexts
  with external services that do not expect a HAProxy-specific name prefix on
  the propagation headers.

  Arguments :
    name-prefix - data name prefix (ie key element prefix)
    use-vars    - data is extracted from HAProxy variables
    use-headers - data is extracted from the HTTP header


  Below is an example of using HAProxy variables to transfer span context data:

  --- test/ctx/otel.cfg -----------------------------------------------
      ...
      otel-scope client_session_start_2
          extract "otel_ctx_1" use-vars
          span "Client session" parent "otel_ctx_1"
      ...
  ---------------------------------------------------------------------


finish <name> ...
  Closing a particular span or span context.  Instead of the name of the span,
  there are several specially predefined names with which we can finish certain
  groups of spans.  So it can be used as the name '*req*' for all open spans
  related to the request channel, '*res*' for all open spans related to the
  response channel and '*' for all open spans regardless of which channel they
  are related to.  Several spans and/or span contexts can be specified in one
  line.

  Arguments :
    name - the name of the span or span context


inject <name-prefix> [use-vars] [use-headers]
  In OpenTelemetry, the transfer of data related to the tracing process between
  microservices that are part of a larger service is done through the
  propagation of the span context.  The basic operations that allow us to access
  and transfer this data are 'inject' and 'extract'.

  'inject' allows us to extract span context so that the obtained data can be
  forwarded to another process (microservice) via the selected carrier. 'inject'
  in the name actually means inject data into carrier.  Carrier is an interface
  here (ie a data structure) that allows us to transfer tracing state from one
  process to another.

  Data transfer can take place via one of two selected storage methods, the
  first is by adding data to the HTTP header and the second is by using HAProxy
  variables (the latter requires OTEL_USE_VARS=1 at compile time).  Only data
  transfer via HTTP header can be used to transfer data to another process (ie
  microservice).  All data is organized in the form of key-value data pairs.

  When neither 'use-vars' nor 'use-headers' is given, injection defaults to
  'use-headers'.  A span carries a single context, so 'inject' may appear at
  most once per span; a second 'inject' on the same span is a configuration
  error.

  Header injection (use-headers) is only available on events that provide a
  writable HTTP header carrier when they fire.  Using it on any other event is
  a configuration error, reported at startup.  It is rejected for these events:

    on-stream-start, on-stream-stop, on-idle-timeout, on-backend-set,
    on-client-session-end, on-server-unavailable, on-server-session-start,
    on-tcp-response, on-http-end-request, on-http-end-response, on-http-reply,
    on-server-session-end

  Variable injection (use-vars) carries no such restriction and may be used
  with any event.

  No matter which data transfer method you use, we need to specify a prefix for
  the key element.  Letters, digits and the characters '_', '.' and '-' can be
  used to construct the data name prefix, with uppercase letters converted to
  lowercase when the prefix is created.  The special prefix '-' can be used to
  generate the name automatically from the scope's event name or the span name.
  The generated name keeps a leading '-', so by the rule below the injected
  headers carry only the bare propagation headers, with no HAProxy-specific
  name prefix.

  When the OpenTelemetry context is transferred via the HTTP header and the
  first character of the context name is '-', the context name is excluded
  from the header.  This behavior enables the use of OpenTelemetry contexts
  with external services that do not expect a HAProxy-specific name prefix on
  the propagation headers.

  Arguments :
    name-prefix - data name prefix (ie key element prefix), or '-' for automatic
                  naming
    use-vars    - HAProxy variables are used to store and transfer data
                  (requires OTEL_USE_VARS=1)
    use-headers - HTTP headers are used to store and transfer data


  Below is an example of using HTTP headers and variables to propagate the span
  context.

  --- test/ctx/otel.cfg -----------------------------------------------
      ...
      otel-scope client_session_start_1
          span "HAProxy session" root
              inject "otel_ctx_1" use-headers use-vars
      ...
  ---------------------------------------------------------------------

  Because HAProxy does not allow the '-' character in the variable name (which
  is automatically generated by the OpenTelemetry API and on which we have no
  influence), it is converted to the letter 'D'.  We can see that there is no
  such conversion in the name of the HTTP header because the '-' sign is allowed
  there.  Due to this conversion, initially all uppercase letters are converted
  to lowercase because otherwise we would not be able to distinguish whether the
  disputed sign '-' is used or not.

  Thus created HTTP headers and variables are deleted when executing the
  'finish' keyword or when detaching the stream from the filter.

  Independently of 'inject', a build with OTEL_USE_VARS=1 registers the HAProxy
  variable 'sess.otel.uuid' on every stream the filter attaches to and stores
  the stream's runtime-context UUID in it, so a configuration can read a stable
  per-stream identifier through 'var(sess.otel.uuid)' (several test examples
  use it as a baggage value).  Without this build option the variable does not
  exist and the fetch yields an empty value.


instrument <type> <name> [aggr <aggregation>] [desc <description>] [unit <unit>] value <sample> [bounds <bounds>] [{ if | unless } <condition>]
instrument update <name> [attr <key> <sample> ...] [{ if | unless } <condition>]
  This keyword allows creating or updating metric instruments within the scope.
  Metric instruments record numerical measurements that are exported alongside
  traces.

  To create a new instrument, give the instrument type, a name, and a sample
  expression for the measurement value, preceded by the 'value' keyword.  An
  optional description ('desc') and unit string ('unit') may also be given.  The
  'aggr', 'desc', 'unit', 'value' and 'bounds' keywords may appear in any order.

  An aggregation type can be specified using the 'aggr' keyword followed by one
  of the supported aggregation types listed below.  When specified, a metrics
  view is registered with the given aggregation strategy.  If no aggregation
  type is specified, the SDK default is used.

  For histogram instruments (hist_int), optional bucket boundaries follow the
  'bounds' keyword as a double-quoted string of space-separated numbers, which
  are sorted internally; any duplicate boundaries are then rejected.  With no
  explicit aggregation type set, histogram aggregation is applied automatically.

  To update an existing instrument (a create form defined in any scope, the
  same one included), use 'update' followed by the instrument name.  Optional
  attributes can be added using the 'attr' keyword, followed by a key and a
  sample expression evaluated at runtime.

  Supported instrument types.  The recorded <value> is applied differently by
  each one, so choose the type that matches how the measurement behaves:

    cnt_int (counter, uint64) -- adds <value> to a monotonically rising total.
    Use it for cumulative counts such as requests served or bytes forwarded;
    <value> is the non-negative amount to add, not the running total.  The
    default aggregation is a sum.

    udcnt_int (up-down counter, int64) -- adds a signed <value> to a total that
    may rise and fall.  Use it for levels tracked as deltas, such as in-flight
    requests or queue depth.  The default aggregation is a non-monotonic sum.

    hist_int (histogram, uint64) -- files each <value> as one observation in a
    bucketed distribution.  Use it for value spreads such as request latencies
    or payload sizes, when percentiles matter.  Bucket boundaries follow the SDK
    default unless set with 'bounds', or switched to a base-2 exponential layout
    with 'aggr exp_histogram'.

    gauge_int (gauge, int64) -- replaces the reported value with the latest
    <value>, a snapshot of a level that rises and falls where only the most
    recent reading matters, such as a current queue length.  The default
    aggregation is the last value.

  Supported aggregation types:
    - drop          : measurements are discarded
    - histogram     : explicit bucket histogram
    - last_value    : last recorded value
    - sum           : sum of recorded values
    - default       : SDK default for the instrument type
    - exp_histogram : base-2 exponential histogram

  Observable (asynchronous) instruments are not supported.  The OpenTelemetry
  SDK invokes their callbacks from an external background thread that is not
  a HAProxy thread.  HAProxy sample fetches rely on internal per-thread-group
  state and return incorrect results when called from a non-HAProxy thread.

  Double-precision types are not supported because HAProxy sample fetches do
  not return double values.

  A create form registers the instrument and supplies the value to record, but
  emits no measurement on its own; a data point is produced only when an
  'update' form for that instrument runs.  An optional trailing 'if'/'unless'
  condition (same syntax as 'otel-event') gates the measurement: on a create
  form it gates every recording of the instrument, on an update form only that
  one.  Registration itself produces no data point and is never gated.

  The update form supplies no value of its own, only attributes and an optional
  condition; to record a different quantity, define a separate instrument.  To
  feed a computed or externally-set value, set the create form's 'value' to a
  'var(...)' fetch and assign that variable with 'set-var'.

  For example:
    instrument cnt_int  "my_counter" desc "Counter" value int(1)
    instrument hist_int "my_hist" aggr exp_histogram desc "Latency" value lat_ns_tot unit "ns"
    instrument hist_int "my_hist2" desc "Latency" value lat_ns_tot unit "ns" bounds "100 1000 10000 100000"
    instrument hist_int "req_bytes" value var(txn.otel_size) unit "By"
    instrument update "my_counter" attr "key1" str("val1")

  Arguments :
    type      - the instrument type (see list above)
    name      - the name of the instrument
    aggr      - optional aggregation type (see list above)
    desc      - optional human-readable description of the instrument
    unit      - optional unit string for the instrument
    value     - sample expression providing the measurement value
    bounds    - optional histogram bucket boundaries (hist_int only)
    attr      - attribute key and sample expression (update form only)
    condition - an optional ACL-based 'if' or 'unless' condition


log-record <severity> [id <integer> event <name>] [time [<unit>] <sample>] [span <ref>] [attr <key> <sample>] ... <sample> ... [{ if | unless } <condition>]
  This keyword emits an OpenTelemetry log record within the scope.  The first
  argument is a required severity level.  Optional keywords follow in any order
  before the trailing sample expressions that form the log record body:

    id <integer>           - numeric event identifier
    event <name>           - event name string
    time [<unit>] <sample> - override the log event timestamp at runtime
    span <ref>             - associate the log record with an open span
    attr <key> <sample>    - add an attribute evaluated at runtime (repeatable)

  The 'id' and 'event' keywords must be specified together or both omitted; any
  other combination is a configuration error.  The identifier must be at least
  1; a zero is accepted by the parser but treated as if no 'id' was given.

  The 'time' keyword takes an optional unit (one of 's' (the default), 'ms',
  'us', or 'ns') followed by a single HAProxy sample expression.  The value
  is interpreted as a count since the Unix epoch in the chosen unit, and the
  sample is evaluated at runtime and must yield an integer (or a string that
  is convertible to one).  The value sets the event timestamp; when evaluation
  or numeric conversion fails, the event timestamp defaults to the wall-clock.
  The observed timestamp always records when the filter emitted the record.

  The remaining arguments at the end are sample fetch expressions.  A single
  sample preserves its native type; multiple samples are concatenated as a
  string.  Alternatively, the body may be a HAProxy log-format string wrapped
  in '%[ ... ]'; the outer wrapper is stripped before parsing, and the inner
  string may then contain log-format aliases such as %ci, %t or %ft.

  Supported severity levels follow the OpenTelemetry specification:
    trace, trace2, trace3, trace4, debug, debug2, debug3, debug4,
    info, info2, info3, info4, warn, warn2, warn3, warn4,
    error, error2, error3, error4, fatal, fatal2, fatal3, fatal4

  The log record is only emitted when the logger is enabled for the configured
  severity (controlled by the 'min_severity' option in the YAML logs signal
  configuration).  If a 'span' reference is given but the named span is not
  found at runtime, the log record is emitted without span correlation.

  An optional trailing 'if'/'unless' condition (same syntax as 'otel-event')
  gates the whole record; when it does not pass at runtime, the record is
  skipped entirely, with neither its body nor its attributes evaluated.  The
  keyword may be repeated to emit several records from one scope.

  For example:
    log-record info str("heartbeat")
    log-record info id 1001 event "http-request" span "Frontend HTTP request" attr "http.method" method method url
    log-record trace id 1000 event "session-start" span "Client session" attr "src_ip" src src str(":") src_port
    log-record warn id 1018 event "server-unavailable" str("503 Service Unavailable")
    log-record info id 1002 event "server-session-end" "%[%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts]"
    log-record info time us date(0,us) str("microsecond-accurate event")
    log-record info time ns hdr(x-event-time-ns) str("client-supplied timestamp")

  Arguments :
    severity  - the log severity level (see list above)
    id        - optional numeric event identifier
    event     - optional event name
    time      - optional explicit timestamp (with optional unit selector)
    span      - optional name of an open span to associate with
    attr      - optional attribute key-value pairs (repeatable)
    sample    - sample fetch expression(s) forming the log record body
    condition - an optional ACL-based 'if' or 'unless' condition


link { <ref> ... | <ref> attr <key> <sample> ... }
  This keyword adds span links to the currently active span.  A span link
  represents a causal relationship to another span without establishing a
  parent-child hierarchy.  Links are useful for connecting spans across
  different traces or for associating related spans within the same trace.

  In the first form, multiple span or context names can be specified in one
  line.  Each name is resolved at runtime by searching for an active span or
  an extracted context with that name.  If a referenced span or context cannot
  be found, the link is silently skipped.

  In the second form, a single link reference is followed by the 'attr' keyword
  and one or more <key> <sample> attribute pairs, which attach attributes to
  that one link.  Each pair gives a key and a single sample expression as the
  value; see the 'attribute' keyword description for the data type conversion
  table.

  Because the 'attr' keyword directly after the first name selects the attribute
  form, a link target in that position cannot itself be the literal word 'attr'.

  For example:
    link "Span A" "Span B"
    link "inbound" attr "link.detail" str("follows-from")

  Arguments :
    ref    - the name of a span or span context to link to
    key    - attribute key attached to a single link
    sample - sample expression giving the attribute value


otel-event <name> [{ if | unless } <condition>]
  Set the event that triggers the 'otel-scope' to which it is assigned.  Only
  one 'otel-event' may be defined per scope; a second is a configuration error.
  Optionally, it can be followed by an ACL-based condition, in which case it
  will only be evaluated if the condition is true.

  ACL-based conditions are executed in the context of a stream that processes
  the client and server connections.  To configure and use the ACL, see section
  7 of the HAProxy Configuration Manual.

  Arguments :
    name      - the event name
    condition - a standard ACL-based condition

  Supported events are (the table gives the names of the events in the OTel
  filter and the corresponding equivalent in the SPOE filter):

    -------------------------------------|------------------------------
      the OTel filter                    |  the SPOE filter
    -------------------------------------|------------------------------
      on-stream-start                    |  -
      on-stream-stop                     |  -
      on-idle-timeout                    |  -
      on-backend-set                     |  -
    -------------------------------------|------------------------------
      on-client-session-start            |  on-client-session
      on-frontend-tcp-request            |  on-frontend-tcp-request
      on-http-wait-request               |  -
      on-http-body-request               |  -
      on-frontend-http-request           |  on-frontend-http-request
      on-switching-rules-request         |  -
      on-backend-tcp-request             |  on-backend-tcp-request
      on-backend-http-request            |  on-backend-http-request
      on-http-tarpit-request             |  -
      on-process-server-rules-request    |  -
      on-http-process-request            |  -
      on-tcp-rdp-cookie-request          |  -
      on-process-sticking-rules-request  |  -
      on-http-headers-request            |  -
      on-http-end-request                |  -
      on-client-session-end              |  -
      on-server-unavailable              |  -
    -------------------------------------|------------------------------
      on-server-session-start            |  on-server-session
      on-tcp-response                    |  on-tcp-response
      on-http-wait-response              |  -
      on-process-store-rules-response    |  -
      on-http-response                   |  on-http-response
      on-http-headers-response           |  -
      on-http-end-response               |  -
      on-http-reply                      |  -
      on-server-session-end              |  -
    -------------------------------------|------------------------------

  The next table shows the filter callback that raises each event.  Events bound
  to a channel analyzer fire from channel_pre_analyze (before the analyzer runs)
  or channel_post_analyze (after it finishes).  The rest fire from a lifecycle
  or data-filtering callback and are not tied to an analyzer:

    -------------------------------------|------------------------------
      the OTel filter event              |  raising filter callback
    -------------------------------------|------------------------------
      on-stream-start                    |  stream_start
      on-stream-stop                     |  stream_stop
      on-idle-timeout                    |  check_timeouts
      on-backend-set                     |  stream_set_backend
    -------------------------------------|------------------------------
      on-client-session-start            |  channel_start_analyze
      on-frontend-tcp-request            |  channel_pre_analyze
      on-http-wait-request               |  channel_post_analyze
      on-http-body-request               |  channel_pre_analyze
      on-frontend-http-request           |  channel_pre_analyze
      on-switching-rules-request         |  channel_pre_analyze
      on-backend-tcp-request             |  channel_pre_analyze
      on-backend-http-request            |  channel_pre_analyze
      on-http-tarpit-request             |  channel_pre_analyze
      on-process-server-rules-request    |  channel_pre_analyze
      on-http-process-request            |  channel_pre_analyze
      on-tcp-rdp-cookie-request          |  channel_pre_analyze
      on-process-sticking-rules-request  |  channel_pre_analyze
      on-http-headers-request            |  http_headers
      on-http-end-request                |  http_end
      on-client-session-end              |  channel_end_analyze
      on-server-unavailable              |  channel_end_analyze
    -------------------------------------|------------------------------
      on-server-session-start            |  channel_start_analyze
      on-tcp-response                    |  channel_pre_analyze
      on-http-wait-response              |  channel_post_analyze
      on-process-store-rules-response    |  channel_pre_analyze
      on-http-response                   |  channel_pre_analyze
      on-http-headers-response           |  http_headers
      on-http-end-response               |  http_end
      on-http-reply                      |  http_reply (never invoked)
      on-server-session-end              |  channel_end_analyze
    -------------------------------------|------------------------------

  --- Proxy mode (HTTP and TCP) ---

  The filter attaches to both HTTP-mode and TCP-mode proxies.  On an HTTP proxy
  it runs HTX-level analysis, so every event above is reachable.  A TCP proxy
  carries no HTTP message and runs no HTTP analysis, so the HTTP-phase events
  never fire on one: a scope bound to such an event on a non-HTTP proxy is inert
  and is reported as a warning at config load.  These events fire on a TCP-mode
  proxy:

    stream lifecycle:
      on-stream-start
      on-stream-stop
      on-idle-timeout
      on-backend-set

    request channel:
      on-client-session-start
      on-frontend-tcp-request
      on-switching-rules-request
      on-backend-tcp-request
      on-process-server-rules-request
      on-tcp-rdp-cookie-request
      on-process-sticking-rules-request
      on-client-session-end
      on-server-unavailable

    response channel:
      on-server-session-start
      on-tcp-response
      on-process-store-rules-response
      on-server-session-end

  The remaining events carry HTTP message data and fire only on an HTTP-mode
  proxy:

    request channel:
      on-http-wait-request
      on-http-body-request
      on-frontend-http-request
      on-backend-http-request
      on-http-tarpit-request
      on-http-process-request
      on-http-headers-request
      on-http-end-request

    response channel:
      on-http-wait-response
      on-http-response
      on-http-headers-response
      on-http-end-response
      on-http-reply

  --- Stream lifecycle events (not tied to a channel analyzer) ---

  The on-stream-start and on-stream-stop events fire from the stream_start and
  stream_stop filter callbacks respectively, before any channel processing
  begins and after all channel processing ends.  No channel is available at
  that point, so context injection/extraction via HTTP headers cannot be used
  in scopes bound to these events.  Sample fetches in these scopes are not
  direction-constrained.

  The on-idle-timeout event fires periodically when the stream has no data
  transfer activity.  It requires the 'idle-timeout' keyword to set the
  interval.  This event is useful for heartbeat spans, idle-time metrics, and
  idle-time log records.  It fires from the check_timeouts filter callback
  using HAProxy's tick-based timer infrastructure.

  The timer is driven by the stream task's own expiry, not the request channel's
  analyse_exp, which analysers overwrite freely (the tarpit analyser, for one,
  assigns its own timeout there).  It therefore keeps firing at its interval
  even while an analyser holds the request, such as during a tarpit timeout, and
  the held request still completes.

  The on-backend-set event fires from the stream_set_backend filter callback
  when a backend is assigned to the stream.  It also fires when the stream stays
  on the same proxy, ie when the frontend and the backend are identical.


  --- Request channel events ---

  Analyzer events (tied to AN_REQ_* bits):

  The on-frontend-tcp-request event fires during frontend TCP content inspection
  (AN_REQ_INSPECT_FE).

  The on-http-wait-request event fires after the complete HTTP request has been
  received (AN_REQ_WAIT_HTTP).  This is a post-analyzer event.

  The on-http-body-request event fires when the request body analyzer is reached
  (AN_REQ_HTTP_BODY).  Unlike on-http-wait-request, it is a pre-analyzer event:
  it fires before the analyzer has buffered the body, so the request body is
  not yet available -- body fetches such as req.body and req.body_len evaluate
  to empty or zero.  Treat it as a marker for the start of body processing, not
  a place to inspect the body; the body is buffered only after this analyzer
  completes, and only when the proxy is configured to hold it (for example with
  'option http-buffer-request').

  The on-frontend-http-request event fires during frontend HTTP request
  processing: header rules, monitoring, statistics and redirects
  (AN_REQ_HTTP_PROCESS_FE).

  The on-switching-rules-request event fires when backend switching rules are
  evaluated (AN_REQ_SWITCHING_RULES).

  The on-backend-tcp-request event fires during backend TCP content inspection
  (AN_REQ_INSPECT_BE).

  The on-backend-http-request event fires during backend HTTP request processing
  (AN_REQ_HTTP_PROCESS_BE).

  The on-http-tarpit-request event fires when a request is held by the tarpit
  analyzer until the configured delay elapses (AN_REQ_HTTP_TARPIT).  Unlike the
  other request events, it is reached only for requests that a tarpit rule has
  marked, never for ordinary traffic.

  The on-process-server-rules-request event fires when use-server rules are
  evaluated (AN_REQ_SRV_RULES).

  The on-http-process-request event fires during inner HTTP request processing
  (AN_REQ_HTTP_INNER).

  The on-tcp-rdp-cookie-request event fires when RDP cookie persistence is
  evaluated (AN_REQ_PRST_RDP_COOKIE).

  The on-process-sticking-rules-request event fires when stick-table persistence
  matching rules are evaluated (AN_REQ_STICKING_RULES).

  Non-analyzer events (not tied to AN_REQ_* bits):

  The on-client-session-start event fires when the request channel analysis
  begins.  It corresponds to the start of a new client session.

  The on-http-headers-request event fires from the http_headers filter callback
  after all HTTP request headers have been parsed and analyzed.

  The on-http-end-request event fires from the http_end filter callback when all
  HTTP request data has been processed and forwarded.

  The on-client-session-end event fires when the request channel analysis ends.

  The on-server-unavailable event fires during request channel end-analysis when
  response analyzers were configured but never executed because the server was
  not reached.


  --- Response channel events ---

  Analyzer events (tied to AN_RES_* bits):

  The on-tcp-response event fires during TCP response content inspection
  (AN_RES_INSPECT).

  The on-http-wait-response event fires after the complete HTTP response has
  been received (AN_RES_WAIT_HTTP).  This is a post-analyzer event.

  The on-process-store-rules-response event fires when stick-table store rules
  are evaluated (AN_RES_STORE_RULES).

  The on-http-response event fires during backend HTTP response processing
  (AN_RES_HTTP_PROCESS_BE).

  Non-analyzer events (not tied to AN_RES_* bits):

  The on-server-session-start event fires when the response channel analysis
  begins, after a server connection has been established.

  The on-http-headers-response event fires from the http_headers filter callback
  after all HTTP response headers have been parsed and analyzed.

  The on-http-end-response event fires from the http_end filter callback when
  all HTTP response data has been processed and forwarded.

  The on-http-reply event is meant to fire from the http_reply filter callback
  when HAProxy generates an internal reply (error page, deny response, redirect)
  on the response channel.  In practice it never fires: HAProxy has not invoked
  the http_reply filter callback since version 2.2-dev8 (2020), when its HTTP
  error handling was reworked to build replies through the http_reply_message()
  helper and the 'struct http_reply' type.  The flt_http_reply() dispatcher
  still exists in the filter API but nothing calls it, so the filter callback
  above never runs and the event never triggers.  To trace internally generated
  replies, attach the scope through an 'http-after-response otel-group' action,
  which runs on every response, including the ones that HAProxy synthesises.

  The on-server-session-end event fires when the response channel analysis ends.


otel-stop [{ if | unless } <condition>]
  Stop the OpenTelemetry filter for the current connection.  When this keyword
  is reached, every span and span context still open on the stream is finished
  and the filter is disabled for the rest of that connection: no further scopes
  run for it, whether triggered by a later event or by an 'otel-group' action.
  It is the per-connection counterpart to the global 'flt-otel disable' CLI
  command that disables the filter for every connection at once.

  The stop is evaluated after the rest of the scope has run, so a stopping scope
  still emits the spans, log records and metrics it defines before tracing ends.

  Without a condition the directive always fires.  An optional 'if' or 'unless'
  condition, using the same syntax as the 'otel-event' keyword, makes the stop
  conditional; the filter is left active when the condition is not met.

  Arguments :
    condition - an optional ACL-based 'if' or 'unless' condition

  Example :
    otel-scope skip_healthchecks
        otel-event on-client-session-start
        otel-stop if { path_beg /health }


idle-timeout <time>
  Set the idle timeout interval for a scope bound to the 'on-idle-timeout'
  event.  The timer fires periodically at the given interval when the stream
  is idle.  This keyword is mandatory for scopes using the 'on-idle-timeout'
  event and cannot be used with any other event.  Only one 'idle-timeout' may be
  defined per scope; a second is a configuration error.

  The <time> argument accepts the standard HAProxy time format: a number
  followed by a unit suffix (us, ms, s, m, h, d).  A value of zero is not
  permitted.

  Arguments :
    time - the idle timeout interval (e.g. 5s, 500ms, 1m)

  Example :
    scopes on_idle_timeout
    ..
    otel-scope on_idle_timeout
        idle-timeout 5s
        span "heartbeat" root
            attribute "idle.elapsed" str("idle-check")
        instrument cnt_int "idle.count" value int(1)
        log-record info str("heartbeat")
        otel-event on-idle-timeout


span <name> [parent <ref>] [link <ref>] [root] [kind <kind>]
  Creating a new span (or referencing an already opened one).  If a new span is
  created, it can have a parent reference to another span or context, an inline
  link to another span, or be marked as a root span.  If no reference is
  specified, the new span will become a root span.  We need to pay attention to
  the fact that in one trace there can be only one root span.  If a non-existent
  span is specified as a reference, a new span will not be created.

  The parent reference is set using the 'parent' keyword followed by the name of
  an existing span or extracted context.  An inline link is set using the 'link'
  keyword followed by a span or context name.  The 'root' keyword explicitly
  marks the span as a root span.  The 'kind' keyword sets the OTel span kind,
  one of internal, server (the default), client, producer or consumer.

  Each 'span' opens a sub-context within the scope: the 'attribute', 'event',
  'baggage', 'status', 'link' and 'inject' keywords that follow it apply to that
  span -- the 'currently active span' -- until the next 'span' or the end of the
  scope.

  For example:
    span "HAProxy session" root
    span "Client session" parent "HAProxy session"
    span "HTTP request" parent "TCP request" link "HAProxy session"
    span "Client session" parent "otel_ctx_1"

  An inline link is meant to be given once per 'span' declaration: the parser
  warns from the second one onward, and the argument limit of the 'span' line
  caps how many can fit.  For multiple links, use the standalone 'link' keyword
  described above.

  Arguments :
    name   - the name of the span being created or referenced
             (operation name)
    parent - 'parent <ref>' references an existing span or extracted
             context as the span's parent
    link   - 'link <ref>' adds an inline link to another span or context
    root   - explicitly marks the span as a root span
    kind   - 'kind <kind>' sets the span kind: internal, server (the
             default), client, producer or consumer


status <code> [<sample> ...] [{ if | unless } <condition>]
  This keyword sets the status for the currently active span.  The status
  indicates the outcome of the operation represented by the span.

  The status code is one of the following predefined values:
    - ignore : do not set any status (default)
    - unset  : explicitly mark status as unset
    - ok     : the operation completed successfully
    - error  : the operation resulted in an error

  An optional description can follow the status code, consisting of one or more
  sample expressions whose values are concatenated as a string.  A description
  is meaningful only with the 'error' status, as required by the OpenTelemetry
  specification; the OTLP exporter sets the status message for 'error' and omits
  it for 'ok' and 'unset', so a description on those codes is accepted but not
  exported.

  An optional trailing 'if'/'unless' condition (same syntax as 'otel-event')
  gates the status.  The keyword may be given more than once per span; the lines
  are evaluated at runtime in configuration order, and the first one whose
  condition holds sets the span status, the remaining lines being skipped.  A
  line without a condition always matches, so an unconditional 'status' acts as
  a default and any lines placed after it are never reached.  This first-match
  behavior is specific to 'status'; 'attribute', 'event' and 'baggage' instead
  apply every line whose condition holds.

  For example:
    status "ok"
    status "error" str("http.status_code: ") status
    status "error" str("server error") if { status ge 500 }

  Arguments :
    code      - the status code (ignore, unset, ok, error)
    sample    - optional sample expression(s) for the status description
    condition - an optional ACL-based 'if' or 'unless' condition


exception <type> [message <sample> ...] [attr <key> <sample> ...] [{ if | unless } <condition>]
  This keyword records an exception on the currently active span.  It is emitted
  via the OpenTelemetry record_exception() operation as a span event named
  'exception' that follows the semantic conventions.  HAProxy has no call stack,
  so 'exception.stacktrace' is never set.

  The first argument is the exception type, a literal string stored as the
  'exception.type' attribute.  The optional 'message' keyword introduces sample
  expressions whose values are concatenated into 'exception.message'.  The
  repeatable 'attr <key> <sample>' clause adds further attributes to the event.

  Recording an exception does not change the span status, so pair it with a
  'status error' line when the exception marks the span as failed.  An optional
  trailing 'if'/'unless' condition (same syntax as 'otel-event') gates the
  record.  The keyword may be repeated to record several exceptions on one span.

  For example:
    exception upstream_error message str("HTTP ") status
    exception bad_gateway attr "http.route" path if { status ge 500 }

  Arguments :
    type      - the exception type, stored as 'exception.type'
    message   - sample expression(s) concatenated into 'exception.message'
    key       - the attribute key for an 'attr' clause
    sample    - sample expression(s) for the message or an attribute value
    condition - an optional ACL-based 'if' or 'unless' condition


set-var <var-name> <sample> ... [{ if | unless } <condition>]
  Set a HAProxy variable to the string value of a sample expression or a
  log-format, evaluated when the scope's event fires.  The value grammar matches
  the 'attribute' keyword: bare sample fetches are concatenated and a '%[ ... ]'
  wrapper marks a HAProxy log-format string.  The result is always stored as a
  string.  The variable name includes its scope (proc, sess, txn, req or res)
  and is registered at parse time.  Assignments run before the scope's spans, so
  span attributes, instruments and log records can read the new value through
  'var(...)'.

  An optional trailing 'if'/'unless' condition (same syntax as 'otel-event')
  gates the assignment.  The keyword may be repeated to set several variables in
  one scope, and every line whose condition holds is applied.

  For example:
    set-var txn.otel_method method
    set-var txn.otel_path str("path=") path
    set-var sess.otel_tid var(sess.otel.uuid)
    set-var txn.otel_internal str("1") if { src 127.0.0.1 }

  Arguments :
    var-name  - the HAProxy variable name (including its scope)
    sample    - sample expression(s) or log-format for the value
    condition - an optional ACL-based 'if' or 'unless' condition


set-var-ctx <var-name> <ref> <field> [{ if | unless } <condition>]
  Set a HAProxy variable from a field of a referenced span or span context,
  evaluated when the scope's event fires.  <ref> names a span or a context
  extracted in this scope; it is resolved against the active spans first and
  then the extracted contexts; an unresolved reference leaves the variable
  unchanged.  Because these assignments run after the scope's spans, a span
  created in the same scope may be referenced.

  An optional trailing 'if'/'unless' condition (same syntax as 'otel-event')
  gates the assignment.  The keyword may be repeated within one scope, and every
  line whose condition holds is applied.

  Supported fields:
    trace-id          - trace id, lowercase hex
    span-id           - span id, lowercase hex
    trace-flags       - W3C trace-flags byte, lowercase hex
    traceparent       - full W3C "00-<trace>-<span>-<flags>" string
    tracestate        - full tracestate header (context references only)
    tracestate(<key>) - a single tracestate entry (context references only)
    baggage           - the whole baggage carrier (all entries)
    baggage(<key>)    - a single baggage entry value
    sampled           - "1" if the sampled flag is set, else "0"
    valid             - "1" if the context is valid, else "0"
    remote            - "1" if the context is remote, else "0"

  For 'baggage(<key>)', a span reference reads the value set on that span, while
  a context reference reads the named entry from the inbound carrier.  Keyless
  'baggage' returns the whole carrier instead: the inbound carrier for a context
  reference, or the serialized outbound baggage for a span reference.  The
  'tracestate' fields are available only when <ref> names a context.

  For example:
    extract "inbound" use-headers
    span "HAProxy session" root
    set-var-ctx txn.in_traceid  "inbound" trace-id
    set-var-ctx txn.traceparent "HAProxy session" traceparent
    set-var-ctx txn.user        "inbound" baggage(userId)
    set-var-ctx txn.local_tid   "HAProxy session" trace-id if { src 127.0.0.1 }

  Arguments :
    var-name  - the HAProxy variable name (including its scope)
    ref       - the referenced span or context name
    field     - the span/context field to read (see list above)
    condition - an optional ACL-based 'if' or 'unless' condition


unset-var <var-name> ... [{ if | unless } <condition>]
  Remove one or more HAProxy variables when the scope's event fires.  A variable
  that is not currently set is silently ignored.  Removal runs after all other
  directives in the scope.

  An optional trailing 'if'/'unless' condition (same syntax as 'otel-event')
  gates the whole directive; when it does not pass, none of the listed variables
  are removed.  The keyword may be repeated within one scope.

  For example:
    unset-var txn.otel_scratch txn.otel_tmp
    unset-var txn.otel_scratch if { src 127.0.0.1 }

  Arguments :
    var-name  - the HAProxy variable name(s) to remove (including scope)
    condition - an optional ACL-based 'if' or 'unless' condition


4.4. "otel-group" section
--------------------------

This section defines a group of OTel scopes that is not triggered by an event
but by HAProxy rules in the configuration file -- specifically 'http-request',
'http-response', 'http-after-response', 'tcp-request content' and 'tcp-response
content'.

The action keyword used in these rules is 'otel-group', and it takes the filter
id and the group name as arguments:

  http-response otel-group <filter-id> <group-name> [{ if | unless } ...]


otel-group <name>
  Creates a new OTel group definition named <name>.

  Arguments :
    name - the name of the OTel group


  The following keywords are supported in this section:
    - scopes


scopes <name> ...
  'otel-scope' sections that are part of the specified group are defined.  If
  the mentioned 'otel-scope' sections are used only in some OTel group, they do
  not have to have defined events.  Several 'otel-scope' sections can be
  specified in one line.

  Arguments :
    name - the name of the 'otel-scope' section


In addition to the 'otel-group' action, the filter registers a sample fetch
usable in these rules and in any ACL:

  otel.context(<name>) : boolean

It returns true when the span context named <name> was extracted on the current
stream with the 'extract' keyword and is valid.  When none was extracted, the
fetch produces no sample, so an ACL using '-m found' reads as false, letting
a configuration act directly on whether an upstream context was propagated,
without first copying a context field into a variable with 'set-var-ctx'.

  Arguments :
    name - the name of the span context to check

The filter also exposes the raw payload byte counts accumulated on the stream:

  otel.bytes_in  : integer
  otel.bytes_out : integer

'otel.bytes_in' counts the payload on the request channel (client to HAProxy)
and 'otel.bytes_out' the payload on the response channel (HAProxy to client).
Both grow as data is forwarded, so they are best read from a close scope --
on-client-session-end or on-server-session-end -- where they hold the final
connection totals.  They are populated in TCP mode; on an HTTP proxy they read
0, because an HTX buffer carries protocol framing rather than raw payload.


5. Examples
------------

Several examples of the OTel filter configuration can be found in the test
directory.  A brief description of the prepared configurations follows:

cmp   - a configuration made for comparison purposes with other tracing
        implementations.

sa    - a standalone configuration that exercises most of the events (their
        exact coverage is listed in the test/README-sa file).  It demonstrates
        spans, attributes, events, links, baggage, status and other features,
        and serves as the reference for the speed tests.

full  - a configuration that extends 'sa' with the remaining lifecycle events;
        it covers every filter event except 'on-http-tarpit-request'.

ctx   - a configuration similar to 'sa', with the difference that the spans are
        opened using extracted span contexts as references instead of direct
        parent span names.  This demonstrates the inject/extract context
        propagation mechanism using HAProxy variables.

fe be - a more complex example of the OTel filter configuration that uses two
        cascaded HAProxy services (frontend and backend).  The span context
        between HAProxy processes is transmitted via the HTTP header using
        inject/extract.

tcp   - a TCP-mode (non-HTTP) configuration that traces the raw connection with
        a single session span, recording one event per TCP hook and counting the
        forwarded payload via the 'otel.bytes_in' and 'otel.bytes_out' sample
        fetches.  It exercises every event that fires on a TCP-mode proxy.

updown - an up-down counter configuration in which a 'udcnt_int' instrument
         tracks the number of active client sessions: a signed delta carried
         through a session variable raises the counter by one when a client
         session starts and lowers it by one when the session ends.  A second
         counter tallies the total sessions started, and the metrics are read
         from an ostream exporter file after a clean shutdown.

err   - a configuration that exercises the runtime error-reporting path: its
        response scope parents a span under an 'orphan span' created only by the
        on-server-unavailable scope, so span creation fails on every response
        and the rate-limited error and warning logging runs on ordinary traffic.
        The counters are read with 'flt-otel status'.

empty - an empty configuration in which the OTel filter is initialized but no
        event is triggered.  It is not very usable, except to check the behavior
        of the OTel filter in the case of a similar configuration.


The OTel filter does not use tracer plugins.  Instead, telemetry data is
exported using the OpenTelemetry protocol (OTLP) directly to any compatible
backend.  The backend is configured through the YAML configuration file
specified by the 'config' keyword.

In order to be able to collect and view trace data we need an OpenTelemetry
compatible backend.  There are many options available, including:

  - Jaeger  : https://www.jaegertracing.io/
  - Grafana : https://grafana.com/oss/tempo/
  - Zipkin  : https://zipkin.io/
  - SigNoz  : https://signoz.io/
  - Datadog : https://www.datadoghq.com/

For quick testing, a simple setup using the OpenTelemetry Collector and Jaeger
can be started with Docker:

  # docker run -d --name jaeger -p 4317:4317 -p 4318:4318 -p 16686:16686 jaegertracing/all-in-one:latest

This starts Jaeger with OTLP/gRPC on port 4317, OTLP/HTTP on port 4318 and the
web UI on port 16686.  If we want to use that container later, it can be started
and stopped using the 'docker container start/stop' commands.

The test configurations use a YAML file that defines an OTLP/HTTP exporter
sending data to localhost:4318.  A typical minimal YAML configuration looks like
this:

  --- otel.yml --------------------------------------------------------
  exporters:
    my_exporter:
      type:     otlp_http
      endpoint: "http://localhost:4318/v1/traces"

  samplers:
    my_sampler:
      type: always_on

  processors:
    my_processor:
      type: batch

  providers:
    my_provider:
      resources:
        - service.name: "haproxy"

  signals:
    traces:
      scope_name: "HAProxy OTel"
      exporters:  my_exporter
      samplers:   my_sampler
      processors: my_processor
      providers:  my_provider
  ---------------------------------------------------------------------

In order to use any of the configurations from the test directory, we can run
one of the pre-configured scripts (from within that directory, because they
resolve the HAProxy binary and their configuration files relative to it):

  % ./run-sa.sh
  % ./run-full.sh
  % ./run-ctx.sh
  % ./run-cmp.sh
  % ./run-fe-be.sh
  % ./run-tcp.sh
  % ./run-updown.sh
  % ./run-err.sh
  % ./run-empty.sh


5.1. Benchmarking results
--------------------------

To check the performance impact of the OTel filter, test configurations located
in the test directory have been benchmarked.  The test results (with the names
README-speed-xxx, where xxx is the name of the configuration being tested) are
also in the test directory.

Testing was done with the wrk utility using 8 threads, 8 connections and a
5-minute test duration.  Detailed results and methodology are documented in the
test/README-test-speed file.

Below is a summary of the 'sa' (standalone) configuration results, the heaviest
of the benchmarked scenarios and thus the worst-case figures:

  ---------------------------------------------------------------
   rate-limit     req/s     avg latency     overhead
  ---------------------------------------------------------------
      100.0%     38,202       213.08 us       21.6%
       50.0%     42,777       190.49 us       12.2%
       25.0%     45,302       180.46 us        7.0%
       10.0%     46,879       174.69 us        3.7%
        2.5%     47,993       170.58 us        1.4%
     disabled    48,788       167.74 us          ~0
     off         48,697       168.00 us      baseline
  ---------------------------------------------------------------

The 'off' baseline is measured with the 'filter opentelemetry' directive
commented out, so the filter is not loaded at all.  The 'disabled' level has
the filter loaded but disabled via 'option disabled', so the initialization
overhead is included but no events are processed.

As the table shows, with the rate limit set to 25% the overhead is about 7%.
At 10% the overhead drops to 3.7%.  In typical production use with a rate limit
of 10% or less, the performance impact should be negligible.


6. OTel CLI
------------

Via the HAProxy CLI interface we can find out the current status of the OTel
filter and change several of its settings.

All supported CLI commands can be found in the following way, using the socat
utility with the assumption that the HAProxy CLI socket path is set to
/tmp/haproxy.sock (of course, instead of socat, nc or other utility can be used
with a change in arguments when running the same):

  % echo "help" | socat - UNIX-CONNECT:/tmp/haproxy.sock | grep flt-otel
  --- command output ----------
  flt-otel debug [level]                  : set the OTEL filter debug level (default: get current debug level)
  flt-otel disable                        : disable the OTEL filter
  flt-otel enable                         : enable the OTEL filter
  flt-otel soft-errors                    : disable hard-errors mode
  flt-otel hard-errors                    : enable hard-errors mode
  flt-otel reset-errors                   : reset runtime-error counters
  flt-otel logging [state]                : set logging state (default: get current logging state)
  flt-otel rate [value]                   : set the rate limit (default: get current rate value)
  flt-otel status                         : show the OTEL filter status
  flt-otel flush                          : force-export buffered telemetry now
  flt-otel instruments                    : show configured metric instruments
  flt-otel scopes                         : show configured scopes and groups
  --- command output ----------

'flt-otel debug' can only be used in case the OTel filter is compiled with the
debug mode enabled.  When invoked without arguments, these commands display the
current value of the respective setting.

The 'flt-otel status' command prints a report for each configured filter
instance.  A typical response looks like this (the 'debug level' line and
the 'counters' block appear only in a debug build):

  % echo "flt-otel status" | socat - UNIX-CONNECT:/tmp/haproxy.sock
  --- command output ----------
   opentelemetry filter status
  ------------------------------------------------------------------------------
     library:       C++ 1.26.0, C wrapper 2.2.0-868
     debug level:   0x7f
     export pipeline
       signal   queued  dropped  exported  failed  last export
       traces   0/2048        0   2400/12     0/0  120 ms
       logs     0/2048        0     300/3     0/0  1500 ms
       metrics  -             -       -/5     -/0  9800 ms
     sdk diagnostics: 0

     proxy fe_main, filter otel
       configuration: otel.cfg
       groups/scopes: 2/29

         instrumentation otel-instr
         configuration: otel.yml
         tracer:        active
         meter:         active
         logger:        active
         rate limit:    100.00 %
         hard errors:   no
         disabled:      no
         logging:       enabled
         runtime err:   hard 0, soft 0 (suppressed 0)
         idle timeout:  1000 ms
         analyzers:     01e03fbe

       counters
         attached: run 0, rate-limit 0, disabled 0, error 0
         disabled: scope 0, hard-error 0
  --- command output ----------

Most lines are self-explanatory.  Per signal, the 'export pipeline' table shows
the batch-processor state: 'queued' is the live queue depth over the configured
capacity, 'dropped' the records dropped because the queue was full, 'exported'
and 'failed' the Export() outcomes as records/calls -- the records moved and the
Export() calls that carried them -- and 'last export' the age of the most recent
successful export.  The call counts reflect the Export() return value: a real
delivery result for synchronous exporters such as ostream and otlp_file, but
only acceptance for the async OTLP/HTTP and OTLP/gRPC exporters, which hand the
batch to a background client and report a connection failure only later.  Those
failures instead appear in 'sdk diagnostics', the internal-message count the SDK
reported; for an OTLP exporter a rising 'sdk diagnostics' with a stale 'last
export' is the signature of an unreachable collector.  Metrics use a periodic
reader rather than a queue, so their 'queued' and 'dropped' read '-', and the
records side of 'exported' and 'failed' reads '-' too because the reader has no
per-record count; the call counts and 'last export' track the periodic push,
which runs on its configured interval whether or not any instrument recorded
data, so the 'exported' count climbs even when no instruments are configured.

'idle timeout' is the smallest scope idle timeout in milliseconds (0 means
off), and 'analyzers' is the channel analyzer bitmask the filter registered.
The 'counters' block tallies how streams attached -- run, rate-limited,
disabled or errored -- and how many later had tracing disabled, by a scope
decision (a failed condition or 'otel-stop') or by a hard error.

'runtime err' reports the data-path errors the filter met while processing
streams: 'hard' counts the hard-error episodes that disabled the filter for a
stream, 'soft' the errors that were swallowed so processing continued, and
'suppressed' the log lines the rate limiter held back.  Both kinds are sent to
the per-instance 'log' target -- hard errors at level err, soft errors at level
warning -- but each emission is edge-triggered and rate-limited per instance, so
a fault recurring on every stream yields only a couple of lines per window while
these counters keep the true tally.  'option dontlog-normal' keeps the error
lines but drops the warnings, and 'flt-otel logging off' silences both; the
counters climb either way.

'flt-otel flush' forces every filter instance to export its buffered telemetry
at once, calling force_flush on the active tracer, meter and logger, each one
bounded by a five-second timeout.  It requires admin access.

'flt-otel reset-errors' clears the 'runtime err' counters and the log
rate-limiter state for every filter instance, so a fresh fault is reported
again instead of folding into the running suppressed tally.  It requires admin
access.

'flt-otel instruments' lists the metric instruments declared in each scope.  A
create-form instrument is shown with its type, unit and creation state -- 'not
created' before first use, 'created (idx N)' once the meter has built it (which
happens lazily, on the first record), or 'pending' while another thread builds
it.  An update-form instrument has no unit or creation state of its own, so
only its name and 'update' type are shown:

  % echo "flt-otel instruments" | socat - UNIX-CONNECT:/tmp/haproxy.sock
  --- command output ----------
   opentelemetry filter instruments
  ------------------------------------------------------------------------------

     proxy otel-test-full-frontend, filter otel-test-full
       scope                  instrument               type       unit          state
       on_stream_start        haproxy.sessions.active  udcnt_int  {session}     created (idx 0)
       on_stream_start        haproxy.fe.connections   gauge_int  {connection}  created (idx 1)
       [...]
       frontend_http_request  haproxy.http.latency     update
  --- command output ----------

A filter that has no instruments configured is shown as '(no instruments)'.

'flt-otel scopes' lists the configured scopes, each with its bound event and
whether the configuration uses it, then the groups with their used state and
scope count.  A debug build appends an 'events' section that reports how often
each event was dispatched; events not dispatched so far are left out:

  % echo "flt-otel scopes" | socat - UNIX-CONNECT:/tmp/haproxy.sock
  --- command output ----------
   opentelemetry filter scopes
  ------------------------------------------------------------------------------

     proxy otel-test-full-frontend, filter otel-test-full
       scopes:
         scope            event            used
         on_stream_start  on-stream-start  yes
         [...]
       groups:
         group                      used  scopes
         http_response_group        yes   2
         http_after_response_group  yes   1
       events:
         event            dispatched
         on-stream-start  3
         [...]
  --- command output ----------

Each section is shown as '(no scopes)', '(no groups)' or '(no events)' when it
has no rows.


7. Known bugs and limitations
-------------------------------

The name of the span context definition can contain only letters, numbers and
the characters '_', '.' and '-'.  Also, all uppercase letters in the name are
converted to lowercase.  The character '-' is converted internally to the 'D'
character; since a HAProxy variable is generated from that name, this should
be taken into account if we want to use the variable somewhere in the HAProxy
configuration.  The above mentioned span context is used in the 'inject' and
'extract' keywords.

An inline span link (using the 'link' keyword within a 'span' declaration) is
meant to be given once per span declaration: the parser warns from the second
inline link onward, and the fixed argument count of the 'span' line (maximum 9
arguments) caps how many can fit.  For multiple links, use the standalone 'link'
keyword instead.

Let's look a little at the example test/fe-be (configurations are in the
test/fe and test/be directories, 'fe' is here the abbreviation for frontend and
'be' for backend).  In case we have the 'rate-limit' set to a value less than
100.0, then distributed tracing will not be started with each new HTTP request.
It also means that the span context will not be delivered (via the HTTP header)
to the backend HAProxy process.  The 'rate-limit' on the backend HAProxy must be
set to 100.0, but because the frontend HAProxy does not send a span context
every time, all such cases will cause an error to be reported on the backend
server.  Therefore, the 'hard-errors' option must be set on the backend server,
so that processing on that stream is stopped as soon as the first error occurs.