http.host as a trace metric

[ericallen]

Is it possible to include the http.host value as a trace metric for nginx's that server multiple sites on it? Currently you can create a metrics from sampled traces and pull that value out. Mostly wondering if it is possible to include it into the datadog default metrics that are create on agent.

Or would be the best solution would be to set a different datadog_service for each site in nginx.conf?

[dgoffredo]

Hi, Eric.

There are three possible places where span-based metrics might be published:

1. From the tracing library itself, e.g. nginx sending data to dogstatsd. This is not supported.
2. From the Datadog Agent. This is what publishes the existing metrics that you mentioned.
3. From the Datadog UI. This is a feature that you also mentioned.

I looked at the Datadog Agent's configuration template to see whether there are options for defining metrics from spans. I do not see any relevant options.

What sort of queries/dashboards/studies are you trying to facilitate by integrating http.host into metrics? Maybe we can find a solution.

[ericallen]

Ultimately i'm trying to do different SLOs for each public facing site. Currently we host a number of different sites on the same nginx server like site1.example.com, api.example.com, site2.example.com, etc... We would like to separate the metrics out by http.host or external site, so that we can get a good breakdown of the health of that one site instead of everything grouped together. One of our sites does websockets, which stay open for a long period of time which then skews the response times for other sites like the restapi, when they are calculated together.

We have tried option 3 that you suggested, it is just the metrics off of the sampled traces don't generate a consistent metrics. The errors are heavily favored which then will change the SLOs so they aren't accurate.

We also tried using the datadog_tag on each server block in nginx but that will only show up in the trace but not the metric. Which I think is confirmed with the line Note: Tags set on spans do not count and will not be available as tags for your traces metrics. from existing metrics

I was hoping for an option maybe in the datadog agent that could add dynamic tags to metrics but I don't think that is possible.

I think the only solution would be to create out each server in nginx to be a different service like datadog_service_name "nginx-site1"; and datadog_service_name "nginx-api";. But i'm not positive that will work since the documentation says that datadog_service_name is only valid in the nginx http block.

[dgoffredo]

As a hack, you might be able to use datadog_operation_name to change the operation name (span name) of spans created by nginx. You could define the operation name to include nginx's $host variable. Trace metrics then contain <SPAN NAME> as part of the metric name itself. I don't know whether part of the name of a metric can be considered a variable in our metrics product, but if so then it is worth considering.

http {
  datadog_operation_name nginx.$host;

  server {
      listen       80;
      server_name  site1.example.org;
      ...
  }

  server {
      listen       80;
      server_name  site2.example.com;
      ...
  }
}

I haven't tried this, and I don't like it. Might break things in your alerts and such.

I'll ask within Datadog whether adding another dimension (tag) to the trace metrics is possible. You're likely not the first to ask.