no data on Functions explorer dashboard with Rust app

[lcmgh]

First of all, thanks for making life easier :) I just gave it a first try and annotated a fn in my Rust app with autometrics. But in the dashboard no data is shown. Also all quantile metrics are set to 0 even the fn got some calls.

Version: autometrics = { version = "0.5.0", features = ["metrics"] }

1. Data from my apps Prometheus endpoint

# TYPE function_calls_duration summary
function_calls_duration{function="batch_insert",module="myapp::mymod::dao",quantile="0"} 0
function_calls_duration{function="batch_insert",module="myapp::mymod::dao",quantile="0.5"} 0
function_calls_duration{function="batch_insert",module="myapp::mymod::dao",quantile="0.9"} 0
function_calls_duration{function="batch_insert",module="myapp::mymod::dao",quantile="0.95"} 0
function_calls_duration{function="batch_insert",module="myapp::mymod::dao",quantile="0.99"} 0
function_calls_duration{function="batch_insert",module="myapp::mymod::dao",quantile="0.999"} 0
function_calls_duration{function="batch_insert",module="myapp::mymod::dao",quantile="1"} 0
function_calls_duration_sum{function="batch_insert",module="myapp::mymod::dao"} 0.01188134
function_calls_duration_count{function="batch_insert",module="myapp::mymod::dao"} 3

2. My data is scraped
3. Dashboard shows no data

[gagbo]

Hello,

Thanks for the detailed bug report. I don't think I can take a deeper look at it before the end of the week, but a preliminary check I can do is:

• It's weird that the function_calls_duration metric is a summary instead of a histogram (not your fault, and since we set the exporter using buckets here it actually might be a bug/issue in the metrics-prometheus-exporter lib we use for you here) – and to be clear, that's why the dashboard is failing; the queries are made for a histogram metric, not a summary one

Is there any specific reason you chose the metrics feature?

Can you try to use the prometheus feature instead and tell us if you have the same issue? If you don't have the issue anymore, I'll create an issue from here to autometrics-rs as it's a bug with the metrics feature. If you still have the issue, can you show the data sent by the Prometheus endpoint again?

[lcmgh]

Hi @gagbo! Thanks for checking back. I have used the metrics feature because my app has already a Prometheus exporter endpoint.

/// use metrics_exporter_prometheus::PrometheusBuilder;
let prom = PrometheusBuilder::new();
prom
    .install()
    .expect("failed to install recorder/exporter");

I was not sure if that feature exposes the Prometheus endpoint at a port and wanted not to have any interference here.

So switching to prometheus makes the autometrics metrics not appear anymore on my existing exporter endpoint.

After switching back to metrics it is similar to before

function_calls_duration{function="batch_insert",module="myapp::mod::dao",quantile="0"} 0
function_calls_duration{function="batch_insert",module="myapp::mod::dao",quantile="0.5"} 0
function_calls_duration{function="batch_insert",module="myapp::mod::dao",quantile="0.9"} 0
function_calls_duration{function="batch_insert",module="myapp::mod::dao",quantile="0.95"} 0
function_calls_duration{function="batch_insert",module="myapp::mod::dao",quantile="0.99"} 0
function_calls_duration{function="batch_insert",module="myapp::mod::dao",quantile="0.999"} 0
function_calls_duration{function="batch_insert",module="myapp::mod::dao",quantile="1"} 0
function_calls_duration_sum{function="batch_insert",module="myapp::mod::dao"} 0.012257653
function_calls_duration_count{function="batch_insert",module="myapp::mod::dao"} 3

[gagbo]

That's weird too (that using the prometheus feature only doesn't work with the existing metrics setup). Are you using the default registry of Prometheus client library, or a separate custom library?

For the record, the prometheus feautures doesn't try to expose a scrapping endpoint, the prometheus-exporter feature does. It's currently documented under " For projects already using custom Prometheus metrics " in the README, not sure what we can do yet to make that clearer.

To go back on topic, can you instead try to change your prometheus exporter initialization:

/// use metrics_exporter_prometheus::PrometheusBuilder;
let prom = PrometheusBuilder::new();
prom
+    .set_buckets(&[0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1.0, 2.5, 5.0, 7.5, 10.0])
+    .expect("Failed to set histogram buckets")
    .install()
    .expect("failed to install recorder/exporter");

and see if it solves at least¹ the "function_calls_durations as summary" issue?

Footnotes

1. I think this might be incompatible with your existing metrics (because this apparently forces all histograms to be exported as histogram instead of summary), but if that's the case I'd rather handle that in a separate issue ↩

[lcmgh]

I have adapted your changes and the metric endpoint now gives

function_calls_duration_bucket{function="batch_insert",module="myapp::mod::dao",le="0.005"} 2
function_calls_duration_bucket{function="batch_insert",module="myapp::mod::dao",le="0.01"} 3
function_calls_duration_bucket{function="batch_insert",module="myapp::mod::dao",le="0.025"} 3
function_calls_duration_bucket{function="batch_insert",module="myapp::mod::dao",le="0.05"} 3
function_calls_duration_bucket{function="batch_insert",module="myapp::mod::dao",le="0.075"} 3
function_calls_duration_bucket{function="batch_insert",module="myapp::mod::dao",le="0.1"} 3
function_calls_duration_bucket{function="batch_insert",module="myapp::mod::dao",le="0.25"} 3
function_calls_duration_bucket{function="batch_insert",module="myapp::mod::dao",le="0.5"} 3
function_calls_duration_bucket{function="batch_insert",module="myapp::mod::dao",le="0.75"} 3
function_calls_duration_bucket{function="batch_insert",module="myapp::mod::dao",le="1"} 3
function_calls_duration_bucket{function="batch_insert",module="myapp::mod::dao",le="2.5"} 3
function_calls_duration_bucket{function="batch_insert",module="myapp::mod::dao",le="5"} 3
function_calls_duration_bucket{function="batch_insert",module="myapp::mod::dao",le="7.5"} 3
function_calls_duration_bucket{function="batch_insert",module="myapp::mod::dao",le="10"} 3
function_calls_duration_bucket{function="batch_insert",module="myapp::mod::dao",le="+Inf"} 3
function_calls_duration_sum{function="batch_insert",module="myapp::mod::dao"} 0.013500508000000001
function_calls_duration_count{function="batch_insert",module="myapp::mod::dao"} 3

[gagbo]

Yeah, that's better, at least the duration related metrics should work. I still have 2 questions:

• don't you see also a function_calls_count metric that is exposed?
• is the function-specific dashboard showing data at least in the latency graph?

[lcmgh]

• There is indeed a function_calls_count metric (forgot to include it in my previous posts)

# HELP function_calls_count Autometrics counter for tracking function calls
# TYPE function_calls_count counter
function_calls_count{function="batch_insert",module="myapp::mod::dao",caller="",result="ok"} 1

# HELP function_calls_duration Autometrics histogram for tracking function call duration
# TYPE function_calls_duration histogram
function_calls_duration_bucket{function="batch_insert",module="myapp::mod::dao",le="0.005"} 1
function_calls_duration_bucket{function="batch_insert",module="myapp::mod::dao",le="0.01"} 1
function_calls_duration_bucket{function="batch_insert",module="myapp::mod::dao",le="0.025"} 1
function_calls_duration_bucket{function="batch_insert",module="myapp::mod::dao",le="0.05"} 1
function_calls_duration_bucket{function="batch_insert",module="myapp::mod::dao",le="0.075"} 1
function_calls_duration_bucket{function="batch_insert",module="myapp::mod::dao",le="0.1"} 1
function_calls_duration_bucket{function="batch_insert",module="myapp::mod::dao",le="0.25"} 1
function_calls_duration_bucket{function="batch_insert",module="myapp::mod::dao",le="0.5"} 1
function_calls_duration_bucket{function="batch_insert",module="myapp::mod::dao",le="0.75"} 1
function_calls_duration_bucket{function="batch_insert",module="myapp::mod::dao",le="1"} 1
function_calls_duration_bucket{function="batch_insert",module="myapp::mod::dao",le="2.5"} 1
function_calls_duration_bucket{function="batch_insert",module="myapp::mod::dao",le="5"} 1
function_calls_duration_bucket{function="batch_insert",module="myapp::mod::dao",le="7.5"} 1
function_calls_duration_bucket{function="batch_insert",module="myapp::mod::dao",le="10"} 1
function_calls_duration_bucket{function="batch_insert",module="myapp::mod::dao",le="+Inf"} 1
function_calls_duration_sum{function="batch_insert",module="myapp::mod::dao"} 0.002298264
function_calls_duration_count{function="batch_insert",module="myapp::mod::dao"} 1

• Dashboard still shows no data in all three panels

With applied code from #66 (comment).

[gagbo]

I see that the output of the scrapping endpoint looks more like what I'm expecting, so this part is solved.

I'm trying to understand what happened to the dashboard, but I think I'll have to test this and reproduce to understand the issue.

• The fact that the dashboard proposes the batch_insert function name as input shows that at least the part extracting valid function labels is working fine (so the dashboard is well connected to the Prometheus data source, and can see the autometrics metrics)
• The fact that none of the graphs show data make me think that maybe the issue is with our build_info metric and how we use it in the query to generate the graph.

Also, the sampling interval used to show the graph is dynamic, and changes with the timespan shown. Do you still have the issue if you change the "Last 6 hours" in the top-right corner to something significantly shorter like "Last 5 minutes"? I hope that it will give enough points (depending on your scrape_interval) to show a curve

[lcmgh]

Do you still have the issue if you change the "Last 6 hours" in the top-right corner to something significantly shorter like "Last 5 minutes"?

Oh, actually I now have found some lines in the Latency (95th and 99th Percentile) panel but still no data in the other ones. The other ones even show an error but only when going back to 7 days.

execution: found duplicate series for the match group {instance="xxxxx:10250", job="kubelet"} on the right hand-side of the operation: [{__name__="up", endpoint="https-metrics", instance="xxxx:10250", job="kubelet", metrics_path="/metrics/cadvisor", namespace="kube-system", node="xxxxxxxxxxx", service="prometheus-operator-kubelet"}, {__name__="up", endpoint="https-metrics", instance="xxxx", job="kubelet", metrics_path="/metrics/probes", namespace="kube-system", node="xxxxx", service="prometheus-operator-kubelet"}];many-to-many matching not allowed: matching labels must be unique on one side

The fact that none of the graphs show data make me think that maybe the issue is with our build_info metric and how we use it in the query to generate the graph.

build_info{commit="",version="1.1.7",branch=""} 1

[gagbo]

The issue might be specifically triggered by your setup then, it will be hard for me to reproduce it.

Can you try to:

• open the function-level dashboard,
• edit the query for the success rate (or call rate) graph
• edit all the on (instance, job) clauses, and replace them with on (instance, job, metrics_path)

and see if that works? I don't understand how you can have (instance, job) pairs that are not-unique, but it looks like that's what the error message is saying.

[lcmgh]

and see if that works? I

Yes this made it work! The legend says {function="batch_insert", module="myapp::mod::dao", version="1.1.7"}

[gagbo]

Thanks for bearing with me! We always assumed that instance, job would be enough to uniquely identify scrape jobs, but it looks like that assumption was wrong :) Now we'll just need to make sure that the metrics_path label is a built-in default label and present in all versions of Prometheus that matter to our use cases. Thanks again :)

[lcmgh]

It's me who need to be thankful! Then I just adapt my dashboard and my team mates are ready to enjoy the new dashboard :) awesome!

[lcmgh]

Huh, I noticed another thing, the duration of my fn is always shown as 0. I'd expect it to be > 0 when there are data points (hovering over the yellow line also shows 0):

[gagbo]

Oh right, because your function calls probably always last less than the smallest bucket which is 0.005s (on average you have 0.0045 according to this comment).

So if you want to have more granularity there, you should change the buckets in your exporter config:

/// use metrics_exporter_prometheus::PrometheusBuilder;
let prom = PrometheusBuilder::new();
prom
-    .set_buckets(&[0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1.0, 2.5, 5.0, 7.5, 10.0])
+    .set_buckets(&[0.0005, 0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1.0, 2.5])
    .expect("Failed to set histogram buckets")
    .install()
    .expect("failed to install recorder/exporter");

There's no hard requirement on the values in the buckets, just:

• adding buckets adds data in your prometheus instance (that's why I removed the top values here, but you can put them back if you want)
• if you ever want to add latency SLOs later, the latency threshold must be one of the bucket value (so if you want to create a SLO that states "99% of queries in this group must last less than 0.02s", then 0.02 must appear in the buckets)

[lcmgh]

Thanks for making this clear to me. In production I will have longer runtimes so this should be fine :).