Add TTFB metric logic and report it to cloudflare analytics

[ankur12-1610]

See filbeam/roadmap#76

[bajtos]

Thank you for the pull request, @ankur12-1610 .

Fixes: filbeam/roadmap#76

That's not correct. Submitting the TTFB data to CF analytics is only the first step, the next step is to visualise them in our dashboards. I updated the description.

I am not familiar with CF analytics, don't we need some sort of authentication for writing data? Or does the URL in CLOUDFLARE_ANALYTICS_URL already include the credentials?

@juliangruber We were discussing using InfluxDB for these analytics. Do you have an opinion whether we are okay to use CF analytics instead?

• Writer is the bot running on Fly.io, so it makes no difference where it writes to
• We will have two readers - the public dasbhoard (Observable build running on CF) and our internal Grafana dashboards

Thoughts?

[juliangruber]

I'm fine with CF analytics, or at least I don't have any concerns for now

[ankur12-1610]

Thank you for the pull request, @ankur12-1610 .

Fixes: filbeam/roadmap#76

That's not correct. Submitting the TTFB data to CF analytics is only the first step, the next step is to visualise them in our dashboards. I updated the description.

I am not familiar with CF analytics, don't we need some sort of authentication for writing data? Or does the URL in CLOUDFLARE_ANALYTICS_URL already include the credentials?

@juliangruber We were discussing using InfluxDB for these analytics. Do you have an opinion whether we are okay to use CF analytics instead?

• Writer is the bot running on Fly.io, so it makes no difference where it writes to
• We will have two readers - the public dasbhoard (Observable build running on CF) and our internal Grafana dashboards

Thoughts?

Hey @bajtos, yes—you’re right, this is just the first step of the issue. I wanted to make sure this part was set up correctly before moving on.

I’ve set up a worker that’s public and accepts any request, then writes the dataset (for testing purpose, which I'll be changing to some kind of secret key based authorisation, after this comment):

// analytics-worker.js
var analytics_worker_default = {
  async fetch(request, env) {
    if (request.method !== "POST") {
      return new Response("Send POST with TTFB data", { status: 405 });
    }
    try {
      const data = await request.json();
      env["analytics-engine"].writeDataPoint({
        blobs: data.blobs,   // [url, location, client, cid]
        doubles: data.doubles // [ttfb, status, bytes]
      });
      return new Response(JSON.stringify({ success: true }), {
        headers: { "Content-Type": "application/json" }
      });
    } catch (error) {
      return new Response(JSON.stringify({
        success: false,
        error: error.message
      }), {
        status: 500,
        headers: { "Content-Type": "application/json" }
      });
    }
  }
};
export {
  analytics_worker_default as default
};

[bajtos]

I’ve set up a worker that’s public and accepts any request, then writes the dataset (for testing purpose, which I'll be changing to some kind of secret key based authorisation, after this comment)

This is IMO adding too much complexity. InfluxDB is simpler to set up because it provides an HTTP API for writing data points.

However, I don't feel too strongly about this. If @juliangruber thinks it's fine to add another worker to record data points to Workers Analytics Engine, then I am okay with following your lead.

I suggest starting with a pull request creating the worker recording data points, so that we can discuss the desired API & request body format, before we move to changing the bot to submit data points via that new API.

[juliangruber]

I'm fine adding a new worker, I don't see worker count as a decisive factor at this point. +1 to opening a PR for the worker, so we can discuss the full scope of the implementation

[ankur12-1610]

@bajtos @juliangruber

Should I keep the worker code in the new PR as it is:

// analytics-worker.js
var analytics_worker_default = {
  async fetch(request, env) {
    if (request.method !== "POST") {
      return new Response("Send POST with TTFB data", { status: 405 });
    }
    try {
      const data = await request.json();
      env["analytics-engine"].writeDataPoint({
        blobs: data.blobs,   // [url, location, client, cid]
        doubles: data.doubles // [ttfb, status, bytes]
      });
      return new Response(JSON.stringify({ success: true }), {
        headers: { "Content-Type": "application/json" }
      });
    } catch (error) {
      return new Response(JSON.stringify({
        success: false,
        error: error.message
      }), {
        status: 500,
        headers: { "Content-Type": "application/json" }
      });
    }
  }
};
export {
  analytics_worker_default as default
};

or should I update it to use secret-based authentication?

[juliangruber]

Let's discuss in that PR please

[ankur12-1610]

Let's discuss in that PR please

@juliangruber @bajtos created a new PR, please take a look at #34