Refactor Performance Tests and add Grafana, Pyroscope and Prometheus monitoring stack#561
Refactor Performance Tests and add Grafana, Pyroscope and Prometheus monitoring stack#561rendre-greyling wants to merge 9 commits into
Conversation
✅ Deploy Preview for porch ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
rendre-greyling
requested review from
Catalin-Stratulat-Ericsson,
efiacor,
kushnaidu,
liamfallon,
mozesl-nokia and
nagygergo
rendre-greyling
force-pushed
the
perf-test-refactor
branch
2 times, most recently
from
640b561 to
82c1eca
Compare
rendre-greyling
force-pushed
the
perf-test-refactor
branch
from
82c1eca to
80f65a5
Compare
|
Rebased onto PR controller. Will add metrics to the controllers as well. Open to suggestions for any further metrics to track and panels in the grafana dashboard |
|
| } | ||
|
|
||
| // Shutdown gracefully shuts down all OpenTelemetry resources. | ||
| func (r *OTelResources) Shutdown(ctx context.Context) error { |
There was a problem hiding this comment.
Do we need a separate explicit shutdown other than the context cancel for SetupOpenTelemetry? As I've seen that shuts down all providers cleanly.
| return nil | ||
| } | ||
|
|
||
| func startMetricsServerIfConfigured(res *OTelResources) error { |
There was a problem hiding this comment.
Why is this needed? If the otelPort and provider a confiugred correctly, then the otel SDK should be responsible for setting up the server.
| } | ||
| }() | ||
|
|
||
| prof := &metrics.Profiling{} |
There was a problem hiding this comment.
Could Pyroscope and metrics initialisation be moved into the otel package? It could be renamed to telemetry.
2 participants
Refactor Performance Tests and add Grafana, Pyroscope and Prometheus monitoring stack
Description
New Performance Tests and Suite
The previous performance tests were split between 2 performance packages, one under
test/performancefolder, the other undertest/e2e/performancewhich was the Iterative test. These tests were outdated and lacked certain metrics. Features of both have now been consolidated under onetest/performancefolder.There are now 2 new tests which replace the previous tests and functionality:
See the
test/performance/README.mdfor all info on the data that is collected by the tests and customization arguments. The data is stored in csv files for later analysis. The data is also exposed to OTel/Prometheus if the-enable-prometheusis set.New Performance Monitoring Stack
The OTel interface has been expanded to expose various new API metrics as well as Memory and CPU data. A new
deploy-monitoring.shscript has been added which will deploy Grafana, Prometheus and Pyroscope pods in the kind cluster for visual analysis of the data.Grafana has 4 dashboards:
Pyroscope UI can also be accessed by http://localhost:4040/ after running the
deploy-monitoring.shfor more granularity.Type of Change
Checklist
Testing Instructions (Optional)
2-function-runner.yamland3-porch-server.yamlin deployments/porchtest/performance/README.mdAdditional Notes (Optional)
Also added the ability for Gitea to be deployed using a docker mirror with the
DOCKERHUB_MIRRORenvvarAI Disclosure
[x] I have used AI in the creation of this PR.