Adding explicit call for apicast-production metrics

[cathal-bailey]

• The test queries Prometheus for metrics for "apicast-staging" and "apicast-production"
• Since production has never received HTTP traffic, request-dependent metrics don't exist and the test fails
• For production metrics, promotes staging config to production and sends a GET request to /get endpoint
• This generates the missing metrics (apicast_status, upstream_status, threescale_backend_calls, and response time histograms - 5 tests now passing)
• prometheus.wait_on_next_scrape so that metrics appear in db, as seen elsewhere for prometheus/apicast in testsuite
• All metrics passing on Alpha with this change, but considerable increase in test time (now ~4 mins)
• Marking apicast-prodiuction as disruptive to ensure apicast restart does not impact other tests.
• Adding idempotent check to prod_client fixture to prevent duplicate promotion errors

[mdujava]

not liking this code reminded me that .promote on a apicast-production will cause restart of apicast, which can affect running tests in parallel, so apicast-production would need to set pytest.mark.disruptive

[cathal-bailey]

Okay understood, my thinking is that .promote is required to in order for the GET request to work.
So would it be safer to split-out test for production metrics and mark disruptive to facilitate the .promote? I think we can leave staging as is, there doesn't seem to be any issues there atm.

[mdujava]

why do you need to promote it by hand? cant you just use api_client or prod_client (depenging on request.param)?

It might be better to just call either way and add # TODO: and github issue to look at this closer later.

[mdujava]

I would not change this, prod_client should throw exception if promote=True and it was already promoted.
this is used across whole testsuite and this behavior should stay as is.

[cathal-bailey]

Gotcha, this change is a proposed fix for an issue I was seeing when I ran the test locally. test_apicast_status_metrics was failing with as it tried to promote production metrics that had already been promoted by metrics. The prod_client change resolved this. I'll add this check into the test file instead so I don't have to touch the contest.py.