Test that pings are deleted following deletion-request

[hackebrot]

Verify that the Data Platform deletes Glean telemetry for a client.

This test is slightly different from other tests as we may need to run a sequence of steps:

• produce Glean telemetry
• verify that it's available in BigQuery tables
• submit deletion-request ping
• verify that the telemetry for this client has been deleted

[jklukas]

verify that the telemetry for this client has been deleted

Note that the pipeline runs shredder on a 28-day schedule, so it could take several weeks for a deletion request to result in pings being deleted, which could make this a difficult case to test.

[hackebrot]

Do you think it makes sense to create a new DAG on a schedule that matches the shredder pipeline schedule?

[jklukas]

Do you think it makes sense to create a new DAG on a schedule that matches the shredder pipeline schedule?

Yes, we could pursue that option. I'm not sure if there's a good way to express "wait several weeks" before running the query task to validate the relevant data is gone. We might need to have two DAGs where the first is for submitting a test deletion request, and the other is offset by several weeks, running the query to validate. Or perhaps we have one DAG, and the task that validates data has been deleted references the previous DAG run.

In any of these cases, the long delays are going to make it very hard to iterate on the tests, particularly when they're first set up.

[mdboom]

Also useful would be to test that once we see a deletion-request ping, we don't see any further pings from that client.

[hackebrot]

Relevant documentation https://mozilla.github.io/glean/book/user/pings/deletion_request.html

The following code is expected to trigger a deletion-request ping:

Glean.set_upload_enabled(False)

Can you confirm that @mdboom, please?

[badboy]

On @mdboom's behalf:
Yes, disabling upload triggers a deletion-request ping (and removes other pending pings and resets the client ID)

[hackebrot]

I created #77 for your suggestion @mdboom.