Separates the storage directives for different worker roles

[Gmerold]

Issue

For instance for Mimir workers: 3 worker types x 3 units of each x 100G for a data volume = 900GAnd from what I'm seeing, the only one that requires a big data volume is mimir-write, then we probably want a little more than 1G for the mimir-backend.

Fixes #183

Solution

Checklist

• I have added or updated relevant documentation.
• PR title makes an appropriate release note and follows conventional commits syntax.
• Merge target is the correct branch, and relevant tandem backport PRs opened.

Context

Testing Instructions

Upgrade Notes

[MichaelThamm]

This makes sense to me. Do not worry about the itests failing, we are resolving those in an upcoming pulse. This implementation somewhat collides with the idea of COS product flavors. E.g. distributed vs. role-all for each coordinated worker so a user can choose to toggle microservices mode in the product. This is not a feature yet, but something to be mindful of.

We still need:

• A tandem PR in Tempo for this

[MichaelThamm]

Now that you have tested the PR, using the temporary refs for: Loki, Mimir, Tempo you will have to:

1. merge the tempo PR
2. revert the refs for Loki, Mimir, Tempo
3. CI for TF testing will fail for READMEs being out of date for Loki and Mimir (because they are local to o11y-stack)
4. It is fine to merge this PR on failing TF lint-terraform-docs.
5. Once merged, run the CI locally on main and confirm this passes

[Gmerold]

@MichaelThamm All merged and cleaned up. Please take one last look, just in case ;)