Review and document the CI/CD matrix

[afrittoli]

I'm relatively new to the team, so forgive me if this already exists in some format.

I would like to propose documenting and reviewing the CI/CD matrix, something along these lines (these are just examples):

Job	Backend	Pythons	OS	Arch	Flavours	Frequency	Special Resources	Weight / Cost	Run filter
Linter					With/without PyTorch	PR		10
Magma Build			Linux			On PR		30	Magma Build Files
Magma Build			Windows			On demand	Windows Node	30	Magma Build Files
E2E tests	CUDA (4)	3.9 - 3.12 (4)	Focal, Jammy		Clang 10, 12	On PR	Nvidia card	50	Code
Build	ROCM	ROCM	3.9 - 3.12 (4)	Manylinux	arm64, amd64. s390x	nightly

Perhaps this can be programmatically extracted from hud?

There may be space for optimisation without renouncing quality.
It would be a useful tool when adding or removing a CI job or a build in understanding the impact it would have.

[malfet]

We have some automatic analysis for CI jobs cost at https://hud.pytorch.org/cost_analysis

[afrittoli]

Thanks @malfet, the CI jobs cost analysis looks really good, and it works across repositories.
It's definitely better than a statically written matrix. Is the source code for the tool available, is it possible for the community to contribute to it?

[ZainRizvi]

Here's the code for the page:
https://github.com/pytorch/test-infra/blob/main/torchci/pages/cost_analysis.tsx

This PR shows where the data itself comes from (it's all stored on ClickHouse)
https://github.com/pytorch/test-infra/pull/5736/files

cc @wdvr