Improve Azure CI performance by enabling pip dependency caching

[vinitjain2005]

Currently, the Azure pipeline installs all dependencies from scratch on every run across all matrix jobs (Windows/Linux, multiple Python versions, wheel/normal builds). Given the large scientific dependency stack (NumPy, SciPy, matplotlib, rdkit, etc.), this significantly increases CI runtime.

Azure DevOps provides a built-in Cache@2 task that can cache pip’s download directory across runs. Introducing pip caching keyed by OS and Python version could:

• Reduce CI runtime
• Lower redundant network usage
• Improve overall CI efficiency
• Not change build behavior or test logic

A possible approach would be:

• Cache the pip download directory (e.g., ~/.cache/pip on Linux, %LocalAppData%\pip\Cache on Windows)
• Key the cache on:
• Agent.OS
• PYTHON_VERSION
• Potentially BUILD_TYPE

Example:-

• task: Cache@2
  inputs:
  key: 'pip | "$(Agent.OS)" | "$(PYTHON_VERSION)"'
  path: $(PIP_CACHE_DIR)

I would be happy to experiment with implementing pip caching in a safe and minimal way that does not affect build reproducibility.

Vinit Jain
GitHub:- @vinitjain2005

[unnatikdm]

Hi @vinitjain2005, I'd like to work on implementing pip caching in Azure Pipelines to reduce CI runtime. The current setup installs all scientific dependencies from scratch on every run across all matrix jobs. Using Cache@2 with keys based on OS and Python version would cache pip's download directory (~/.cache/pip on Linux, %LocalAppData%\pip\Cache on Windows), significantly speeding up builds without changing any logic. I've checked and didn't see an existing issue for this. Could you assign me? I'm ready to open an issue and submit a PR.

[unnatikdm]

Hi @vinitjain2005,

I’ve opened a PR for implementing pip caching in Azure Pipelines to speed up CI runs.

Could you please take a look when you get a chance? I’d really appreciate your feedback and a review. If everything looks good, we can proceed further.

Thanks!

[IAlibay]

As detailed in #5267 I am not sure that I agree that significant improvements will be seen here, the install costs are very low compared to the costs associated with running the tests.