Add functional tests for Pod Count feature

[khansaad]

Description

This PR contains tests for the pod count API added as part of #1879
Fixes # (issue)

Type of change

• Bug fix
• New feature
• Docs update
• Breaking change (What changes might users need to make in their application due to this PR?)
• Requires DB changes

How has this been tested?

Please describe the tests that were run to verify your changes and steps to reproduce. Please specify any test configuration required.

• New Test X
• Functional testsuite

Test Configuration

• Kubernetes clusters tested on:

Checklist 🎯

• Followed coding guidelines
• Comments added
• Dependent changes merged
• Documentation updated
• Tests added or updated

Additional information

Include any additional information such as links, test results, screenshots here

Summary by Sourcery

Add functional tests for the v1 recommendations API, covering the new recommendation schema including replicas, nested resource structures, and pod count metrics for container and namespace experiments.

Tests:

• Introduce pytest-based sanity, negative, and extended tests for GET and POST /recommendations v1, validating replicas fields, nested resources, pod_count metrics, and various parameter combinations and failure scenarios.
• Document the new recommendations v1 test coverage in the remote monitoring test README, describing scenarios for pod count, replicas, and multiple experiments.

[sourcery-ai]

Reviewer's Guide

Adds functional tests around the new /recommendations v1 API and pod count-related behavior, including helper wrappers and documentation of the new test coverage.

File-Level Changes

Change	Details	Files
Introduce helper wrappers for the v1 recommendations API used by remote monitoring tests.	Import the v1 recommendations API path constant from the new test module into the shared utils helper module. Add list_recommendations_v1 helper to perform GET /recommendations with optional experiment_name and monitoring_end_time parameters. Add generate_recommendations_v1 helper to perform POST /recommendations with experiment_name, interval times, and target parameters.	tests/scripts/helpers/utils.py
Document new Recommendations API (v1) functional test scenarios, including pod_count and replicas validation.	Describe sanity, negative, and extended test scenarios for GET /recommendations v1 focusing on new schema, replicas, nested resources, and pod_count metrics. Describe sanity, negative, and extended test scenarios for POST /recommendations v1 including target handling, time range parameters, and pod_count aggregation.	tests/scripts/remote_monitoring_tests/Remote_monitoring_tests.md
Add pytest-based functional tests for the v1 recommendations API validating new schema, pod count metrics, and error handling.	Define RECOMMENDATIONS_API_V1 path and import shared helpers to create experiments, update results, and form the Kruize URL. Add sanity tests for GET /recommendations v1 for single, namespace, and multiple experiments, verifying api_version, kubernetes_objects, replicas fields, nested resources, and podCount metrics and aggregation_info. Add sanity tests for POST /recommendations v1 with target=remote/local and default target, including long-running result ingestion and validation of recommendations presence. Add negative tests for invalid target, missing experiment_name, missing interval_end_time, invalid experiment names, and invalid timestamps, asserting ERROR_STATUS_CODE and message contents where appropriate.	tests/scripts/remote_monitoring_tests/rest_apis/test_recommendation_resource_api.py

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[khansaad]

@sourcery-ai review

[khansaad]

Archive.zip
Test Results

[chandrams]

@khansaad I don't see any validation for pod count here, can you please point me to where it is being validated

[khansaad]

Added now

[chandrams]

@khansaad - If this new v1/recommendations is going to be replacing the older API, all existing recommendations tests should work with this new v1/recommendations API too.

Can you run the scale test as well using this new API?

[khansaad]

@khansaad - If this new v1/recommendations is going to be replacing the older API, all existing recommendations tests should work with this new v1/recommendations API too.

Can you run the scale test as well using this new API?

Summary of the test run
exp_count / results_count / reco_count = 5000 / 7200000 / 300000
Postgres DB size in MB = 22034
Results count - 7200000
total_kruizeMetrics-20.csv
Update Reco Latency Max / Avg value: 0.89 / 0.49
Update Results Latency Max / Avg value: 0.14 / 0.11
LoadResultsByExpName Latency Max / Avg value: 0.46 / 0.32
Generate Plots Latency Max / Avg value: 0.0 / 0.0
Kruize memory Max value: 44.27 GB
Kruize cpu Max value: 7.92
Execution time - 04:36:12

https://ci.app-svc-perf.corp.redhat.com/job/ExternalTeams/job/Autotune/job/kruize_scale_tests/126/

[chandrams]

@khansaad I don't see this being called for local monitoring reco validation

[khansaad]

For local monitoring, the validate_config_local_monitoring method is being used instead for both current and recommended config.

[chandrams]

@khansaad Can you please trigger the runs on jenkins and share the results after the latest changes

[khansaad]

@khansaad Can you please trigger the runs on jenkins and share the results after the latest changes

Started this run : https://ci.app-svc-perf.corp.redhat.com/job/ExternalTeams/job/Autotune/job/kruize_functional_tests/552/