From b5ab88fab6dc50500195a7ca0efe97ba5166ca16 Mon Sep 17 00:00:00 2001 From: Adrian Edwards Date: Wed, 20 May 2026 15:13:09 -0400 Subject: [PATCH 1/4] Document the types of testing being done and how to run them Signed-off-by: Adrian Edwards --- docs/source/development-guide/testing/toc.rst | 40 ++++++++++++++++++- 1 file changed, 38 insertions(+), 2 deletions(-) diff --git a/docs/source/development-guide/testing/toc.rst b/docs/source/development-guide/testing/toc.rst index cbe469805..dd28c64a7 100644 --- a/docs/source/development-guide/testing/toc.rst +++ b/docs/source/development-guide/testing/toc.rst @@ -1,8 +1,44 @@ Testing =============== -**THIS SECTION IS UNDER CONSTRUCTION.** +CollectOSS aims to have a comprehensive set of tests to enable more rapid iteration and greater confidence that changes have not caused new breakage. + +These tests fall into one of several general types. +* unit tests - standalone tests that are simple to run and test single units of functionality (often individual functions or classes) +* integration tests - small subsystem tests that require bringing up additional pieces, such as redis or a database, to perform the test +* end-to-end tests - complete system tests that require running everything + +Unit Tests +----------- + +Unit tests are implemented via pytest and tagged as ``unit`` to make them easy to run. + +To run the unit tests, clone the CollectOSS repository and run ``uv run pytest -m unit`` + + +Integration Tests +------------------ +Unit tests are also implemented via pytest and tagged as ``integration``. +Because they require additional components, they are not quite as easy to run. + + +To run the integration tests you will need to start up the associated services. This can be done as follows: + +1. Enter the tests directory with ``cd tests/``, this ensures you use the correct dockerfile. +2. Bring up the associated services using the ``docker-compose.yml`` file by running ``docker compose up`` or the podman equivalent. +3. The tests can now be run in a new terminal using ``uv run pytest -m integration`` + +End to End Tests +------------------ + +The end to end tests are currently run as part of a CI job in github actions that is run on pull request. + +The main form of end to end test is the smoke test. This test brings up and runs the full container stack for three minutes. +A script monitors the output logs and looks for specific log statements that indicate that CollectOSS is coming up and behaving as expected. + +Future end to end tests may also run CollectOSS to the point of fully collecting on some smaller repositories and validating that the database is as expected. + If you have questions or would like to help please open an issue on GitHub_. -.. _GitHub: https://github.com/chaoss/collectoss/toss/issues +.. _GitHub: https://github.com/chaoss/collectoss/issues From da2856bc96e743866a39a503ed1b53cd0836600b Mon Sep 17 00:00:00 2001 From: Adrian Edwards Date: Wed, 20 May 2026 17:10:31 -0400 Subject: [PATCH 2/4] Adjust headings so another top level heading can be added Signed-off-by: Adrian Edwards --- docs/source/development-guide/testing/toc.rst | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/docs/source/development-guide/testing/toc.rst b/docs/source/development-guide/testing/toc.rst index dd28c64a7..8d3515b60 100644 --- a/docs/source/development-guide/testing/toc.rst +++ b/docs/source/development-guide/testing/toc.rst @@ -1,15 +1,20 @@ Testing =============== + CollectOSS aims to have a comprehensive set of tests to enable more rapid iteration and greater confidence that changes have not caused new breakage. -These tests fall into one of several general types. + +Types of Testing +----------------- + +The tests of the CollectOSS app fall into one of several general types. * unit tests - standalone tests that are simple to run and test single units of functionality (often individual functions or classes) * integration tests - small subsystem tests that require bringing up additional pieces, such as redis or a database, to perform the test * end-to-end tests - complete system tests that require running everything Unit Tests ------------ +~~~~~~~~~~~ Unit tests are implemented via pytest and tagged as ``unit`` to make them easy to run. @@ -17,7 +22,7 @@ To run the unit tests, clone the CollectOSS repository and run ``uv run pytest - Integration Tests ------------------- +~~~~~~~~~~~~~~~~~~ Unit tests are also implemented via pytest and tagged as ``integration``. Because they require additional components, they are not quite as easy to run. @@ -28,8 +33,8 @@ To run the integration tests you will need to start up the associated services. 2. Bring up the associated services using the ``docker-compose.yml`` file by running ``docker compose up`` or the podman equivalent. 3. The tests can now be run in a new terminal using ``uv run pytest -m integration`` -End to End Tests ------------------- +End to End (E2E) Tests +~~~~~~~~~~~~~~~~~~~~~~~ The end to end tests are currently run as part of a CI job in github actions that is run on pull request. From b4360e3d2463773a5e624f992ab6ff05c1e86fb8 Mon Sep 17 00:00:00 2001 From: Adrian Edwards Date: Wed, 20 May 2026 17:10:51 -0400 Subject: [PATCH 3/4] include slack in testing signoff Signed-off-by: Adrian Edwards --- docs/source/development-guide/testing/toc.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/development-guide/testing/toc.rst b/docs/source/development-guide/testing/toc.rst index 8d3515b60..b9c2c2170 100644 --- a/docs/source/development-guide/testing/toc.rst +++ b/docs/source/development-guide/testing/toc.rst @@ -44,6 +44,6 @@ A script monitors the output logs and looks for specific log statements that ind Future end to end tests may also run CollectOSS to the point of fully collecting on some smaller repositories and validating that the database is as expected. -If you have questions or would like to help please open an issue on GitHub_. +If you have questions about testing in CollectOSS or would like to help please reach out via the `CHAOSS Slack `_ (in the #wg-collectoss-8knot channel) or open an issue on GitHub_. .. _GitHub: https://github.com/chaoss/collectoss/issues From a79b67b432afc81047dce342632e5cb349ffa762 Mon Sep 17 00:00:00 2001 From: Adrian Edwards Date: Wed, 20 May 2026 17:10:58 -0400 Subject: [PATCH 4/4] add a testing standards section Signed-off-by: Adrian Edwards --- docs/source/development-guide/testing/toc.rst | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/docs/source/development-guide/testing/toc.rst b/docs/source/development-guide/testing/toc.rst index b9c2c2170..5a973f209 100644 --- a/docs/source/development-guide/testing/toc.rst +++ b/docs/source/development-guide/testing/toc.rst @@ -44,6 +44,25 @@ A script monitors the output logs and looks for specific log statements that ind Future end to end tests may also run CollectOSS to the point of fully collecting on some smaller repositories and validating that the database is as expected. +Testing Standards +----------------- + +Different parts of the CollectOSS codebase are held to different standards when it comes to how thoroughly changes are expected to be tested/validated before being allowed to merge. + +An approximate, non-exhaustive list of the various levels of testing include: + +* **Code Review** - only a code review is needed to make sure things look okay (spelling/grammar, formatting etc). Typically used for README changes or changes to other simple, non-functional text files in the repo +* **Sanity Check** - a simple, automated check, such as a build job, should be run to ensure that syntax is correct and that the changes aren't causing a build failure. Typically used for documentation (what you are reading now) +* **Automated Functional Test** - A more complex automated check, such as unit tests, integration tests, E2E smoke tests, etc should be run to ensure that CollectOSS can at least start up successfully with the new code. Typically used for trivial changes to subcomponents that already have automated tests +* **Manual Functional Test Procedure** - A set of pre-defined testing steps designed to exercise the specific code/problem being changed. This will usually be derived from the reproduction steps for the bug being solved or documented in the related issue/PR before testing so others can reproduce it. Typically used to test fixes for specific bugs +* **Full Collection Test** - The change should be built and run on a small instance (with relevant repos being added to the collection set if necessary) and the instance should be allowed to run to full collection (all collection stages for all repos marked as "success" in the ``collection_status`` operations table). Typically used for basic/generalized behavior changes +* **Difficult Repo Test** - Either the manual functional test or the full collection test can be made more "difficult" by including one or more known-difficult repositories, such as `chaoss/jank `_ (an artificial repo intended to contain a bunch of examples of problematic git data), or any other repo demonstrating a relevant and extreme/difficult scenario (huge overall size, huge commit count, 50-100k+ commits, etc). Typically used for parsing/performance tests +* **Stress/Scale Test** - the change should be run on an instance (likely pre-existing) with at least 10k diverse repositories for at least one or more full cycles of the collection interval (about 1-2 weeks) to ensure that nothing breaks under load or other scaling-related conditions. Typically used for performance issues, bugs unique to large scale repos, and code thats important enough to require testing on a wide range of different repositories. + +Both the final merge decisions as well as decisions about which level of testing is appropriate for a given PR rests with the project maintainers. + + + If you have questions about testing in CollectOSS or would like to help please reach out via the `CHAOSS Slack `_ (in the #wg-collectoss-8knot channel) or open an issue on GitHub_. .. _GitHub: https://github.com/chaoss/collectoss/issues