Add new command 'launchable compare subsets' by gayanW · Pull Request #1107 · cloudbees-oss/smart-tests-cli

gayanW · 2025-08-14T06:56:22Z

This adds a new command launchable compare subsets which can be used to compare two subset files.

 % launchable compare subsets --help
Usage: launchable compare subsets [OPTIONS] FILE_BEFORE FILE_AFTER

  Compare two subset files and display changes in test order positions.

Options:
  --help  Show this message and exit.

Example usage:

 % pr -m -t subset-before.txt subset-after.txt 
src/test/java/example/DivTest.java  src/test/java/example/Add2Test.java
src/test/java/example/DB1Test.java  src/test/java/example/MulTest.java
src/test/java/example/MulTest.java  src/test/java/example/AddTest.java
src/test/java/example/Add2Test.java src/test/java/example/File1Test.jav
src/test/java/example/File1Test.jav src/test/java/example/DivTest.java
src/test/java/example/File0Test.jav src/test/java/example/File0Test.jav
src/test/java/example/SubTest.java  src/test/java/example/DB1Test.java
src/test/java/example/DB0Test.java  src/test/java/example/DB0Test.java
src/test/java/example/AddTest.java  src/test/java/example/SubTest.java

% launchable compare subsets subset-before.txt subset-after.txt
|   Before |   After |   Order Change | Test Path                            |
|----------|---------|----------------|--------------------------------------|
|        9 |       3 |             -6 | src/test/java/example/AddTest.java   |
|        2 |       7 |             +5 | src/test/java/example/DB1Test.java   |
|        1 |       5 |             +4 | src/test/java/example/DivTest.java   |
|        4 |       1 |             -3 | src/test/java/example/Add2Test.java  |
|        7 |       9 |             +2 | src/test/java/example/SubTest.java   |
|        3 |       2 |             -1 | src/test/java/example/MulTest.java   |
|        5 |       4 |             -1 | src/test/java/example/File1Test.java |
|        6 |       6 |             +0 | src/test/java/example/File0Test.java |
|        8 |       8 |             +0 | src/test/java/example/DB0Test.java   |

When there're new test paths in the subset-after.txt, new file paths will be a marked as 'NEW':

% pr -m -t subset-before.txt subset-after.txt 
src/test/java/example/SubTest.java  src/test/java/example/NewTest.java
src/test/java/example/DivTest.java  src/test/java/example/SubTest.java
src/test/java/example/Add2Test.java src/test/java/example/File0Test.jav
src/test/java/example/File0Test.jav src/test/java/example/DB1Test.java
src/test/java/example/AddTest.java  src/test/java/example/DivTest.java
src/test/java/example/File1Test.jav src/test/java/example/MulTest.java
src/test/java/example/MulTest.java  src/test/java/example/File1Test.jav
src/test/java/example/DB0Test.java  src/test/java/example/DB0Test.java
src/test/java/example/DB1Test.java  src/test/java/example/Add2Test.java
				                    src/test/java/example/AddTest.java

% launchable compare subsets subset-before.txt subset-after.txt
| Before   |   After | Order Change   | Test Path                            |
|----------|---------|----------------|--------------------------------------|
| -        |       1 | NEW            | src/test/java/example/NewTest.java   |
| 3        |       9 | +6             | src/test/java/example/Add2Test.java  |
| 9        |       4 | -5             | src/test/java/example/DB1Test.java   |
| 5        |      10 | +5             | src/test/java/example/AddTest.java   |
| 2        |       5 | +3             | src/test/java/example/DivTest.java   |
| 1        |       2 | +1             | src/test/java/example/SubTest.java   |
| 4        |       3 | -1             | src/test/java/example/File0Test.java |
| 7        |       6 | -1             | src/test/java/example/MulTest.java   |
| 6        |       7 | +1             | src/test/java/example/File1Test.java |
| 8        |       8 | +0             | src/test/java/example/DB0Test.java   |

When there're deleted test paths between the two subsets, deleted test paths will be marked as 'DELETED':

pr -m -t subset-before.txt subset-after.txt
src/test/java/example/NewTest.java  src/test/java/example/DB1Test.java
src/test/java/example/SubTest.java  src/test/java/example/DB0Test.java
src/test/java/example/File0Test.jav src/test/java/example/File1Test.jav
src/test/java/example/DB1Test.java  src/test/java/example/SubTest.java
src/test/java/example/DivTest.java  src/test/java/example/AddTest.java
src/test/java/example/MulTest.java  src/test/java/example/MulTest.java
src/test/java/example/File1Test.jav src/test/java/example/File0Test.jav
src/test/java/example/DB0Test.java  src/test/java/example/Add2Test.java
src/test/java/example/Add2Test.java src/test/java/example/DivTest.java
src/test/java/example/AddTest.java

% launchable compare subsets subset-before.txt subset-after.txt
|   Before | After   | Order Change   | Test Path                            |
|----------|---------|----------------|--------------------------------------|
|        1 | -       | DELETED        | src/test/java/example/NewTest.java   |
|        8 | 2       | -6             | src/test/java/example/DB0Test.java   |
|       10 | 5       | -5             | src/test/java/example/AddTest.java   |
|        7 | 3       | -4             | src/test/java/example/File1Test.java |
|        3 | 7       | +4             | src/test/java/example/File0Test.java |
|        5 | 9       | +4             | src/test/java/example/DivTest.java   |
|        4 | 1       | -3             | src/test/java/example/DB1Test.java   |
|        2 | 4       | +2             | src/test/java/example/SubTest.java   |
|        9 | 8       | -1             | src/test/java/example/Add2Test.java  |
|        6 | 6       | +0             | src/test/java/example/MulTest.java   |

ono-max · 2025-08-15T00:22:07Z

launchable/commands/compare/subsets.py

+    changes.sort(key=lambda x: (abs(x[3]) if isinstance(x[3], int) else float('inf')), reverse=True)
+
+    # Display results in a tabular format
+    headers = ["Before", "After", "Order Change", "Test Path"]


Probably, "Order Change" sounds a little bit unclear to me. Can we rename it to the following names?

After - Before

Order Δ

Order Diff

Diff (After - Before)

Δ (After - Before)

I'll change it to 'After - Before'

ono-max · 2025-08-15T00:28:48Z

launchable/commands/compare/subsets.py

+        after_tests = f.read().splitlines()
+    after_index_map = {test: idx for idx, test in enumerate(after_tests)}
+
+    changes = []


Can you please define the type for this variable?

e.g.

smart-tests-cli/launchable/utils/http_client.py

Lines 24 to 25 in e1ff1d3

DEFAULT_TIMEOUT: Tuple[int, int] = (5, 60)

DEFAULT_GET_TIMEOUT: Tuple[int, int] = (5, 15)

kohsuke · 2025-08-15T00:33:58Z

With this command, we are trying to draw attention to tests that boosted in ranks. So it is critical that we sort the result like that. IOW, sort them in the ascending order of diff.

kohsuke · 2025-08-15T00:46:37Z

"Test path" is a technical term in our product to refer to a certain structured test name, so I think it's best to avoid that, since what we are showing is not that.

Maybe just "test"?

gayanW · 2025-08-15T00:56:40Z

"Test path" is a technical term in our product to refer to a certain structured test name, so I think it's best to avoid that, since what we are showing is not that.

Maybe just "test"?

I think we can name it 'Test'

Btw these are the names used in inspect subset command

smart-tests-cli/launchable/commands/inspect/subset.py

Line 57 in e1ff1d3

header = ["Order", "Test Path", "In Subset", "Estimated duration (sec)"]

ono-max · 2025-08-15T01:05:29Z

launchable/commands/compare/subsets.py

+        if test in before_index_map:
+            before_idx = before_index_map[test]
+            order_diff = after_idx - before_idx
+            changes.append((test, before_idx + 1, after_idx + 1, order_diff))


May I ask why you're using tuple, instead of map?

I think tuple is a suitable choice for representing a fixed-size row in a table. It should be also slightly more efficient compared to a map in this case.

I declared the variable with types, and also added a comment to make it clear on what goes in the tuple.

gayanW · 2025-08-15T03:38:37Z

I updated the PR so that the results are sorted in ascending order of the diff value.
And changed the column name to 'Test'.

% launchable compare subsets subset-before.txt subset-after.txt         
|   Before |   After |   After - Before | Test                                 |
|----------|---------|------------------|--------------------------------------|
|        9 |       3 |               -6 | src/test/java/example/AddTest.java   |
|        4 |       1 |               -3 | src/test/java/example/Add2Test.java  |
|        3 |       2 |               -1 | src/test/java/example/MulTest.java   |
|        5 |       4 |               -1 | src/test/java/example/File1Test.java |
|        6 |       6 |               +0 | src/test/java/example/File0Test.java |
|        8 |       8 |               +0 | src/test/java/example/DB0Test.java   |
|        7 |       9 |               +2 | src/test/java/example/SubTest.java   |
|        1 |       5 |               +4 | src/test/java/example/DivTest.java   |
|        2 |       7 |               +5 | src/test/java/example/DB1Test.java   |

kohsuke

Woot!

gayanW added 2 commits August 14, 2025 15:06

Add new command 'launchable compare subsets'

a3e6b12

Refactor 'compare subsets' command to display deleted tests

3d1fabe

gayanW requested review from kohsuke and ono-max August 14, 2025 06:56

Merge methods in commands/compare/subsets.py for readability

65699d7

ono-max reviewed Aug 15, 2025

View reviewed changes

Refactor 'compare subsets' to sort results in ASC order

6e82703

kohsuke approved these changes Aug 15, 2025

View reviewed changes

kohsuke merged commit 4e46793 into main Aug 15, 2025
13 checks passed

kohsuke deleted the AIENG-217 branch August 15, 2025 15:18

github-actions bot mentioned this pull request Aug 14, 2025

Release for v1.110.1 #1100

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add new command 'launchable compare subsets'#1107

Add new command 'launchable compare subsets'#1107
kohsuke merged 4 commits intomainfrom
AIENG-217

gayanW commented Aug 14, 2025

Uh oh!

ono-max Aug 15, 2025 •

edited

Loading

Uh oh!

gayanW Aug 15, 2025

Uh oh!

ono-max Aug 15, 2025 •

edited

Loading

Uh oh!

kohsuke commented Aug 15, 2025

Uh oh!

kohsuke commented Aug 15, 2025

Uh oh!

gayanW commented Aug 15, 2025

Uh oh!

ono-max Aug 15, 2025

Uh oh!

gayanW Aug 15, 2025

Uh oh!

gayanW commented Aug 15, 2025 •

edited

Loading

Uh oh!

kohsuke left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	DEFAULT_TIMEOUT: Tuple[int, int] = (5, 60)
	DEFAULT_GET_TIMEOUT: Tuple[int, int] = (5, 15)

Conversation

gayanW commented Aug 14, 2025

Uh oh!

ono-max Aug 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gayanW Aug 15, 2025

Choose a reason for hiding this comment

Uh oh!

ono-max Aug 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kohsuke commented Aug 15, 2025

Uh oh!

kohsuke commented Aug 15, 2025

Uh oh!

gayanW commented Aug 15, 2025

Uh oh!

ono-max Aug 15, 2025

Choose a reason for hiding this comment

Uh oh!

gayanW Aug 15, 2025

Choose a reason for hiding this comment

Uh oh!

gayanW commented Aug 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kohsuke left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ono-max Aug 15, 2025 •

edited

Loading

ono-max Aug 15, 2025 •

edited

Loading

gayanW commented Aug 15, 2025 •

edited

Loading