Project: S3779 LZW ColGroup | ASML by MasterBrain2000 · Pull Request #2560 · apache/systemds

MasterBrain2000 · 2026-07-19T19:28:04Z

Group Project for the AMLS Module

Closes apache#2420.

…nd downstream CLA operations

janniklinde · 2026-07-20T08:35:54Z

Your code does not build (see error logs). Could you please fix those issues to have a prototype that compiles?

janniklinde · 2026-07-29T07:44:29Z

@MasterBrain2000 @m-ollka @Mancer1 could you please address the issues causing the tests to fail? Thanks

Mancer1 · 2026-08-03T07:41:02Z

@janniklinde, all the build problems have been addressed.

codecov · 2026-08-03T09:56:03Z

Codecov Report

❌ Patch coverage is 90.77118% with 73 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.52%. Comparing base (d5f6950) to head (3284f6f).
⚠️ Report is 29 commits behind head on main.

Files with missing lines	Patch %	Lines
...ss/colgroup/ColGroupPiecewiseLinearCompressed.java	90.17%	31 Missing and 35 partials ⚠️
...ress/colgroup/functional/PiecewiseLinearUtils.java	92.85%	3 Missing and 3 partials ⚠️
...he/sysds/runtime/compress/colgroup/ColGroupIO.java	88.88%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2560      +/-   ##
============================================
- Coverage     71.61%   71.52%   -0.10%     
- Complexity    50132    50650     +518     
============================================
  Files          1614     1631      +17     
  Lines        193986   196297    +2311     
  Branches      37935    38267     +332     
============================================
+ Hits         138925   140396    +1471     
- Misses        44155    44899     +744     
- Partials      10906    11002      +96

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Mancer1 · 2026-08-03T12:15:32Z

@janniklinde, How much code coverage is accepted? We have some null functions that are required to be implemented but never called.

janniklinde · 2026-08-03T13:17:53Z

Generally, all functions that are expected to provide a working implementation should be tested, @Mancer1. Coverage also includes nested function calls, so you should be able to achieve high coverage by testing the publicly available functions of the compressed column group. Please use the tests for existing AColGroup implementations as a reference.

Also, please avoid reimplementing helper functionality that is already available in the test utilities, such as random matrix generation.

…o improve coverage

# Conflicts: # src/main/java/org/apache/sysds/runtime/compress/colgroup/ColGroupPiecewiseLinearCompressed.java # src/test/java/org/apache/sysds/test/component/compress/colgroup/ColGroupPiecewiseLinearCompressedOperationsTest.java

Mancer1 · 2026-08-07T19:07:24Z

@janniklinde, we have improved the coverage to 95%. I hope that suffices

Mancer1 · 2026-08-08T11:37:46Z

@janniklinde, we have already reformatted our files that we edited. The files that java Codestyle check is showing to reformat are those not part of our PR.

rice@Arnor:~/MasterBrain2000-Project-S3779-LZW-ColGroup$  dev/format-changed.sh
All PR-edited Java lines are correctly formatted.
rice@Arnor:~/MasterBrain2000-Project-S3779-LZW-ColGroup$ git status
On branch main
Your branch is up to date with 'origin/main'.

So, should we leave it be, since we were working on a stale version of main?

janniklinde · 2026-08-10T07:57:23Z

You can simply run the following @Mancer1:

git fetch upstream
dev/format-changed.sh upstream/main

Mancer1 · 2026-08-10T08:09:09Z

You can simply run the following @Mancer1:
git fetch upstream
dev/format-changed.sh upstream/main

@janniklinde, I believe I tried it before. It didn't affect anything.

rice@Arnor:~/MasterBrain2000-Project-S3779-LZW-ColGroup$ git fetch upstream
dev/format-changed.sh upstream/main
remote: Enumerating objects: 299, done.
remote: Counting objects: 100% (172/172), done.
remote: Compressing objects: 100% (90/90), done.
remote: Total 299 (delta 105), reused 82 (delta 82), pack-reused 127 (from 2)
Receiving objects: 100% (299/299), 262.07 KiB | 507.00 KiB/s, done.
Resolving deltas: 100% (122/122), completed with 16 local objects.
From https://github.com/apache/systemds
 * [new branch]            dependabot/github_actions/docker/login-action-4.6.0 -> upstream/dependabot/github_actions/docker/login-action-4.6.0
   7a0e5ca0ac..9dccbc3e60  main       -> upstream/main
All PR-edited Java lines are correctly formatted.
rice@Arnor:~/MasterBrain2000-Project-S3779-LZW-ColGroup$ git status
On branch main
Your branch is up to date with 'origin/main'.
```

janniklinde

Thanks for the updates. I left comments in the code which should still be fixed. Most of them are minor and easy to patch.

Please address those remaining issues to pass the project.

janniklinde · 2026-08-10T13:20:22Z

+	public static AColGroup compressPiecewiseLinearFunctional(IColIndex colIndexes, MatrixBlock in,
+		CompressionSettings cs) {
+
+		final int numRows = in.getNumRows();
+		final int numCols = colIndexes.size();
+		int[][] breakpointsPerCol = new int[numCols][];
+		double[][] slopesPerCol = new double[numCols][];
+		double[][] interceptsPerCol = new double[numCols][];
+
+		for(int col = 0; col < numCols; col++) {
+			final int colIdx = colIndexes.get(col);
+			double[] column = PiecewiseLinearUtils.getColumn(in, colIdx);
+			PiecewiseLinearUtils.SegmentedRegression fit = PiecewiseLinearUtils
+				.compressSuccessivePiecewiseLinear(column, cs);
+			breakpointsPerCol[col] = fit.getBreakpoints();
+			interceptsPerCol[col] = fit.getIntercepts();
+			slopesPerCol[col] = fit.getSlopes();
+
+		}
+		return ColGroupPiecewiseLinearCompressed.create(colIndexes, breakpointsPerCol, slopesPerCol, interceptsPerCol,
+			numRows);
+
+	}


Incorrect if cs.transposed = true, ther e you need to iterate over rows

Done. Just need Tests for coverage

janniklinde · 2026-08-10T14:18:03Z

A few recurring patterns I noticed here:

Several tests only verify return types, non-null values, or that unsupported methods throw

Some expected results derived from the implementation itself instead of being computed independently

Error tolerances sometimes very loose relative to input range

Missing edge cases (e.g., non-commutative ops, transposed input, zero target loss, serialization through the normal ColGroupIO)

Leftover DP tests

…l statements turn to throwNotImplenmented functions

…sposed matrices

janniklinde · 2026-08-13T13:20:53Z

Let me know once all issues have been addressed for me to make a final pass over the code @Mancer1.

Mancer1 · 2026-08-13T20:16:50Z

@janniklinde, I'll tell you when we're fully done with the changes. I'm just replying "Done" to your comments for better team coordination and to know which changes have been made.

Mancer1 · 2026-08-15T20:07:03Z

@janniklinde, we have made the changes you have requested. You may rerun the workflows.

Mancer1 · 2026-08-16T14:40:18Z

@janniklinde, as mentioned before, the Java Code Style failure happens on files that are not part of our PR. We have done both dev/format-changed.sh and dev/format-changed.sh upstream/main before we pushed our code.

janniklinde

I did another review pass and looks much better now. Most of the comments have been resolved. Have a look at the comments that are still open and the ones that I added now.

janniklinde · 2026-08-17T09:21:47Z

+/**
+ * This class represents a new ColGroup which is compresses column into segments (piecewise linear) to represent the
+ * original Data each column is approximate by a set of linear segments defined by breakpoints, slopes and intercepts
+ */
+
+public class ColGroupPiecewiseLinearCompressed extends AColGroupCompressed {
+	/**
+	 * breakpoints indices per column to define the segment boundaries slopes of the regression line per segment per
+	 * column intercepts of the regression line per segment per column
+	 */


Weird comment, maybe one above class

janniklinde · 2026-08-17T09:22:09Z

+	int[][] breakpointsPerCol;
+	double[][] slopesPerCol;
+	double[][] interceptsPerCol;
+	int numRows;


Why not private?

janniklinde · 2026-08-17T09:29:02Z

+/**
+ * Performance benchmark for piecewise linear compression. Successive is benchmarked across large matrices to show
+ * scalability. DP is only used as a quality reference on small matrices due to quadratic complexity
+ *
+ */


Outdated comment

janniklinde · 2026-08-17T09:29:22Z

+		return mb;
+	}
+
+	/// returns a average number of segments per column


Unnecessary comment

janniklinde · 2026-08-17T09:31:33Z

+/**
+ * Tests for ColGroupPiecewiseLinearCompressed operations.
+ */


Unnecessary

janniklinde · 2026-08-17T09:35:02Z

+		assertTrue("disk size should be positive", colGroupPLC.getExactSizeOnDisk() > 0);
+		assertTrue("num values should be positive", colGroupPLC.getNumValues() > 0);
+	}


Weak assert

MaryAbdull and others added 11 commits July 19, 2026 15:51

[SYSTEMDS-3543] Piece-wise linear compression of column groups

d767502

Closes apache#2420.

PiecewiseLinearUtils.computeBreakpointSuccessive is compressed in O(n)

d73a371

Old DP algorithm and its usages are removed

f442491

add DDCLZW enum value to ColGroupType

6dea132

new ColGroupPiecewiseLinearCompressed methods; fix testComputeSUm

5bd5180

new replace & unitaryOperation

0263992

ColGroupIO support

f3fe67f

[Minor] fixed mistake in previous commit

ff38235

Performance tests and Benchmarks for sucussive Piece wise algorithm a…

5e50b92

…nd downstream CLA operations

added operation tests

37ef00d

using tab instead of spaces for intentations

86f671a

github-project-automation Bot added this to SystemDS PR Queue Jul 19, 2026

github-project-automation Bot moved this to In Progress in SystemDS PR Queue Jul 19, 2026

Mancer1 added 12 commits August 2, 2026 21:29

Implement missing abstract method in ColGroupPiecewiseLinearCompressed

07be457

ci: configure GPG loopback pinentry for release script workflow

5cc6487

ci: configure GPG loopback pinentry for release script workflow

10e86e8

ci: configure GPG loopback pinentry for release script workflow

f99b824

ci: configure GPG loopback pinentry for release script workflow

4a9a13a

Implement missing abstract method in ColGroupPiecewiseLinearCompressed

29153b1

added missing import

de890dc

added all missing abstract methods

4da1965

ci: restore release-scripts.yml workflow to original state

e51a38f

added the missing licenses and imports

5cc8a39

style: fix checkstyle violations in test classes

82c8bd9

added the missing ValueFunction import

1298867

style: apply eclipse line formatting fixes via format-changed.sh

02b90bf

Mancer1 and others added 7 commits August 3, 2026 16:43

Remove all DP functions and using testUtils helper function instead t…

f3d419d

…o improve coverage

deleting unused imports, and formatting

bfdbcf0

added skeleton for remaining 0% coverage functions

1bf3737

updated skeleton for 0% coverage functions

69a1013

testing write

02cf453

# Conflicts: # src/main/java/org/apache/sysds/runtime/compress/colgroup/ColGroupPiecewiseLinearCompressed.java # src/test/java/org/apache/sysds/test/component/compress/colgroup/ColGroupPiecewiseLinearCompressedOperationsTest.java

tests 16-30

7004f24

tests 01-15

83d9bee

Mancer1 force-pushed the main branch from abfa731 to 83d9bee Compare August 6, 2026 22:29

Mancer1 added 2 commits August 7, 2026 00:30

Add unit tests and coverage fixes and some comments

448642c

correcting Java codestyle

5d0b8b4

janniklinde self-requested a review August 10, 2026 11:10

janniklinde requested changes Aug 10, 2026

View reviewed changes

github-project-automation Bot moved this from In Progress to In Review in SystemDS PR Queue Aug 10, 2026

Mancer1 added 3 commits August 10, 2026 23:54

DP code and related and legacy comments deleted or edited, return nul…

ffd45bf

…l statements turn to throwNotImplenmented functions

some codestyles and compressPiecewiseLinearFunctional can tackle tran…

bf2d2fc

…sposed matrices

SegmentSumSq guard deleted

90777aa

fixed the compression error

66a3836

Mancer1 added 2 commits August 15, 2026 20:22

Done the remaining requested changes

b76eec7

removing unused imports

3284f6f

janniklinde reviewed Aug 17, 2026

View reviewed changes

Uh oh!

Conversation

MasterBrain2000 commented Jul 19, 2026

Uh oh!

janniklinde commented Jul 20, 2026

Uh oh!

janniklinde commented Jul 29, 2026

Uh oh!

Mancer1 commented Aug 3, 2026

Uh oh!

codecov Bot commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Mancer1 commented Aug 3, 2026

Uh oh!

janniklinde commented Aug 3, 2026

Uh oh!

Mancer1 commented Aug 7, 2026

Uh oh!

Mancer1 commented Aug 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

janniklinde commented Aug 10, 2026

Uh oh!

Mancer1 commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

janniklinde left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

janniklinde commented Aug 13, 2026

Uh oh!

Mancer1 commented Aug 13, 2026

Uh oh!

Mancer1 commented Aug 15, 2026

Uh oh!

Mancer1 commented Aug 16, 2026

Uh oh!

janniklinde left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov Bot commented Aug 3, 2026 •

edited

Loading

Mancer1 commented Aug 8, 2026 •

edited

Loading

Mancer1 commented Aug 10, 2026 •

edited

Loading