Initial implementation of memory-efficient Excel writing for large documents

[gamerover98]

I have implemented an initial version that does not use memory to write a document. This solution is useful in cases where it’s necessary to create large Excel documents sequentially, meaning there’s no need for a second manipulation of the cells already added to the document.

The issue is described here: #480

---

Currently, this code only allows text to be written into a cell, and there is no functionality yet for changing its style. I would also like to point out that, due to the rush in which this modification was made, you should see this pull request not as a final change but as a way to demonstrate that even more can be achieved with Fastexcel!

You can compare the memory usage difference using VisualVM with the images from the issue:

• Memory usage:
  

• Memory Profiling:
  

The difference is remarkable, isn’t it? You can now create huge files without wasting memory!

Here’s the code used for the tests:

package org.example;

import lombok.extern.slf4j.Slf4j;
import org.dhatim.fastexcel.Workbook;
import org.dhatim.fastexcel.Worksheet;

import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;

@Slf4j
public class Main {

    private static final String EXCEL_RELATIVE_PATH = "./target/data/example.xlsx";
    private static final int COLUMNS = 100;
    private static final int ROWS = 10000000;

    public static void main(String... args) throws Exception {
        try (var outputStream = new FileOutputStream(getExcelFile());
             var workbook = new Workbook(outputStream, "MyApplication", "1.0");
             var worksheet = workbook.newStreamWorksheet("Sheet 1")) {

           // Start writing on the document.
            worksheet.start(COLUMNS);

            log.debug("Generating data...");

            for (int rowIndex = 0; rowIndex < ROWS; rowIndex++) {
                worksheet.appendRow(column -> "test"); // Append a new row and set "test" at each cell.
            }

           // The autoclosable will close the worksheet and the workbook.
        }
    }

    public static File getExcelFile() throws IOException {
        var file = new File(EXCEL_RELATIVE_PATH);

        if (file.mkdirs()) {
            log.debug("Excel file created");
        }

        if (file.exists()) {
            Files.delete(Path.of(file.toURI()));
            log.debug("Excel file deleted");
        }

        return file;
    }
}

Lastly, I’d like to mention that test cases haven’t been added due to the nature of this pull request. If this approach moves forward, they will be included.

Thank you all <3

[tiago-s-vieira-alb]

Hi @gamerover98 , how can I test with multiple different values per row?

This replicates the same value in all the columns:
worksheet.appendRow(column -> "test"); // Append a new row and set "test" at each cell.

How can I append to that row different values?

[gamerover98]

Hi @tiago-s-vieira-alb,

in worksheet.appendRow(column -> "test"); the column is the index of the column.

9adf2ae#diff-70d50d3cdef0a64c1a61b46213e78789bfd3f304132a741dcc93bb87afde5627R128

If I remember correctly, the Cell class is not exposed in the API, so I only included the index.

---

So, to answer your question:

for (int rowIndex = 0; rowIndex < ROWS; rowIndex++) {
    worksheet.appendRow(column -> {
        return switch (column) {
            case 0 -> "first column";
            case 1 -> "second column";
            ...
            case n-th-column: -> "latest column";
            default: ...
        }
    });
}

[tiago-s-vieira-alb]

Thank you.. we already tested with some lambda functions and it works!

Did you tried to apply some styles? Merging cells?

[gamerover98]

Did you tried to apply some styles? Merging cells?

Nope, this pull request is only meant to demonstrate that it is possible to do better than what Fastexcel currently offers. However, if I remember correctly, I left comments in the source to point out the lack of decorative features for the cells.

[tiago-s-vieira-alb]

We migrate to another library (poi-streaming) for writing, that works well for our purpose.
We maintained fastexcel for the reading parts.

[rzymek]

This is already supported in the writer for some time now, as demonstrated in this e2e test:

fastexcel/e2e/src/test/java/org/dhatim/fastexcel/MemoryUsageE2E.java

Lines 43 to 57 in 4a1a60c

	try(Workbook wb = new Workbook(out, "test", "1.0")){
	for (int s = 0; s < SHEETS; s++) {
	try(Worksheet sheet = wb.newWorksheet("sheet " + s)){
	for (int r = 0; r < ROWS; r++) {
	printProgress("writing", s, r);
	for (int c = 0; c < COLS; c++) {
	sheet.value(r, c, valueFor(r, c));
	}
	if (r % FLUSH_EVERY_NR_OR_ROWS == 0) {
	sheet.flush();
	}
	}
	}
	}
	}

Though I get that the API (need to call flush()) is hard to discover.
On the other hand, with the way values can be written at any position via ws.value(row,col,value) it's hard to determine inside the library when it's a good time to flush.