Reducing boost file set

[gsrohde]

Somewhere (and I can't remember where) I mentioned that the uncompressed size of this file set could be more that halved if we got rid of files that aren't actually used by BioCro. This isn't so urgent now that we've reduced the size of the BioCro R package by other means, but it is something to consider going forward.

In any case, I thought it worthwhile making a record of the method I used to find and dispense with unused Boost files. I was working on a Mac using the Z shell, but I'm sure this would work without modification in the Bash shell and on any Linux machine, and probably also on any Windows machine that has Cygwin or MinGW installed.

The first step was to add the -H flag to line at the top of the Makevars (or Makevars.win) file that appends to the PKG_CPPFLAGS variable so that the line became

PKG_CPPFLAGS+=-I../src/inc -DR_NO_REMAP -H

Then, starting in the src directory, I executed the following sequence of commands. There are probably a few extraneous things left in here, which I'm not bothering to weed out at this point:

# This line could be omitted if you start with a clean working directory or if you use the `--preclean` option in the `R CMD INSTALL` command immediately following.
for file in `grep --exclude-dir boost --exclude-dir .deps "<boost/" --include "*.h" --include "*.cpp" -rl .`; do touch $file; done

cd ..
R CMD INSTALL . 2>&1 | tee install_log
grep -E "^\.+ \.\./src/inc" install_log > boost_includes
sed 's|^.*\.\./src/inc/||' boost_includes | sort -u > boost_unique_includes
cd src/inc
mv ../../boost_unique_includes .
find boost -type f | sort > all_boost_files
for file in `find boost -type f`; do
    if ! grep $file -q boost_unique_includes; then rm $file; fi
done
cd -
R CMD INSTALL --preclean . 2>&1 | tee install_log_2
cd tests
Rscript testthat.R

[justinmcgrath]

This would be nice. Boost takes a lot of space, and I assumed that the export tool was bringing only the minimum requirement. The Boost installation instructions seems like the most sensible place for this. https://github.com/biocro/inc/blob/04c4598924930ee2ff7e84df669640cf35180d44/adding_the_boost_libraries.md

[gsrohde]

This would be nice. Boost takes a lot of space, and I assumed that the export tool was bringing only the minimum requirement. The Boost installation instructions seems like the most sensible place for this. https://github.com/biocro/inc/blob/04c4598924930ee2ff7e84df669640cf35180d44/adding_the_boost_libraries.md

I looked briefly at those instructions, and one line gave me pause:

Note that we do not actually use boost/units, but it is included whenever boost/numeric/odeint.hpp is included, and BioCro will not compile on some operating systems without it.

That made me wonder if there were issues I'm not considering and whether what works on my OS might not work elsewhere. (That said, the units-related files, or at least some of them, remain, even after I've eliminated the files unused on my system.)

If you all thought it worthwhile, I could make a branch with the files trimmed down for testing, and then maybe a branch of BioCro that uses that branch to do such testing.

[justinmcgrath]

We're right at 5 MB and we've already removed things I felt were valuable. Making any additions is going to be very difficult now, so if we can remove things that are actually unnecessary that would be great.

[justinmcgrath]

If things do work fine, possibly it worth reporting your findings to the Boost authors. I feel like their tool isn't working properly if unnecessary stuff is exported.

[justinmcgrath]

Or maybe we need to double check that we're choosing the right things to export.

[gsrohde]

If things do work fine, possibly it worth reporting your findings to the Boost authors. I feel like their tool isn't working properly if unnecessary stuff is exported.

My guess is that their tool is designed to export whole Boost libraries as a unit (plus any dependencies of the chosen libraries). What files from any particular library actually get used by a code base is likely to change during the course of development. We, on the other hand, have a fairly stable code base (I think), so it is less consequential to throw out the stuff we don't actually use. And we are packaging the stuff we do use along with our own code, which is probably not the usual use case.

[justinmcgrath]

My guess is that their tool is designed to export whole Boost libraries as a unit (plus any dependencies of the chosen libraries)

I wondered if their documentation gave details. It does have a section about how it can give "fat dependencies" if a term matches a library name. It says you can avoid that by giving specific headers, which we do, so I'm not sure that what we're doing is a problem. However, there's a "--scan" argument, that will read files for dependencies. That would be more robust than what we're doing now, which requires updating the docs whenever we change dependencies. The page also implies that it gives a smaller set of dependencies, " If you want a more minimal list, then try using the names of the headers you are actually including, or use the --scan option to scan your source code.", but "more minimal" isn't reassuring wording.

https://www.boost.org/doc/libs/1_78_0/tools/bcp/doc/html/index.html

And we are packaging the stuff we do use along with our own code, which is probably not the usual use case.

This is one of the specific use cases they give, "... and for Boost users who want to distribute a subset of Boost with their application", which is what makes me think if it's not giving a minimal set then something is not as expected.

[justinmcgrath]

I don't see newton_raphson_boost.h anywhere, but it's in the docs as including several Boost files. Maybe this is where the extra files are from? @eloch216 is this one of the files that only you were using that you moved out?

[gsrohde]

However, there's a "--scan" argument, that will read files for dependencies. That would be more robust than what we're doing now, which requires updating the docs whenever we change dependencies.

This is very true. newton_raphson_boost.h no longer exists. Conversely, parameters.h uses boost/algorithm/string.hpp, which isn't mentioned in the Developer's Manual documentation.

The page also implies that it gives a smaller set of dependencies, " If you want a more minimal list, then try using the names of the headers you are actually including, or use the --scan option to scan your source code.", but "more minimal" isn't reassuring wording.

Yes, unfortunately. Although the --scan option is considerably simpler than going through the steps I outline, it also results in a considerably larger set of files. Still, even if we do want to use the steps I outline, using the --scan option to produce an initial file set is probably a good option. Moreover, I could probably come up with a script to do most of the work automatically.

[eloch216]

I don't see newton_raphson_boost.h anywhere, but it's in the docs as including several Boost files. Maybe this is where the extra files are from? @eloch216 is this one of the files that only you were using that you moved out?

Yes, I think I was the only one (potentially) using it. (I see to remember that the boost Newton-Raphson solver was only able to handle one dimension, and I ended up writing my own, though.) It could probably be removed if it's contributing to a space issue, since I could always add it back in if I need it.

[gsrohde]

It could probably be removed if it's contributing to a space issue, since I could always add it back in if I need it.

... or you could install that part of Boost on your own machine unless and until you want to make it available to others to use.