Add CPU & MEM composite score?

[al6x]

Just wondering if it may be worth to add a third column "CPU & MEM". So you can see languages that are best at both CPU & MEM?

Score calculated as 0.5 * (lang_cpu/all_lang_median_cpu + lagn_mem/all_lang_median_mem) the lower the better.

So you can easily spot problem with language like "Racket" that has C-like CPU time but consumes x100 more memory.

Feel free to close ticket if not interested :)

[nuald]

The metrics (not formula though) you've described fits the value we already have - Energy. For Intel RAPL (https://blog.chih.me/read-cpu-power-with-RAPL.html) the whole (package) energy consumption is based on CPU (PP0), GPU (PP1) and DRAM usage. However, the current benchmarks results have PP0 (CPU) only as my previous server that I've used for the benchmarks has an old Intel CPU that doesn't provide additional RAPL data, and I've implemented PP0 reading only.

However, I've just tried rapl-read (https://web.eece.maine.edu/~vweaver/projects/rapl/index.html), and my current server has all the data:

$ sudo ./rapl-read -s
...
	Package 0
		package-0	: 2.297906J
		core	: 1.646846J
		uncore	: 0.046143J
		dram	: 0.518615J

So I guess now I can add the relevant data, but it would require some time to implement the proper solution (the current one, based on powercap, doesn't provide the package data, only PP0 and PP1: https://www.kernel.org/doc/html/latest/power/powercap/powercap.html ). As always, all contributions are welcome.

[nuald]

Please dismiss my notes about current implementation not supporting whole package. It reads /sys/class/powercap/intel-rapl/intel-rapl:0/energy_uj that is the whole package (with PP0 etc are within that directory, e.g. /sys/class/powercap/intel-rapl/intel-rapl:0/intel-rapl:0:0/energy_uj ). I just need to update the README as it still has the old info.

So, you already have the metrics available. As for the given example, RAPL metrics are not quite accurate on short time intervals (as it could be some external factors involved). If you take a look to mandel.b tests, you could see that tests which are slower than Racket's could consume less energy (because use less DRAM), therefore CPU&MEM-wise are little bit better. It's not straightforward correlation like in your formula (as it depends on the DRAM and CPU models), but more realistic. I guess real server platforms (as an opposite to my laptop "server") could give even better numbers (like you could have in data centers), but I don't have any, sorry.

Another note - bf-tests are CPU intensive, not memory. If you take a look into other tests, like Primes, you could the that memory usage indeed affects the energy consumption:

• Ruby (--jit) runs in 1.9 secs and use 418Mb total with energy consumption 31J
• Ruby/truffleruby runs in 2 secs and use 694Mb total with energy consumption 52J
• despite almost same performance, the latter uses (694-418)/418=66% more DRAM, and has (52-31)/31=67% more energy consumption (almost direct correlation here).

[al6x]

Energy could be statistically correlated, but it's not looks like it's the same metric. Example, from the "Mandel in Brainfuck" table.

Difference between C/gcc and Racket - energy 1.4, memory 105. Not just slightly different, two orders of magnitude difference.

Language	Time, s	Memory, MiB	Energy, J
C/gcc	12.523±0.183	0.71±00.03 + 1.01±00.02	257.16±05.69
Racket (Syntax Objects)	16.950±0.862	112.16±00.33 + 68.29±00.23	361.35±27.86

[nuald]

Right, energy consumption doesn't necessary correlate to RAM usage (I guess the correlation could be caused by more CPU work that could be unaccounted for as we track only the benchmarked process, but not, for example, kernel).

I'd like to note two things though:

• the majority of the tests are compiled/run with the "speed" option. Obviously, that could affect the RAM consumption (for example, inlining or loop unrolling could increase memory consumption in GC-ed languages).
• the current tables are already big and difficult to read, adding other metrics is quite problematic.

I think I can store the raw results though in the version control, and additional scripts could analyze them with any metrics wanted. That would require some work, and I'm not sure when I can implement that. All PRs are welcomed as usually.