Proposal: change CSV format

[dutow]

Currently the CSV format is limited by:

• adding many information about the test setup into a single field, instead of using separate columns for the separate features
• doesn't include some parameters used during the run, such as the PG_TDE variable

This is just a draft/proposal, not an actual ready to be merged PR, as it only hardcodes the PG_TDE variable instead of providing something more generic, I just want to validate the idea with it first.

The usecase for this script is that a separete simple CSV processing script can go through the CSV files, and upload them to a database used by grafana, without:

• relying on information not existing in the csv, such as shell variables
• doing extensive string processing to try to recover all the information which was put into a single field

Toughts?

[inikep]

Toughts:

1. From my perspective I need only

WORKLOAD, 8 THDS, 16 THDS, 32 THDS, 64 THDS
COUNT_RANGES, 32376.96, 64667.49, 96826.10, 136624.87
DISTINCT_RANGES, 17525.07, 35048.50, 53367.04, 73789.13
ORDER_RANGES, 22491.88, 44684.53, 66624.92, 93439.61

and a set of parameters

NODE=node4
SERVER_VERSION=PS845-5
WRITES_TIME_SECONDS=300
NUM_TABLES=16
DATASIZE=10M
ENGINE=innodb
ENGINE_CACHE=96G
CONFIG_BASE=stable8x-noACID
BENCH_NAME=release-8.4.5-5@3d3abca6bf6_2025-06-23_07:43

which are duplicated now.
2. What output format will be the best (most convenient) for grafana?
3. If we can't make the output good for both grafana and readabilty by humans, we may output separate file(s) for grafana.

[dutow]

We could also store the parameters in a separate file in the same directory, and then the upload script could read that too - bu currently I don't see a file like that. We have build_info.txt, but it's missing some of the parameters.

My main reason for this change was:

1. BENCH_NAME=release-8.4.5-5@3d3abca6bf6_2025-06-23_07:43 - this currently concatenates multiple parameters into a single string. In the grafana db, I have branch, commit hash and time in 3 separate fields. I could have written a parser for this string from the CSV field, but since we already have a CSV file, it seemed a better idea to split these into 3 fields
2. The PG_TDE variable is missing from both the csv and the build_info, and we might have other similar parameters in the future. Now that I look at this, maybe it would make more sense to include it in build_info, and parse that file too.

And maybe I could include the separate branh/commit/time fields in that file too.

What do you think about that approach?

[inikep]

1. Currently all variables (including new "START_DATE" "SERVER_REPO_URL" "SERVER_BRANCH" "SERVER_GIT_HASH") will be written to variables.inc.sh which may be included in bash with source:
   8cf0d9d1b2906
2. I introduced print_postgres_variables() which allows to add additional variables to variables.inc.sh. Currently it adds only PG_TDE:
   af276476cf38dd