allow for parameter ordering control by the user

[k-rister]

We need to drop any parameter sorting that is being done and allow for the following:

We have the ability to potentially combine from 3 different places:

1. required
2. includes
3. sets

I would probably like the ability to at least do this:

1. Force the required block to be first
2. Force the include block to be second
3. Force the set block to be third

However, I could see myself really wanting to do this:

1. Force the required block to be first
2. Force one include block to be second
3. Force the set block to be third
4. Force additional includes to be inserted at specific points in the set block

[k-rister]

Today we generate something like this:

fio --clocksource=gettimeofday --ramp_time=5s --runtime=10s --time_based=1 --norandommap --name=doio --ioengine=sync --unlink=1 --filename=/tmp/fio.foo --size=10240k --rw=read --write_hist_log=fio --log_hist_msec=10000 --write_bw_log=fio --write_iops_log=fio --write_lat_log=fio --log_avg_msec=1000 --log_unix_epoch=1 --output-format=json+ --output=fio-result.json fio.job

when it needs to be this:

fio --name=global --clocksource=gettimeofday --ramp_time=5s --runtime=10s --time_based=1 --norandommap --write_hist_log=fio --log_hist_msec=10000 --write_bw_log=fio --write_iops_log=fio --write_lat_log=fio --log_avg_msec=1000 --log_unix_epoch=1 --output-format=json+ --output=fio-result.json --name=doio --ioengine=sync --unlink=1 --filename=/tmp/fio.foo --size=10240k --rw=read

Some of this is because of multiplex and some of this is because of bench-fio.

[k-rister]

An high level overview of what we need for bench-fio to work properly (from the multiplex perspective):

fio job file:

-----------
[global]
rw=randread
size=128m

[job1]

[job2]
-----------

equivalent CLI command:

$ fio --name=global --rw=randread --size=128m --name=job1 --name=job2


What we need:

from fio's multiplex.json:
  we have the essentials parameters         2 |

mv-params:

include set for globals
  name=global                               1 |
  size=128                                    |

set
  includes globals

  name=job1                                 3 |
  rw=read                                     |
  bs=[ 4K, 8K ]                               |
  name=job2                                   |
  rw=write                                    |
  bs=[ 4K, 8K ]                               |


4 iterations:

    1                         2           3
    ------------------------- ----------- ------------------------------------------------------------

fio --name=global --size=128m <essential> --name=job1 --rw=read --bs=4K --name=job2 --rw=write --bs=4K

fio --name=global --size=128m <essential> --name=job1 --rw=read --bs=8K --name=job2 --rw=write --bs=4K

fio --name=global --size=128m <essential> --name=job1 --rw=read --bs=4K --name=job2 --rw=write --bs=8K

fio --name=global --size=128m <essential> --name=job1 --rw=read --bs=8K --name=job2 --rw=write --bs=8K

[rafaelfolco]

fio.json

{
    "global-options": [
        {
            "name": "common-params",
            "params": [
                { "arg": "duration", "vals": [ "300" ] },
                { "arg": "ioengine", "vals": [ "libaio" ] }
            ]
        }
    ],
    "sets": [
        {
            "enabled": "yes",
            "include": "common-params",
            "params": [
                { "arg": "name", "vals": [ "job1" ] },
                { "arg": "bs", "vals": [ "4K", "8K" ] },
                { "arg": "rw", "vals": [ "read", "randread" ] },
                { "arg": "name", "vals": [ "job2" ] },
                { "arg": "rw", "vals": [ "read" ] },
                { "arg": "ioengine", "vals": [ "sync" ] }
            ]
        }
    ]
}

fio-req.json

{
    "presets": {
        "essentials": [
            { "arg": "ioengine", "vals": ["sync"] }
        ],
        "defaults": [
            { "arg": "rw", "vals": ["write"] },
            { "arg": "bs", "vals": ["1K"] },
            { "arg": "duration", "vals": ["60"] }
        ],
        "sequential-read": [
            { "arg": "rw", "vals": [ "read" ] },
            { "arg": "bs", "vals": [ "4K" ] }
        ]
    },
    "validations": {
        "size_BKMG": {
            "description": "bytes in B/K/M/G, converted to K, shown in KiB",
            "args": [ "bs" ],
            "vals": "^([1-9][0-9]*)+[BKMG]?",
            "convert": "K",
            "transform": {
                "search": "^([1-9][0-9]*).$",
                "replace": "\\1KiB"
            }
        },
        "log_types": {
            "description": "all possible log types",
            "args": [ "write_bw_log", "write_hist_log", "write_iolog", "write_iops_log", "write_lat_log" ],
            "vals": "^fio$"
        },
        "rw_types": {
            "description": "all possible testtypes",
            "args": [ "rw" ],
            "vals": "^(|rand)(read|write|trim)$|^readwrite$|^randrw$|^trimwrite$"
        },
        "ioengine": {
            "description": "all possible ioengine types",
            "args": [ "ioengine" ],
            "vals": "^sync$|^libaio$"
        },
        "duration": {
            "description": "duration",
            "args": [ "duration" ],
            "vals": "^([1-9][0-9]*)+[smh]?"
        },
        "name": {
           "description": "name",
           "args": [ "name" ],
           "vals": "^.+$"
        }
    },
    "units": {
       "size_BKMG": {
            "": { "": "1", "B": "1", "K": "1024", "M": "1024*1024", "G": "1024*1024*1024" },
            "B": { "": "1", "B": "1", "K": "1024", "M": "1024*1024", "G": "1024*1024*1024" },
            "K": { "": "1/1024", "B": "1/1024", "K": "1", "M": "1024", "G": "1024*1024" },
            "M": { "": "1/1024/1024", "B": "1/1024/1024", "K": "1/1024", "M": "1", "G": "1024" },
            "G": { "": "1/1024/1024/1024", "B": "1/1024/1024/1024", "K": "1/1024/1024", "M": "1/1024", "G": "1" }
        }
    }
}

Patch

The proposed change to be tested:

diff --git a/multiplex.py b/multiplex.py
index ba77c8b..cd2f071 100755
--- a/multiplex.py
+++ b/multiplex.py
@@ -65,7 +65,7 @@ def dump_json(obj, format = 'readable'):
         sep += ' '

     return json.dumps(obj, indent = indentation, separators = (',', sep),
-                      sort_keys = True)
+                      sort_keys = False)

 def param_enabled(param_obj):
     """Return True if param is enabled, False otherwise"""
@@ -153,12 +153,8 @@ def load_param_sets(sets_block):
         if 'params' in set:
             for param in set['params']:
                 if param_enabled(param):
-                    replace_param = param_exists(param, param_set)
-                    if replace_param:
-                        idx = param_set.index(replace_param)
-                        param_set[idx] = param
-                    else:
-                        param_set.append(param)
+                    #(rfolco): Append param (allow dup)
+                    param_set.append(param)

         # mv_array is the outter array containing the inner sets
         mv_array.append(param_set)
@@ -347,9 +343,9 @@ def override_presets(json_obj):
                     del presets_dict["essentials"][idx]

             if len(_json) > 0:
-                # append essentials (new/undefined ones)
                 for _ess in presets_dict["essentials"]:
-                    _json.append(_ess)
+                    #(rfolco): Insert essential param into the beginning
+                    _json.insert(0, _ess)
             else:
                 idx = json_obj.index(_json)
                 json_obj[idx] = copy.deepcopy(presets_dict["essentials"])

Test

./multiplex.py --input ~/fio/fio.json --requirements ~/fio/fio-req.json | less

Output JSON:

[
    [
        {
            "arg": "duration",
            "role": "client",
            "val": "300"
        },
        {
            "arg": "ioengine",
            "role": "client",
            "val": "sync"
        },
        {
            "arg": "name",
            "role": "client",
            "val": "job1"
        },
        {
            "arg": "bs",
            "role": "client",
            "val": "4KiB"
        },
        {
            "arg": "rw",
            "role": "client",
            "val": "read"
        },
        {
            "arg": "name",
            "role": "client",
            "val": "job2"
        },
        {
            "arg": "rw",
            "role": "client",
            "val": "read"
        },
        {
            "arg": "ioengine",
            "role": "client",
            "val": "sync"
        }
    ],
    [
        {
            "arg": "duration",
            "role": "client",
            "val": "300"
        },
        {
            "arg": "ioengine",
            "role": "client",
            "val": "sync"
        },
        {
            "arg": "name",
            "role": "client",
            "val": "job1"
        },
        {
            "arg": "bs",
            "role": "client",
            "val": "4KiB"
        },
        {
            "arg": "rw",
            "role": "client",
            "val": "randread"
        },
        {
            "arg": "name",
            "role": "client",
            "val": "job2"
        },
        {
            "arg": "rw",
            "role": "client",
            "val": "read"
        },
        {
            "arg": "ioengine",
            "role": "client",
            "val": "sync"
        }
    ],
    [
        {
            "arg": "duration",
            "role": "client",
            "val": "300"
        },
        {
            "arg": "ioengine",
            "role": "client",
            "val": "sync"
        },
        {
            "arg": "name",
            "role": "client",
            "val": "job1"
        },
        {
            "arg": "bs",
            "role": "client",
            "val": "8KiB"
        },
        {
            "arg": "rw",
            "role": "client",
            "val": "read"
        },
        {
            "arg": "name",
            "role": "client",
            "val": "job2"
        },
        {
            "arg": "rw",
            "role": "client",
            "val": "read"
        },
        {
            "arg": "ioengine",
            "role": "client",
            "val": "sync"
        }
    ],
    [
        {
            "arg": "duration",
            "role": "client",
            "val": "300"
        },
        {
            "arg": "ioengine",
            "role": "client",
            "val": "sync"
        },
        {
            "arg": "name",
            "role": "client",
            "val": "job1"
        },
        {
            "arg": "bs",
            "role": "client",
            "val": "8KiB"
        },
        {
            "arg": "rw",
            "role": "client",
            "val": "randread"
        },
        {
            "arg": "name",
            "role": "client",
            "val": "job2"
        },
        {
            "arg": "rw",
            "role": "client",
            "val": "read"
        },
        {
            "arg": "ioengine",
            "role": "client",
            "val": "sync"
        }
    ]
]

Simplified JSON check

[ 
   [ duration=300, ioengine=sync, name=job1, bs=4Kib, name=job2, rw=read, ioengine=sync],
   [ duration=300, ioengine=sync, name=job1, bs=4Kib, rw=randread, name=job2, rw=read, ioengine=sync],
   [ duration=300, ioengine=sync, name=job1, bs=8Kib, rw=read, name=job2, rw=read, ioengine=sync],
   [ duration=300, ioengine=sync, name=job1, bs=8Kib, rw=randread, name=job2, rw=read, ioengine=sync]
]

• global duration=300 overrides default 60
• global ioengine=libioengine is overriden by essential ioengine=sync
• after globals/essentials, job1 and job2 params are placed in the same order they appear in the sets
• ioengine from set is always sync (same as essential)
• bs and rw permutations = 4, bs=[4,8] and rw=[read,randread] --> 2x2

[rafaelfolco]

@k-rister pls see if this solves the fio multi-job case.