How do you use run_gen.py?

[ychen306]

I get the following error while running /scripts/run_gen.py --target='gcc clang' -j 1

Running 16 test sets: ['gcc_no_opt', 'gcc_opt', 'gcc_skx_no_opt', 'gcc_skx_opt', 'gcc_icx_no_opt', 'gcc_icx_opt', 'gcc_tgl_no_opt', 'gcc_tgl_opt', 'clang_no_opt', 'clang_opt', 'clang_skx_no_opt', 'clang_skx_opt', 'clang_icx_no_opt', 'clang_icx_opt', 'clang_tgl_no_opt', 'clang_tgl_opt']
Traceback (most recent call last):
  File "./scripts/run_gen.py", line 1778, in <module>
    prepare_env_and_start_testing(os.path.abspath(args.out_dir), args.timeout, args.target, args.num_jobs,
  File "./scripts/run_gen.py", line 1580, in prepare_env_and_start_testing
    print_online_statistics_and_cleanup(lock, stat, targets, task_threads, num_jobs, no_tmp_cln)
  File "./scripts/run_gen.py", line 1399, in print_online_statistics_and_cleanup
    stat_str, verbose_stat_str, prev_len = form_statistics(stat, targets, prev_len, task_threads)
  File "./scripts/run_gen.py", line 1322, in form_statistics
    verbose_stat_str += "\tcpu time: " + strfdelta(stat.get_target_duration(i.name),
  File "<string>", line 2, in get_target_duration
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/managers.py", line 850, in _callmethod
    raise convert_to_error(kind, result)
KeyError: 'gcc_no_opt'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 126, in _main
    self = reduction.pickle.load(from_parent)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/synchronize.py", line 110, in __setstate__
    self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory

[Vsevolod-Livinskij]

I can't reproduce the error, so I'll need more info.

Which OS do you use?
Do you use yarpgen from the top of the trunk?

Have you set YARPGEN_HOME envienvironment variable to point to the root of yarpgen?
Otherwise, you have to use run_gen.py inside the scripts directory:

cd yarpgen/scripts/
./run_gen.py --target='gcc clang' -j 1

[ychen306]

I've set YARPGEN_HOME to the root of yarpgen. The git hash is 67080d151580c8707019c7deda1e8a2630089722, I am using python 3.8.3 on Mac Catalina (10.15.6)

[ychen306]

I also ran it on a Linux machine and the error is gone. Seems like it has something to do with mac

[Vsevolod-Livinskij]

Ok, it is probably the known problem with macOS. The issue is that Python on macOS uses the spawn method, and on Linux, it uses fork. The spawn method does not inherit the global state, so the run_gen.py fails.

There is a branch that works on macOS in my repo, but it is not thoroughly tested yet. You can try to use it.
I'll work on bringing it to the trunk, and I'll let you know when that happens.