Open
Conversation
05b5ff0 to
97d901c
Compare
- enable user to extend plotman's functionality / plotting flow, by providing external scripts, without need to touch plotmans internals - many of current feature requests can be solved with lighweight scripts (bash/python/...), e.g. ericaltendorf#712, ericaltendorf#711, ericaltendorf#677, ericaltendorf#638, ericaltendorf#582 ..., - process cpu and io niceness can also be manipulated, affinity to cpu or NUMA on big server hardware can be set differently based on stage or plotter - all new code is in new library 'hooks' + two new shell scripts directly inside hooks.d directory (located inside plotman's own config directory - supplied hook serves as reference implementation / example, providing functionality for ericaltendorf#677 - modification of existing plotman's code is only to call entrypoint in hooks.py and pass current jobs list containing jobs objects - currently manager.py maybe_start_new_plot() and plotman.py kill is injected with call into hooks.try_run() and hooks.run() respectively - try_run consumes fully refreshed jobs[], compares phase to previous job phase and if it is changed, calls hooks.run() - run() takes plotmans environment, extends it with particular job's metadata and calls all executable files from hooks.d directory having extension .sh or .py - scripts are called synchronously, cmd exec is waiting until the script process returns. that means the implementation is not suitable for LONG running actions. - anyhow, plotman CLI can be called without issues from within the hooks, recursion in job_refresh -> try_run -> hooks script -> plotman cmd is being checked for new: src/plotman/hooks.py src/plotman/resources/hooks.d update: src/plotman/configuration.py src/plotman/manager.py setup.cfg
hook scripts run
- to prevent expansion of paths / params to simple *
e.g. $PLOTMAN_DSTDIR/*$PLOTMAN_PLOTID* -> /*
if $VARS would be missing
through dst/tmp/tmp2 dirs (ericaltendorf#582,...)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
enable user to extend plotman's functionality / plotting flow,
by providing external scripts, without need to touch plotmans
internals
many of current feature requests can be solved with lighweight
scripts (bash/python/...), e.g. Start Rsync automatically when Plotman starts #712, Clean temp dirs if no plos are running #711, Automatically suspend jobs on stage 3:6 if not enought space in dst disk #677, Stop multiple plots copying over to the HDD at the same time. #638, plotman kill doesn't remove all temp files #582 ...
process cpu and io niceness can also be manipulated, affinity
to cpu or NUMA on big server hardware can be set differently
based on stage or plotter
all new code is in new library 'hooks' + two new shell scripts
directly inside hooks.d directory (located inside plotman's
own config directory
supplied hook serves as reference implementation / example,
providing functionality for Automatically suspend jobs on stage 3:6 if not enought space in dst disk #677
modification of existing plotman's code is only to call entrypoint
in hooks.py and pass current jobs list containing jobs objects
currently manager.py maybe_start_new_plot() and plotman.py kill
is injected with call into hooks.try_run() and hooks.run() respectively
try_run consumes fully refreshed jobs[], compares phase to previous
job phase and if it is changed, calls hooks.run()
run() takes plotmans environment, extends it with particular job's
metadata and calls all executable files from hooks.d directory
having extension .sh or .py
scripts are called synchronously, cmd exec is waiting until the
script process returns. that means the implementation is not suitable
for LONG running actions.
anyhow, plotman CLI can be called without issues from within the hooks,
recursion in job_refresh -> try_run -> hooks script -> plotman cmd
is being checked for
stdout/stderr of running the hook scripts is currently forced to /tmp/plotman-hooks.log
new: src/plotman/hooks.py
src/plotman/resources/hooks.d
update: src/plotman/configuration.py
src/plotman/manager.py
setup.cfg