-
Notifications
You must be signed in to change notification settings - Fork 0
Modernize project: migrate to pyproject.toml and refactor IPC #7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| name: CI | ||
|
|
||
| on: | ||
| push: | ||
| branches: [master, main] | ||
| pull_request: | ||
|
|
||
| jobs: | ||
| test: | ||
| runs-on: ubuntu-latest | ||
| strategy: | ||
| matrix: | ||
| python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"] | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - uses: actions/setup-python@v6 | ||
| with: | ||
| python-version: ${{ matrix.python-version }} | ||
| - run: pip install . pytest | ||
| - run: pytest tests.py -v |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| __pycache__/ | ||
| *.py[cod] | ||
| *.egg-info/ | ||
| *.egg | ||
| dist/ | ||
| build/ | ||
| .eggs/ | ||
| *.so | ||
| .pytest_cache/ | ||
| .tox/ |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,47 +1,56 @@ | ||
| Limit the function execution time | ||
| ================================= | ||
| # kill-timeout | ||
|
|
||
| Limit the function execution time. | ||
|
|
||
| This module provides a decorator which runs the function in a separate | ||
| multiprocessing.Process and sends SIGKILL after the specified timeout if the | ||
| function didn't complete. | ||
| `multiprocessing.Process` and sends `SIGKILL` after the specified timeout if | ||
| the function didn't complete. | ||
|
|
||
| **Note:** This uses `SIGKILL` and `multiprocessing.Process`, so it only works | ||
| on POSIX systems (Linux, macOS, etc.), not on Windows. | ||
|
|
||
| Requirements | ||
| ------------ | ||
| ## Requirements | ||
|
|
||
| * Python 3.7+ (needed for multiprocessing.Process.kill) | ||
| * tblib (to pass the traceback from separate process) | ||
| * Python 3.10+ | ||
| * [tblib](https://pypi.org/project/tblib/) (to preserve tracebacks from the subprocess) | ||
|
|
||
| Install | ||
| ------- | ||
| ## Install | ||
|
|
||
| ```bash | ||
| pip install kill-timeout | ||
| ``` | ||
|
|
||
| Usage | ||
| ----- | ||
| ## Usage | ||
|
|
||
| ```python | ||
| from kill_timeout import kill_timeout | ||
|
|
||
|
|
||
| limit_in_seconds = 5 | ||
|
|
||
|
|
||
| @kill_timeout(limit_in_seconds) | ||
| def long_running_function(**parameters): | ||
| """Function which makes some computations | ||
| """Function which makes some computations. | ||
|
|
||
| It could take too long for some parameters. | ||
| """ | ||
| ... | ||
|
|
||
| try: | ||
| result = long_running_function(iterations=9001) | ||
| print("Function returned: %r" % result) | ||
| print(f"Function returned: {result!r}") | ||
| except TimeoutError: | ||
| print("Function didn't finished in %d seconds" % limit_in_seconds) | ||
| print(f"Function didn't finish in {limit_in_seconds} seconds") | ||
| except Exception: | ||
| print("Function failed with its internal error! Its original traceback:") | ||
| raise | ||
| ``` | ||
|
|
||
| ## How it works | ||
|
|
||
| Each call to a decorated function: | ||
|
|
||
| 1. Spawns a new `multiprocessing.Process` to run the function. | ||
| 2. Waits for the process to complete within the timeout using `process.join(seconds)`. | ||
| 3. If the process is still alive after the timeout, sends `SIGKILL` and reaps it. | ||
| 4. If the function raised an exception, re-raises it in the caller with the original traceback (via `tblib`). |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -3,6 +3,8 @@ | |
|
|
||
| from tblib.decorators import return_error, Error | ||
|
|
||
| _fork_ctx = multiprocessing.get_context("fork") | ||
|
|
||
|
|
||
| class TimeoutError(TimeoutError): | ||
| """Custom TimeoutError | ||
|
|
@@ -13,52 +15,61 @@ class TimeoutError(TimeoutError): | |
| pass | ||
|
|
||
|
|
||
| def _target(func, child_conn, args, kwargs): | ||
| """Top-level target function for the subprocess (must be picklable).""" | ||
| try: | ||
| child_conn.send(return_error(func)(*args, **kwargs)) | ||
| finally: | ||
| child_conn.close() | ||
|
|
||
|
|
||
| def kill_timeout(seconds): | ||
| """Decorator to limit function execution time | ||
|
|
||
| It runs the function in separate multiprocessing.Process and sends SIGKILL after | ||
| the specified timeout if the function didn't complete. | ||
|
|
||
| Note: return values larger than the OS pipe buffer (~64KB on Linux) may cause | ||
| a false timeout because the child blocks on send() while the parent waits on | ||
| join(). Keep return values small or return None and communicate large results | ||
| through files or shared memory. | ||
| """ | ||
|
|
||
| def decorator(func): | ||
|
|
||
| manager = multiprocessing.Manager() | ||
| results = manager.dict() | ||
|
|
||
| def target(key, *args, **kwargs): | ||
| results[key] = return_error(func)(*args, **kwargs) | ||
|
|
||
| @wraps(func) | ||
| def wrapper(*args, **kwargs): | ||
| # key object would be uniq in the parent process, but it can't be | ||
| # used directly, because when it goes to manager the new object is | ||
| # created, so the `id(key)` should be used as the real key | ||
| key = object() | ||
| process = multiprocessing.Process( | ||
| target=target, | ||
| args=(id(key),) + args, | ||
| kwargs=kwargs, | ||
| parent_conn, child_conn = _fork_ctx.Pipe() | ||
| process = _fork_ctx.Process( | ||
| target=_target, | ||
| args=(func, child_conn, args, kwargs), | ||
| ) | ||
| process.start() | ||
| child_conn.close() | ||
| process.join(seconds) | ||
| if process.is_alive(): | ||
| # NOTE: killing the process without getting its return code via | ||
| # .join() would cause the zombie process to stay until the next | ||
| # multiprocessing.Process() run, it is acceptable, since it | ||
| # shouldn't end up with many zombie processes, and the | ||
| # alternatives like creating the daemon threads to catch the | ||
| # zombies don't look like a good fit | ||
| process.kill() | ||
| if id(key) in results: | ||
| result = results.pop(id(key)) | ||
| process.join() | ||
| parent_conn.close() | ||
| raise TimeoutError( | ||
| f"function {func.__name__} didn't complete in {seconds} seconds" | ||
| ) | ||
| if parent_conn.poll(): | ||
| result = parent_conn.recv() | ||
| parent_conn.close() | ||
|
Comment on lines
49
to
+59
|
||
| if isinstance(result, Error): | ||
| result.reraise() | ||
| else: | ||
| return result | ||
| else: | ||
| raise TimeoutError("function %s didn't complete in %s seconds" % ( | ||
| func.__name__, seconds | ||
| )) | ||
| return result | ||
| parent_conn.close() | ||
| if process.exitcode and process.exitcode < 0: | ||
| raise ChildProcessError( | ||
| f"function {func.__name__} was killed by signal " | ||
| f"{-process.exitcode}" | ||
| ) | ||
| raise ChildProcessError( | ||
| f"function {func.__name__} exited unexpectedly " | ||
| f"(exit code {process.exitcode})" | ||
| ) | ||
|
Comment on lines
63
to
72
|
||
|
|
||
| return wrapper | ||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| [build-system] | ||
| requires = ["setuptools>=68.0"] | ||
| build-backend = "setuptools.build_meta" | ||
|
|
||
| [project] | ||
| name = "kill-timeout" | ||
| version = "0.1.0" | ||
| description = "Limit the function execution time" | ||
| readme = "README.md" | ||
| license = "MIT" | ||
ei-grad marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| requires-python = ">=3.10" | ||
| authors = [ | ||
| {name = "Andrew Grigorev", email = "andrew@ei-grad.ru"}, | ||
| ] | ||
| keywords = ["kill", "timeout", "multiprocessing", "decorator"] | ||
| classifiers = [ | ||
| "Development Status :: 4 - Beta", | ||
| "Intended Audience :: Developers", | ||
| "Operating System :: POSIX", | ||
| "Programming Language :: Python :: 3", | ||
| "Programming Language :: Python :: 3.10", | ||
| "Programming Language :: Python :: 3.11", | ||
| "Programming Language :: Python :: 3.12", | ||
| "Programming Language :: Python :: 3.13", | ||
| "Programming Language :: Python :: 3.14", | ||
| ] | ||
| dependencies = [ | ||
| "tblib>=3.0", | ||
| ] | ||
|
|
||
| [project.urls] | ||
| Homepage = "https://github.com/ei-grad/kill-timeout" | ||
| "Bug Tracker" = "https://github.com/ei-grad/kill-timeout/issues" | ||
| "Source Code" = "https://github.com/ei-grad/kill-timeout" | ||
|
|
||
| [tool.setuptools] | ||
| py-modules = ["kill_timeout"] | ||
|
|
||
| [tool.pytest.ini_options] | ||
| testpaths = ["tests.py"] | ||
This file was deleted.
This file was deleted.
This file was deleted.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pipe.send()can block if the payload is large enough to fill the OS pipe buffer, and the parent waits injoin()before reading. This can deadlock or produce a false timeout (child stuck sending, parent waiting to join). A more robust approach is to receive while waiting (poll loop with remaining timeout), or usemultiprocessing.Queue(which uses a feeder thread) to avoid blocking the child on large writes.