Home

Welcome to the proclib wiki!

Proclib is a high-level library that wraps over the standard library subprocess module. It provides helpful conveniences like automatic splitting of commands using shlex.split. Mandatory code example:

>>> from proclib.api import spawn
>>> r = spawn(['cat'], data='stdin\n')
>>> r.out
'stdin\n'
>>> r.status_code
0

Streaming

"But it reads the entire stdout!", you say. For long running, expensive processes it is recommended that you stream them- streaming processes allow you to read from stdout and stderr asynchronously (the process need not exit and can run while you consume the output). For example:

>>> from contextlib import closing
>>> with closing(spawn(['yes', 'head -n 2'])) as r:
...     for item in r.stdout:
...         print(item)
...
y
y

With long running processes you often need to pass huge files of data (preferrably not XML!) in. To do that you simply need to pass a file object to the data parameter. Or you can also pass in any iterable that yields data that can be written, e.g. generator functions, lists, StringIO objects, etc.

>>> spawn(['head -n 20', 'tail -n 4'], data=open('README.rst'))
<Response [tail]>

The streaming feature will hopefully cater more to users with heavier workloads than normal. With the streaming feature, Proclib allows you to use a sane language to glue together components of your system to create a data processing pipeline that is written in a variety of languages. Also note that proclib is streaming by default- so the things that look "non streaming" are usually conveniences added for common usage patterns.