RFC: Parallel shadow map generation

[buma]

I started working on Paralelizing shadow map generation with joblib in my paralel branch. I added new parameter n_jobs and it works already. It is a little faster then with only one job.

Joblib is only needed if more than one job is requested.

This height map:

python heightmap.py --projection EPSG:3857 --elevation-dir data/ --output my.heightmap10004 --geojson export2.geojson --save-image my10004.png 46.55 15.65 4 1000

with this render:

 python render.py my.heightmap10004 "2014-08-04 03:00" "2014-08-04 20:00" 5 rendered5 --background-map heightmap1004_image.png --opacity 0.7 --n_jobs=4

Renders in 14 seconds with 4 jobs and 42 seconds with one on 4 core Intel(R) Core(TM) i5-3570K CPU @ 3.40GHz

The problem is that now height map is pickled for each job. Running time is strangely longer if I save height map myself and do mmap of height map for each job.

Another problem is background image. Previously it was opened once, now it is opened for each job, because It can't be pickled, I have to optimize it.

And parallel could use threads instead of multiprocessing if GIL isn't needed in C code. With no gil I don't know if this is possible here.

What do you think?

[perliedman]

I guess we need to break out one time initialization, like loading heightmap, background image etc., and then use something like this: https://pythonhosted.org/joblib/parallel.html#working-with-numerical-data-in-shared-memory-memmaping

I'm a real novice when it comes to calling C in Python, this is the first time I've done it, so I don't know too much about what we can do about the GIL. Reading this: http://docs.cython.org/src/userguide/external_C_code.html, it seems hard to work without the GIL:

You can specify nogil in a C function header or function type to declare that it is safe to call without the GIL.:

[...]
If you are implementing such a function in Cython, it cannot have any Python arguments, Python local variables, or Python return type, and cannot manipulate Python objects in any way or call any function that does so without acquiring the GIL first.

I might be missing some obvious workaround for how we currently both pass and return numpy arrays from C.

[perliedman]

A thought is if it wouldn't be easier to put the parallelization parts inside the C code, since most of the time is spent inside a quite tight loop that could be easy to split into multiple threads: https://github.com/perliedman/shadow-mapper/blob/master/shadowmap.c#L70