Tukukkk

I wrote a class that uses a Cython utility module. I then tried to speed things up with Multiprocessing to process multiple instances of the class simultaneously but got an error. Error sending result: '(0, <MemoryView of 'ndarray' at 0x19de04081f0>)'. Reason: 'TypeError('no default __reduce__ due to non-trivial __cinit__',)' I've looked into writing a __reduce__ function but about everything I've seen pertains to pickling classes, not methods or modules. I also looked into writing a __cinit__ method but saw even less that seemed relevant.

Below is a simplistic represenntation of the package and module layouts that generates the error (there'll really be hundreds of DNG objects to process, each referencing a unique 20ish MB file, and ljpeg really has hundreds of lines and is called tens to hundreds of times for each DNG). In the example the error can be fixed by removing array type declarations but if I was to do that in the real thing the performance hit would be orders of magnitude larger than the multiprocessing gains.

Can this be fixed without slowing it down appreciably or major refactoring, and if so, how?

sequence.py

import multiprocessing



import numpy as np



from dng import DNG





def test_decode():

    input_file = np.zeros(3000, dtype=np.intc)



    pool = multiprocessing.Pool()

    tasks = 



    for i in range(10):

        task = pool.apply_async(thread, (i, input_file))

        tasks.append(task)



    pool.close()

    pool.join()



    for task in tasks:

        print(task.get())





def thread(i, input_file):

    dng = DNG(input_file)

    return i, dng.image





if __name__ == '__main__':

    test_decode()

dng.py

import numpy as np



import ljpeg





class DNG:

    def __init__(self, input_file):



        self.image = ljpeg.decode(input_file)

ljpeg.pyx

cpdef int[:] decode(int[:] encoded_image):

    encoded_image = __bar(encoded_image, 10000, 1000)

    return encoded_image





cdef int[:] __bar(int[:] array, int i, int ii):

    for j in range(i):

        for jj in range(ii):

            array = __foo(array)

    return array





cdef int[:] __foo(int[:] array):

    array[0] += 1

    return array

output:

Traceback (most recent call last):

  File "F:/Documents/Python/threading_multi/sequence.py", line 31, in <module>

    test_decode()

  File "F:/Documents/Python/threading_multi/sequence.py", line 22, in test_decode

    print(task.get())

  File "C:Python36libmultiprocessingpool.py", line 644, in get

    raise self._value

multiprocessing.pool.MaybeEncodingError: Error sending result: '(0, <MemoryView of 'ndarray' at 0x19de04081f0>)'. Reason: 'TypeError('no default __reduce__ due to non-trivial __cinit__',)'



Process finished with exit code 1

I'm pretty sure that this an error returning the memoryviews generated by each thread to the main thread (because the memoryview can't be pickled). However, the memoryview itself wraps another Python object that probably can be pickled.

There's no real need to specify the return type of decode (or make it cpdef) since it's only called from Python. At the end of decode return the .base of the memoryview to get the underlying object that it wraps:

def decode(int[:] encoding_image):

    # ...

    return encoding_image.base