Tukukkk

I'm trying to write a program that operates on a long list of elements (the list is called training_set in the code example). Each row of the list contains two numbers that have to be found on another list called IDs: hence, my program iterates on training_set's rows and, for each of them, finds the corresponding 2 numbers in IDs and then performs some more computation (not shown in the code).

With sequential execution, this requires about 300s. Since each row of training_set is indipendend from the others, I was thinking of parallelizing the computation by splitting the input by the #cpu_cores, using multiprocessing.Pool.
However, the parallelized version is slower than the sequential one.

num_procs = int(multiprocessing.cpu_count())



with open("training_set.txt", "r") as f:

    reader = csv.reader(f)

    training_set  = list(reader)

training_set = [element[0].split(" ") for element in training_set]



with open("node_information.csv", "r") as f:

    reader = csv.reader(f)

    node_info  = list(reader)

IDs = [element[0] for element in node_info]



batch_size = int(len(training_set)/num_procs)

inputs=



# split list into batches to feed to the different threads

for i in range(num_procs):

    if i == (num_procs-1): inputs.append(list(training_set[int(i*batch_size):(len(training_set)-1)]))

    else: inputs.append(list(training_set[int(i*batch_size): int((i+1)*batch_size)]))



def init(IDs):

    global identities

    identities = copy.deepcopy(IDs)



def analyze_pairs(partialList):

    pairsSet = copy.deepcopy(partialList)

    for i in range(len(pairsSet)):

        source = pairsSet[i][0] # an ID of edges

        target = pairsSet[i][1] # an ID of edges

        ## find an index maching to the source ID

        index_source = identities.index(source)

        index_target = identities.index(target)

        ***additional computation***



if __name__ == '__main__':

    pool = Pool(num_procs, initializer=init, initargs=(IDs,))

    training_features = pool.map(analyze_pairs, inputs)

I'm not showing the rest of the code of the for loop (at the end of analyze_pairs()) because the problem persists even if i remove that code, hence it's not there that the problem resides.)

I know that there are already many questions on this topic, but I couldn't find a solution for my case.
I don't think that here parallelism introduces more overhead than speedup because the input of each thread is large (on a 8 threads cpu, each thread should take at least 35s) and there is no explicit message passing. I also tried to use copy.deepcopy to make sure that each thread works on a separate list(althought it shouldn't be a problem since each thread only performs 'read' actions on the list), but it didn't work.

What could the problem be? Thanks in advance.