Tukukkk

I have a file with word frequency logs from a very messy corpus of scraped Polish text that I am trying to clean to get accurate word frequencies. Since this is a big text file, I divided it into batches.

Here is a snippet from the original file:

 1 środka(byłe

 1 środka.było

 1 środkacccxli.

 1 (środkach)

 1 „środkach”

 1 środ­kach

 1 środkach...

 1 środkach.",

 1 środkach"

 1 środkach".

 1 środkachwzorem

 1 środkach.życie

 1 środkajak

 1 "środkami"

 1 (środkami)

 1 „środkami”)

 1 środkami!"

 1 środkami”

 1 środkami)?

 1 środkami˝.

My goal is to clean true word labels and remove noisy word labels (e.g. collocations of words concatenated through punctuation). This is what is achieved by the first part of the script.

#-*- coding: utf-8 -*-



import io

import pandas as pd

import numpy as np



num_batches = 54



for i in range(1,num_batches+1):



infile_path = r'input_batch_' + str(i) + r'.txt'

outfile_path = r'output_batch_' + str(i) + r'.txt'



with io.open(infile_path, 'r', encoding = 'utf8') as infile,

     io.open(outfile_path, 'w', encoding='utf8') as outfile:



     entries_raw = infile.readlines()

     entries_single = [x.strip() for x in entries_raw]

     entries = [x.split('t') for x in entries_single]



     data = pd.DataFrame({"word": , "freq": })



     for j in range(len(entries)):

        data.loc[j] = entries[j][1], entries[j][0]

As you can see in the data sample above, several noisy entries belong to the same true label. Once cleaned, their frequencies should be added. This is what I try to achieve in the second part of my script:

     freq_dict = dict()

     keys = np.unique(data['word'])



     for key in keys:

        for x in range(len(data)):

             if data['word'][x] == key:

                 if key in freq_dict:

                     prior_freq = freq_dict.get(key)

                     freq_dict[key] = prior_freq + data['freq'][x]

                 else:

                    freq_dict[key] = data['freq'][x]



    file_name = u'sample_' + str(i) + u'.csv'

    data.to_csv(file_name, index=False)

The problem with this code is that it is either buggy, running into an infinite loop or sth, or is very slow, even for processing a single batch, to the point of being impractical. Are there ways to streamline this code to make it computationally tractable? In particular, can I achieve the same goal without using for loops? Or by using a different data structure for word-frequency lookup than a dictionary?

Here is the code in one piece with fixed indentation, in case you are able to reproduce my issues on your end:

# -*- coding: utf-8 -*-



import io

import pandas as pd

import numpy as np



num_batches = 54



for i in range(1 ,num_batches +1):



    infile_path = r'input_batch_' + str(i) + r'.txt'

    outfile_path = r'output_batch_' + str(i) + r'.txt'



    with io.open(infile_path, 'r', encoding = 'utf8') as infile, 

         io.open(outfile_path, 'w', encoding='utf8') as outfile:



        entries_raw = infile.readlines()

        entries_single = [x.strip() for x in entries_raw]

        entries = [x.split('t') for x in entries_single]



        data = pd.DataFrame({"word": , "freq": })



        for j in range(len(entries)):

            data.loc[j] = entries[j][1], entries[j][0]



            freq_dict = dict()

            keys = np.unique(data['word'])



            for key in keys:

                for x in range(len(data)):

                    if data['word'][x] == key:

                        if key in freq_dict:

                            prior_freq = freq_dict.get(key)

                            freq_dict[key] = prior_freq + data['freq'][x]

                        else:

                            freq_dict[key] = data['freq'][x]



        file_name = u'sample_' + str(i) + u'.csv'

        data.to_csv(file_name, index=False)