Tukukkk

Caveat: I'm about a year in to teaching myself how to code as a supplement to work so please don't puke all over this; I know it's not great.

Anyway, I'm trying to write a code to download pdf's from about 1500000 web pages that go up in order by a single integer and then extract text from only those that contain certain keywords. I'll remove some of the names in the following, but I've gotten this to run with multithreading (I'm using Oracle VirtualBox and 10 cores from my machine) and it can handle about 25/minute, but that would take me about 41 days which is basically infeasible. Looking for help to make this code block much faster so that I can get this done in a reasonable amount of time. Any help would be appreciated.

Other notes: not all of the files when iterating through websites are actually in pdf form and some of the webpages are just empty. Also, I've taken a lot of this from other people so there's likely a lot of useless code in here. I've created a file (costSummary.txt) that collects the Serial Number of the item in question and the cost of that item, and it can likely be optimized.

#Load libraries

from wand.image import Image as Img

import requests

from PIL import Image

import pytesseract

import cv2

import os

from os import path

from os import listdir

from os import stat

import multiprocessing as multi

import numpy as np

import sys



#Create function to extract the information

def webIterate():

    myfile = open('costSummary.txt', 'w')

    for i in range(20):



        id = 4766120 + i

        image_url = 'http://website{}.pdf'.format(id)

        r = requests.get(image_url, stream = True)

        with open('python{}.pdf'.format(id), 'wb') as pdf:

            for chunk in r.iter_content(chunk_size=1024):

                if chunk:

                    pdf.write(chunk)

        if stat('python{}.pdf'.format(id)).st_size > 0:

            with Img(filename='python{}.pdf'.format(id), resolution = 300) as img:

                img.compression_quality = 99

                img.save(filename='sample_scan{}.jpg'.format(id))

                if path.exists('sample_scan{}-0.jpg'.format(id)):

                    os.rename('sample_scan{}-0.jpg'.format(id), 'sample_scan{}.jpg'.format(id))

            text = ''

            if stat('sample_scan{}.jpg'.format(id)).st_size > 0:

                text = pytesseract.image_to_string(Image.open('sample_scan{}.jpg'.format(id)))

            if stat('python{}.pdf'.format(id)).st_size == 0:

                text = ''

            if (text.find('HORIZONTAL WELL') == -1):

                text = ''       

            for item in text.split('n'):

                if 'SERIAL' in item:

                    sn = item.strip()

            for item in text.split('n'):

                if '$' in item:

                    capex = item.strip()       

            myfile.write('%sn' % sn) 

            myfile.write('%sn' % capex)

            test = os.listdir('/home/myDirectory')

            for item in test:

                if item.endswith('{}.jpg'.format(id)):

                    os.remove(item)

                elif item.endswith('{}.pdf'.format(id)):

                    os.remove(item)

        else:

            test = os.listdir('/home/myDirectory')

            for item in test:

                if item.endswith('{}.jpg'.format(id)):

                    os.remove(item)

                elif item.endswith('{}.pdf'.format(id)):

                    os.remove(item)



    myfile.close()

    file_name = multi.current_process().name +'.txt'

    test = os.listdir('/home/myDirectory')

    for item in test:

        if item.endswith('.jpg'):

            os.remove(item)

        elif item.endswith('.pdf'):

            os.remove(item)



cpus = multi.cpu_count()

workers = 



for cpu in range(cpus):

    sys.stdout.write('CPU' + str(cpu) + 'n')

    worker = multi.Process(name = str(cpu),

                           target = webIterate,

                           args = ())

    worker.start()

    workers.append(worker)



for worker in workers:

    worker.join()