5
\$\begingroup\$

I am trying to build a useable NLP corpus but getting bottlenecked by how long the program takes (200 hours). With so much data I know that optimizing my code even a little bit will net me huge time savings down the road, so I wanted to post this code and ask for some advice on speeding it up. I added a parameter to load the dataset if I have already made it, resulting in much faster times afterward, and tested on a small portion of the dataset, not the whole thing. When generating the tables, kwargs expects a size parameter and for loading it expects two file directory parameters. The features "full_text" is the raw full text of an academic article and "full_text_cleaned" has had the text set to lowercase and punctuation stripped. I can also provide how I created this dataset if requested.

Sample Corpus: 29000 entries

paper_id title abstract full_text full_text_cleaned
string string string string string

Relevant code

class Doc_Finder():
    def __init__(self, corpus_file):
        self.corpus = pd.read_csv(corpus_file)
        self.corpus_dir = corpus_file

    def make_keywords(self, save=False, load=False, **kwargs):
        if save and load:
            raise Exception("Invalid Parameters")
        if save:
            self.keyword_index = pd.DataFrame(columns=['id','keyword'])
            total = kwargs['size']
            self.keyword_map = []
            for _, row in tqdm(self.corpus.head(total).iterrows(), total=total):
                s = []
                for i in str(row['full_text_cleaned']).split(' '): 
                    if i not in s and not i.isnumeric() and i not in stopwords.words('english') and i.isalnum() and not isHyperlink(i):
                        s.append(i)
                        if i not in self.keyword_map:
                            self.keyword_map.append(i)
                for i in s:
                    self.keyword_index.loc[len(self.keyword_index.index)] = [row['paper_id'], i]
            
            if save:
                idx_dir = './keyword_index_' + str(total) + '.csv'
                self.keyword_index.to_csv(idx_dir, index=False)
                tempdf = pd.DataFrame(data=self.keyword_map, columns=['keyword'], dtype=str)
                map_dir = './keyword_map_' + str(total) + '.csv'
                tempdf.to_csv(map_dir, index=False)
        elif load:
            self.keyword_index = pd.read_csv(kwargs['index_dir'])
            self.keyword_map = pd.read_csv(kwargs['map_dir'])['keyword'].tolist()

I run the code using this snippet

doc_finder = Doc_Finder(sample_corpus)
doc_finder.make_keywords(save=True, size=len(doc_finder.corpus['paper_id']))

The result is two csvs:

The keyword index has a paper_id and keyword pair. Listing each keyword and each paper.

paper_id keyword
00033d5a12240a8684cfe943954132b43434cf48 extraction
00033d5a12240a8684cfe943954132b43434cf48 expected

Then the set of all keywords from all the documents. This is in an attempt to make the document retrieval I will be doing after this faster, as I can match against the set of keywords and then query all the papers that have those keywords in the index table.

Keyword list

keyword
expected
...

I recognize that an SQL table of some kind would probably be a better solution, but I am limited to my local machine and do not know SQL very well. I also found a library called Dask but I think I would need to rethink how I perform this creation process since not all of the techniques would transfer. I am a little out of my comfort zone but this has been a rewarding experience.

\$\endgroup\$
3
  • \$\begingroup\$ Please do not edit the question, especially the code, after an answer has been posted. Changing the question may cause answer invalidation. Everyone needs to be able to see what the reviewer was referring to. What to do after the question has been answered. \$\endgroup\$
    – pacmaninbw
    Dec 18, 2023 at 1:37
  • \$\begingroup\$ It was appended to the question for that reason but okay \$\endgroup\$
    – evader110
    Dec 18, 2023 at 3:29
  • 1
    \$\begingroup\$ If someone provides an answer and you want to update the code, you can ask a follow up question with a link back to the existing question. \$\endgroup\$
    – pacmaninbw
    Dec 18, 2023 at 13:01

2 Answers 2

4
\$\begingroup\$

measurements

This submission is about performance, yet it includes no profile measurements, and almost no performance data. We are told only that each of 29k articles takes an expected 25 seconds to extract keywords, without learning how many megabytes that is.

representation

We are told we need to "build a useable NLP corpus" without learning how subsequent stages of the pipeline will use this first stage. Discarding full_text in favor of just full_text_cleaned would be an easy way to cut I/O delays in half, if the pipeline only needs the cleaned text.

Using parquet format compression offers similar benefits.

From your mention of keywords, almost certainly you should be using an RDBMS that offers full text indexing.

memory use

You're storing 29k articles in RAM. Twice.

Maybe you have lots of memory. And your article count will not increase in the coming months. Cross your fingers.

Consider adopting a streaming approach, where you continuously stream data out to disk, never holding onto the text of more than one article at a time. Process an article, spit out its data, forget the article, and move on to the next.

Organizing your workflow around enormous memory-hungry data frames is the exact opposite of streaming. If you're not paging to disk now, you will be once the corpus gets bigger.

constructor

class Doc_Finder():

No.

Pep-8 is very clear on how to spell this: DocFinder

Similarly, spell it def is_hyperlink.

    def __init__(self, corpus_file):
        self.corpus = ...
        self.corpus_dir = corpus_file

That second object attribute appears to be misnamed as a directory when really it's a ..._file

The language doesn't require this, but it is polite to the Gentle Reader to introduce all object attributes up in the constructor, as an inventory of what's important. Please add:

        self.keyword_index = None
        self.keyword_map = None

stuttering if

        if save:
            ...            
            if save:

It's unclear what good that second clause is doing. Recommend you combine the two of them.

meaningful identifiers

                s = []

Wow. You're just not going to give me anything, are you? Not a hint, not a comment.

Maybe call this vocabulary?

Also, based on the i not in s conjunct, it looks like you really want a set rather than a list. The in operator runs in O(N) linear time for vocabulary list of size N, and in O(1) constant time for a set of any size. As it stands, you have a O(N²) quadratic loop which is very slow for "large" articles.

Similar remarks for the keyword_map list, which should be a set. Also there's no need to test membership, you can just unconditionally add word i to that set. If it's already there, no harm done, there's no effect.

pre-allocate

    for i in s:
        self.keyword_index.loc[len(self.keyword_index.index)] = [row['paper_id'], i]

We are extending the keyword_index DataFrame one row at a time. Don't do that, as it is slower than necessary.

Build up a giant list of dicts, and feed that all at once to pd.DataFrame to create a giant DataFrame all at once. Or have a generator yield those dicts.

\$\endgroup\$
1
  • \$\begingroup\$ I made some major revisions thanks to your advice. Look it over and let me know what else could be improved \$\endgroup\$
    – evader110
    Dec 17, 2023 at 21:04
1
\$\begingroup\$

After J_H Comments

At first, I was a little hurt by his tone but I realized I had been slacking in doing good programming. After switching to Dask, using apply instead of a nasty for loop. I brought down the computation of 100 articles (see profile metrics below) from 14 minutes to 64s.

def is_hyperlink(input_string):
    return len(input_string) >= 4 and input_string[:4] == 'http'

def create_document_keywords(document):
    return [word for word in document.split(' ') if not word.isnumeric() and word.isalnum() and not is_hyperlink(word) and word not in stopwords.words('english')]

# DocFinder will traverse the provided corpus files using a keyword map and index 
# to find relevant documents for a given user query (a question to be more specific)
class DocFinder():
    def __init__(self, corpus_dir):
        # corpus_dir is the relative path to the corpus
        self.corpus_dir = corpus_dir
        
        # corpus_columns are a list of the columns from the corpus
        self.corpus_columns = []
        # Get the names of the columns of the corpus
        with open(corpus_dir, 'r') as file:
            self.corpus_columns = file.readline().strip(' \n').split(',')

        # ALL Columns in the corpus are assumed to be string types
        self.corpus_dtypes = dict(zip(self.corpus_columns, itertools.repeat('string')))

        # keyword_index_dir & keyword_map_dir are relative filepaths 
        self.keyword_index_dir = ''
        self.keyword_map_dir = ''
        

    # Generator to access a specific number of rows with specific columns if requested
    def get_corpus_data(self, number_rows=None, columns=None):
        # If the user provided no specific columns then return all the columns
        if columns:
            # If there is a specified number of rows yield those otherwise get everything
            return dd.read_csv(self.corpus_dir, header=0, dtype=self.corpus_dtypes)[columns].head(number_rows)
        else:
            return dd.read_csv(self.corpus_dir, header=0, dtype=self.corpus_dtypes).head(number_rows)


    # This will populate the index and map files if they do not exist
    # and set the keyword_index_dir & keyword_map_dir members
    def generate_keyword_files(self, save=False, load=False, **kwargs):
        if save and load:
            raise Exception("Invalid Parameters")
        if save:
            total_rows = kwargs['size']

            # Create the filepaths for the index and map
            self.keyword_index_dir = './keyword_index_' + str(total_rows) + '.csv'
            self.keyword_map_dir = './keyword_map_' + str(total_rows) + '.csv'
            
            # Create the actual index and map
            needed_data = self.get_corpus_data(columns=['paper_id','full_text_cleaned'], number_rows=total_rows) #len(self.corpus.index)):
            needed_data['full_text_cleaned'] = needed_data['full_text_cleaned'].apply(create_document_keywords)
            needed_data = needed_data.rename(columns = {'full_text_cleaned' : 'keyword'})
            keyword_index = needed_data.explode(column='keyword', ignore_index=True)
            keyword_map = keyword_index['keyword'].unique()
            
            # Write the index and map to disk
            keyword_index.to_csv(self.keyword_index_dir, index=False)
            map_as_csv_string = ',\n'.join(keyword_map)
            map_as_csv_string = 'keyword,\n' + map_as_csv_string
            with open(self.keyword_map_dir, 'w', encoding="utf-8") as output_file:
                output_file.write(map_as_csv_string)

        elif load:
            self.keyword_index_dir = kwargs['index_dir']
            self.keyword_map_dir = kwargs['map_dir']

After Actually Implementing Dask

I realized none of the operations being performed were utilizing Dask because in the header method, I still needed to set compute=False and change the map/index creation code.

# Create the actual index and map
needed_data_ddf = self.get_corpus_data(columns=['paper_id','full_text_cleaned'], number_rows=total_rows)
needed_data_ddf = needed_data_ddf.map_partitions(dask_apply_keywords, 'full_text_cleaned', meta=needed_data_ddf)
needed_data_ddf = needed_data_ddf.explode(column='full_text_cleaned')
needed_data = needed_data_ddf.compute()

and implement a wrapper function for create_document_keywords called dask_apply_keywords

def dask_apply_keywords(df, column):
    df[column] = df[column].apply(create_document_keywords)
    return df

Old Profile:

         458701173 function calls (453365372 primitive calls) in 840.286 seconds
   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000  840.286  840.286 project.py:498(test_profile)
        1   29.772   29.772  812.344  812.344 project.py:428(make_keywords)
    86037   41.761    0.000  720.342    0.008 indexing.py:683(__setitem__)
    86037    1.477    0.000  579.871    0.007 indexing.py:1528(_setitem_with_indexer)     
    86036    3.118    0.000  577.779    0.007 indexing.py:1864(_setitem_with_indexer_missing)
    86036    4.542    0.000  527.726    0.006    frame.py:7849(append)
    86036    0.732    0.000  198.572    0.002    concat.py:82(concat)
    86036    2.199    0.000  132.524    0.002    concat.py:469(get_result)
   430187    2.787    0.000  124.552    0.000    frame.py:502(__init__)
    86036    1.790    0.000  116.282    0.001     concat.py:35(concatenate_block_managers)
 774340/344151   17.862    0.000  104.420    0.000 base.py:250(__new__)
    86037    0.689    0.000   98.465    0.001 indexing.py:611(_get_setitem_indexer)       
    86037    1.534    0.000   96.516    0.001 indexing.py:1147(_convert_to_indexer)       
    86036    0.265    0.000   95.280    0.001 frame.py:2927(T)
    86036    2.350    0.000   95.015    0.001 frame.py:2805(transpose)
   258207    1.723    0.000   94.801    0.000 base.py:3036(get_loc)
   258207   90.551    0.000   90.572    0.000 {method 'get_loc' of 'pandas._libs.index.IndexEngine' objects}
 1806763/1462619   72.512    0.000   83.022    0.000 {built-in method numpy.core._multiarray_umath.implement_array_function}
   258108    2.856    0.000   82.904    0.000 concat.py:101(concat_compat)
  602257    1.463    0.000   72.062    0.000 <__array_function__ internals>:2(concatenate)
    86036    0.435    0.000   69.896    0.001 series.py:1560(to_frame)
   860472    3.852    0.000   66.970    0.000 blocks.py:2711(make_block)
    86038    1.016    0.000   66.446    0.001 construction.py:241(init_dict)
    86036    2.826    0.000   65.316    0.001 concat.py:306(__init__)
    96693    0.963    0.000   56.535    0.001 wordlist.py:18(words)

Second Profile after implementing @J_H 's feedback:

         105232492 function calls (105231779 primitive calls) in 64.858 seconds

   Ordered by: cumulative time
   List reduced from 1572 to 157 due to restriction <0.1>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.008    0.008   64.857   64.857 project.py:530(test_profile)
        1    0.000    0.000   64.849   64.849 project.py:460(generate_keyword_files)      
        1    0.000    0.000   63.799   63.799 series.py:4246(apply)
        1    0.000    0.000   63.799   63.799 apply.py:1022(apply)
        1    0.000    0.000   63.799   63.799 apply.py:1073(apply_standard)
        1    0.000    0.000   63.798   63.798 {pandas._libs.lib.map_infer}
      100    0.000    0.000   63.798    0.638 project.py:410(create_document_keywords)    
      100    1.241    0.012   63.783    0.638 project.py:411(<listcomp>)
   233214    0.902    0.000   62.290    0.000 wordlist.py:18(words)
   233214    0.736    0.000   41.868    0.000 api.py:206(raw)
   233214    0.423    0.000   33.540    0.000 api.py:222(open)
   466429    0.639    0.000   20.017    0.000 compat.py:39(_decorator)
   233214    0.464    0.000   17.385    0.000 data.py:323(open)
   233214    0.465    0.000   15.610    0.000 data.py:332(join)
   233215    0.324    0.000   13.148    0.000 data.py:302(__init__)
   233219   10.535    0.000   10.535    0.000 {built-in method io.open}
   233216    0.229    0.000   10.471    0.000 genericpath.py:16(exists)
   233240   10.242    0.000   10.242    0.000 {built-in method nt.stat}
   233214    0.271    0.000   10.040    0.000 simple.py:136(line_tokenize)
   233214    0.311    0.000    9.680    0.000 simple.py:112(tokenize)
   233214    4.669    0.000    9.303    0.000 wordlist.py:19(<listcomp>)
   233214    4.614    0.000    7.335    0.000 simple.py:116(<listcomp>)

Third Profile (after actually implementing Dask)

 39429 function calls (38600 primitive calls) in 47.527 seconds

   Ordered by: cumulative time
   List reduced from 1597 to 160 due to restriction <0.1>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.002    0.002   47.526   47.526 project.py:536(test_profile)       
        1    0.000    0.000   47.524   47.524 project.py:462(generate_keyword_files)
        1    0.000    0.000   47.123   47.123 base.py:290(compute)
        1    0.000    0.000   47.123   47.123 base.py:538(compute)
        1    0.000    0.000   47.121   47.121 threaded.py:36(get)
        1    0.000    0.000   47.120   47.120 local.py:350(get_async)
     1724   47.101    0.027   47.101    0.027 {method 'acquire' of '_thread.lock' objects}
        3    0.002    0.001   47.020   15.673 local.py:127(queue_get)
      427    0.006    0.000   47.018    0.110 queue.py:153(get)
      430    0.004    0.000   47.011    0.109 threading.py:270(wait)
        1    0.000    0.000    0.356    0.356 generic.py:3294(to_csv)
        1    0.000    0.000    0.356    0.356 format.py:1056(to_csv)
        1    0.000    0.000    0.355    0.355 csvs.py:232(save)
        1    0.000    0.000    0.353    0.353 csvs.py:259(_save)
        1    0.003    0.003    0.353    0.353 csvs.py:292(_save_body)
        5    0.000    0.000    0.350    0.070 csvs.py:302(_save_chunk)
        5    0.260    0.052    0.260    0.052 {pandas._libs.writers.write_csv_rows}
        1    0.000    0.000    0.097    0.097 progress.py:106(_finish)
        1    0.000    0.000    0.097    0.097 threading.py:979(join)
        1    0.000    0.000    0.097    0.097 threading.py:1017(_wait_for_tstate_lock)
        6    0.000    0.000    0.071    0.012 base.py:1262(_format_native_types) 
        5    0.000    0.000    0.071    0.014 numeric.py:303(_format_native_types)
       22    0.070    0.003    0.070    0.003 {method 'astype' of 'numpy.ndarray' objects}
```
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.