Skip to main content
12 votes

Relative frequency of words in tree of documents

There are two reasons for the poor performance of the code in the post: first, it performs much unnecessary work, and, second, the operations that it does perform are carried out using inefficient ...
Gareth Rees's user avatar
  • 49.6k
8 votes

NLTK sentence / word tokenize

1. Quick review: The collections.Counter class has an update method that adds counts for items in an iterable. So instead of: <...
Gareth Rees's user avatar
  • 49.6k
8 votes
Accepted

Splitting text into n-grams and analyzing statistics on them

Code organisation Your code seems to be splitted into small-ish functions which is good. However, there is something that could easily be improved : you could move your code actually doing something (...
SylvainD's user avatar
  • 29.2k
7 votes
Accepted

Make a summary from a larger text-file

Use a linter The first thing I do when reviewing JavaScript code is run it through a linter. There several online linters, JSHint for example. JSHint will complain about ES6 features: 'arrow ...
janos's user avatar
  • 111k
7 votes

Separating a String of Text into Separate Words in Python

While your program is functional and solid, there are plenty of improvements to be made. How I would solve it. A better way to solve this would probably be: ...
sbottingota's user avatar
6 votes

Markov chain text generation in Python

Holy incomprehensible comprehensions! There is no good justification for defining seq, freq, and ...
200_success's user avatar
6 votes
Accepted

Tokenizing texts from Gutenberg archive for analysis

This is the ideal place for a class. Each book is its own object with its own method of returning its tokens. I would make a method tokens, which I would make a ...
Graipher's user avatar
  • 41k
6 votes
Accepted

Preprocessing steps to follow while cleaning and extracting text data from tweets

Copying my answer from SO: You can use pandas vectorized string methods to do your processing and it also removes the for loop ...
umutto's user avatar
  • 176
6 votes

Using lots of regex substitutions to tokenize text

3x to 4x speedup using str.translate() Based on a quick test (see below), str.translate() is an order of magnitude faster than a regular expression for replacing a ...
RootTwo's user avatar
  • 10k
6 votes
Accepted

Wordcloud from all answers of a user here on CR

Quick bits You have some issues that some linters would pick up: I would suggest moving your main code into a function. So that it doesn't pollute the global namespace. You've got some trailing ...
Peilonrayz's user avatar
  • 43.3k
5 votes
Accepted

Search in a big dictionary Python

I'd suggest turning the problem around. A dictionary is really good for looking up the key, but not for finding a key for a specific value. First, you need to convert your dictionary to a dictionary ...
Sjoerd Job Postmus's user avatar
5 votes
Accepted

Labeling modified words

When there're comments in the code that tell what it does, it usually indicates that the following piece of code should be a separate function with a meaningful name. I'd create a separate function ...
kraskevich's user avatar
  • 5,580
5 votes
Accepted

Flag words that would be difficult for an early reader

The code you presented is not readable and definitely needs refactoring. You can simplify several code constructions and apply the "Extract Variable" refactoring method to improve readability of the ...
alecxe's user avatar
  • 17.3k
5 votes
Accepted

Substitute IDs to word tokens

There is nothing wrong in general with for loops. If you have to iterate over some elements there is no magic that can avoid this. So why is your function "not very efficient"? And why is this because ...
stefan's user avatar
  • 2,925
5 votes
Accepted

Remove determiners in a string

Thanks for sharing your code. It's a nice project you have there. Naming You should take some time to choose carefully your variable. l1 is not obvious, maybe <...
Julien Rousé's user avatar
5 votes
Accepted

Tokenizing SGML text for NLTK analysis

Regex compilation If performance is a concern, this: arr = [re.sub(pattern, '', i) for i in arr] is a problem. You're re-compiling your regex on every function ...
Reinderien's user avatar
5 votes

Separating a String of Text into Separate Words in Python

_char_to_class First of all, let's take a look at your _char_to_class method. ...
DeathIncarnate's user avatar
4 votes
Accepted

Sanitizing words extracted from text files and writing them to a database

Memory usage What I notice most, is that this is bound to use a lot of memory. First you load all the files (at once) into memory. Then you copy over (with modifications) the data into another ...
Sjoerd Job Postmus's user avatar
4 votes

Using lots of regex substitutions to tokenize text

Is there a way to make the substitution faster? E.g. ... Combine some of the regexes Both of (NON_BREAKING, ONE_SPACE) substitute the same replacement expression, as does the triple ({OPEN,CLOSE}...
J_H's user avatar
  • 25.6k
4 votes

Recursive right-to-left segmenting (tokenizing) of strings in Python

Here are some general comments: Having a function mutate a global variable is a bad practice. In this case, it may seem easy enough to reason about what is going on, but I guarantee that at some ...
Jared Goguen's user avatar
4 votes
Accepted

Analysis of the most common and salient words in a text

First, I will start with PEP 8 specifications. The PEP 8 analysis shows the following: ...
Jishan's user avatar
  • 156
4 votes

Substitute IDs to word tokens

The code can be packed into a list comprehension: ...
Jean-François Fabre's user avatar
4 votes
Accepted

Simple natural language classifier

Welcome to Code Review! This is an interesting program; thanks for sharing! To help you maintain it ... baseDict appears to be unused, and can be removed. Ditto ...
AJNeufeld's user avatar
  • 33.5k
4 votes
Accepted

Syllabification function for Turkish words

First of all you should know that the script doesn't syllabize properly for every word. For example if you give the word authenticated the function returns ['aut', 'hen', 'ti', 'ca', 'ted'] which is ...
Giannis Papaioannou's user avatar
4 votes
Accepted

Finding word association strengths from an input text

Review Styling Import should be at the top of the file Use a if __name__ == '__main__': guard Split some functionality into function, keeping everything in the ...
Ludisposed's user avatar
  • 11.5k
4 votes

Extracting all nouns, verbs and adjectives from a large text dataset

Welcome to Code Review, here some suggestions about your code: public class review { ... } Java classnames always begin with uppercase letter so rename it to <...
dariosicily's user avatar
  • 3,961
4 votes

Text Normalizer

I'd suggest to do some profiling, or simply using timeit for measuring which part of code takes the long time, and then focus on that: ...
yedpodtrzitko's user avatar
4 votes
Accepted

Creating csvs using Pandas on large dataset for document retrieval

measurements This submission is about performance, yet it includes no profile measurements, and almost no performance data. We are told only that each of 29k articles takes an expected 25 seconds to ...
J_H's user avatar
  • 25.6k
3 votes

Negation detection in sentiment analysis

Here are a few suggestions on how to improve the code: State the intent What is the purpose of the function? What is the input, what is the output? In human words, what does the algorithm do? A ...
paulw's user avatar
  • 91
3 votes

Song lyric generator using Markov Chains - Python

Some suggestions: def unique(s): u = [] for x in s: if x not in u: u.append(x) else: pass return u may become ...
MarianD's user avatar
  • 1,876

Only top scored, non community-wiki answers of a minimum length are eligible