12
votes
Relative frequency of words in tree of documents
There are two reasons for the poor performance of the code in the post: first, it performs much unnecessary work, and, second, the operations that it does perform are carried out using inefficient ...
- 49.7k
8
votes
NLTK sentence / word tokenize
1. Quick review:
The collections.Counter class has an update method that adds counts for items in an iterable. So instead of:
<...
- 49.7k
7
votes
Separating a String of Text into Separate Words in Python
While your program is functional and solid, there are plenty of improvements to be made.
How I would solve it.
A better way to solve this would probably be:
...
- 1,063
6
votes
Using lots of regex substitutions to tokenize text
3x to 4x speedup using str.translate()
Based on a quick test (see below), str.translate() is an order of magnitude faster than a regular expression for replacing a ...
- 10k
6
votes
Accepted
Preprocessing steps to follow while cleaning and extracting text data from tweets
Copying my answer from SO:
You can use pandas vectorized string methods to do your processing and it also removes the for loop ...
- 176
6
votes
Accepted
Tokenizing texts from Gutenberg archive for analysis
This is the ideal place for a class. Each book is its own object with its own method of returning its tokens. I would make a method tokens, which I would make a ...
- 41k
6
votes
Accepted
Wordcloud from all answers of a user here on CR
Quick bits
You have some issues that some linters would pick up:
I would suggest moving your main code into a function. So that it doesn't pollute the global namespace.
You've got some trailing ...
- 43.4k
5
votes
Accepted
Flag words that would be difficult for an early reader
The code you presented is not readable and definitely needs refactoring.
You can simplify several code constructions and apply the "Extract Variable" refactoring method to improve readability of the ...
- 17.3k
5
votes
Accepted
Labeling modified words
When there're comments in the code that tell what it does, it usually indicates that the following piece of code should be a separate function with a meaningful name. I'd create a separate function ...
- 5,580
5
votes
Accepted
Substitute IDs to word tokens
There is nothing wrong in general with for loops. If you have to iterate over some elements there is no magic that can avoid this. So why is your function "not very efficient"? And why is this because ...
- 2,980
5
votes
Accepted
Remove determiners in a string
Thanks for sharing your code.
It's a nice project you have there.
Naming
You should take some time to choose carefully your variable. l1 is not obvious, maybe <...
- 821
5
votes
Accepted
Tokenizing SGML text for NLTK analysis
Regex compilation
If performance is a concern, this:
arr = [re.sub(pattern, '', i) for i in arr]
is a problem. You're re-compiling your regex on every function ...
- 63.1k
5
votes
Separating a String of Text into Separate Words in Python
_char_to_class
First of all, let's take a look at your _char_to_class method.
...
- 1,685
4
votes
Using lots of regex substitutions to tokenize text
Is there a way to make the substitution faster? E.g. ... Combine some of the regexes
Both of (NON_BREAKING, ONE_SPACE) substitute the same replacement expression, as does the triple ({OPEN,CLOSE}...
- 32.6k
4
votes
Recursive right-to-left segmenting (tokenizing) of strings in Python
Here are some general comments:
Having a function mutate a global variable is a bad practice. In this case, it may seem easy enough to reason about what is going on, but I guarantee that at some ...
- 872
4
votes
Accepted
Analysis of the most common and salient words in a text
First, I will start with PEP 8 specifications. The PEP 8 analysis shows the following:
...
- 156
4
votes
Substitute IDs to word tokens
The code can be packed into a list comprehension:
...
4
votes
Accepted
Simple natural language classifier
Welcome to Code Review!
This is an interesting program; thanks for sharing!
To help you maintain it ...
baseDict appears to be unused, and can be removed. Ditto ...
- 33.9k
4
votes
Accepted
Syllabification function for Turkish words
First of all you should know that the script doesn't syllabize properly for every word. For example if you give the word authenticated the function returns ['aut', 'hen', 'ti', 'ca', 'ted'] which is ...
4
votes
Accepted
Finding word association strengths from an input text
Review
Styling
Import should be at the top of the file
Use a if __name__ == '__main__': guard
Split some functionality into function, keeping everything in the ...
- 11.6k
4
votes
Extracting all nouns, verbs and adjectives from a large text dataset
Welcome to Code Review, here some suggestions about your code:
public class review { ... }
Java classnames always begin with uppercase letter so rename it to <...
- 3,971
4
votes
Text Normalizer
I'd suggest to do some profiling, or simply using timeit for measuring which part of code takes the long time, and then focus on that:
...
- 691
4
votes
Accepted
Creating csvs using Pandas on large dataset for document retrieval
measurements
This submission is about performance, yet it includes no
profile
measurements, and almost no performance data.
We are told only that each of 29k articles takes an expected 25 seconds
to ...
- 32.6k
3
votes
Accepted
Random name generator in Java
Consider Regex
Your WordChecker class has numerous helper methods that all bubble up to just pattern matching. This is exactly why Regex exists, Java has ...
- 9,829
3
votes
Accepted
Summarize a document as a key-phrase or key-words
I didn't comb through your code in detail, partly because I doubt the people who gave you this task did either. I presume it actually does what it is supposed to do, and properly accomplishes the ...
- 1,259
3
votes
Accepted
Preprocessing text input to a machine-learning algorithm
Given that you are already using Python, I would highly recommend using Spacy (base text parsing & tagging) and Textacy (higher level text processing built on top of Spacy). It can do everything ...
- 146
3
votes
Negation detection in sentiment analysis
Here are a few suggestions on how to improve the code:
State the intent
What is the purpose of the function? What is the input, what is the output? In human words, what does the algorithm do? A ...
- 91
3
votes
Song lyric generator using Markov Chains - Python
Some suggestions:
def unique(s):
u = []
for x in s:
if x not in u:
u.append(x)
else:
pass
return u
may become ...
- 1,876
3
votes
Interpreting tweets about football
Here's some little nitpicks:
There really shouldn't be a space here:
self.football_teams .append(name)
Should be:
...
- 4,500
3
votes
Accepted
Calculate LIX value of a text
Your code could be simplified and better organized.
You seem to be expecting HTML, as evidenced by various calls that look like ...
- 144k
Only top scored, non community-wiki answers of a minimum length are eligible