Questions tagged [natural-language-processing]

The field of natural language processing covers attempts to make sense of text in a human language using computers

Filter by
Sorted by
Tagged with
1
vote
0answers
63 views

Linear SVM(in R) giving suspiciously high CV accuracy

Here is my code that reads a text dataset, featurizes the text column and then does CV using Linear SVM. Could I be leaking labels from somewhere as the accuracy I get is suspiciously high and way ...
1
vote
0answers
1k views

Extracting names from a body of text

I am trying to extract names from a body of text to use as stopwords. I tried a few different approaches to identifying names (or proper nouns in general) below. The second approach is much faster ...
3
votes
0answers
111 views

Converting a sentence string to a number in Scheme

I wrote a scheme procedure that converts strings such as "Two hundred and fifty" into a number like "250" that I can use in math calculations (It's for some Natural Language Processing project). Is ...
3
votes
0answers
212 views

Naive implementation of Naive Bayes in Haskell

A simple implementation of Naive Bayes, as I'm a Haskell beginner I've tried to put an emphasis on clarity and documentation. I've tried to incorporate the feedback from a similar question. This is ...
2
votes
2answers
2k views

Negation detection in sentiment analysis

This basic function to detect negation while performing sentiment analysis. So that not good kind of words can be considered a negative. This code is working fine. ...
4
votes
1answer
587 views

Extract entities from sentences

I have a function that extracts basic parts-of-speech words (NN,NNP,JJ,VBS etc.) from about 2000 sentences. I want to understand if there is a way to optimize so I can bring down the execution time (...
2
votes
1answer
342 views

Telegram Bot That Retrieves News Articles Based On User Requests

This code is for a Telegram bot I created. Basically if the user types in Whats the news,Whats the technology news,...
3
votes
0answers
138 views

Read in file, Filter out records, output results

I have been working on a project for some time and have let one Python file grow into a huge file that tries to do everything. The structure is pretty flat and I don't like it. I've used other ...
4
votes
1answer
152 views

Haikuifier (Or at least Haiku Identifier)

All the usual stuff. Style, substance, algorithm please! I'm vaguely considering plugging it into a bot, hence the lacklustre catching of exceptions right now. ...
5
votes
2answers
317 views

Calculating cooccurrence probabilities for pairs of words in a document

It is a 1.5 hour coding test, started the moment when the question was sent by email. My solution was done under the strict condition. I was not told anything before the test. The question is about ...
1
vote
1answer
751 views

Entity resolution with NLTK

I am trying to write a script of Python code, for entity extraction and resolution. The excerpts of the algorithm: It is trying to extract the entity as PoS Tag with Hidden Markov Model(HMM). After ...
4
votes
1answer
2k views

Sanitizing words extracted from text files and writing them to a database

This code reads text files from some path, tokenizes, removes stop words, lowercases, removes punctuation and numbers, then writes the result to a database. ...
5
votes
2answers
19k views
2
votes
1answer
1k views

Clustering similar tweets in a corpus

I am attempting to write a statistical program using an LDA model I've trained/created using Gensim. I am very new to Python and am a student level programmer. This current program is working and ...
12
votes
1answer
415 views

Make a summary from a larger text-file

This code makes summaries from larger texts. I have searched around for an algorithm and found the following: Associate words with their grammatical counterparts. (e.g. "city" and "cities") ...
3
votes
3answers
6k views

Search in a big dictionary Python

I have a big Python dictionary with more then 150,000 keys every key has list value. This dictionary contains infinitive words as key and all grammatical forms of these words as values: ...
8
votes
2answers
14k views

Splitting text into n-grams and analyzing statistics on them

I have made the algorithm that split text into n-grams (collocations) and it counts probabilities and other statistics of this collocations. When file is more then 50 megabytes it takes long time to ...
6
votes
3answers
501 views

Korean word segmentation using frequency heuristic

This a continuation of a previous question. I want to thank Joe Wallis for his help with increasing the readability of my code. Although the changes made by Joe Wallis did increase the speed of the ...
8
votes
1answer
2k views

Fuzzy / approximate text matching program in Python

I'm trying to build a program that will find approximate text matches between two texts. Basically, I'd like to find a way of identifying quotations, so that when an author quotes, say, the King James ...
5
votes
1answer
561 views

Markov chain text generation in Python

I'm willing to listen to any advice you have for improving it. ...
6
votes
1answer
154 views

TrieSet with Wildcard Search

What I'm ultimately trying to accomplish is a tool in which I pour some documents. Then I mark #x documents as desired and #y documents as undesired. My tool now is to analyse and compare the ...
6
votes
1answer
170 views

Part of Speech Tagger

I have a Part of Speech Tagger written in PHP that currently takes a relatively long and unpredictable time to execute - just wanted to know if any of you knew of any quick ways I can improve time ...
8
votes
2answers
301 views

Port of NLTK tokenizing code from Python to Rust

I'm working on a port of NLTK to Rust. I am fairly new to Rust, so I wanted to post a small file to check if I was writing idiomatic Rust. I've included the original Python. The Python has docstrings ...
3
votes
1answer
173 views

Verb conjugator for French

I had written with very few PHP skills a conjugator script. It was not good, so I decided to start it from scratch with a help from a friend. I'm using the code formatter from eclipse. Now I need ...
8
votes
1answer
1k views

Three-way conversion between Japanese writing systems

When my go-to Japanese transcription site went down for a while, I decided to write my own. My application converts between Romaji, Hiragana and Katakana — however, unlike any other converter I'...
3
votes
2answers
117 views

Turkish word filtering

I need to filter Turkish words according to vovel alternation rules. For example -ler is plural but can have form of -lar thus I use metaphonemes for these transitive allomorphs as vovel_dict in my ...
3
votes
1answer
1k views

Latent Dirichlet Allocation in Python

I've recently finished writing a "simple-as-possible" LDA code in Python. The theory from which I've developed my code can be found in the book Computer Vision by Simon Prince, free (courtesy of ...
13
votes
2answers
709 views

Speeding up Python depth-first search in large graph

I'm building a Markov text poetry generator. The main method, which is attached below, performs a depth-first search through the Markov chain, which is implemented as a NetworkX digraph of Word ...
2
votes
2answers
1k views

Finding top most N common character grams in English

I suspect there might be a much more efficient way to accomplish this task. I would appreciate a more clever hint for performance increase. I am also looking for feedback on style and simplification ...
7
votes
1answer
396 views

Implementing a (small) full-text search in Ruby

I'm trying to implement a small full-text search in Ruby. I feel like I've laid down the foundation but I've encountered a blocker that has gotten me thinking that something with the design is ...
16
votes
4answers
789 views

checkmate - C spelling corrector 2.0

Since I posted my first version of the spelling corrector here, I've been working on improving it a little in my free time. I've also gone ahead and put the project up on Github so that others can ...
6
votes
1answer
59 views

Bayesian spam filter interface

In essence, what I'm looking to do is split a string into one-word and two-word segments, and train classifiers against them. First it lowercasifies the string, then strips all non-alphanumeric ...
0
votes
1answer
973 views

Converting from POS tagged word-tokens to POS tagged Phrases

I have sentences expressed as Parts of speech (POS) tagged words. I want to have all the short phrases in them to be joined up with underscores. I want them to have the Part of speech tag of the final ...
4
votes
2answers
160 views

Bayes rating selective term filtering

I am working on a project, and at the moment I have just finished the majority of the Bayes Rating/Classifier. I have two sets of methods I am most concerned about. Method set 1, will select a ...
19
votes
1answer
825 views

Spell Corrector in C

I recently stumbled across this article on how to write a spelling corrector, and figured I'd try to have a go at it in C (mainly because the link at the end of the page for the C code is broken). ...
7
votes
1answer
160 views

Korean Romanization to Hangeul library

After a little work, I've finally ironed this out... Though could I get someone to go over this and see if there is a better way to present this library within the code, before I publish it? Premise: ...
4
votes
1answer
134 views

Simple string-root detection in a string-family

(This problem is related to Simple string-split by root and sufix algorithm) There are many ways to find a "common root" of a list of similar strings, that begins with the same substring... The ...
3
votes
1answer
97 views

“Toy” human-language detection software

I've written a small program to detect the human language of a document or text fragment. I tried to stick to good design principles and I tried to make it pretty robust. I would be generously ...
5
votes
1answer
452 views

Assigning sentiment to each tweet - Twitter trend

Below assignment is taken from here. Introduction In this project, you will develop a geographic visualization of twitter data across the USA. You will need to use dictionaries, lists, and ...
4
votes
1answer
65 views

A system for determination of human language from input text:: robustification

This program works, but it's pretty clunky and fragile. I want to make it more dynamic / solid. For instance some of the functionality depends on the order I entered things into an ...
6
votes
1answer
16k views

Removing all stopwords from a list of words

What is the fastest Pythonic way to remove all stopwords from a list of words in a document? Right now I am using a list comprehension that contains a for loop. <...
3
votes
0answers
80 views

Generation and parsing of English numerals (cardinal and ordinal)

I need to use the English numerals (American) in Haskell, so I looked for a library that did it. I have not found one, therefore I developed the program that I am presenting here. There are actually ...
3
votes
1answer
35 views

Formatting output

This code has loops under loops which affects the performance badly. Please help me to optimize the code to improve its performance. ...
4
votes
1answer
355 views

Script to generate documents based on conditionals and CSV data

I wrote this script a while back to generate press releases. It consumes a csv of data where each row contains the data for a country and fills in values in pre-written text based on the data. In its ...
4
votes
1answer
930 views

Generate and store hypernyms for all words in a hashmap

I have a system which reads in a clause in the form of a prolog "fact", i.e. 'is'('a sentence', 'this').. I want to generalize this up into higher-order classes and ...
2
votes
1answer
7k views

Identifying “Matching Bigrams” in Large Text Collection

I have a large number of plain text files (north of 20 GB), and I wish to find all "matching" "bigrams" between any two texts in this collection. More specifically, my workflow looks like this: for ...
2
votes
2answers
2k views

NLTK language detection code in Python

I need to write some code that checks thousands of websites, to determine if they are in English or not. Below is the source code. Any improvements would be appreciated. ...
8
votes
2answers
1k views

Using Viterbi algorithm to analyze sentences

I've probably done some pretty horrendous things here, but I'm throwing it out for people to give me some feedback that I can start using to immediately improve my Clojure coding style. Additional ...
4
votes
1answer
461 views

Back propagation neural network

This is the first time I tried to write a back propagation ANN and I would like to know what more experienced people think of it. The code is meant to distinguish if text is written in English, French ...
6
votes
2answers
1k views

N-gram generation

I have the following code, but it's too slow. How can I make it faster? ...