Questions tagged [natural-language-processing]

The field of natural language processing covers attempts to make sense of text in a human language using computers

Filter by
Sorted by
Tagged with
19
votes
1answer
819 views

Spell Corrector in C

I recently stumbled across this article on how to write a spelling corrector, and figured I'd try to have a go at it in C (mainly because the link at the end of the page for the C code is broken). ...
16
votes
4answers
789 views

checkmate - C spelling corrector 2.0

Since I posted my first version of the spelling corrector here, I've been working on improving it a little in my free time. I've also gone ahead and put the project up on Github so that others can ...
14
votes
2answers
2k views

Song lyric generator using Markov Chains - Python

I have written a pop song generator which uses the Markovify library to produce lyrics based on (just for testing purposes) songs by Avril Lavigne. In order to make the generator a bit more ...
13
votes
2answers
707 views

Speeding up Python depth-first search in large graph

I'm building a Markov text poetry generator. The main method, which is attached below, performs a depth-first search through the Markov chain, which is implemented as a NetworkX digraph of Word ...
12
votes
1answer
414 views

Make a summary from a larger text-file

This code makes summaries from larger texts. I have searched around for an algorithm and found the following: Associate words with their grammatical counterparts. (e.g. "city" and "cities") ...
12
votes
1answer
481 views

Using lots of regex substitutions to tokenize text

I authored a piece of code that was merged into the nltk codebase. It is full of regex substitutions: ...
10
votes
3answers
1k views

Analysis of the most common and salient words in a text

I've been trying to get more understanding and experience in the Natural Language Processing space and, to get some more practice, decided to attempt to make a simple high-level analysis of the "...
8
votes
2answers
1k views

Using Viterbi algorithm to analyze sentences

I've probably done some pretty horrendous things here, but I'm throwing it out for people to give me some feedback that I can start using to immediately improve my Clojure coding style. Additional ...
8
votes
2answers
14k views

Splitting text into n-grams and analyzing statistics on them

I have made the algorithm that split text into n-grams (collocations) and it counts probabilities and other statistics of this collocations. When file is more then 50 megabytes it takes long time to ...
8
votes
2answers
300 views

Port of NLTK tokenizing code from Python to Rust

I'm working on a port of NLTK to Rust. I am fairly new to Rust, so I wanted to post a small file to check if I was writing idiomatic Rust. I've included the original Python. The Python has docstrings ...
8
votes
1answer
5k views

NLTK sentence / word tokenize

I have a method that takes in a String parameter, and uses NLTK to break the String down to sentences, then into words. Afterwards, it converts each word into lowercase, and finally creates a ...
8
votes
1answer
1k views

Three-way conversion between Japanese writing systems

When my go-to Japanese transcription site went down for a while, I decided to write my own. My application converts between Romaji, Hiragana and Katakana — however, unlike any other converter I'...
8
votes
1answer
193 views

Summarize a document as a key-phrase or key-words

A few days ago I finished a coding challenge for a potential job. I was super happy with my code, till I got the response that my code wasn't good enough. :( So, apparently I'm still making mistakes, ...
8
votes
1answer
2k views

Fuzzy / approximate text matching program in Python

I'm trying to build a program that will find approximate text matches between two texts. Basically, I'd like to find a way of identifying quotations, so that when an author quotes, say, the King James ...
7
votes
1answer
42 views

Labeling modified words

I'm working on a Python function that takes a piece of text and indicates if words are modified by words like very or not. I'm looking for instance to label very nice differently from not nice. This ...
7
votes
3answers
219 views

Linking two databases based on street addresses

For my work, I wrote a python script to link 2 files. Since I am an autodidact and since no one of my colleagues writes code, I ask the question here. My code takes an unbelievable time to run. Is it ...
7
votes
1answer
160 views

Korean Romanization to Hangeul library

After a little work, I've finally ironed this out... Though could I get someone to go over this and see if there is a better way to present this library within the code, before I publish it? Premise: ...
7
votes
1answer
396 views

Implementing a (small) full-text search in Ruby

I'm trying to implement a small full-text search in Ruby. I feel like I've laid down the foundation but I've encountered a blocker that has gotten me thinking that something with the design is ...
6
votes
1answer
16k views

Removing all stopwords from a list of words

What is the fastest Pythonic way to remove all stopwords from a list of words in a document? Right now I am using a list comprehension that contains a for loop. <...
6
votes
2answers
1k views

N-gram generation

I have the following code, but it's too slow. How can I make it faster? ...
6
votes
1answer
140 views

Tokenizing texts from Gutenberg archive for analysis

I am writing a program to analyze books from the Gutenberg archive. The program takes the title and URL and finds the text and downloads it. Then it goes through the text and tokenizes it. Here is the ...
6
votes
3answers
501 views

Korean word segmentation using frequency heuristic

This a continuation of a previous question. I want to thank Joe Wallis for his help with increasing the readability of my code. Although the changes made by Joe Wallis did increase the speed of the ...
6
votes
1answer
106 views

Mark V. Shaney: a script to produce gibberish

What follows in an attempt at implementing Mark V. Shaney using contemporary Python. One question has already been asked while working on a generator in the code, but included here is the entire ...
6
votes
1answer
154 views

TrieSet with Wildcard Search

What I'm ultimately trying to accomplish is a tool in which I pour some documents. Then I mark #x documents as desired and #y documents as undesired. My tool now is to analyse and compare the ...
6
votes
1answer
170 views

Part of Speech Tagger

I have a Part of Speech Tagger written in PHP that currently takes a relatively long and unpredictable time to execute - just wanted to know if any of you knew of any quick ways I can improve time ...
6
votes
1answer
59 views

Bayesian spam filter interface

In essence, what I'm looking to do is split a string into one-word and two-word segments, and train classifiers against them. First it lowercasifies the string, then strips all non-alphanumeric ...
6
votes
0answers
50 views

Document classfier

Description: I am working on a classifier which categorizes the text based on some criteria, at present, it is a map of category and list of words if any of the words appear in the text, a category ...
5
votes
5answers
146 views

Calculate LIX value of a text

I've been building a simple tool to calculate the LIX value of a text (a standard measurement of the readability of a text). My approach was to have every step of the calculation as seperate ...
5
votes
1answer
561 views

Markov chain text generation in Python

I'm willing to listen to any advice you have for improving it. ...
5
votes
2answers
19k views
5
votes
1answer
104 views

Flag words that would be difficult for an early reader

This is part of a project I made a couple of years ago and was looking at again. Its purpose is to check text for words that an early reader (about a late kindergarten or first grade level) would ...
5
votes
2answers
965 views

Preprocessing text input to a machine-learning algorithm

I have written the following function to preprocess some text data as input to machine learning algorithm. It lowercase, tokenises, removes stop words and lemmatizes, returning a string of space-...
5
votes
1answer
100 views

Regex, match the most informative pattern

I have a function that is designed to parse an utterance (or typed in string) and identify the intent as a yes or no answer. There are many ways of saying "yes" to something, and similarly for "no" ...
5
votes
2answers
317 views

Calculating cooccurrence probabilities for pairs of words in a document

It is a 1.5 hour coding test, started the moment when the question was sent by email. My solution was done under the strict condition. I was not told anything before the test. The question is about ...
5
votes
1answer
452 views

Assigning sentiment to each tweet - Twitter trend

Below assignment is taken from here. Introduction In this project, you will develop a geographic visualization of twitter data across the USA. You will need to use dictionaries, lists, and ...
5
votes
1answer
115 views

Program to count common words in documents

I finished my first program in Ruby and I would like to share it with you so I can get some suggestions or recommendations. I'd really like to hear them because I am learning and I want to have a ...
4
votes
1answer
436 views

Relative frequency of words in tree of documents

I have a tree structure where at every node there is a list of documents (document length can vary from 5 to 500), and each document contains a number of words. I want to calculate relative frequency ...
4
votes
4answers
8k views

Cleaning and extracting meaningful text from tweets

I have a dataset of around 200,000 tweets. I am running a classification task on them. Dataset has two columns - class label and the tweet text. In the preprocessing step I am passing the dataset ...
4
votes
2answers
160 views

Bayes rating selective term filtering

I am working on a project, and at the moment I have just finished the majority of the Bayes Rating/Classifier. I have two sets of methods I am most concerned about. Method set 1, will select a ...
4
votes
1answer
461 views

Back propagation neural network

This is the first time I tried to write a back propagation ANN and I would like to know what more experienced people think of it. The code is meant to distinguish if text is written in English, French ...
4
votes
1answer
94 views

Remove determiners in a string

I'm working on a program that aim to take sentences (currently in french) and compact them to a length of 38 characters while retaining as much information as possible. You can find another part of ...
4
votes
1answer
1k views

Random name generator in Java

I wrote working random name generator in Java. Here's my code: NameGenerator.java: ...
4
votes
1answer
2k views

Sanitizing words extracted from text files and writing them to a database

This code reads text files from some path, tokenizes, removes stop words, lowercases, removes punctuation and numbers, then writes the result to a database. ...
4
votes
1answer
355 views

Script to generate documents based on conditionals and CSV data

I wrote this script a while back to generate press releases. It consumes a csv of data where each row contains the data for a country and fills in values in pre-written text based on the data. In its ...
4
votes
1answer
65 views

A system for determination of human language from input text:: robustification

This program works, but it's pretty clunky and fragile. I want to make it more dynamic / solid. For instance some of the functionality depends on the order I entered things into an ...
4
votes
1answer
930 views

Generate and store hypernyms for all words in a hashmap

I have a system which reads in a clause in the form of a prolog "fact", i.e. 'is'('a sentence', 'this').. I want to generalize this up into higher-order classes and ...
4
votes
1answer
151 views

Haikuifier (Or at least Haiku Identifier)

All the usual stuff. Style, substance, algorithm please! I'm vaguely considering plugging it into a bot, hence the lacklustre catching of exceptions right now. ...
4
votes
1answer
587 views

Extract entities from sentences

I have a function that extracts basic parts-of-speech words (NN,NNP,JJ,VBS etc.) from about 2000 sentences. I want to understand if there is a way to optimize so I can bring down the execution time (...
4
votes
1answer
134 views

Simple string-root detection in a string-family

(This problem is related to Simple string-split by root and sufix algorithm) There are many ways to find a "common root" of a list of similar strings, that begins with the same substring... The ...
4
votes
0answers
57 views

Ternary Search Tree / N-Gram Model in Python

I implemented a word n-gram model using a character ternary search tree. It is intended to be passed a generator that yields a long sequence of words (from a corpus) and its requirements are that it ...