Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [natural-language-processing]

The field of natural language processing covers attempts to make sense of text in a human language using computers

4
votes
0answers
47 views

Ternary Search Tree / N-Gram Model in Python

I implemented a word n-gram model using a character ternary search tree. It is intended to be passed a generator that yields a long sequence of words (from a corpus) and its requirements are that it ...
2
votes
0answers
33 views

Define the scope of negation with the Dependency Parser of spaCy

Sentiment words behave very differently when under the semantic scope of negation. I want to use a slightly modified version of Das and Chen (2001) They detect words such as no, not, and never and ...
2
votes
1answer
81 views

Finding word association strengths from an input text

I have the written the following (crude) code to find the association strengths among the words in a given piece of text. ...
2
votes
0answers
40 views

Fetching and modifying CNN headlines

This is a project I came up just as an exercise with to familiarize myself with Python syntax and data types. I am just learning how to code and would like to avoid establishing bad habits. On the ...
3
votes
0answers
40 views

Download and analyze PDFs of Congressional records

This is built with Python 2.7.15. The goal of this script is to count the number of words spoken by each Senator on the floor of Congress between given dates. It pulls from the Congressional Record, ...
3
votes
1answer
50 views

Reduce the length of words in a sentence

This function's goal is to reduce the length of a sentence to exactly max_length characters by cutting each word in the sentence ...
2
votes
1answer
44 views

Counting lower vs non-lowercase tokens for tokenized text with several conditions

Assuming that the text is tokenized with whitespace for a natural language processing task, the goal is to check the count of the words (regardless of casing) and check them through some conditions. ...
1
vote
1answer
71 views

Replacing words with their abbreviations - Follow up

This is the follow up for a question you can find here: replacing words with their abbreviations The goal here is to compare two petential way of answering said question that was: This particular ...
1
vote
0answers
63 views

Language-detection heuristic (English, French or German) based on Unigram and Bigram models

Given a string, for example "I hate AI", I need to find out if the sentence is in English, German or French. Unigram Model makes the prediction on the basis of each character frequency in a training ...
1
vote
2answers
416 views

replacing words with their abbreviations

I'm working on a program that aim to take sentences (currently in french) and compact them to a length of 38 characters while retaining as much information as possible. You can find another part of ...
4
votes
1answer
89 views

Remove determiners in a string

I'm working on a program that aim to take sentences (currently in french) and compact them to a length of 38 characters while retaining as much information as possible. You can find another part of ...
3
votes
1answer
51 views

Syllabification function for Turkish words

I wrote an NLP script for processing Turkish language. Yesterday I added syllabication but I wonder if it could be done better. It is kinda hard-coded, so I would like to know if I can improve it. ...
3
votes
1answer
60 views

Simple natural language classifier

This program estimates the likelihood for a string to belong to a certain natural language by computing the cosine similarity between an input string's and several natural languages' letter frequency, ...
1
vote
0answers
33 views

Function for root matching between two paragraphs

I created an algorithm that match roots of two texts, a question and a paragraph made of sentences. I aim at predicting in which sentence it exists the answer of a question. Yet It seems that I really ...
4
votes
1answer
386 views

Relative frequency of words in tree of documents

I have a tree structure where at every node there is a list of documents (document length can vary from 5 to 500), and each document contains a number of words. I want to calculate relative frequency ...
1
vote
2answers
130 views

Substitute IDs to word tokens

I writing a function in Python to substitute word tokens in a sentence with IDs. The sentence is a list of tokens (list_of_tokens). The IDs are provided in a dictionary mapping tokens with an IDs (...
2
votes
1answer
65 views

Haskell sentence segregation

I am trying to implement sentence segregation using Haskell, I have achieved a decent bulk of it using the NLP.FullStop library, but this doesn't seem to account ...
1
vote
1answer
3k views

Generating a word bigram co-occurrence matrix

I have written a method which is designed to calculate the word co-occurrence matrix in a corpus, such that element(i,j) is the number of times that word i follows word j in the corpus. Here is my ...
5
votes
1answer
115 views

Program to count common words in documents

I finished my first program in Ruby and I would like to share it with you so I can get some suggestions or recommendations. I'd really like to hear them because I am learning and I want to have a ...
1
vote
0answers
146 views

Analysis of the most common words in a text

I have been trying to gain more understanding and experience in the NPL area and to get some more practice. I decided to attempt to create a simple high-level ...
10
votes
3answers
1k views

Analysis of the most common and salient words in a text

I've been trying to get more understanding and experience in the Natural Language Processing space and, to get some more practice, decided to attempt to make a simple high-level analysis of the "...
3
votes
1answer
2k views

Identify and extract URLs from text corpus

I'm working on a project that requires POS Tagging of paragraphs. The text contains lot of URLs which contain various punctuation marks such as . ...
6
votes
1answer
105 views

Mark V. Shaney: a script to produce gibberish

What follows in an attempt at implementing Mark V. Shaney using contemporary Python. One question has already been asked while working on a generator in the code, but included here is the entire ...
2
votes
1answer
2k views

Removing stop words from a Spark Dataframe

I am trying to apply a function to two Spark Dataframes (in Zeppelin): ...
1
vote
0answers
33 views

NER and its F Measure Calculation

I am trying to write one Name Entity Recognition in Hindi. I have primarily used NLTK of Python. I have used HMM Module with its supervised training. The data is annotated and saved in .pos files ...
3
votes
2answers
310 views

Recursive right-to-left segmenting (tokenizing) of strings in Python

I want to segment a list of strings based on a custom list of words/forms, from right-to-left. Here I've illustrated it with made-up English examples, but the point is to segment/tokenize using a ...
5
votes
5answers
139 views

Calculate LIX value of a text

I've been building a simple tool to calculate the LIX value of a text (a standard measurement of the readability of a text). My approach was to have every step of the calculation as seperate ...
4
votes
0answers
70 views

Optimally splitting a text into strings from a set

I am trying to write an algorithm that starts with a corpus of texts (e.g., a Wikipedia dump). It first builds an array of individual characters (e.g., "a", ...
3
votes
1answer
68 views

Interpreting tweets about football

I am trying to process the football tweets and extract information like goals, cards, corners, player name, team name. I write the code which works, but I may be missing some better python ...
7
votes
3answers
218 views

Linking two databases based on street addresses

For my work, I wrote a python script to link 2 files. Since I am an autodidact and since no one of my colleagues writes code, I ask the question here. My code takes an unbelievable time to run. Is it ...
5
votes
1answer
103 views

Flag words that would be difficult for an early reader

This is part of a project I made a couple of years ago and was looking at again. Its purpose is to check text for words that an early reader (about a late kindergarten or first grade level) would ...
4
votes
1answer
1k views

Random name generator in Java

I wrote working random name generator in Java. Here's my code: NameGenerator.java: ...
3
votes
0answers
51 views

Time calculation for NLTK tagging

I am trying to calculate the time required to tag one sentence/file by one trained NLTK HMM Tagger. To do this I am writing the following code, please suggest if I need to revise anything here. ...
1
vote
0answers
126 views

Naïve Bayes classifier to group questions by intent

I am trying to train a question-answer system, where I am trying to group similar questions, and identify the most apt response. The program should identify the intent/focus. To do it, I have tagged ...
8
votes
1answer
180 views

Summarize a document as a key-phrase or key-words

A few days ago I finished a coding challenge for a potential job. I was super happy with my code, till I got the response that my code wasn't good enough. :( So, apparently I'm still making mistakes, ...
12
votes
1answer
456 views

Using lots of regex substitutions to tokenize text

I authored a piece of code that was merged into the nltk codebase. It is full of regex substitutions: ...
4
votes
4answers
8k views

Cleaning and extracting meaningful text from tweets

I have a dataset of around 200,000 tweets. I am running a classification task on them. Dataset has two columns - class label and the tweet text. In the preprocessing step I am passing the dataset ...
3
votes
2answers
653 views

Define if post extract from a bilingual Facebook page are in English using Python

I am currently extracting post on a biligual page on Facebook. Therefore, I have the problem of splitting the post in French and english before starting analysing them. I have construct a function ...
6
votes
1answer
129 views

Tokenizing texts from Gutenberg archive for analysis

I am writing a program to analyze books from the Gutenberg archive. The program takes the title and URL and finds the text and downloads it. Then it goes through the text and tokenizes it. Here is the ...
2
votes
0answers
40 views

Classes to help transform “How hard is it to …?” questions into replies like “It's really hard to …”

I've written a Twitter bot, @answering_yelp, which responds to another Twitter bot, @hard_to_yelp. They scrape yelp reviews for sentences which start "How hard is it to..." and my bot answer with some ...
7
votes
1answer
42 views

Labeling modified words

I'm working on a Python function that takes a piece of text and indicates if words are modified by words like very or not. I'm looking for instance to label very nice differently from not nice. This ...
3
votes
1answer
236 views

Multiword Expression Tagging in Python

I am trying to write a small python code,where I am reading a text file-which contains both Multiwords (MWEs) and singular words (NMWEs). I am trying to tag each one, as follows. I have a ...
5
votes
2answers
918 views

Preprocessing text input to a machine-learning algorithm

I have written the following function to preprocess some text data as input to machine learning algorithm. It lowercase, tokenises, removes stop words and lemmatizes, returning a string of space-...
8
votes
1answer
5k views

NLTK sentence / word tokenize

I have a method that takes in a String parameter, and uses NLTK to break the String down to sentences, then into words. Afterwards, it converts each word into lowercase, and finally creates a ...
2
votes
0answers
280 views

Traning and testing of sentiment analysis

Here is my code which takes two files of positive and negative comments and creates a training and testing set for sentiment analysis using nltk, sklearn, Python and statistical algorithms. The ...
5
votes
1answer
97 views

Regex, match the most informative pattern

I have a function that is designed to parse an utterance (or typed in string) and identify the intent as a yes or no answer. There are many ways of saying "yes" to something, and similarly for "no" ...
1
vote
0answers
96 views

Using polymorphic objects to represent parts of a deconstructed sentence

I am trying to make a chatbot using javascript by deconstructing a sentence into its intent by separating the Noun's Verb's, Adjectives, Durations's, Numbers etc... to construct a data query to try to ...
14
votes
2answers
2k views

Song lyric generator using Markov Chains - Python

I have written a pop song generator which uses the Markovify library to produce lyrics based on (just for testing purposes) songs by Avril Lavigne. In order to make the generator a bit more ...
0
votes
1answer
7k views

Read a CSV file and do natural language processing on the data

I am studying the techniques of data mining and data processing. I'm doing this through data I've collected and stored in a csv file. The problem is that this filed was very large, to the point of ...
1
vote
0answers
63 views

Linear SVM(in R) giving suspiciously high CV accuracy

Here is my code that reads a text dataset, featurizes the text column and then does CV using Linear SVM. Could I be leaking labels from somewhere as the accuracy I get is suspiciously high and way ...