Questions tagged [natural-language-processing]

The field of natural language processing covers attempts to make sense of text in a human language using computers

Filter by
Sorted by
Tagged with
3
votes
0answers
38 views

Django project for events of the day, grouped by keywords in common

I have made a website where people write about their day and see it analyzed. In particular, the website takes event titles and groups them together if they have any common words (after stemming the ...
7
votes
2answers
77 views

Document classfier

Description: I am working on a classifier which categorizes the text based on some criteria, at present, it is a map of category and list of words if any of the words appear in the text, a category ...
6
votes
1answer
72 views

Ternary Search Tree / N-Gram Model in Python

I implemented a word n-gram model using a character ternary search tree. It is intended to be passed a generator that yields a long sequence of words (from a corpus) and its requirements are that it ...
2
votes
0answers
82 views

Define the scope of negation with the Dependency Parser of spaCy

Sentiment words behave very differently when under the semantic scope of negation. I want to use a slightly modified version of Das and Chen (2001) They detect words such as no, not, and never and ...
2
votes
1answer
362 views

Finding word association strengths from an input text

I have the written the following (crude) code to find the association strengths among the words in a given piece of text. ...
2
votes
0answers
43 views

Fetching and modifying CNN headlines

This is a project I came up just as an exercise with to familiarize myself with Python syntax and data types. I am just learning how to code and would like to avoid establishing bad habits. On the ...
3
votes
0answers
46 views

Download and analyze PDFs of Congressional records

This is built with Python 2.7.15. The goal of this script is to count the number of words spoken by each Senator on the floor of Congress between given dates. It pulls from the Congressional Record, ...
3
votes
1answer
52 views

Reduce the length of words in a sentence

This function's goal is to reduce the length of a sentence to exactly max_length characters by cutting each word in the sentence ...
2
votes
1answer
46 views

Counting lower vs non-lowercase tokens for tokenized text with several conditions

Assuming that the text is tokenized with whitespace for a natural language processing task, the goal is to check the count of the words (regardless of casing) and check them through some conditions. ...
1
vote
1answer
125 views

Replacing words with their abbreviations - Follow up

This is the follow up for a question you can find here: replacing words with their abbreviations The goal here is to compare two petential way of answering said question that was: This particular ...
1
vote
0answers
70 views

Language-detection heuristic (English, French or German) based on Unigram and Bigram models

Given a string, for example "I hate AI", I need to find out if the sentence is in English, German or French. Unigram Model makes the prediction on the basis of each character frequency in a training ...
1
vote
2answers
1k views

replacing words with their abbreviations

I'm working on a program that aim to take sentences (currently in french) and compact them to a length of 38 characters while retaining as much information as possible. You can find another part of ...
4
votes
1answer
109 views

Remove determiners in a string

I'm working on a program that aim to take sentences (currently in french) and compact them to a length of 38 characters while retaining as much information as possible. You can find another part of ...
3
votes
1answer
56 views

Syllabification function for Turkish words

I wrote an NLP script for processing Turkish language. Yesterday I added syllabication but I wonder if it could be done better. It is kinda hard-coded, so I would like to know if I can improve it. ...
3
votes
1answer
65 views

Simple natural language classifier

This program estimates the likelihood for a string to belong to a certain natural language by computing the cosine similarity between an input string's and several natural languages' letter frequency, ...
1
vote
0answers
35 views

Function for root matching between two paragraphs

I created an algorithm that match roots of two texts, a question and a paragraph made of sentences. I aim at predicting in which sentence it exists the answer of a question. Yet It seems that I really ...
4
votes
1answer
452 views

Relative frequency of words in tree of documents

I have a tree structure where at every node there is a list of documents (document length can vary from 5 to 500), and each document contains a number of words. I want to calculate relative frequency ...
1
vote
2answers
158 views

Substitute IDs to word tokens

I writing a function in Python to substitute word tokens in a sentence with IDs. The sentence is a list of tokens (list_of_tokens). The IDs are provided in a dictionary mapping tokens with an IDs (...
2
votes
1answer
89 views

Haskell sentence segregation

I am trying to implement sentence segregation using Haskell, I have achieved a decent bulk of it using the NLP.FullStop library, but this doesn't seem to account ...
1
vote
1answer
3k views

Generating a word bigram co-occurrence matrix

I have written a method which is designed to calculate the word co-occurrence matrix in a corpus, such that element(i,j) is the number of times that word i follows word j in the corpus. Here is my ...
5
votes
1answer
116 views

Program to count common words in documents

I finished my first program in Ruby and I would like to share it with you so I can get some suggestions or recommendations. I'd really like to hear them because I am learning and I want to have a ...
1
vote
0answers
163 views

Analysis of the most common words in a text

I have been trying to gain more understanding and experience in the NPL area and to get some more practice. I decided to attempt to create a simple high-level ...
10
votes
3answers
2k views

Analysis of the most common and salient words in a text

I've been trying to get more understanding and experience in the Natural Language Processing space and, to get some more practice, decided to attempt to make a simple high-level analysis of the "...
3
votes
1answer
3k views

Identify and extract URLs from text corpus

I'm working on a project that requires POS Tagging of paragraphs. The text contains lot of URLs which contain various punctuation marks such as . ...
6
votes
1answer
109 views

Mark V. Shaney: a script to produce gibberish

What follows in an attempt at implementing Mark V. Shaney using contemporary Python. One question has already been asked while working on a generator in the code, but included here is the entire ...
2
votes
1answer
3k views

Removing stop words from a Spark Dataframe

I am trying to apply a function to two Spark Dataframes (in Zeppelin): ...
1
vote
0answers
34 views

NER and its F Measure Calculation

I am trying to write one Name Entity Recognition in Hindi. I have primarily used NLTK of Python. I have used HMM Module with its supervised training. The data is annotated and saved in .pos files ...
3
votes
2answers
353 views

Recursive right-to-left segmenting (tokenizing) of strings in Python

I want to segment a list of strings based on a custom list of words/forms, from right-to-left. Here I've illustrated it with made-up English examples, but the point is to segment/tokenize using a ...
5
votes
5answers
157 views

Calculate LIX value of a text

I've been building a simple tool to calculate the LIX value of a text (a standard measurement of the readability of a text). My approach was to have every step of the calculation as seperate ...
4
votes
0answers
72 views

Optimally splitting a text into strings from a set

I am trying to write an algorithm that starts with a corpus of texts (e.g., a Wikipedia dump). It first builds an array of individual characters (e.g., "a", ...
3
votes
1answer
70 views

Interpreting tweets about football

I am trying to process the football tweets and extract information like goals, cards, corners, player name, team name. I write the code which works, but I may be missing some better python ...
7
votes
3answers
220 views

Linking two databases based on street addresses

For my work, I wrote a python script to link 2 files. Since I am an autodidact and since no one of my colleagues writes code, I ask the question here. My code takes an unbelievable time to run. Is it ...
5
votes
1answer
104 views

Flag words that would be difficult for an early reader

This is part of a project I made a couple of years ago and was looking at again. Its purpose is to check text for words that an early reader (about a late kindergarten or first grade level) would ...
4
votes
1answer
2k views

Random name generator in Java

I wrote working random name generator in Java. Here's my code: NameGenerator.java: ...
3
votes
0answers
53 views

Time calculation for NLTK tagging

I am trying to calculate the time required to tag one sentence/file by one trained NLTK HMM Tagger. To do this I am writing the following code, please suggest if I need to revise anything here. ...
1
vote
0answers
128 views

Naïve Bayes classifier to group questions by intent

I am trying to train a question-answer system, where I am trying to group similar questions, and identify the most apt response. The program should identify the intent/focus. To do it, I have tagged ...
8
votes
1answer
195 views

Summarize a document as a key-phrase or key-words

A few days ago I finished a coding challenge for a potential job. I was super happy with my code, till I got the response that my code wasn't good enough. :( So, apparently I'm still making mistakes, ...
12
votes
1answer
494 views

Using lots of regex substitutions to tokenize text

I authored a piece of code that was merged into the nltk codebase. It is full of regex substitutions: ...
4
votes
4answers
9k views

Cleaning and extracting meaningful text from tweets

I have a dataset of around 200,000 tweets. I am running a classification task on them. Dataset has two columns - class label and the tweet text. In the preprocessing step I am passing the dataset ...
2
votes
2answers
1k views

Define if post extract from a bilingual Facebook page are in English using Python

I am currently extracting post on a biligual page on Facebook. Therefore, I have the problem of splitting the post in French and english before starting analysing them. I have construct a function ...
6
votes
1answer
145 views

Tokenizing texts from Gutenberg archive for analysis

I am writing a program to analyze books from the Gutenberg archive. The program takes the title and URL and finds the text and downloads it. Then it goes through the text and tokenizes it. Here is the ...
2
votes
0answers
40 views

Classes to help transform “How hard is it to …?” questions into replies like “It's really hard to …”

I've written a Twitter bot, @answering_yelp, which responds to another Twitter bot, @hard_to_yelp. They scrape yelp reviews for sentences which start "How hard is it to..." and my bot answer with some ...
7
votes
1answer
42 views

Labeling modified words

I'm working on a Python function that takes a piece of text and indicates if words are modified by words like very or not. I'm looking for instance to label very nice differently from not nice. This ...
2
votes
1answer
251 views

Multiword Expression Tagging in Python

I am trying to write a small python code,where I am reading a text file-which contains both Multiwords (MWEs) and singular words (NMWEs). I am trying to tag each one, as follows. I have a ...
5
votes
2answers
980 views

Preprocessing text input to a machine-learning algorithm

I have written the following function to preprocess some text data as input to machine learning algorithm. It lowercase, tokenises, removes stop words and lemmatizes, returning a string of space-...
8
votes
1answer
6k views

NLTK sentence / word tokenize

I have a method that takes in a String parameter, and uses NLTK to break the String down to sentences, then into words. Afterwards, it converts each word into lowercase, and finally creates a ...
1
vote
0answers
293 views

Traning and testing of sentiment analysis [closed]

Here is my code which takes two files of positive and negative comments and creates a training and testing set for sentiment analysis using nltk, sklearn, Python and statistical algorithms. The ...
5
votes
1answer
104 views

Regex, match the most informative pattern

I have a function that is designed to parse an utterance (or typed in string) and identify the intent as a yes or no answer. There are many ways of saying "yes" to something, and similarly for "no" ...
1
vote
0answers
97 views

Using polymorphic objects to represent parts of a deconstructed sentence

I am trying to make a chatbot using javascript by deconstructing a sentence into its intent by separating the Noun's Verb's, Adjectives, Durations's, Numbers etc... to construct a data query to try to ...
14
votes
2answers
2k views

Song lyric generator using Markov Chains - Python

I have written a pop song generator which uses the Markovify library to produce lyrics based on (just for testing purposes) songs by Avril Lavigne. In order to make the generator a bit more ...