Questions tagged [natural-language-processing]

The field of natural language processing covers attempts to make sense of text in a human language using computers

Filter by
Sorted by
Tagged with
3
votes
1answer
162 views

Reduce run time of NLP approximate matching code

The code below matches a list of features to a large corpus and returns the sub-query match with a score above 80. The challenge is the list of features on the full data-set is > 5,000 and comparing ...
2
votes
2answers
93 views

Simplified Pig Latin translator in APL

I wrote a simplified pig latin translator in APL and I would like some feedback on it, as I am not sure my implementation is neat enough. The simplified pig latin translation follows the following ...
7
votes
2answers
249 views

Extracting all nouns, verbs and adjectives from a large text dataset

For a project I am using the yelp dataset (found here: https://www.yelp.com/dataset) to create a Hashset of all verbs, nouns and adjectives found in the restaurant reviews. I have it up and running ...
10
votes
1answer
110 views

Wordcloud from all answers of a user here on CR

Since I haven't really used Python's new async features yet, I took some older code of mine, which took all of my answers here on Code Review and generated a word cloud from them, and updated it to ...
8
votes
2answers
392 views

NLP sentiment analysis in Norwegian

Please keep in mind that I am very new to data science and completely new to NLP! I am trying to create a model to classify customer reviews as either negative or positive. However, my main problem is ...
6
votes
2answers
252 views

Tokenizing SGML text for NLTK analysis

I have an NLTK parsing function that I am using to parse a ~2GB text file of a TREC dataset. The goal for this dataset is tokenize the entire collection, perform some calculations (such as calculating ...
3
votes
0answers
48 views

Django project for events of the day, grouped by keywords in common

I have made a website where people write about their day and see it analyzed. In particular, the website takes event titles and groups them together if they have any common words (after stemming the ...
7
votes
2answers
80 views

Document classfier

Description: I am working on a classifier which categorizes the text based on some criteria, at present, it is a map of category and list of words if any of the words appear in the text, a category ...
6
votes
1answer
123 views

Ternary Search Tree / N-Gram Model in Python

I implemented a word n-gram model using a character ternary search tree. It is intended to be passed a generator that yields a long sequence of words (from a corpus) and its requirements are that it ...
2
votes
0answers
169 views

Define the scope of negation with the Dependency Parser of spaCy

Sentiment words behave very differently when under the semantic scope of negation. I want to use a slightly modified version of Das and Chen (2001) They detect words such as no, not, and never and ...
2
votes
1answer
813 views

Finding word association strengths from an input text

I have the written the following (crude) code to find the association strengths among the words in a given piece of text. ...
2
votes
0answers
46 views

Fetching and modifying CNN headlines

This is a project I came up just as an exercise with to familiarize myself with Python syntax and data types. I am just learning how to code and would like to avoid establishing bad habits. On the ...
4
votes
0answers
49 views

Download and analyze PDFs of Congressional records

This is built with Python 2.7.15. The goal of this script is to count the number of words spoken by each Senator on the floor of Congress between given dates. It pulls from the Congressional Record, ...
3
votes
1answer
54 views

Reduce the length of words in a sentence

This function's goal is to reduce the length of a sentence to exactly max_length characters by cutting each word in the sentence ...
2
votes
1answer
63 views

Counting lower vs non-lowercase tokens for tokenized text with several conditions

Assuming that the text is tokenized with whitespace for a natural language processing task, the goal is to check the count of the words (regardless of casing) and check them through some conditions. ...
1
vote
1answer
182 views

Replacing words with their abbreviations - Follow up

This is the follow up for a question you can find here: replacing words with their abbreviations The goal here is to compare two petential way of answering said question that was: This particular ...
1
vote
0answers
78 views

Language-detection heuristic (English, French or German) based on Unigram and Bigram models

Given a string, for example "I hate AI", I need to find out if the sentence is in English, German or French. Unigram Model makes the prediction on the basis of each character frequency in a training ...
1
vote
2answers
3k views

replacing words with their abbreviations

I'm working on a program that aim to take sentences (currently in french) and compact them to a length of 38 characters while retaining as much information as possible. You can find another part of ...
4
votes
1answer
130 views

Remove determiners in a string

I'm working on a program that aim to take sentences (currently in french) and compact them to a length of 38 characters while retaining as much information as possible. You can find another part of ...
3
votes
1answer
60 views

Syllabification function for Turkish words

I wrote an NLP script for processing Turkish language. Yesterday I added syllabication but I wonder if it could be done better. It is kinda hard-coded, so I would like to know if I can improve it. ...
3
votes
1answer
75 views

Simple natural language classifier

This program estimates the likelihood for a string to belong to a certain natural language by computing the cosine similarity between an input string's and several natural languages' letter frequency, ...
1
vote
0answers
36 views

Function for root matching between two paragraphs

I created an algorithm that match roots of two texts, a question and a paragraph made of sentences. I aim at predicting in which sentence it exists the answer of a question. Yet It seems that I really ...
4
votes
1answer
567 views

Relative frequency of words in tree of documents

I have a tree structure where at every node there is a list of documents (document length can vary from 5 to 500), and each document contains a number of words. I want to calculate relative frequency ...
1
vote
2answers
213 views

Substitute IDs to word tokens

I writing a function in Python to substitute word tokens in a sentence with IDs. The sentence is a list of tokens (list_of_tokens). The IDs are provided in a dictionary mapping tokens with an IDs (...
2
votes
1answer
111 views

Haskell sentence segregation

I am trying to implement sentence segregation using Haskell, I have achieved a decent bulk of it using the NLP.FullStop library, but this doesn't seem to account ...
1
vote
1answer
4k views

Generating a word bigram co-occurrence matrix

I have written a method which is designed to calculate the word co-occurrence matrix in a corpus, such that element(i,j) is the number of times that word i follows word j in the corpus. Here is my ...
5
votes
1answer
125 views

Program to count common words in documents

I finished my first program in Ruby and I would like to share it with you so I can get some suggestions or recommendations. I'd really like to hear them because I am learning and I want to have a ...
1
vote
0answers
167 views

Analysis of the most common words in a text

I have been trying to gain more understanding and experience in the NPL area and to get some more practice. I decided to attempt to create a simple high-level ...
10
votes
3answers
2k views

Analysis of the most common and salient words in a text

I've been trying to get more understanding and experience in the Natural Language Processing space and, to get some more practice, decided to attempt to make a simple high-level analysis of the "...
3
votes
1answer
4k views

Identify and extract URLs from text corpus

I'm working on a project that requires POS Tagging of paragraphs. The text contains lot of URLs which contain various punctuation marks such as . ...
6
votes
1answer
124 views

Mark V. Shaney: a script to produce gibberish

What follows in an attempt at implementing Mark V. Shaney using contemporary Python. One question has already been asked while working on a generator in the code, but included here is the entire ...
2
votes
1answer
3k views

Removing stop words from a Spark Dataframe

I am trying to apply a function to two Spark Dataframes (in Zeppelin): ...
1
vote
0answers
35 views

NER and its F Measure Calculation

I am trying to write one Name Entity Recognition in Hindi. I have primarily used NLTK of Python. I have used HMM Module with its supervised training. The data is annotated and saved in .pos files ...
3
votes
2answers
438 views

Recursive right-to-left segmenting (tokenizing) of strings in Python

I want to segment a list of strings based on a custom list of words/forms, from right-to-left. Here I've illustrated it with made-up English examples, but the point is to segment/tokenize using a ...
5
votes
5answers
224 views

Calculate LIX value of a text

I've been building a simple tool to calculate the LIX value of a text (a standard measurement of the readability of a text). My approach was to have every step of the calculation as seperate ...
5
votes
0answers
76 views

Optimally splitting a text into strings from a set

I am trying to write an algorithm that starts with a corpus of texts (e.g., a Wikipedia dump). It first builds an array of individual characters (e.g., "a", ...
3
votes
1answer
73 views

Interpreting tweets about football

I am trying to process the football tweets and extract information like goals, cards, corners, player name, team name. I write the code which works, but I may be missing some better python ...
7
votes
3answers
224 views

Linking two databases based on street addresses

For my work, I wrote a python script to link 2 files. Since I am an autodidact and since no one of my colleagues writes code, I ask the question here. My code takes an unbelievable time to run. Is it ...
5
votes
1answer
105 views

Flag words that would be difficult for an early reader

This is part of a project I made a couple of years ago and was looking at again. Its purpose is to check text for words that an early reader (about a late kindergarten or first grade level) would ...
4
votes
1answer
4k views

Random name generator in Java

I wrote working random name generator in Java. Here's my code: NameGenerator.java: ...
3
votes
0answers
62 views

Time calculation for NLTK tagging

I am trying to calculate the time required to tag one sentence/file by one trained NLTK HMM Tagger. To do this I am writing the following code, please suggest if I need to revise anything here. ...
1
vote
0answers
130 views

Naïve Bayes classifier to group questions by intent

I am trying to train a question-answer system, where I am trying to group similar questions, and identify the most apt response. The program should identify the intent/focus. To do it, I have tagged ...
9
votes
1answer
215 views

Summarize a document as a key-phrase or key-words

A few days ago I finished a coding challenge for a potential job. I was super happy with my code, till I got the response that my code wasn't good enough. :( So, apparently I'm still making mistakes, ...
16
votes
2answers
658 views

Using lots of regex substitutions to tokenize text

I authored a piece of code that was merged into the nltk codebase. It is full of regex substitutions: ...
4
votes
4answers
10k views

Cleaning and extracting meaningful text from tweets

I have a dataset of around 200,000 tweets. I am running a classification task on them. Dataset has two columns - class label and the tweet text. In the preprocessing step I am passing the dataset ...
2
votes
2answers
1k views

Define if post extract from a bilingual Facebook page are in English using Python

I am currently extracting post on a biligual page on Facebook. Therefore, I have the problem of splitting the post in French and english before starting analysing them. I have construct a function ...
6
votes
1answer
193 views

Tokenizing texts from Gutenberg archive for analysis

I am writing a program to analyze books from the Gutenberg archive. The program takes the title and URL and finds the text and downloads it. Then it goes through the text and tokenizes it. Here is the ...
2
votes
0answers
40 views

Classes to help transform “How hard is it to …?” questions into replies like “It's really hard to …”

I've written a Twitter bot, @answering_yelp, which responds to another Twitter bot, @hard_to_yelp. They scrape yelp reviews for sentences which start "How hard is it to..." and my bot answer with some ...
7
votes
1answer
43 views

Labeling modified words

I'm working on a Python function that takes a piece of text and indicates if words are modified by words like very or not. I'm looking for instance to label very nice differently from not nice. This ...
2
votes
1answer
273 views

Multiword Expression Tagging in Python

I am trying to write a small python code,where I am reading a text file-which contains both Multiwords (MWEs) and singular words (NMWEs). I am trying to tag each one, as follows. I have a ...