Questions tagged [natural-language-processing]

The field of natural language processing covers attempts to make sense of text in a human language using computers

Filter by
Sorted by
Tagged with
5 votes
1 answer
83 views

Local Search Engine in Rust

I made a simple search engine using the xkcd API in Rust which turned out better than I'd hoped for! I decided to use tf-idf as a way to rank results, which I feel like has some room for improvement. ...
joeymalvinni's user avatar
5 votes
2 answers
97 views

Creating csvs using Pandas on large dataset for document retrieval

I am trying to build a useable NLP corpus but getting bottlenecked by how long the program takes (200 hours). With so much data I know that optimizing my code even a little bit will net me huge time ...
evader110's user avatar
  • 143
2 votes
0 answers
87 views

NLP pre-processing function optimization as it is extremely low on 92Mb data set

I have a data set that is of 300,000 rows approximately and two columns, each row contains a string, some might be larger than others. All in all, the data set in a ...
Louis's user avatar
  • 121
7 votes
4 answers
428 views

Separating a String of Text into Separate Words in Python

Occasionally, we want to do a rudimentary parsing on English text; we separate the text into separate words. ...
Samuel Muldoon's user avatar
0 votes
1 answer
212 views

Short Text Pre-processing

For educational purpose I am preprocessing multiple short texts containing the description of the symptoms of cars fault. The text is written by humans and is rich in misspelling, capital letters and ...
Andrea Ciufo's user avatar
1 vote
1 answer
97 views

Python voice assistant that acts on trigger phrases

I made a Python voice assistant. It takes the user's voice input and there are multiple if-else statements that specify a condition and if it satisfies that condition it executes a specific function. ...
Rohith Nambiar's user avatar
2 votes
1 answer
106 views

Remove duplicates from a Pandas dataframe taking into account lowercase letters and accents

I have the following DataFrame in pandas: code town district suburb 02 Benalmádena Málaga Arroyo de la Miel 03 Alicante Jacarilla Jacarilla, Correntias Bajas (Jacarilla) 04 Cabrera d'Anoia ...
Carola's user avatar
  • 143
3 votes
0 answers
675 views

Rust code implementing cosine similarity

I've been trying to create a piece of code which consists of looping through each element of a list of questions, preprocess it, and then calculate the Cosine similarity with the rest of the elements (...
Shodai Thox's user avatar
2 votes
2 answers
325 views

Markov text generator program in Python

This is my first non-trivial program in my Python. I am coming from a Java background and I might have messed up or ignored some conventions. I would like to hear feedback on my code. ...
BovineScatologist's user avatar
1 vote
1 answer
163 views

Finding a path from one wikipedia page to another using semantic similarity of links (Spacy)

I've just picked coding back up for the first time in a long time, so I understand if your eyes bleed looking at this code. It all works, but I'd be grateful for any tips (how to improve the python ...
Will's user avatar
  • 111
1 vote
1 answer
100 views

IDF Function with a list of list

I wanted to build a Inverse Document Frequency function, because in my opinion was not easy to do with scikit and I wanted also to show how it works for educational reasons. Also reading this question ...
Andrea Ciufo's user avatar
2 votes
0 answers
49 views

Looping over files to create a dataframe

As part of my NLP project at work, I want to loop over all files that are either PDF of docx in the same directory. The end purpose is to create a dataframe with text content of the files in one ...
Sam.H's user avatar
  • 143
2 votes
2 answers
232 views

Text Normalizer

I am working on a text normalizer. It works just fine with small text files but takes a very long time with large text files such as 5 MB or more. Is there anything to change in the code to make it ...
mehio hatab's user avatar
3 votes
1 answer
187 views

Reduce run time of NLP approximate matching code

The code below matches a list of features to a large corpus and returns the sub-query match with a score above 80. The challenge is the list of features on the full data-set is > 5,000 and comparing ...
Rtimeseries's user avatar
2 votes
2 answers
391 views

Simplified Pig Latin translator in APL

I wrote a simplified pig latin translator in APL and I would like some feedback on it, as I am not sure my implementation is neat enough. The simplified pig latin translation follows the following ...
RGS's user avatar
  • 1,058
7 votes
2 answers
1k views

Extracting all nouns, verbs and adjectives from a large text dataset

For a project I am using the yelp dataset (found here: https://www.yelp.com/dataset) to create a Hashset of all verbs, nouns and adjectives found in the restaurant reviews. I have it up and running ...
Ruben Eschauzier's user avatar
10 votes
1 answer
166 views

Wordcloud from all answers of a user here on CR

Since I haven't really used Python's new async features yet, I took some older code of mine, which took all of my answers here on Code Review and generated a word cloud from them, and updated it to ...
Graipher's user avatar
  • 40.9k
8 votes
2 answers
619 views

NLP sentiment analysis in Norwegian

Please keep in mind that I am very new to data science and completely new to NLP! I am trying to create a model to classify customer reviews as either negative or positive. However, my main problem is ...
Grevioos's user avatar
  • 291
6 votes
2 answers
365 views

Tokenizing SGML text for NLTK analysis

I have an NLTK parsing function that I am using to parse a ~2GB text file of a TREC dataset. The goal for this dataset is tokenize the entire collection, perform some calculations (such as calculating ...
artemis's user avatar
  • 193
3 votes
0 answers
67 views

Django project for events of the day, grouped by keywords in common

I have made a website where people write about their day and see it analyzed. In particular, the website takes event titles and groups them together if they have any common words (after stemming the ...
Bo Work's user avatar
  • 391
7 votes
2 answers
88 views

Document classfier

Description: I am working on a classifier which categorizes the text based on some criteria, at present, it is a map of category and list of words if any of the words appear in the text, a category ...
CodeYogi's user avatar
  • 5,107
6 votes
1 answer
348 views

Ternary Search Tree / N-Gram Model in Python

I implemented a word n-gram model using a character ternary search tree. It is intended to be passed a generator that yields a long sequence of words (from a corpus) and its requirements are that it ...
Christian Adam's user avatar
2 votes
0 answers
642 views

Define the scope of negation with the Dependency Parser of spaCy

Sentiment words behave very differently when under the semantic scope of negation. I want to use a slightly modified version of Das and Chen (2001) They detect words such as no, not, and never and ...
HJA24's user avatar
  • 219
3 votes
1 answer
3k views

Finding word association strengths from an input text

I have the written the following (crude) code to find the association strengths among the words in a given piece of text. ...
Kristada673's user avatar
2 votes
0 answers
51 views

Fetching and modifying CNN headlines

This is a project I came up just as an exercise with to familiarize myself with Python syntax and data types. I am just learning how to code and would like to avoid establishing bad habits. On the ...
GCM's user avatar
  • 31
2 votes
1 answer
235 views

Python Program Generating N-Gram Language Model

I am pretty new to Python. I am writing this program to randomly generate sentences based on the n-gram language. It takes me very long to run this with the large input file I have, so it is very hard ...
Yuhe Zhu's user avatar
4 votes
0 answers
108 views

Download and analyze PDFs of Congressional records

This is built with Python 2.7.15. The goal of this script is to count the number of words spoken by each Senator on the floor of Congress between given dates. It pulls from the Congressional Record, ...
Sebastian's user avatar
  • 141
3 votes
1 answer
350 views

Reduce the length of words in a sentence

This function's goal is to reduce the length of a sentence to exactly max_length characters by cutting each word in the sentence ...
Comte_Zero's user avatar
2 votes
1 answer
154 views

Counting lower vs non-lowercase tokens for tokenized text with several conditions

Assuming that the text is tokenized with whitespace for a natural language processing task, the goal is to check the count of the words (regardless of casing) and check them through some conditions. ...
alvas's user avatar
  • 705
1 vote
1 answer
909 views

Replacing words with their abbreviations - Follow up

This is the follow up for a question you can find here: replacing words with their abbreviations The goal here is to compare two petential way of answering said question that was: This particular ...
Comte_Zero's user avatar
1 vote
2 answers
9k views

replacing words with their abbreviations

I'm working on a program that aim to take sentences (currently in french) and compact them to a length of 38 characters while retaining as much information as possible. You can find another part of ...
Comte_Zero's user avatar
4 votes
1 answer
319 views

Remove determiners in a string

I'm working on a program that aim to take sentences (currently in french) and compact them to a length of 38 characters while retaining as much information as possible. You can find another part of ...
Comte_Zero's user avatar
3 votes
1 answer
161 views

Syllabification function for Turkish words

I wrote an NLP script for processing Turkish language. Yesterday I added syllabication but I wonder if it could be done better. It is kinda hard-coded, so I would like to know if I can improve it. ...
MeteHan's user avatar
  • 350
3 votes
1 answer
89 views

Simple natural language classifier

This program estimates the likelihood for a string to belong to a certain natural language by computing the cosine similarity between an input string's and several natural languages' letter frequency, ...
Passa's user avatar
  • 33
4 votes
1 answer
840 views

Relative frequency of words in tree of documents

I have a tree structure where at every node there is a list of documents (document length can vary from 5 to 500), and each document contains a number of words. I want to calculate relative frequency ...
Ankita Patnaik's user avatar
1 vote
2 answers
283 views

Substitute IDs to word tokens

I writing a function in Python to substitute word tokens in a sentence with IDs. The sentence is a list of tokens (list_of_tokens). The IDs are provided in a dictionary mapping tokens with an IDs (...
Gabrer's user avatar
  • 141
2 votes
1 answer
174 views

Haskell sentence segregation

I am trying to implement sentence segregation using Haskell, I have achieved a decent bulk of it using the NLP.FullStop library, but this doesn't seem to account ...
danbroooks's user avatar
1 vote
1 answer
5k views

Generating a word bigram co-occurrence matrix

I have written a method which is designed to calculate the word co-occurrence matrix in a corpus, such that element(i,j) is the number of times that word i follows word j in the corpus. Here is my ...
quanty's user avatar
  • 285
5 votes
1 answer
145 views

Program to count common words in documents

I finished my first program in Ruby and I would like to share it with you so I can get some suggestions or recommendations. I'd really like to hear them because I am learning and I want to have a ...
H4ml3tt3d's user avatar
1 vote
0 answers
187 views

Analysis of the most common words in a text

I have been trying to gain more understanding and experience in the NPL area and to get some more practice. I decided to attempt to create a simple high-level ...
user67809's user avatar
10 votes
3 answers
3k views

Analysis of the most common and salient words in a text

I've been trying to get more understanding and experience in the Natural Language Processing space and, to get some more practice, decided to attempt to make a simple high-level analysis of the "...
alecxe's user avatar
  • 17.3k
3 votes
1 answer
4k views

Identify and extract URLs from text corpus

I'm working on a project that requires POS Tagging of paragraphs. The text contains lot of URLs which contain various punctuation marks such as . ...
Thirupathi Thangavel's user avatar
4 votes
1 answer
446 views

Attention matrix in Python with PyTorch

I want to implement Q&A systems with attention mechanism. I have two inputs; context and query which shapes are ...
jef's user avatar
  • 141
6 votes
1 answer
179 views

Mark V. Shaney: a script to produce gibberish

What follows in an attempt at implementing Mark V. Shaney using contemporary Python. One question has already been asked while working on a generator in the code, but included here is the entire ...
Noctis Skytower's user avatar
2 votes
1 answer
4k views

Removing stop words from a Spark Dataframe

I am trying to apply a function to two Spark Dataframes (in Zeppelin): ...
schoon's user avatar
  • 131
3 votes
2 answers
902 views

Recursive right-to-left segmenting (tokenizing) of strings in Python

I want to segment a list of strings based on a custom list of words/forms, from right-to-left. Here I've illustrated it with made-up English examples, but the point is to segment/tokenize using a ...
arjan's user avatar
  • 131
5 votes
5 answers
476 views

Calculate LIX value of a text

I've been building a simple tool to calculate the LIX value of a text (a standard measurement of the readability of a text). My approach was to have every step of the calculation as seperate ...
Mark Buskbjerg's user avatar
5 votes
0 answers
88 views

Optimally splitting a text into strings from a set

I am trying to write an algorithm that starts with a corpus of texts (e.g., a Wikipedia dump). It first builds an array of individual characters (e.g., "a", ...
wizplum's user avatar
  • 151
3 votes
1 answer
83 views

Interpreting tweets about football

I am trying to process the football tweets and extract information like goals, cards, corners, player name, team name. I write the code which works, but I may be missing some better python ...
user2129623's user avatar
7 votes
3 answers
232 views

Linking two databases based on street addresses

For my work, I wrote a python script to link 2 files. Since I am an autodidact and since no one of my colleagues writes code, I ask the question here. My code takes an unbelievable time to run. Is it ...
francois's user avatar