Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [data-mining]

Data mining is the process of analyzing large amounts of data in order to find patterns and commonalities.

2
votes
1answer
60 views

Analyzing patient treatment data using Pandas

I work in the population health industry and get contracts from commercial companies to conduct research on their products. This is the general code to identify target patient groups from a provincial ...
2
votes
1answer
113 views

Finding word association strengths from an input text

I have the written the following (crude) code to find the association strengths among the words in a given piece of text. ...
2
votes
2answers
32 views

Python program to rank based on the frequency of names that appears in text files

I've written a python program to rank the names that appear in the file(s) based on their frequency. In other words, there are multiple files and want to rank the frequency of the names that appears ...
2
votes
1answer
155 views

Data scraping from Internet with Excel-VBA

I have built an Excel application, which gets all the titles of books from Amazon.com, which it is asked to do and scrapes the following data out of them: Book Title Author Price What do you need to ...
3
votes
1answer
820 views

k-means using numpy

This is k-means implementation using Python (numpy). I believe there is room for improvement when it comes to computing distances (given I'm using a list comprehension, maybe I could also pack it in a ...
1
vote
1answer
90 views

R function to generate predictions from ratings

I am trying to improve the run time of a program I wrote in R. Generally, what I am doing is feeding a function a data frame of values and generating a prediction off of operations on specific columns....
3
votes
0answers
148 views

Pandas data extraction task taking too much memory. How to optimize for memory usage?

I need to process some data (one of its columns contains a json/dict with params- I need to extract those params to individual columns of their own; catch- some rows have some parameters, others have ...
3
votes
1answer
83 views

Large dataset with pyspark - optimizing join, sort, compare between rows and group by with aggregation

I have a csv file with more than 700,000,000 records in this structure: ...
5
votes
0answers
162 views

Code for training machine learning linear regression and SVM

Ok , for my final year project I've wrote this piece of code to train my machine learning model on a this dataset , here the code i used ...
4
votes
1answer
146 views

General class for basic Statistical Measures mode, arithmetic mean,geometric mean, median , variance ,and stander division functions

I am trying to do general class for basic Statistical Measures mode, arithmetic mean,geometric mean, median , variance ,and stander division functions. I am looking for some general feedback on how I ...
1
vote
3answers
391 views

General Java class to find mode

I am trying to make general class to find mode. I am looking for some general feedback on how I can improve the structure and efficiency of my code. ...
4
votes
2answers
791 views

Calculate Geometric and Arithmetic Mean

I am trying to make general class to calculate Geometric mean using exponential of the arithmetic mean of logarithms , and Arithmetic Mean. I am looking for some general feedback on how I can improve ...
5
votes
2answers
7k views

Gradient descent for linear regression using numpy/pandas

I currently follow along Andrew Ng's Machine Learning Course on Coursera and wanted to implement the gradient descent algorithm in python3 using ...
9
votes
1answer
835 views

Cosine Similarity on Huge Dataset

I have a very large data file full of movie ratings that I am looking at for work. I wanted to do this in a clean and very effective manner. The ratings file contains on a per column by column basis: ...
5
votes
0answers
5k views

Implementation of Single Layer Perceptron Learning Algorithm in C

I have implemented a working version of perceptron learning algorithm in C. Right now, it only works on single layer perceptrons and only takes two inputs. I plan on making it work with more than two ...
3
votes
0answers
36 views

Calculating frequencies of each obs in the data

I am currently attempting to make some code more maintainable for a research project I am working on. I am definitely looking to create some more functions, and potentially create a general class to ...
3
votes
1answer
77 views

Structuring flat html structure into a tree

Given an html page with heading tags such h1, h2 through h6 and ...
7
votes
1answer
433 views

PANDAS DataFrame operations to analyze top Server Fault tags

I am working on learning how to do frequency analysis of Server Fault question tags to see if there is any useful data that I can glean from them. I'm storing the raw data in Bitbucket for global ...
3
votes
1answer
33 views

Efficient implementation of aggregating test/train data

Here is a short python snippet to ingest train data: ...
0
votes
1answer
80 views

Code for finding repeated entries with different data

project is the data frame. For the purpose of the code, HOUSE.NO is a column of the type character, and ...
3
votes
2answers
5k views

Mining association rules in Java

Let \$I = \{ i_1, i_2, \dots, i_d \}\$ be the set of all possible items, and \$T = \{ t_1, t_2, \dots, t_N \}\$ be the (multi)set of all given transactions, where \$t_i \subseteq I\$ for all \$i \in \{...
2
votes
1answer
464 views

Nested loops - Random Forest, multiple parameters

I'm writing a code which task is to grow Random Forest trees based on multiple parameters. In short: Firstly, I declare a data frame in which model parameters and some stats will be saved. Secondly, ...
5
votes
2answers
266 views

Document term matrix in Clojure

This is my very first foray into Clojure (I'm normally a Python-pushing data-type). I'm trying to create a simple term-document matrix as a vector of vectors, out of a vector of strings. For those ...
5
votes
1answer
229 views

Data analytics on static file of 50,000+ tweets

I'm trying to optimize the main loop portion of this code, as well as learn any "best practices" insights I can for all of the code. This script currently reads in one large file full of tweets (50MB ...
2
votes
1answer
20k views

Apriori algorithm for frequent itemset generation in Java

I have this algorithm for mining frequent itemsets from a database. In that problem, a person may acquire a list of products bought in a grocery store, and he/she wishes to find out which product ...
4
votes
3answers
188 views

Analyze very large sets of engineering data from Excel files

I am an electrical power engineer with some programming skills. My boss asked me to make a program which could analyze very large data, make some calculations and give the result. The task looks like ...
2
votes
1answer
2k views

File parser to extract data from text file

I am trying to extract the data from input file and store it for plotting. I have tested this code for a few files of same format. I am not sure if the code works correctly with the little change in ...
2
votes
0answers
175 views

C# port of data mining algorithm much slower than reference implementation

I was trying to implement the algorithm specified in this research paper (please ignore the math, since it's irrelevant to the question). This algorithm is very basic in formal concept analysis. The ...
3
votes
0answers
317 views

Frequent subgraph mining program

I'm trying to make a programme that reads graphs from a .txt file, puts them in a vector, and finally puts the frequent closed graphs in another resulting file. I'...
4
votes
1answer
4k views

Implementation of KNN in R

I have implemented the K-Nearest Neighbor algorithm with Euclidean distance in R. It works fine but takes tremendously huge time than the library function (get.knn). Please point out the possibility ...
2
votes
1answer
201 views

CSMR for large-scale text-prcessing

I'm working on a project for large-scale text-processing, which is a first implementation of the basic idea of CSMR. CSMR is an algorithm that measures the similarity between documents by calculating ...
5
votes
2answers
5k views

AutoComplete program using the n-gram model

For my Advanced Data Mining class (undergrad) we were to design a program that would predict the next word a user is likely to type via automatic text classification using the n-gram model. The ...
6
votes
1answer
22k views

Apriori algorithm using Pandas

I want to optimize my Apriori algorithm for speed: ...
5
votes
1answer
6k views

Alternative to Python's Naive Bayes Classifier for Twitter Sentiment Mining

I am doing sentiment analysis on tweets. I have code that I developed from following an online tutorial (found here) and adding in some parts myself, which looks like this: ...