Questions tagged [data-mining]
Data mining is the process of analyzing large amounts of data in order to find patterns and commonalities.
44
questions
9
votes
1
answer
1k
views
Cosine Similarity on Huge Dataset
I have a very large data file full of movie ratings that I am looking at for work. I wanted to do this in a clean and very effective manner. The ratings file contains on a per column by column basis:
...
- 91
7
votes
1
answer
9k
views
Implementation of Single Layer Perceptron Learning Algorithm in C
I have implemented a working version of perceptron learning algorithm in C. Right now, it only works on single layer perceptrons and only takes two inputs.
I plan on making it work with more than two ...
- 383
7
votes
1
answer
463
views
PANDAS DataFrame operations to analyze top Server Fault tags [closed]
I am working on learning how to do frequency analysis of Server Fault question tags to see if there is any useful data that I can glean from them. I'm storing the raw data in Bitbucket for global ...
- 453
6
votes
2
answers
292
views
Document term matrix in Clojure
This is my very first foray into Clojure (I'm normally a Python-pushing data-type). I'm trying to create a simple term-document matrix as a vector of vectors, out of a vector of strings.
For those ...
- 295
6
votes
1
answer
26k
views
6
votes
3
answers
10k
views
Gradient descent for linear regression using numpy/pandas
I currently follow along Andrew Ng's Machine Learning Course on Coursera and wanted to implement the gradient descent algorithm in python3 using ...
- 351
5
votes
2
answers
6k
views
AutoComplete program using the n-gram model
For my Advanced Data Mining class (undergrad) we were to design a program that would predict the next word a user is likely to type via automatic text classification using the n-gram model.
The ...
- 53
5
votes
1
answer
6k
views
k-means using numpy
This is k-means implementation using Python (numpy). I believe there is room for improvement when it comes to computing distances (given I'm using a list comprehension, maybe I could also pack it in a ...
- 381
5
votes
1
answer
269
views
Data analytics on static file of 50,000+ tweets
I'm trying to optimize the main loop portion of this code, as well as learn any "best practices" insights I can for all of the code. This script currently reads in one large file full of tweets (50MB ...
- 153
5
votes
1
answer
6k
views
Alternative to Python's Naive Bayes Classifier for Twitter Sentiment Mining
I am doing sentiment analysis on tweets. I have code that I developed from following an online tutorial (found here) and adding in some parts myself, which looks like this:
...
- 439
5
votes
1
answer
200
views
A simple probabilistic AI for generating random sentences in Java
Motivation
I have this repository. It contains a program that analyzes an input text file and builds a word graph: in the graph, each node represents a word in the analyzed text. Now, if there are two ...
- 28.2k
5
votes
0
answers
204
views
Code for training machine learning linear regression and SVM
Ok , for my final year project I've wrote this piece of code to train my machine learning model on a this dataset , here the code i used
...
- 229
4
votes
2
answers
1k
views
Calculate Geometric and Arithmetic Mean
I am trying to make general class to calculate Geometric mean using exponential of the arithmetic mean of logarithms , and Arithmetic Mean.
I am looking for some general feedback on how I can improve ...
- 433
4
votes
3
answers
196
views
Analyze very large sets of engineering data from Excel files
I am an electrical power engineer with some programming skills. My boss asked me to make a program which could analyze very large data, make some calculations and give the result.
The task looks like ...
4
votes
1
answer
171
views
FPGrowth Algorithm Implementation
This is my implementation of the FPGrowth algorithm where, as an optimisation, I avoid re-creating the tree at each extension of the prefix, while I use a view representation that I think would be ...
- 113
4
votes
1
answer
5k
views
Implementation of KNN in R
I have implemented the K-Nearest Neighbor algorithm with Euclidean distance in R. It works fine but takes tremendously huge time than the library function (get.knn). Please point out the possibility ...
- 43
4
votes
1
answer
263
views
General class for basic Statistical Measures mode, arithmetic mean,geometric mean, median , variance ,and stander division functions
I am trying to do general class for basic Statistical Measures mode, arithmetic mean,geometric mean, median , variance ,and stander division functions.
I am looking for some general feedback on how I ...
- 433
3
votes
1
answer
3k
views
Finding word association strengths from an input text
I have the written the following (crude) code to find the association strengths among the words in a given piece of text.
...
- 317
3
votes
1
answer
40
views
Efficient implementation of aggregating test/train data
Here is a short python snippet to ingest train data:
...
3
votes
1
answer
544
views
Large dataset with pyspark - optimizing join, sort, compare between rows and group by with aggregation
I have a csv file with more than 700,000,000 records in this structure:
...
- 33
3
votes
0
answers
510
views
Pandas data extraction task taking too much memory. How to optimize for memory usage?
I need to process some data (one of its columns contains a json/dict with params- I need to extract those params to individual columns of their own; catch- some rows have some parameters, others have ...
- 131
3
votes
0
answers
50
views
Calculating frequencies of each obs in the data
I am currently attempting to make some code more maintainable for a research project I am working on. I am definitely looking to create some more functions, and potentially create a general class to ...
- 73
3
votes
0
answers
359
views
Frequent subgraph mining program
I'm trying to make a programme that reads graphs from a .txt file, puts them in a vector, and finally puts the frequent closed graphs in another resulting file.
I'...
- 43
2
votes
2
answers
126
views
Data mining in Java: finding undrawn lottery rows
(See the next iteration.)
Introduction
Suppose Evil Lottery Inc is interested in not paying millions of dollars back to players. They gather the drawn lottery rows first, after which they mine rows ...
- 28.2k
2
votes
1
answer
2k
views
File parser to extract data from text file
I am trying to extract the data from input file and store it for plotting.
I have tested this code for a few files of same format. I am not sure if the code works correctly with the little change in ...
- 123
2
votes
1
answer
472
views
Analyzing patient treatment data using Pandas
I work in the population health industry and get contracts from commercial companies to conduct research on their products. This is the general code to identify target patient groups from a provincial ...
- 225
2
votes
1
answer
23k
views
Apriori algorithm for frequent itemset generation in Java
I have this algorithm for mining frequent itemsets from a database. In that problem, a person may acquire a list of products bought in a grocery store, and he/she wishes to find out which product ...
- 28.2k
2
votes
2
answers
304
views
Python program to rank based on the frequency of names that appears in text files
I've written a python program to rank the names that appear in the file(s) based on their frequency. In other words, there are multiple files and want to rank the frequency of the names that appears ...
- 163
2
votes
2
answers
7k
views
Mining association rules in Java
Let \$I = \{ i_1, i_2, \dots, i_d \}\$ be the set of all possible items, and \$T = \{ t_1, t_2, \dots, t_N \}\$ be the (multi)set of all given transactions, where \$t_i \subseteq I\$ for all \$i \in \{...
- 28.2k
2
votes
1
answer
109
views
R function to generate predictions from ratings
I am trying to improve the run time of a program I wrote in R. Generally, what I am doing is feeding a function a data frame of values and generating a prediction off of operations on specific columns....
- 137
2
votes
1
answer
103
views
Structuring flat html structure into a tree
Given an html page with heading tags such h1, h2 through h6 and ...
- 151
2
votes
1
answer
1k
views
Nested loops - Random Forest, multiple parameters
I'm writing a code which task is to grow Random Forest trees based on multiple parameters. In short:
Firstly, I declare a data frame in which model parameters and some stats will be saved.
Secondly, ...
- 131
2
votes
1
answer
207
views
CSMR for large-scale text-prcessing
I'm working on a project for large-scale text-processing, which is a first implementation of the basic idea of CSMR. CSMR is an algorithm that measures the similarity between documents by calculating ...
- 131
2
votes
0
answers
180
views
producer-consumer Pipeline problem implementation in asyncio
I wrote this code to make a non-blocking manager along with pipeline operations using asyncio, my main concern is to catch received items producer, and when the received operation is complete. I want ...
- 21
2
votes
0
answers
721
views
How to avoid bottlenecks json processing with Apache Beam?
I have a input with some transaction data in json input (in this case a file)
...
- 253
2
votes
0
answers
186
views
C# port of data mining algorithm much slower than reference implementation
I was trying to implement the algorithm specified in this research paper (please ignore the math, since it's irrelevant to the question). This algorithm is very basic in formal concept analysis. The ...
1
vote
3
answers
796
views
General Java class to find mode
I am trying to make general class to find mode. I am looking for some general feedback on how I can improve the structure and efficiency of my code.
...
- 433
1
vote
1
answer
113
views
Data mining in Java: finding undrawn lottery rows - follow-up
(See the previous (initial) iteration.)
This time, I have substantially reduced the usage of the final and this keywords. Also, ...
- 28.2k
1
vote
1
answer
1k
views
Data scraping from Internet with Excel-VBA
I have built an Excel application, which gets all the titles of books from Amazon.com, which it is asked to do and scrapes the following data out of them:
Book Title
Author
Price
What do you need to ...
- 319
1
vote
1
answer
290
views
Better way to create a contingency table with pandas for film genres from a Film DataFrame
From a public dataset available on film rating I created a contingency table as follow.
Honestly I don't like all these "for-loops" I think the quality of the code can be definitely improved ...
- 579
1
vote
0
answers
37
views
Relational join of two datasets
Front Matter
I'm learning Scala and have not gotten used to functional programming and the language. I'm hoping a review of my naively implemented code can help me bridge my object-oriented ways to ...
- 111
1
vote
0
answers
40
views
Python 3 script for statistiscal analysis of English vocabulary
This is a Python 3 script that statistically analyzes 105,230 words with a total of 913,935 characters, its results will be used for pseudoword generation purposes.
You will need this file for it to ...
- 3,086
0
votes
1
answer
86
views
Code for finding repeated entries with different data
project is the data frame. For the purpose of the code, HOUSE.NO is a column of the type character, and ...
- 315
-4
votes
0
answers
16
views
Request for Assistance with DFS, PMI, and Pairwise Constraints Implementation (Python) [closed]
I've been working on a Python implementation combining Depth-first Search (DFS depth = 2), Pointwise Mutual Information (PMI), and Pairwise Constraints (ML must link and CL cannot link). While I've ...
- 1