Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [clustering]

Cluster analysis or clustering is the task of grouping a set of objects in such a way that objects in the same group (called a cluster) are more similar (in some sense or another) to each other than to those in other groups (clusters).

2
votes
1answer
41 views

Welford's online variance calculation algorithm for vectors

I'm developing a face recognizing application using the face_recognition Python library. The faces are encoded as 128-dimension floating-point vectors. In addition to this, each named known person ...
3
votes
0answers
45 views

Locality Sensitive Hash (similar to k-Nearest Neighbor), in Python+Numpy

I've tried implementing Locality Sensitive Hash, the algorithm that helps recommendation engines, and powers apps like Shazzam that can identify songs you heard at restaurants. LSH is supposed to run ...
2
votes
0answers
38 views

Extracting face features from selfies faster

My code takes two .jpg files (two selfies) and extracts a vector of 128 features from each selfie. After this extraction we take these two vectors and get their cosine distance. I timed the script ...
0
votes
1answer
40 views

Analyzing distances between clusters of orders [closed]

I wrote the Python class below, which does what I want it to do, but the data structure is a mess. Was wondering if there was a better structure I could use to get the same results but with better ...
1
vote
0answers
32 views

Cluster correlations for many (> 500k) features by threshold

I have a matrix with roughly 500,000 features that I'd like to correlate and find features that have correlations >= 0.90, then cluster these into one group. Is there an efficient algorithm to do this?...
2
votes
2answers
141 views

Top k closest pairs in a set of million 128-dimensional points [closed]

I have a set of 1 million points in 128-dimensional space. Among all trillion pairs of the points in the set, I need to get a subset of 100 million pairs whose cosine distances are less than that of ...
1
vote
1answer
46 views

Efficiently determining maximum allowed euclidean distance between lists of colors

I was recently tasked with determining which hex RGB colors in list color_list are nearest to each hex RGB color in list target_colors, using euclidean distance as the measuring stick, and only ...
3
votes
1answer
467 views

k-means using numpy

This is k-means implementation using Python (numpy). I believe there is room for improvement when it comes to computing distances (given I'm using a list comprehension, maybe I could also pack it in a ...
5
votes
1answer
718 views

Closest Pair algorithm implementation in C++

I had been working on the implementation of closest pair algorithm in a 2-D plane. My approach has been that of divide and conquer O(nlogn) : ...
3
votes
1answer
57 views

Simple natural language classifier

This program estimates the likelihood for a string to belong to a certain natural language by computing the cosine similarity between an input string's and several natural languages' letter frequency, ...
0
votes
0answers
21 views

Normal Distribution and K-means clustering demonstration

I have two classes, the 1st one aims to display the Normal Distribution, and the 2nd one aims to perform K-means clustering. The emphasis is to build classes of the mathematical methods as examples ...
4
votes
1answer
68 views

Haskell K-means implementation

The following is a Haskell backend to a K-means visualisation: I have omitted the API code (exists in a separate module), the relevant endpoints simply call ...
1
vote
1answer
85 views

R function to generate predictions from ratings

I am trying to improve the run time of a program I wrote in R. Generally, what I am doing is feeding a function a data frame of values and generating a prediction off of operations on specific columns....
2
votes
0answers
46 views

Regression on Pandas DataFrame

I am working on the following assignment and I am a bit lost: Build a regression model that will predict the rating score of each product based on attributes which correspond to some very common ...
4
votes
1answer
122 views

K-Nearest Neighbors in pure Python

I want a general criticism on this code. Using external modules is not an option, I can only use what comes with CPython. ...
0
votes
1answer
312 views

Determining the similarity between two documents

I've made some code that reads in text files (which hold quite large vectors of word frequencies), which in turn stores each index of a vector within an ArrayList ...
13
votes
2answers
1k views

Removing neighbors in a point cloud

I have written a program to optimize a point cloud in dependency of their distances to each other. The code works very well for smaller number of points. For 1700 points it takes ca. 6 minutes. But I ...
11
votes
2answers
2k views

K-Means image segmentation algorithm

I am a new C++ programmer and I have some experience in Python and C but I was almost completely self taught (I learned C++ with OpenClassrooms). I would like to learn the conventions and how things ...
2
votes
0answers
3k views

Fuzzy c Means in Python

This is my implementation of Fuzzy c-Means in Python. In the main section of the code, I compared the time it takes with the sklearn implementation of kMeans. ...
2
votes
0answers
185 views

Calculation of clustering metric in Python

When I try to run the following code for arrays with more than 10k elements, it takes hours and I don't know how to make it in the most efficient way. Any ideas? ...
3
votes
0answers
298 views

Compute distance matrix using DTW acceptable for scipy.cluster.hierarchy

I am new to both data science and python. I have a dataset of the time-dependent samples, which I want to run agglomerative hierarchical clustering on them. I have found that Dynamic Time Warping (DTW)...
2
votes
0answers
129 views

Cluster tweet texts

I am using using the following code to cluster tweet texts. The input is a dictionary containing tweet-id and tweet text as key value pairs. Example: ...
5
votes
0answers
500 views

k-means implementation in python

This is the first mini-project that I'm working on python, where I implement k-means. I'm planning to parallelize it as soon as I've written a good serial version. Code description: Below you will ...
3
votes
1answer
391 views

R - Outlier Detection Algorithm

I am trying to implement an algorithm for detecting outliers in R and I am pretty new to the language. The outlier algorithm is described in this paper in detail on page 10-11, but to summarize it ...
4
votes
1answer
246 views

DBSCAN “region query” too slow; implement a tree?

My current DBSCAN in Python works...but its indexing is far too slow; its a linear scan: ...
2
votes
0answers
349 views

A Simple K-Means Cluster Analyzer v0.2

This code is a revision of a previous post and works well. This my first attempt at creating a robust, computationally lean K-Means Cluster analyzer. I first saw this algorithm in an intermediate ...
4
votes
1answer
105 views

Grouping orders by similarity (cluster) of their items

My task is to write an algorithm for grouping list of orders into batches, each batch consisting of 4 orders. Orders are grouped by similarity of their items, which means the more items from Order X ...
3
votes
0answers
1k views

A Simple K-Means Cluster Analyzer v0.1

[NOTE] This question can be depreciated in favor of version 0.2. This code works well. This my first attempt at creating a robust, computationally lean K-Means Cluster analyzer. I first saw this ...
1
vote
0answers
78 views

A Simple Cluster Generator v0.32

This is a code revision of a previous post and works well. The purpose of this code is to produce a universe of points, randomly generated around predetermined centroids, provided from the user as ...
5
votes
1answer
119 views

A Simple Cluster Generator v0.31

[NOTE] This question can be depreciated in favor of version 0.32. This is a code revision of a previous post and works well. The purpose of this code is to produce a universe of points, randomly ...
3
votes
1answer
117 views

A Simple Cluster Generator v0.3

[NOTE] This question can be depreciated in favor of version 0.31. This is a code revision of a previous post and works well. The purpose of this code is to produce a universe of points, randomly ...
5
votes
2answers
111 views

A Simple Cluster Generator v0.2

[NOTE] This question can be depreciated in favor of version 0.3. This is a code revision of a previous post and works well. Code has been reworked to be far more clear and concise, thanks to ...
1
vote
1answer
83 views

A Simple Cluster Generator v0.1

[NOTE] This question can be depreciated in favor of version 0.2. The purpose of this code is to produce a universe of points, randomly generated around predetermined centroids, provided as a vector ...
5
votes
1answer
135 views

Java find minimum range

Question Description: Given a list of companies, eg {ABC, BBC} and a large list of data looks like below: ...
7
votes
1answer
561 views

PANDAS nearest site algorithm

I have got CSVs full of property transactions in the UK from 1995 to 2017, separated by year such as "RS2015.csv". I have a 2nd CSV with a list of wind turbines in the UK. Both have coordinates in WGS ...
3
votes
0answers
45 views

Getting hex colours from a image

I am trying to get the hex colours from an image. The problem I am having for some reason randomly the code causes high CPU usage, which times out the browser and I am not sure how to optimise the ...
2
votes
0answers
33 views

Identifying local peaks among some 2D points

The following MATLAB code takes in multiple peak coordinates and heights and eliminates lesser peaks that are within a certain distance threshold of the highest peak of the vicinity. Is there a better ...
2
votes
1answer
195 views

Predicting outliers in network data

I'm implementing KNN algorithm to predict outliers in network data which contains the following columns: source IP address, source port number, protocol and total bytes transferred. To achieve this, I'...
7
votes
1answer
119 views

Speeding up maximum self-similarity test for heavy tail-exponents

I am trying to reproduce results from a research paper using python. I've checked my method and it works on relatively small sample datasets. However, the code does not run for my actual dataset, ...
1
vote
1answer
2k views

DBSCAN c++ implementation

For work I had to implement the DBSCAN algorithm in the 3D space for clusters finding. It works, now I wonder how is the quality of the code. I'm especially concerned about incrementing the size of ...
4
votes
1answer
87 views

Nearest Neighbor Classifier for CIFAR-10 in Haskell

To get myself into functional programming, I implemented a simple nearest neighbor classifier using Haskell. The code works but is incredibly slow. Profiling tells me that most of the time is spent ...
8
votes
1answer
736 views

Cosine Similarity on Huge Dataset

I have a very large data file full of movie ratings that I am looking at for work. I wanted to do this in a clean and very effective manner. The ratings file contains on a per column by column basis: ...
7
votes
3answers
584 views

Cosine similarity of one vector with many

I'm keen to hear ideas for optimising R code to compute the cosine similarity of a vector x (with length l) with ...
8
votes
2answers
2k views

Given a collection of points on a 2D plane, find the pair that is closest to each other

Full disclosure: I'm working on this for an online course. However, my goal is really just to get a pointer to where the issue is. The goal is to implement the closest points problem, that is, given ...
5
votes
1answer
952 views

Pole (Hackerrank)

Problem Description Kevin was thinking about telephone poles and came up with an idea for a fun programming challenge. There are n telephone poles ascending a mountain and each pole has a weight ...
2
votes
0answers
320 views

Single-pass clustering algorithm for sparse matrices

I have written single pass clustering algo for reading sparse matrices passed from scikit tfidfvectoriser but the speed is king of average for medium size matrix. How can I scale for large size ...
1
vote
2answers
137 views

How to speed up execution of single pass clustering algorithm?

This is my code for clustering 111272 by 29987 tfidf vector but it is taking long time .How can I speed up the code execution. The tfidf matrix is sparse and works well for 10000 records. ...
5
votes
1answer
331 views

Clustering points on a sphere

I have written a short Python program which does the following: loads a large data file (\$10^9+\$ rows) where each row is a point on a sphere. The code then loads a pre-determined triangular grid on ...
4
votes
1answer
2k views

KNN algorithm implemented in Python

This is the first time I tried to write some code in Python. I think it gives proper answers but probably some "vectorization" is needed ...