Questions tagged [numpy]

NumPy is the fundamental package for scientific computing with the programming language Python.

122 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
8 votes
0 answers
129 views

Calculation of independence test statistic

This code (~30lines) detects independence between variables in a dataset, by using a bootstrap of a statistic. This is fully-broadcasted, but it's still long. Reading it, or making it run, do you ...
user avatar
  • 195
7 votes
0 answers
94 views

Using get_dummies to create a Simple Recommender System - Cold Start

Question: was using get_dummies a good choice for converting categorical strings? I used get_dummies to convert categorical ...
user avatar
7 votes
0 answers
156 views

Multithreaded HD Image Processing + Logistic reg. Classifier + Visualization

[I'm awaiting suggestions for improvement/optimization/more speed/general feedback ...] This code takes a label and a folder path of subfolders as input that have certain labels ex: trees, cats with ...
user avatar
7 votes
0 answers
265 views

Numerical integration in cython

I have a set of nested functions that I need to call multiple times. I know scipy.quad is pretty fast, but I will need to call the integrator recursively and want ...
user avatar
6 votes
0 answers
92 views

Code optimisation: Converting dataframe to numpy's ndarray

I am working with a dataframe of over 21M rows. ...
user avatar
  • 161
6 votes
0 answers
68 views

Subtracting elements of datasets of an HDF5 file

I am trying to solve the following problem: Input: Input is two arrays (Nx4, sorted in column-2) stored in datasets-1 and 2 in HDF5 file (input.h5). N is huge (...
user avatar
  • 61
6 votes
0 answers
4k views

Cubic spline interpolation in Python from scratch

I implemented the cubic spline interpolation explained in https://en.wikipedia.org/wiki/Spline_interpolation as a Python class. Of course, such an interpolation should exist already in some Python ...
user avatar
6 votes
0 answers
2k views

Calculating T-Test within Large Pandas Dataframes

The below code runs a t-statistic within a large dataframe (rnadf) based on masked values from another dataframe (cnvdf_maked). ...
user avatar
5 votes
0 answers
104 views

Python 3 script to make photo collages

I have written a Python script that takes a list of filenames of pictures and generates a photo collage of them. This is my most complex project yet, and I have written it completely by myself (like I ...
user avatar
5 votes
0 answers
78 views

Neural Network Written in Python is Extremely Slow

I coded a basic feedforward neural network with all pure python with the exception of numpy in order to better understand how neural networks work. It works, but the only problem is it is extremely ...
user avatar
5 votes
0 answers
675 views

Customizable Multi-Agent Predator/Prey Simulation

I am trying to create a highly customizable "battle simulator". The default setup is a predator/prey relationship (no births); however, I am trying to create code such that it's easy to add and ...
user avatar
  • 51
5 votes
0 answers
667 views

Calculating complex exponential numbers using OpenMP in Cython

I am trying to create a function that can either beat numexpr or perform comparably for the vectorized mathematical operation ...
user avatar
  • 151
4 votes
0 answers
53 views

How to structure and write exchangable calculation models (in python3 modules)?

I do research and compare different numeric models to each other. Therefore, I want to easily switch out the numeric models in my simulations and evaluation code. Some models work in cartesian ...
user avatar
4 votes
0 answers
481 views

Slow copying of memmap array to numpy array

I have multiple binary (structured) file, each of 2GB, which I am currently reading in pair, using memmap to cross-correlate the same. I want to minimise the time required by this IO process, in the ...
user avatar
4 votes
0 answers
132 views

Text-based tic-tac-toe game in Python

I'm working on a text-based tic-tac-toe game, mostly for fun and because I'm learning programming. I would appreciate any comments you could give me to improve it. Main structure: Uses a Board class ...
user avatar
4 votes
0 answers
67 views

Importer for binary datafile of PI88 Nanoindenter

As an exercise, I want to rewrite a C++ program to import measurement files (and later create excelsheets, reports ...). The measurement files split in 2 files, a binary file, and a xml-file. Because ...
user avatar
  • 288
4 votes
0 answers
2k views

Distance transform on image using NumPy

I would like to find the find the distance transform of a binary image in the fastest way possible without using the scipy function distance_transform_edt(). The ...
user avatar
4 votes
0 answers
830 views

numpy 2d array anti diagonal averaging

The following piece of code averages the ante-diagonal elements of a 2d array x of size (m,n) with, for my purpose, m > n. ...
user avatar
  • 228
4 votes
0 answers
2k views

Pandas/Numpy statistical odds ratio test

I'm looking for a faster way to do odds ratio tests on a large dataset. I have about 1200 variables (see var_col) I want to test against each other for mutual exclusion/ co-occurrence. An odds ratio ...
user avatar
4 votes
0 answers
3k views

Applying hysteresis to a signal

Using python 3.6. I have written this code (below) to apply hysteresis to a signal (a numpy array). My aim was to create a function that didn't require any loop to remember the previous state of the ...
user avatar
4 votes
0 answers
633 views

Color image classification and labeling algorithm

I have some code I've been using to de-noise some images and then label all unique and contiguous elements. The function zoneMap() de-noises the image by picking a ...
user avatar
  • 181
4 votes
0 answers
213 views

Finding where column slices of elements in multi-dimensional array are equal

Skippable Intro: I have a dataset that corresponds to an observed time-series. The dataset is organized into a dictionary - which contains an array of years, an array of months, an array of days, an ...
user avatar
4 votes
0 answers
3k views

Reducing memory consumption for Numpy array

I'm running the following code using numpy arrays, I get a MemoryError in Ubuntu, while the same code runs on Mac OSX. (Pagination is automatically setup in Mac) This process consumes around 30 GB. ...
user avatar
  • 601
4 votes
0 answers
662 views

Find the closest parametric values corresponding to a BSpline's control points

On a curve generated by scipy.interpolate.BSpline I want to find the closest parameters relative to each control point, so that the given parametric range is ...
user avatar
  • 383
4 votes
0 answers
1k views

2-D Kalman Filtering in Python using Numpy

I am practicing Kalman filtering and wrote a short python class that uses Numpy to calculate the 2-D kalman filter for position and velocity along the X axis: assume that the object is only moving ...
user avatar
4 votes
0 answers
224 views

Simple Stateful LSTM example with arbitrary sequence length

I'm trying to make a simple example showing how to create a RNN with Keras that accepts as input a sequence of arbitrary length. In my example, the LSTM is trying to classify whether a sequence has ...
user avatar
  • 1,433
4 votes
0 answers
433 views

Plant/Residue Detection project, How to structure it?

Introduction I have wanted to develop this project for more than two years. I just started writing it a couple days ago. I use it to detect plant or residue pixels in crop row images but it provides ...
user avatar
4 votes
0 answers
536 views

Python Code For Drawing a Fractal Tree using pyglet

So I've just finished doing a coding challenge that involved drawing a fractal tree. Instead of doing it recursively(which is what I did initially but it proved to be to slow for branches greater than ...
user avatar
4 votes
0 answers
286 views

Generalized NumPy reshape function to imitate einsum syntax

I often perform a lot of reshape(), transpose() or squeeze() operations on ...
user avatar
  • 206
3 votes
0 answers
56 views

Implementation of a crypto formula in Python

I need to implement in Python the formula in the image. The set B is a set of real numbers \$-1 \le t \le 1\$. Also, \$\mathbb{F}_2\$ is a set with the elements 0 and 1. \$y_i\$ is the \$i\$-th ...
user avatar
  • 139
3 votes
0 answers
43 views

Nearest neighbor search algorithm of quantization

I'm doing some research on Nearest Neighbor Search of quantization, where running speed is very important. As I've just started using Python, I wonder if the following code for multi-processes can be ...
user avatar
3 votes
0 answers
45 views

Generating basic Fractals

Trying to generate fractals image with given IFS equations. I have two classes: Shape - Gives co-ordinate to form a shape. Ifs -...
user avatar
  • 634
3 votes
0 answers
129 views

Implement vectorization instead of nested loops on dataframe

I have a dataset ('sample_data.csv') of the form below: ...
user avatar
  • 31
3 votes
0 answers
307 views

Clean and fast MiniMax for numpy implementation of Tic-Tac-Toe

Tic-tac-toe is small enough to be solved completely but since I have a slow computer and this is an interesting exercise, I want to build the fastest possible minimax tic-tac-toe implementation in ...
user avatar
  • 333
3 votes
0 answers
250 views

Implementing a tic tac toe board in python

This is my attempt at making a class to represent a game board for tic tac toe. I normally hate while loops but I used one because I cant seem to make the ...
user avatar
  • 139
3 votes
0 answers
61 views

TensorFlow regressor to predict probability of winning, given a current score

This is a TensorFlow regressor that tells me that if you get x score, you will have x% to win a game in x game(game is not important). Doing this as a project to learn tensorflow. ...
user avatar
  • 31
3 votes
0 answers
416 views

N-D Kalman Filter in Python + Numpy

Just implemented this Kalman Filter in Python + Numpy keeping the Wikipedia notation It's a pretty straightforward implementation of the original algorithm, the goals were develop skills related to ...
user avatar
3 votes
0 answers
59 views

Compare every combination of variables within the powerset

I have some survey data that represents individual's responses to multiple survey questions. There are about 10,000 people in my actual dataset, and each person answered 35 questions. From these 35 ...
user avatar
  • 600
3 votes
0 answers
1k views

Local maxima 3D array python

I'm trying to find the local maxima in a 3D numpy array, but I can't seem to find an easy way to do that using numpy, scipy, or anything else. For now I implemented it using ...
user avatar
  • 31
3 votes
0 answers
368 views

Locality Sensitive Hash (similar to k-Nearest Neighbor), in Python+Numpy

I've tried implementing Locality Sensitive Hash, the algorithm that helps recommendation engines, and powers apps like Shazzam that can identify songs you heard at restaurants. LSH is supposed to run ...
user avatar
  • 187
3 votes
0 answers
143 views

Computing Doppler delay on a meshgrid

Objective Plot the contour of the iso-doppler and iso-delay lines for a transmitter-receiver reflection on a specular plane. Implementation This Doppler shift can be expressed as follows: $$f_{D,0}(...
user avatar
  • 773
3 votes
0 answers
66 views

Minimization problem solving and its step limits

I am trying to solve the following minimization problem: $$ \min\lvert\lvert{x}\rvert\rvert_1 + \beta\lvert\lvert{x}\rvert\rvert^2_2 s.t. \sum_{m = 1}^M (y - \lvert{c}^{H} . x\rvert^2)^2 \le \epsilon ...
user avatar
3 votes
0 answers
85 views

Variable bit length lossy floating point compression

I am implementing a new compression algorithm for the weights of a neural network for the Leela Chess project. the weights are roughly 100Mb of ...
user avatar
  • 3,512
3 votes
0 answers
90 views

Data set generation with Python and Matplotlib

I want to use Python and Matplotlib to create a data set of images. I want to do this by transforming the plot to a numpy matrix. At the moment it takes about 0.1 second to generate 100 images of size ...
user avatar
  • 1,003
3 votes
0 answers
576 views

Searching for the location of a sub-image within an image

I have a number of images that were cropped and their backgrounds removed. I've tried to write a function that will search the original unmodified image for the cropped image so that I can restore the ...
user avatar
  • 181
3 votes
0 answers
226 views

Merging bin-data via a bin count threshold

When performing a chi-squared test, one takes the square of the differences of the expected counts per bin and observed counts per bin, and divides these per-bin differences by the expected counts per ...
user avatar
3 votes
0 answers
86 views

Numpy versus Theano GPU parallelization

I am learning Theano to accelerate my image processing functions. As a start, I am trying to reimplement the function to turn images from colors to black and white (with the same number of channels): ...
user avatar
  • 131
3 votes
0 answers
43 views

Deriving incremental feature representations for bit flips

Given a boolean 3D matrix and a set of actions specifying bit flips at certain positions, a set of resulting matrices can be obtained, one matrix for each action, as if executing each bit flip ...
user avatar
  • 285
3 votes
0 answers
3k views

Reading a csv file and making a histogram in Python using NumPy and Matplotlib

I have written the following code which reads a csv file that contains a bunch of words and their sentiment value. Words like abandon may have a value of -1, while words like progress and freedom have ...
user avatar
3 votes
0 answers
407 views

Applying Scikit-Learn Linear Kernel with a generator for low memory pressure

matrix = # a sparse matrix of TFIDF vectors - 28458x3218988 My aim was to create a similarity matrix comparing all documents to all documents using ...
user avatar