Questions tagged [numpy]

NumPy is the fundamental package for scientific computing with the programming language Python.

Filter by
Sorted by
Tagged with
5
votes
2answers
645 views

Defensive programming type-checking

I have issues with dynamically typed languages, and I tend to worry about type a lot. Numpy has different behaviour depending on if something is a matrix or a plain ndarray, or a list. I didn't ...
3
votes
2answers
499 views

How can I make my ant simulation faster?

I was not sure if I should post about 200 lines here. I want to make this ant simulation faster. The bottleneck is at Ant.checkdistancebetweenantsandfood(). It ...
3
votes
1answer
20k views

Reading columns and rows in a .csv file

I have some data in a .csv file, which looks roughly like this: ...
17
votes
2answers
84k views

Fastest way to iterate over Numpy array

I wrote a function to calculate the gamma coefficient of a clustering. The bottleneck is the comparison of values from dist_withing to ...
3
votes
1answer
4k views

Help improve my my python code for Exponential Smoothing Sumitted to Statsmodel

I just submitted this pull request to statsmodels (a Python toolkit for statistical and econometric modeling). It adds support for exponential smoothing of time ...
1
vote
1answer
7k views

Fastest way for working with itertools.combinations

I need to speed up the function below: ...
6
votes
1answer
24k views

Apriori algorithm using Pandas

I want to optimize my Apriori algorithm for speed: ...
1
vote
2answers
100 views

Elegantly avoid columns containing NaNs?

The next bit of code does exactly what i want it to do, but it is 'rather' unelegant and unpythonic, so the question is quite simple: how can this code be changed into something more elegant ...
5
votes
1answer
102 views

Organization of complex empirical equations

I am working on coding up a set of models that are based on empirical models. These models tend to be quite complicated. In some cases, the logic used by the equations makes it impossible to compute ...
1
vote
1answer
2k views

Downsampling n-dimensional data from bins in one dimension

So I'm writing some code to perform a quite specific task given a large numpy array with N rows and 3 columns representing points in 3D. The 3D points are to be binned along one of the dimensions ...
2
votes
1answer
161 views

How to reduce runtime of gene data processing program?

This program takes a huge data set as input, processes it, calculates and then writes the output to an array. Most calculations may be quite simple, such as summation. In the input file, there are ...
3
votes
3answers
594 views

Distinct nearest pixels for floating-point coordinates

dem_rows and dem_cols contain float values for a number of things i can identify in an image, but I need to get the nearest ...
1
vote
3answers
1k views

NumPy Scipy optimization

I have an optimization issue and I'm not sure if I can improve the overall speed of my function. The function draw_w is my actual implementation that gives me the ...
6
votes
1answer
2k views

Using the SSIM method for large images

I'm try to implement the SSIM method. This method has already been implemented in Python (source code), but my goal is implement it with using only Python and NumPy. My goal is also to use this ...
5
votes
1answer
590 views

Calculating the nonlinear reflection coefficient of a crystalline silicon slab

The purpose of this script is to calculate the nonlinear reflection coefficient of a crystalline silicon slab. It takes some input files (columns of data separated by whitespace), converts that data ...
2
votes
1answer
123 views

What would be the optimal way to address these arrays

I think the arrays here are slowing down my code due to incorrect usage. There is a lot of stuff going on here, it's understandable if nobody responds. Just looking for some pointers. Thanks. ...
4
votes
1answer
1k views

Statistical samples and distributions

It is usually inefficient to find bugs in a longer code. Unfortunately, after spending too much time in debugging my code, I realize that a good programming habit is important. Please give me some ...
44
votes
2answers
89k views

Finding the closest point to a list of points

I'm trying to find the closest point (Euclidean distance) from a user-inputted point to a list of 50,000 points that I have. Note that the list of points changes all the time. and the closest distance ...
5
votes
0answers
275 views

Summing 2D NumPy array by multiple labels

Each element of the array is a coordinate (x, y). Each coordinate has two labels. Goal: sum the elements that have the same two labels. How can ...
4
votes
1answer
373 views

Eliminate for loops in numpy implementation

I have the following dataset in numpy ...
5
votes
2answers
2k views

Portable Python CSV class

I have been working on a project where I needed to analyze multiple, large datasets contained inside many CSV files at the same time. I am not a programmer but an engineer, so I did a lot of searching ...
2
votes
1answer
2k views

Using numpy polynomial module - is there a better way?

I'm working on a project where I need to solve for one of the roots of a quartic polymonial many, many times. Is there a better, i.e., faster way to do this? Should I write my own C-library? The ...
3
votes
2answers
543 views

Specialized version of the cross correlation function

I am writing a specialized version of the cross correlation function as used in neuroscience. The function below is supposed to take a time series data and ask how ...
3
votes
1answer
3k views

Are these list-comprehensions written the fastest possible way?

This is a simple repeatable question regarding the usage of Python3's comprehensions: Could the Python3 syntax be used in another way to speed up the process, further so the gap between Python3 and ...
2
votes
1answer
705 views

Performance problems with 1D cellular automata experiment in Python with Numpy

I'm an experienced programmer but relatively new to Python (a month or so) so it's quite likely that I've made some gauche errors. I very much appreciate your time spent taking a look at my code. My ...
1
vote
1answer
6k views

Figuring out the structure of a shape tuple

I have some code that uses a multidimensional look-up table (LUT) to do interpolation. The typical application is a color space conversion, where 3D inputs (RGB) are converted to 4D (CMYK), but the ...
3
votes
2answers
2k views

A pythonic way of de-interleaving a list (i.e. data from a generator), into multiple lists

I've recently discovered the wonders of the Python world, and am quickly learning. Coming from Windows/C#/.NET, I find it refreshing working in ...
2
votes
1answer
700 views

Reading from a file and connect all data in one big data than to use generators

Is there better way to read from a file and connect all data in one big data than to use generators? At the moment, I do the following: use generators to read data from files. use NumPy to pack all ...
3
votes
1answer
849 views

Using Gibbs sampling to segment an image

I have been reading some NumPy guides but can't seem to figure it out. My TA told me I should be able to speed up my code by using a NumPy array instead of a for ...
6
votes
1answer
2k views

Implementing the CutHill-McKee Algorithm

I am very much interested in the Reverse Cuthil McKee Algorithm. I have seen Fortran and C or C++ implementations of it, and I decided that it would be a nice exercise to implement it in Python. I ...
4
votes
1answer
1k views

Mean shift image processing algorithm for color segmentation

I'm implementing a version of the mean shift image processing algorithm for color segmentation in Python/NumPy. I've written a pure NumPy version of the actual mean shifting per pixel (which I ...
6
votes
2answers
5k views

Python / Numpy running 15x slower than MATLAB - am I using Numpy effeciently?

Here is some code I wrote in Python / Numpy that I pretty much directly translated from MATLAB code. When I run the code in MATLAB on my machine, it takes roughly 17 seconds. When I run the code in ...
2
votes
2answers
186 views

Simple arithmetic in Python

As a beginner, I wrote the following python script that solves warrant 1 of this document - pp 436-438. My solution, although works, seems to me as poorly designed and highly unmaintanable. I was ...
1
vote
1answer
3k views

Element-comparison NumPy matrix

I need a NumPy matrix with comparisons among all elements in a sequence: ...
2
votes
2answers
450 views

Compound assignment operators in Python's NumPy library

The "vectorizing" of fancy indexing by Python's NumPy library sometimes gives unexpected results. For example: ...
1
vote
1answer
2k views

Data cleaning - am I using classes correctly here?

I have a data processing application that does some data cleaning as the first step. This is the module I import for that purpose. The only 'public' part of the api is the 'clean' function which ...
2
votes
1answer
153 views

Simple utility/convenience module - am I doing it right?

I have a large library (four packages) of data processing code that I have written over the last 12 months. I am quite new to python and would like to see if I am doing things correctly. This utility/...
1
vote
1answer
1k views

Linear Regression and data manipulation

How could I improve the following code that runs a simple linear regression using matrix algebra? I import a .csv file (link here) called 'cdd.ny.csv', and perform the matrix calculations that solve ...
0
votes
1answer
1k views

Moving average of a data series

How can I rework the code up that ema(x, 5) returns the data now in emaout? I'm trying to enclose everything in one def. Right ...
7
votes
1answer
457 views

Code Review of small scientific project, particuarly array vs. list perform

I'm new to python but not to programming in general. This is my first project beyond the basics. I was wondering if I could get some feedback on the code in general, in particular on any bad Java/C++ ...
2
votes
2answers
1k views

Rolling loop with an array of data points

I have a NumPy array of about 2500 data points. This function is called on a rolling basis where 363 data points is passed at a time. ...
6
votes
5answers
8k views

Calculating the maximum drawdown of a set of returns

I wrote a simple function that calculates and returns the maximum drawdown of a set of returns. I am trying to squeeze as much efficiency for speed out of the code as possible. I've got it down to ...
5
votes
1answer
1k views

Reading a binary file containing periodic samples

I have the following code for reading HTK feature files. The code below is working completely correct (verified it with unit tests and the output of the original HTK toolkit). ...

1
8 9 10 11
12