3
\$\begingroup\$

I've found that the function genfromtxt from numpy in Python is very slow.

Therefore I decided to wrap a subroutine with f2py to read my data. The data is a matrix.

subroutine genfromtxt(filename, nx, ny, a)
implicit none
    character(100):: filename
    real, dimension(ny,nx) :: a 
    integer :: row, col, ny, nx
    !f2py character(100), intent(in) ::filename
    !f2py integer, intent(in) :: nx
    !f2py integer, intent(in) :: ny
    !f2py real, intent(out), dimension(nx,ny) :: a

    !Opening file
    open(5, file=filename)

    !read data again
    do row = 1, ny
        read(5,*) (a(row,col), col =1,nx) !reading line by line 
    end do
    close (5)
end subroutine genfromtxt

The length of the filename is fixed to 100 because f2py can't deal with dynamic arrays. The code works for filenames shorter than 100, otherwise the code in Python crashes.

This is how I call the function in Python

import Fmodules as modules
w_map=modules.genfromtxt(filename,100, 50)

Any idea on how to do this dynamically without sending nx and ny as parameters?

\$\endgroup\$
  • \$\begingroup\$ because f2py can't deal with dynamic arrays Documentation says it can (cf. this scipy docs page), but I've never been able to get it to work (though, admittedly, not much attempts at it. If you can get it to work, I have a solution for the allocatable arrays (though it's hella ugly). \$\endgroup\$ – Kyle Kanos Jul 3 '15 at 20:52
4
\$\begingroup\$

Numpy's genfromtxt is indeed slow which is due to it's flexibility. It tries very hard to figure out what the layout of your data file is. So you will not get this flexibility by implementing it yourself in fortran.

Have you tried using loadtxt instead?

Depending on the conditions, for the same matrix genfromtxt may be more than 20 times slower than loadtxt.

BTW a simple implementation in python which is in my case faster than both loadtxt and genfromtxt:

with open("matrix.txt",'r') as f:
    a=array([fromstring(s,dtype=float,sep=' ') for line in f])

I guess the speed comes from the fact that I do not have to read in the whole file to check how many lines it has.

EDIT: I realise this does not really answer your question, but I believe that the speed-up that you may get from using fortran does not really warrant the loss of flexibility.

\$\endgroup\$
  • \$\begingroup\$ Thanks in the end i decided to port my data to HDF5. I think is one of the fastest formats to read and write data on the fly. \$\endgroup\$ – ilciavo Aug 25 '15 at 11:08
  • 1
    \$\begingroup\$ That is indeed a nice format to work with. An as soon as you work with binary and not text data, it is going to be much faster than converting text based files. FYI Numpy can also save matrices in a binary format if you ever have a smaller project where going full HDF5 is too much of a hassle. Cheers \$\endgroup\$ – Ben K. Aug 25 '15 at 12:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.