7
\$\begingroup\$

I have written a python function that outputs scatter plots using Matplotlib after processing the data a little. It works but it's painfully slow. I was wondering if anybody had any suggestions as to how I can improve the efficiency of this function or whether its slow just because it's processing a lot of data (a 30x43 element Pandas data frame).

def scatter_x_yi(self,filtered_data,x_parameter):
    '''
   takes some prefiltered data (len(x) by len(Y)) and the name of a
    data column (x_parameter), plots it against every other column (yi)
    '''
    PE=Parameter_Estimation_Tools() #custom class 
    number_of_full_subplots=len(filtered_data)/16   
    remainder=len(filtered_data)-(number_of_full_subplots*16)-1
    os.path.dirname(str(filtered_data))     
    '''
    some code for isolating the x parameter column
    and all the y's (yi)
    '''
    yi=[]
    for i in range(len(filtered_data)):
        if filtered_data[i].name!=x_parameter:
            yi.append(filtered_data[i])      
    '''
    code for plotting the scatter plots (43 in total)
    using matplotlib in 'units' of 4 by 4, then plotting 
    the remainder in a last figure

    '''
    try:
        for i in range(len(filtered_data)):
            for j in range(number_of_full_subplots+1):
                fig=plt.figure(j)
                txt = fig.suptitle('{} , {} of {}'.format(str(x_parameter),str(j),str(number_of_full_subplots)),fontsize='18')
                txt.set_text('{} , {} of {}'.format(str(x_parameter),str(j),str(number_of_full_subplots)))                    
                for k in range(16):
                    plt.subplot(4,4,k)
                    PE.scatter_x_y(filtered_data,x_parameter,yi[16*j+k].name)
                    plt.subplots_adjust(wspace=0.2, hspace=0.25,top=0.9)
            mng=plt.get_current_fig_manager()
            mng.window.showMaximized()
            plt.ioff()
            fig.savefig(str(j)+'.png',bbox_inches='tight',orientation='landscape')
            plt.close(fig)
            plt.ion()
    except IndexError:
        for i in range(len(filtered_data)-remainder,len(filtered_data)-1):
            PE.scatter_x_y(filtered_data,x_parameter,yi[i].name)
            plt.subplots_adjust(wspace=0.2, hspace=0.25,top=0.9)
            mng=plt.get_current_fig_manager()
            mng.window.showMaximized()
            plt.ioff()
            fig.savefig('2.png',bbox_inches='tight',orientation='landscape')
            plt.close(fig)
            plt.ion()
        print '{} has been plotted against all other parameters'.format(x_parameter)
\$\endgroup\$

closed as off-topic by Toby Speight, Graipher, Zeta, Gerrit0, IEatBagels Mar 12 at 20:29

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Lacks concrete context: Code Review requires concrete code from a project, with sufficient context for reviewers to understand how that code is used. Pseudocode, stub code, hypothetical code, obfuscated code, and generic best practices are outside the scope of this site." – Toby Speight, Graipher, Zeta, Gerrit0, IEatBagels
If this question can be reworded to fit the rules in the help center, please edit the question.

  • 5
    \$\begingroup\$ This code doesn't run. What is self? What is Parameter_Estimation_Tools() \$\endgroup\$ – Curt F. Jul 20 '15 at 6:03
3
\$\begingroup\$

Based on the code provided, here are a few suggestions for where decreasing the time it takes to run it:

Too many ranges, iterators

The first thing that pops out as an area for improvement is the number of basic iterators being used, like for x in range()... and when you are using the Pandas library, it's likely there's a speed improvement there.

The problem with using for x in range() is that the script has to go through every single item in the range, one at a time. Pandas has a number of more efficient approaches to handling this, namely df.groupby(), df[colname].apply(), and in the worst case scenario for index, row in df.iterrows() that will very likely cut down your time.

Here's one example:

yi = []
for i in range(len(filtered_data)):
    if filtered_data[i].name != x_parameter:
        yi.append(filtered_data[i])

It appears you are simply trying to make a subset of filtered_data where the column name doesn't match the x_parameter. Assuming that filtered_data is a Pandas dataframe object, you can replace the code block above with this and it should be faster:

yi = filtered_data.loc[filtered_data.name != x_parameter]

Parallelize the chart generation

Instead of having the code blocked from processing data while it waits for matplotlib to create and save the next chart, making this into a separate function and process that can be run concurrently may save you some time. Breaking up the different steps in your pipeline allows you to run the slow loops separately, and also makes testing easier. For a specific example of how to use multiprocessing to run your charting code concurrently, check out this answer on Stackoverflow.

Use ipython magic functions

It is very helpful to use ipython and the magic functions it contains to debug time issues in code. For instance, if you have broken up your one big script into smaller functions, you can use %timeit and to find out what is taking too long.

Lose your self

This isn't really a speed optimization, but definitely helps with readability.

self is the first argument to this function, however it doesn't seem this variable is used elsewhere. My guess is that the function being asked about is actually a method within a class. One quick and easy optimization would be to remove self from the argument list, since it doesn't appear that you need it, and then add @staticmethod to the just before your function. So this will look like:

@staticmethod
def scatter_x_yi(filtered_data, x_parameter):
    [... rest of the method...]

More Possibilities

  • There may be some additional ways to optimize this code hidden in the Parameter_Estimation_Tools(). What is that process doing? Take a look if that is taking too much time.
  • It appears this isn't a standalone function but a method contained in a class. What if this function were broken out to run in isolation?
\$\endgroup\$

Not the answer you're looking for? Browse other questions tagged or ask your own question.