2
\$\begingroup\$

I have the following function and code snippet to extract cell values for multiple years, format it, and save to a list. Each raster has 365 bands — one for each day. A separate operation is performed on each flattened list pr_flt which contains daily cell values of XY for 1991-2015. The cursor contains XY coordinates for each cell. The following code works but it takes hours to finish due to the huge amount of data and possible use of lists instead of multidimensional arrays. I am wondering if there are any suggestions to reduce the runtime of the code, if possible.

import arcpy

def row_to_values(row):
    values = []
    for col in row:
        if isinstance(col, unicode) and col != u'f':
            # split and convert all entries to float
            values += (float(v) for v in col.split(','))
        else:
            values.append(col)
    return values


with arcpy.da.SearchCursor(fc, fields) as cursor:
    for idx, row in enumerate(cursor):
        pt_annual_lists = []
        pt_loc = str(row[0]) +  str(" ") + str(row[1])
        for y in xrange(1991,2016,1):
            pt_year_list = []
            result = arcpy.GetCellValue_management(in_raster = "D:/temp/ras" + str(y) + ".tiff", location_point = pt_loc, band_index="")
            cell_value = result.getOutput(0)
            pt_year_list.append(cell_value)
            cell_value = [s.replace('\\n', ',') for s in pt_year_list]            
            pt_annual_lists.append(row_to_values(cell_value)) 

        # flatten list of lists
        pr_str = [val for sublist in pt_annual_lists for val in sublist]
        pr_flt = [float(i) for i in pr_str]
\$\endgroup\$

migrated from gis.stackexchange.com May 20 at 0:21

This question came from our site for cartographers, geographers and GIS professionals.

  • \$\begingroup\$ If you're truly interested in performance don't code in python, use C++ or C# if you find C++ too daunting. If the total size of the raster is smaller than your available memory you can read the whole raster into an array (RasterToNumpyArray resources.arcgis.com/en/help/main/10.2/index.html#//…) then index the array rather than trying to read a single cell at a time; if your raster is compressed the GetCellValue needs to work very hard to get a single cell value. \$\endgroup\$ – Michael Stimson May 19 at 23:11
  • \$\begingroup\$ You would be more likely to get an answer if you included enough information to actually run the code. \$\endgroup\$ – 200_success May 20 at 4:17
  • 1
    \$\begingroup\$ Michael, that's not really true. Numpy is typically about as fast as handwritten c code due to being a heavily optimized library. Numpy is slow when people think it is C and do loops \$\endgroup\$ – Oscar Smith May 20 at 5:28
  • \$\begingroup\$ @200_success I have yearly NetCDF files with 365 bands. Please suggest how to include that information here for anyone to try and run my code. \$\endgroup\$ – Ibe May 20 at 8:41
  • \$\begingroup\$ Use of coordinates from *.csv file instead of using arcpy.da.SearchCursor has reduced run time by 150%. Other bottleneck is arcpy.GetCellValue_management which is taking 95% of the processing time in each iteration. Please suggest any open source library to extract pixel values faster. \$\endgroup\$ – Ibe May 20 at 18:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.