2
\$\begingroup\$

I am using Python 3.8 with ArcGIS Pro 2.5. This code creates a joined table where I match values for plant greenness in the current month with rainfall X number of months into the past. Each loop iterates through about 600 files (the number of files in ws1 and ws2). The code works perfectly but slowly, taking approximately four hours to run. For my work (a grad student in plant biology) I will need to run this code multiple times.

I am interested in executing the first loop in parallel, however, I've been stumped about how to do so. Any advice about how to optimize this code to run in parallel (using the multiprocessing or other package) would be appreciated. I am using an 8-core 16-thread CPU.

Here is what the code does:

The code first iterates through ws1, which contains tables of plant greenness. The date is isolated from the file name, and then I create a new variable X months before this date. The second loop iterates through ws2, which contains tables of precipitation. Similarly, I isolate the date from the filename, and then create a new variable X months into the past. The final loop iterates through ws2 again and finds the historic filename that matches the one created in the second loop.

r from the first loop is associated with plant greenness at the current month, r2 is associated with precipitation at the current month, while r3 from the last loop is associated with historic precipitation. When the dates from all three loops match then I do the add join.

import arcpy, os, glob, datetime
arcpy.env.addOutputsToMap = 0
from datetime import datetime

ws1=glob.glob(r'D:\GIS_Files\NEW_LA_FULL_SERIES\ALL_DATA_2000_2020\CITY_POLYs\*.shp')
ws2 = glob.glob(r'D:\GIS_Files\NEW_LA_FULL_SERIES\ALL_DATA_2000_2020\PRECIP_preprocessed_2000_2020\CITY_ZONAL_TABLES\*.dbf')

for r in ws1:
    basename_BOUND = os.path.basename(r)[35:41]
    basename_BOUND1=str(basename_BOUND)
    D_POLY=datetime.strptime(basename_BOUND1,'%Y%m').date()
    from datetime import datetime
    import dateutil.relativedelta
    EARLIER_POLY=D_POLY-dateutil.relativedelta.relativedelta(months=2)
    STR_PREV_POLY="{:%Y%m}".format(EARLIER_POLY)   
    print(STR_PREV_POLY)    
    for r2 in ws2:
        basename_ZONAL=os.path.basename(r2)[12:18]
        basename_ZONAL1=str(basename_ZONAL)    
        D_PRECIP= datetime.strptime(basename_ZONAL1,'%Y%m').date()
        from datetime import datetime
        import dateutil.relativedelta
        EARLIER_PRECIP=D_PRECIP-dateutil.relativedelta.relativedelta(months=2)        
        STR_PREV_PRECIP="{:%Y%m}".format(EARLIER_PRECIP)        
        for r3 in ws2:
            basename_FINAL=os.path.basename(r3)[12:18]
            print(basename_FINAL)
            if STR_PREV_POLY==STR_PREV_PRECIP:
                if STR_PREV_PRECIP==basename_FINAL:
                    result = arcpy.JoinField_management(in_data=r, in_field="GEO_id",join_table=r3, join_field="GEO_id", fields=["MEAN","STD"])[0]
\$\endgroup\$
1
  • \$\begingroup\$ Welcome to CodeReview@SE. Can you provide a handful of lines each from input files in your question, and hyperlinks to as much sample input as you see fit? I see surprises coming up. \$\endgroup\$ – greybeard Mar 28 '20 at 5:59
2
\$\begingroup\$

You should do a maintainability pass before you start optimizing. In that spirit:

Imports

from datetime import datetime
import dateutil.relativedelta

should be at the top of the file, and should not be repeated. Also, this:

arcpy.env.addOutputsToMap = 0

should be moved past the imports section.

Pathlib

These:

ws1=glob.glob(r'D:\GIS_Files\NEW_LA_FULL_SERIES\ALL_DATA_2000_2020\CITY_POLYs\*.shp')
ws2 = glob.glob(r'D:\GIS_Files\NEW_LA_FULL_SERIES\ALL_DATA_2000_2020\PRECIP_preprocessed_2000_2020\CITY_ZONAL_TABLES\*.dbf')

should get a little love from pathlib:

Factor out the common directory -

data_dir = Path(r'D:\GIS_Files\NEW_LA_FULL_SERIES\ALL_DATA_2000_2020')
ws1 = (data_dir / 'CITY_POLYs').glob('*.shp')
ws2 = (data_dir / 'PRECIP_preprocessed_2000_2020' / 'CITY_ZONAL_TABLES').glob('*.dbf')

Hard-coded string slices

These:

[35:41]
[12:18]

are a nightmare. I don't know what they're actually selecting, but I can nearly guarantee that there's a better way. It seems like a slice out of a path, and paths are structured - you should not have to do indexing like this.

Nomenclature

You, currently, understand what these variables mean:

ws1
ws2
r
r2
r3
result

But no one else does, and you might not in six months. Do yourself a favour and give these meaningful names.

Logical combination

        if STR_PREV_POLY==STR_PREV_PRECIP:
            if STR_PREV_PRECIP==basename_FINAL:

should simply be

if STR_PREV_POLY == basename_FINAL:

That intermediate variable has no effect.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.