3
\$\begingroup\$

This is for an API and I want to optimize this because it has 2 for loops followed by another 2 for loops and it is taking 4 hours for 40000 rows in the database.

The object obtained after the request to the API is a nested dictionary. Is there a faster way to go through a nested dictionary with lists and more dictionaries with more lists within, than using nested for loops?

I have to analyze more than 500 000 files and for different transportation modes so it will take me 1 week to run this code for all the modes. Is there some way to optimize it?

I am looking for a guideline or a methodology to follow. I am not a developer or software engineer and new to Python.

import urllib
import json
import requests
import csv
import pandas as pd
def get_car(tripgokey,database):
    tripgo_key='xxxxxx'
    mode ='me_car'
    database=pd.read_csv('tripgo_get_car.csv')
    for ind, row in database.iterrows():
        orig_cord=row[1]
        dest_cord=row[2]
        dep_time=row[3]
#     print([orig_cord,dest_cord,dep_time])
        URL= 'https://api.tripgo.com/v1/routing.json?from=('+orig_cord+')&to=('+dest_cord+')&modes='+mode+'&v=11&locale=en&departAfter='+str(dep_time)+'&ir=true'
        headers={
            'Accept': 'application/json',
            'X-TripGo-Key': tripgo_key
        }
        TripGoData=requests.get(URL,headers=headers).json()
        print (URL)
        if 'groups'and'segmentTemplates'in TripGoData:
            bestTrip={}
    #Choose the best Trip
            for Group in TripGoData['groups']:        
                for Trip in Group['trips']:
                    if  Trip['depart'] >=dep_time:
                        if bestTrip == {}:
                            bestTrip=Trip
                        elif Trip['arrive']<bestTrip['arrive']: # It is prefer to arrive early rather than a longer trip
                            bestTrip=Trip
    #Get the General Information for the  BEST TRIP
            database.loc[ind,"CO2"]= bestTrip['carbonCost']
            database.loc[ind,"Total_Trip_Time"]=(int(bestTrip['arrive'])-int(bestTrip['depart']))/60
            database.loc[ind,"TotalCost"]=bestTrip['moneyCost']
    #Get the individual information for each segment in the BEST TRIP
            for Segment in bestTrip['segments']:
                    for Template in TripGoData['segmentTemplates']:
                        if Template['hashCode']==Segment['segmentTemplateHashCode']:
                            if Template['modeInfo']['identifier']=='me_car':
                                database.loc[ind,"Car_Distance"]=int(Template['metres'])/1000
                                database.loc[ind,"Car_Cost"]=Template['localCost']['cost']
                                database.loc[ind,"Car_Travel_Time"]=(int(Segment['endTime'])-int(Segment['startTime']))/60
                            elif "stationary_parking"in Template['modeInfo']['identifier']:
                                if "onstreet"in Template['modeInfo']['identifier']:
                                    database.loc[ind,"Parking"]="onstreet"
                                else:
                                    database.loc[ind,"Parking"]="offstreet"    
                                if 'localCost' in Template:
                                    database.loc[ind,"Cost_Parking"]=Template['localCost']['cost']
                                else:
                                    database.loc[ind,"Cost_Parking"]=0
                            else:
                                database.loc[ind,"Walking_Distance"]=int(Template['metres'])/1000
                                database.loc[ind,"Walking_Travel_Time"]=(int(Segment['endTime'])-int(Segment['startTime']))/60
        elif 'error' in TripGoData:
            if 'errorCode' in TripGoData:
                database.loc[ind,"DATA_STATUS"]="Error: "+str(TripGoData['errorCode'])+"-->"+TripGoData['error']
            else:
                database.loc[ind,"DATA_STATUS"]="Error: "+TripGoData['error']
        else:
            database.loc[ind,"DATA_STATUS"]="Data no available"
    database.to_csv('Car_Data.csv',index=False)
\$\endgroup\$
1
  • 2
    \$\begingroup\$ The current question title, which states your concerns about the code, is too general to be useful here. Please edit to the site standard, which is for the title to simply state the task accomplished by the code. Please see How to get the best value out of Code Review: Asking Questions for guidance on writing good question titles. \$\endgroup\$
    – BCdotWEB
    Commented Mar 5, 2020 at 11:54

2 Answers 2

4
\$\begingroup\$

Some suggestions for optimizing:

  • Did you consider the option that the API request could be taking the longest time? Maybe it's worth it to check whether the API has an option to extract multiple records at once.
  • It might be possible that pd.read_json will suit your needs, but I never worked with it so you might have to figure out if nested dictionaries can be read, and if yes, how.
  • Setting fields using .loc is much slower than creating a list and setting it as a column at once.
  • A general tip: create some timers to check which part of your code is the slowest. Start with optimizing at the slowest part. You can also make use of a profiler instead of creating timers, but that can be complicated.
\$\endgroup\$
2
\$\begingroup\$

One way to (slightly) speed up repeated connections to the same server is to use a Session, which keeps the connection alive.

In addition, requests can take a parameter dictionary which will even take care of urlencoding for you:

import requests

URL = "https://api.tripgo.com/v1/routing.json"
HEADERS = {'Accept': 'application/json',
           'X-TripGo-Key': "XXXX"}

with requests.Session() as session:
    ...
    for ind, (_, orig_cord, dest_cord, dep_time) in database.iterrows():
        params = {"from": f"({orig_cord})",
                  "dest": f"({dest_cord})",
                  "modes": mode,
                  "departAfter": dep_time,
                  "v": 11, "locale": "en", "ir": "true"}

        data = session.get(URL, params=params, headers=headers).json()
        ...

Note that I followed Python's official style-guide, PEP8, by having spaces around = when using it for assignment and after commas, used tuple unpacking to directly set the variables in the loop instead of having to do it manually right away and used the relatively new f-string to get the parenthesis around the parameters where needed.

If the API does not support making multiple requests at once, you might also have to look into making multiple requests in parallel, e.g. using aiohttp like I did in this recent question of mine.


For your nested for loop, I'm not sure there is a better way to check all nested subentries. But you can at least make it into a generator expression that flattens it and then use the built-in min:

from operator import itemgetter

trips = (trip for group in data["groups"] for trip in group
         if trip["depart"] >= dep_time)
best_trip = min(trips, key=itemgetter("arrive"))

Note that PEP8 recommends using lower_case for variables (and functions).


This line does not do what you think it does:

if 'groups'and'segmentTemplates'in TripGoData:

This is parsed as

if ('groups' and 'segmentTemplates') in TrupGoData:

which is

if 'segmentTemplates' in TrupGoData:

because non-empty strings are truthy and and returns the last value if all values are truthy (and the first falsey value if not).

Instead you have to use

if 'groups' in TrupGoData and 'segmentTemplates' in TrupGoData:

And for writing to a csv it is probably easier to use csv.DictWriter and return dictionaries from each row:

import csv

def get_best_trip_info(orig_cord, dest_cord, dep_time):
    data = ...
    best_trip = ...
    out = {}
    out["carbonCost"] = best_trip['carbonCost']
    ...
    return out


database = ...
with open('Car_Data.csv', "w") as out_file
    writer = csv.DictWriter(out_file, ["carbonCost", ...])
    writer.writerows(get_best_trip_info(orig_cord, dest_cord, dep_time)
                     for ind, (_, orig_cord, dest_cord, dep_time) in database.iterrows())
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.