Calculating the total daily amount of confirmed cases of Coronavirus

Question

I'm writing a small program to plot new COVID-19 infections. As of right now, I have it so the program reads the given data file, pulls out the daily cases and dates for each country, and adds together all the cases for a given date. However, because both of the lists generated have lengths of over 2000, it currently runs extremely slowly. Is there any change I can make to improve the speed of my program?

import pylab as pl

cases = pd.read_csv("daily-cases-covid-19.csv")
dc = cases.loc[:,'Daily confirmed cases (cases)']
dd = cases.loc[:,'Date']




worldCases = []

for i in range(0,len(dd)):
    count = 0
    for j in range(0,len(dd)):
        if dd[j]==dd[i]:
            count+=dc[i]
    worldCases.append(count)

Here is an example of the CSV I am reading through. The purpose of the nested loops is to add together all of the confirmed cases in each country on a given date.

Afghanistan,AFG,"Jan 1, 2020",0
Afghanistan,AFG,"Jan 2, 2020",0
Afghanistan,AFG,"Jan 3, 2020",0
Afghanistan,AFG,"Jan 4, 2020",0
Afghanistan,AFG,"Jan 5, 2020",0
Afghanistan,AFG,"Jan 6, 2020",0
Afghanistan,AFG,"Jan 7, 2020",0
Afghanistan,AFG,"Jan 8, 2020",0
Afghanistan,AFG,"Jan 9, 2020",0
Afghanistan,AFG,"Jan 10, 2020",0
Afghanistan,AFG,"Jan 11, 2020",0
Afghanistan,AFG,"Jan 12, 2020",0
Afghanistan,AFG,"Jan 13, 2020",0
Afghanistan,AFG,"Jan 14, 2020",0
Afghanistan,AFG,"Jan 15, 2020",0
Afghanistan,AFG,"Jan 16, 2020",0
Afghanistan,AFG,"Jan 17, 2020",0
Afghanistan,AFG,"Jan 18, 2020",0
Afghanistan,AFG,"Jan 19, 2020",0

Answer 1

So it seams like you have a Pandas dataframe you are working with. At the moment it looks like this section of code does the following

worldCases = []                   

for i in range(0,len(dd)):               #iterate through all dates
    count = 0                            #setup a counter
    for j in range(0,len(dd)):           #iterate through all dates
        if dd[j]==dd[i]:                 
            count+=dc[i]                 #add one to counter if inner date == outer date
    worldCases.append(count)             #track number of times a unique date occurs

You are effectively binning your data, getting a count of the number of times each unique date occurs. Pandas offers you more efficient and convenient tools for doing this. Specifically look into the groupby method.

A much faster way to get the same output for worldCases would be to do the following:

# group the daily cases data by the date and then compute 
# the sum of cases within each date group

dc = 'Daily confirmed cases (cases)'
worldCases = cases.loc[:, dc].groupby(cases['Date']).sum()

Also, if you find yourself writing lots of loops like the above, you may want to check out the user guide for "group by" operations in the pandas documentation. It is a very powerful way of handling situations like this, and more, but can take a bit of getting used to. It can feel like a whole different way of thinking about your data.