2
\$\begingroup\$

So this is a project I have been working on for the last few weeks. Just started learning Python. Started out with bash scripting and got the itch to learn more. Anyway code fetches covid-19 data from the internet, assembles data the way I need it to, plots it on graphs that are saved as .png files, writes all the data and graphs to a 'README.md' file and pushes it to github. As I am self taught I am posting this for feedback so I dont develop bad habits and can learn the best practices. Any advice or criticism would be appreciated. Thank you.

#!/usr/bin/env python3
import requests
import pandas as pd
import io
import numpy as np
import matplotlib.pyplot as plt
import os

# **** Build data ****

# Fetch data
url = 'https://raw.githubusercontent.com/nytimes/covid-19-data/master/us.csv'
download = requests.get(url).content
df = pd.read_csv(io.StringIO(download.decode('utf-8')))

# Extract each column individually
date = df['date']
cases = df['cases']
deaths = df['deaths']

# Calculate new cases
total_cases = np.array(cases)
new_cases = np.diff(total_cases)
new_cases = np.insert(new_cases, 0, 1)

# Calculate new deaths
total_deaths = np.array(deaths)
new_deaths = np.diff(total_deaths)
new_deaths = np.insert(new_deaths, 0, 0)

# Create csv for total cases and deaths
df = pd.DataFrame({'date': date, 'total cases': total_cases,
    'total deaths': total_deaths})
df.to_csv('data/us_covid-19_total.csv', index=False)

# Create csv for new cases and deaths
df = pd.DataFrame({'date': date, 'new cases': new_cases, 
    'new deaths': new_deaths})
df.to_csv('data/us_covid-19_new.csv', index=False)

# Create csv for all aggregated data
df = pd.DataFrame({'date': date, 'total cases': total_cases, 
    'total deaths': total_deaths, 'new cases': new_cases, 'new deaths': new_deaths})
df.to_csv('data/us_covid-19_data.csv', index=False)

# **** Plot data ****

# x axis for all plots
x = np.array(date, dtype='datetime64')

# Plot Total Cases
y = total_cases / 1000000
plt.figure('US Total COVID-19 Cases', figsize=(15, 8))
plt.title('US Total COVID-19 Cases')
plt.ylabel('Cases (in millions)')
plt.grid(True, ls='-.')
plt.yticks(np.arange(min(y), max(y) + 10))
plt.plot(x, y, color='b')
plt.savefig('plots/US_Total_COVID-19_Cases.png')

# Plot Total Deaths
y = total_deaths / 1000
plt.figure('US Total COVID-19 Deaths', figsize=(15, 8))
plt.title('US Total COVID-19 Deaths')
plt.ylabel('Deaths (in thousands)')
plt.grid(True, ls='-.')
plt.yticks(np.arange(min(y), max(y) + 100, 50))
plt.plot(x, y, color='b')
plt.savefig('plots/US_Total_COVID-19_Deaths.png')

# Plot New Cases
y = new_cases / 1000
plt.figure('US New COVID-19 Cases', figsize=(15, 8))
plt.title('US New COVID-19 Cases')
plt.ylabel('Cases (in thousands)')
plt.grid(True, ls='-.')
plt.yticks(np.arange(min(y), max(y) + 100, 50))
plt.plot(x, y, color='b')
plt.savefig('plots/US_New_COVID-19_Cases.png')

# Plot New Deaths
y = new_deaths
plt.figure('US New COVID-19 Deaths', figsize=(15, 8))
plt.title('US New COVID-19 Deaths')
plt.ylabel('Deaths')
plt.grid(True, ls='-.')
plt.yticks(np.arange(min(y), max(y) + 1000, 500))
plt.plot(x, y, color='b')
plt.savefig('plots/US_New_COVID-19_Deaths.png')

# **** Write to README.md ****

# New cases and deaths in the last 24 hours
cases = new_cases[-1]
deaths = new_deaths[-1]

# 7-day mean for new cases and deaths
cmean = np.mean(new_cases[-7:])
dmean = np.mean(new_deaths[-7:])

# Date
date = np.array(date, dtype='datetime64')
date = date[-1]

# DataFrame for new cases and deaths in the last 24 hours 
df_24 = pd.DataFrame({'New cases': [f'{cases:,d}'], 'New deaths': [f'{deaths:,d}']})
df_24 = df_24.to_markdown(index=False, disable_numparse=True)

# DataFrame for 7-day average
df_avg = pd.DataFrame({'Cases': [f'{int(cmean):,d}'], 'Deaths': [f'{int(dmean):,d}']})
df_avg = df_avg.to_markdown(index=False, disable_numparse=True)

# Write to 'README.md'
f = open('README.md', 'w')
f.write(f'''# US COVID-19 [Data](https://github.com/drebrb/covid-19-data/blob/master/data/us_covid-19_data.csv)
###### Reported numbers for {str(date)} 
{df_24}
###### 7-day average 
{df_avg}
## [Total Cases and Deaths](https://github.com/drebrb/covid-19-data/blob/master/data/us_covid-19_total.csv)
### Cases
![Plot](https://github.com/drebrb/covid-19-data/blob/master/plots/US_Total_COVID-19_Cases.png)
### Deaths
![Plot](https://github.com/drebrb/covid-19-data/blob/master/plots/US_Total_COVID-19_Deaths.png)
## [New Cases and Deaths](https://github.com/drebrb/covid-19-data/blob/master/data/us_covid-19_new.csv) 
### Cases
![Plot](https://github.com/drebrb/covid-19-data/blob/master/plots/US_New_COVID-19_Cases.png)
### Deaths
![Plot](https://github.com/drebrb/covid-19-data/blob/master/plots/US_New_COVID-19_Deaths.png)''')
f.close()

# **** push to github ****

os.system('git add . && git commit -m "Updating data." && git push')
```
\$\endgroup\$
2
  • \$\begingroup\$ Please follow the guidelines: codereview.stackexchange.com/help/how-to-ask . \$\endgroup\$
    – BCdotWEB
    Feb 27 at 10:45
  • 1
    \$\begingroup\$ Just a quick tip: Since version 3.6 of Python you can write 1_000_000 instead of 1000000. The underscore in numbers is actually ignored by the Python interpreter, but it makes big numbers more readable for the human eye and can therefore prevent errors. \$\endgroup\$
    – Flursch
    Mar 2 at 13:58
3
\$\begingroup\$

Your code is currently a linear sequence of steps. Even for fairly short programs, that's an inflexible structure that is difficult to experiment with, debug, test, and evolve. The solution is to break your code apart into separate functions, each having a narrow focus. Here's a rough sketch. The basic idea is to place all code (other than imports and constants) inside of functions.

# Imports and constants.

import sys
...

DATA_URL = 'https://raw.githubusercontent.com/nytimes/covid-19-data/master/us.csv'

# The program entry point

def main(args):
    ...

# Functions to do various things.

def fetch_data():
    ...

def write_csv_file(():
    ...

# Call main().

if __name__ == '__main__':
    main(sys.arvg[1:])

Making that change will instantly open up other opportunities. For example, you might want to add a new behavior in the future. That new behavior can be easily added without requiring you to revisit the entire program. Instead, you could simply use a command-line argument (args in the sketch above) to trigger the new behavior.

Your code also has some repetitiveness -- for example, CSV file creation and plot creation. The solution again involves using functions: identify the parts that are repetitive; store the parameters that vary in a data structure; and move the repeated action into a function. For example, for CSV file creation you might consider something along the lines sketched below. A similar set of tactics could be applied to plot creation as well.

def main(args):
    ...
    write_csv_files(...)

def write_csv_files(date, total_cases, total_deaths, new_cases, new_deaths):
    # Prepare the parameters that vary.
    csv_cols = {
        'date': date,
        'total cases': total_cases,
        'total deaths': total_deaths,
        'new cases': new_cases,
        'new deaths': new_deaths,
    }
    # A data structure to drive the iteration.
    csv_file_params = {
        'total': ['date', 'total cases', 'total deaths']
        'new': ['date', 'new cases', 'new deaths'],
        'data': ['date', 'total cases', 'total deaths', 'new cases', 'new deaths'],
    }
    # Iterate.
    for suffix, col_names in csv_file_params.items():
        file_path = f'data/us_covid-19_{suffix}.csv'
        d = {nm : csv_cols[nm] for nm in col_names}
        write_csv_file(file_path, d)

def write_csv_file(file_path, d, index=False):
    df = pd.DataFrame(d)
    df.to_csv(file_path, index=index)

Your code is starting to get a proliferation of variables. For example, in the sketch above, write_csv_files() requires 5 arguments. That's not terrible, but it is a bit of an early-warning sign. If those variables need to move around together in your program, you could consider various ways to bundle them together. There are many options, such as a dict, a namedtuple, or a dataclass.

Large multi-line strings are a hindrance to code readability and understanding. Rather than cluttering up the README writing logic with a giant string, separate the boring stuff (the bulk of the text than never changes) from the coding logic:

from textwrap import dedent

# Define the template as a constant.
README_TEMPLATE = dedent('''
    # US COVID-19 [Data](https://github.com/drebrb/covid-19-data/blob/master/data/us_covid-19_data.csv)
    ###### Reported numbers for {} 
    {}
    ###### 7-day average 
    {}
    ## [Total Cases and Deaths](https://github.com/drebrb/covid-19-data/blob/master/data/us_covid-19_total.csv)
    ### Cases
    ![Plot](https://github.com/drebrb/covid-19-data/blob/master/plots/US_Total_COVID-19_Cases.png)
    ### Deaths
    ![Plot](https://github.com/drebrb/covid-19-data/blob/master/plots/US_Total_COVID-19_Deaths.png)
    ## [New Cases and Deaths](https://github.com/drebrb/covid-19-data/blob/master/data/us_covid-19_new.csv) 
    ### Cases
    ![Plot](https://github.com/drebrb/covid-19-data/blob/master/plots/US_New_COVID-19_Cases.png)
    ### Deaths
    ![Plot](https://github.com/drebrb/covid-19-data/blob/master/plots/US_New_COVID-19_Deaths.png)
''')

def write_readme(date, df_24, df_avg):
    with open('README.md', 'w') as fh:
        fh.write(README_TEMPLATE.format(date, df_24, df_avg))

HTTP requests and git operations can fail. They should be wrapped in try blocks, with failures handled one way or another, as noted already in the good review from Reinderien.

\$\endgroup\$
1
  • 1
    \$\begingroup\$ Wow thank you so much. Some of what you explained I do not understand YET as I am still learning but you have provided me with enough information for me to now research and learn. Thank you. \$\endgroup\$
    – drebrb
    Feb 28 at 0:54
3
\$\begingroup\$

You should add a requirements.txt containing something like

tabulate
pandas
requests
numpy
matplotlib

The first requirement was hidden and bit me when I attempted to run your code.

Introduce error checking to your requests call, and let it handle encoding for you:

url = 'https://raw.githubusercontent.com/nytimes/covid-19-data/master/us.csv'
with requests.get(url) as response:
    response.raise_for_status()
    with io.StringIO(response.text) as download:
        df = pd.read_csv(download)

You should be creating the data and plot directories if they don't exist, something like

data = Path('data')
if data.exists():
    assert data.is_dir()
else:
    data.mkdir()
# ...
df.to_csv(data / 'us_covid-19_total.csv', index=False)

Use a context manager for your readme writing:

with open('README.md', 'w') as f:
    f.write( # ...

Try to move your large heredoc-style README content to a template file and use the built-in templating facility.

Don't call os.system, don't use shell statements, and don't concatenate them with &&. Instead, use check_call and call directly into /usr/bin/git.

\$\endgroup\$
2
  • 1
    \$\begingroup\$ Wow thank you. That's a lot of great information. Definitely going to add the requirements.txt, sorry about that! As for the data and plots directory I was on the fence on whether I should warn the user if the directories are missing or if I should just create them. I'll need to research templating and check_call as I am not familiar with them, but I do understand your request call block, and context manager and will be implementing those. Thank you, still have lots to learn lol. \$\endgroup\$
    – drebrb
    Feb 27 at 17:42
  • \$\begingroup\$ Throwing if the directories don't already exist is a fine option too; whatever works for your use case. \$\endgroup\$
    – Reinderien
    Feb 27 at 19:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.