# Web scraping JSON-data from API put into Dataframe

The web-scraping process currently takes quite a bit of time, and I wonder if I can structure the code otherwise or improve it in any way.

Code looks like this:

import numpy as np
import pandas as pd
import requests
import json
from sklearn import preprocessing
from sklearn.preprocessing import OneHotEncoder

results_2017 = []
results_2018 = []
for game_id in range(2017020001, 2017021271, 1):
url = 'https://statsapi.web.nhl.com/api/v1/game/{}/boxscore'.format(game_id)
r_2017 = requests.get(url)
game_data_2017 = r_2017.json()

for homeaway in ['home','away']:

game_dict_2017 = game_data_2017.get('teams').get(homeaway).get('teamStats').get('teamSkaterStats')
game_dict_2017['team'] = game_data_2017.get('teams').get(homeaway).get('team').get('name')
game_dict_2017['homeaway'] = homeaway
game_dict_2017['game_id'] = game_id
results_2017.append(game_dict_2017)

df_2017 = pd.DataFrame(results_2017)

for game_id in range(2018020001, 2018020667, 1):
url = 'https://statsapi.web.nhl.com/api/v1/game/{}/boxscore'.format(game_id)
r_2018 = requests.get(url)
game_data_2018 = r_2018.json()

for homeaway in ['home','away']:

game_dict_2018 = game_data_2018.get('teams').get(homeaway).get('teamStats').get('teamSkaterStats')
game_dict_2018['team'] = game_data_2018.get('teams').get(homeaway).get('team').get('name')
game_dict_2018['homeaway'] = homeaway
game_dict_2018['game_id'] = game_id
results_2018.append(game_dict_2018)

df_2018 = pd.DataFrame(results_2018)

df = df_2017.append(df_2018)

• Welcome to Code Review! What task does this code accomplish? Please tell us, and also make that the title of the question via edit. Maybe you missed the placeholder on the title element: "State the task that your code accomplishes. Make your title distinctive.". Also from How to Ask: "State what your code does in your title, not your main concerns about it.". Jan 9 '19 at 18:42

• You could mention multiprocessing.Pool.map that is particularly helpful in this case. Jan 9 '19 at 22:03