3
\$\begingroup\$

I have two folders that contain multiple json files. The first folder contains game logs for all MLB teams and is structured like this.

{"Team": "PIT", "Games": [{"Date": "Jul 24, 2020", "Opponent": "@ San Diego", "Results": "L", "Score": "7-2", "Line": "+128", "Over_Under": "O", "Total": "8", "Players": []}, {"Date": "Jul 25, 2020", "Opponent": "@ San Diego", "Results": "L", "Score": "5-1", "Line": "+115", "Over_Under": "U", "Total": "8", "Players: []}]}

I have a second folder containing json files and the files are structured like this

[{"StatID": 2593242, "TeamID": 4, "PlayerID": 10002075, "SeasonType": 1, "Season": 2019, "Name": "Colin Moran", "Team": "PIT", "Position": "3B", "PositionCategory": "IF", "Started": 1, "InjuryStatus": null, "GameID": 54207, "OpponentID": 31, "Opponent": "STL", "Day": "2019-04-01T00:00:00", "DateTime": "2019-04-01T13:05:00", "HomeOrAway": "HOME", "Games": 1, "FantasyPoints": 12.0, "AtBats": 3.0, "Runs": 1.0, "Hits": 2.0, "Singles": 0.0, "Doubles": 1.0, "Triples": 0.0, "HomeRuns": 1.0, "RunsBattedIn": 3.0, "BattingAverage": 0.667, "Outs": 1.0, "Strikeouts": 0.0, "Walks": 2.0, "HitByPitch": 0.0, "Sacrifices": 0.0, "SacrificeFlies": 0.0, "GroundIntoDoublePlay": 0.0, "StolenBases": 0.0, "CaughtStealing": 0.0, "OnBasePercentage": 0.8, "SluggingPercentage": 2, "OnBasePlusSlugging": 2.8, "Wins": 0.0, "Losses": 0.0, "Saves": 0.0, "InningsPitchedDecimal": 0.0, "TotalOutsPitched": 0.0, "InningsPitchedFull": 0.0, "InningsPitchedOuts": 0.0, "EarnedRunAverage": 0, "PitchingHits": 0.0, "PitchingRuns": 0.0, "PitchingEarnedRuns": 0.0, "PitchingWalks": 0.0, "PitchingStrikeouts": 0.0, "PitchingHomeRuns": 0.0, "PitchesThrown": 0.0, "PitchesThrownStrikes": 0.0, "WalksHitsPerInningsPitched": 0, "PitchingBattingAverageAgainst": 0, "FantasyPointsFanDuel": 37.7, "FantasyPointsDraftKings": 27.0, "WeightedOnBasePercentage": 0.8, "PitchingCompleteGames": 0.0, "PitchingShutOuts": 0.0, "PitchingOnBasePercentage": 0, "PitchingSluggingPercentage": 0, "PitchingOnBasePlusSlugging": 0, "PitchingStrikeoutsPerNineInnings": 0, "PitchingWalksPerNineInnings": 0, "PitchingWeightedOnBasePercentage": 0}]

I want to add the players stats from the second set of files to the first set. Right now I'm using a nested for loop and matching the two dicts based on date and team like this if player['Day'].split('T')[0] == obj['Date'] and player['Team'] == team_data['Team']:

However, this is incredibly slow and inefficient. Is there a faster and more efficient way to do this?

Here is my code:

import json
import pandas as pd
import os

path_to_json = '/Users/aus10/MLB/Combined_Clean_Team_Data'
Game_logs_json_files = [pos_json for pos_json in os.listdir(path_to_json) if pos_json.endswith('.json')]

path_to_json = '/Users/aus10/MLB/FPTS_Data'
FPTS_json_files = [pos_json for pos_json in os.listdir(path_to_json) if pos_json.endswith('.json')]

for file in Game_logs_json_files:
    with open('/Users/aus10/MLB/Combined_Clean_Team_Data/'+file+'') as json_file:
        team_data = json.load(json_file)

    for file_1 in FPTS_json_files:
        with open('/Users/aus10/MLB/FPTS_Data/'+file_1+'') as json_file:
            fantasy_data = json.load(json_file)

            for obj in team_data['Games']:
                for player in fantasy_data:
                    if player['Day'].split('T')[0] == obj['Date'] and player['Team'] == team_data['Team']:
                        obj['Players'].append(player)
            
            for obj in team_data['Games']:
                for player in fantasy_data:
                    if player['Day'].split('T')[0] == obj['Date'] and player['Opponent'] == team_data['Team']:
                        obj['Opposing_P_ERA'] = player["EarnedRunAverage"]
                        obj['Opposing_P_WoBA'] = player["PitchingWeightedOnBasePercentage"]
                        
    with open('/Users/aus10/MLB/Combined_Clean_Team_Data/'+file+'', 'w') as my_file:
        json.dump(team_data, my_file)
\$\endgroup\$
2
  • 1
    \$\begingroup\$ How are the files organized in the directories? One file per team for the game logs? One file per player for the fantasy data? How much data in total (Mega or Giga bytes)? Does the end result need to be a bunch of json files? \$\endgroup\$ – RootTwo Sep 29 '20 at 20:19
  • \$\begingroup\$ @RootTwo One file per team and then files for players in every player who played on a certain date. Team data is 358.7 MB and player data is 164 MB. The end results should be one json file for every team with all of the team game stats plus player stats included in the player array. \$\endgroup\$ – Austin Johnson Oct 1 '20 at 12:15
2
\$\begingroup\$

Use pathlib. It has a very nice interface for file/directory operations. The / operator joins path components together. A pathlib.Path has a .glob() method for finding filenames that match a pattern, and an .open() for opening files.

It is wise to save updated data to a new file or directory. At least until you verify the code is working correctly.

The player data is small enough that it can all be loaded into memory at once. Use a data structure that makes things easier. In this case, use dicts to group the player data by team and date.

team_game_player_data = {'STL':{date(7, 24, 2020):[player_1, player_2, ...],
                                date(7, 25, 2020):[player_1, player_2, ...],
                                ...
                                },
                         'PIT':{date(7, 24, 2020):[player_1, player_2, ...],
                                date(7, 25, 2020):[player_1, player_2, ...],
                                ...
                                },
                         }
                     

Then open each team file and add the player data to the games.

The ERA and WoBA code didn't look right in the original (or I didn't understand it), so I collected if, but didn't save it.

I don't have any data files, so this code is not tested. But, it should give you an idea on how to proceed.

You didn't state your goals for the code, but I suspect you might be better off with a database of some kind rather than a bunch of json files.

import datetime
import json

from collections import defaultdict
from pathlib import Path


BASE_DIR = Path('/Users/aus10/MLB/')
GAME_DIR = BASE_DIR / 'Combined_Clean_Team_Data'
FPTS_DIR = BASE_DIR / 'FPTS_Data'
SAVE_DIR = BASE_DIR / 'Saved_Team_Data'

team_game_player_data = defaultdict(lambda:defaultdict(list))
opposing_player_data = defaultdict(lambda:defaultdict(dict))

for path in FPTS_DIR.glob('*.json'):
    with path.open() as fpts_file:
        fantasy_data = json.load(fpts_file)

        for player_game_data in fantasy_data:
            date = datetime.date.fromisoformat(player_game_data['Day'][:10])
            team = player_game_data['Team']
            opponent = player_game_data['Opponent']

            team_game_player_data[team][date].append(player_game_data)

            opposing_player_data[team][date]['Opposing_P_ERA'] = player_game_data["EarnedRunAverage"]
            opposing_player_data[team][date]['Opposing_P_WoBA'] = player_game_data["PitchingWeightedOnBasePercentage"]
        

for path in GAME_DIR.glob('*.json'):
    with path.open() as team_file:
        data = json.load(team_file)
        
        team = data['Team']
    
        for game in data['Games']:
            game_day = dt.datetime.strptime(game['Date'], "%b %d, %Y").date()
            game['Players'] = team_game_player_data[team][game_day]
            
    filename = path.name
    with (SAVE_DIR / filename).open('w') as save_file:
        json.dump(data, save_file)
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.