0
\$\begingroup\$

The project I've working on is designed with the intent of being able to quickly and efficiently identify discrepancies in live sports betting data that comes from web API's in json format. The first step is to retrieve the json's and I have incorporated async for that. Each json is structured differently so it gets parsed within it's own class called GamesDict. Once parsed it gets inserted to a database table via sqlalchemy and is ready for evaluation. A key point here is that the json files contain basic data for every game, but we also need more data that can only be obtained via a separate json URL. URL's for individual games need event_id's so that is obtained within the db_query script. Once the individual game json's for each source are obtained they can be parsed and inserted to tables similar to how it was done in the first process.

This is the first iteration of the project and I know I have a lot to learn and would appreciate any feedback.

These are the issues I'm dealing with right now:

  1. Stick with core or use ORM?
  2. How to organize the data - more tables or less?
  3. How to move/replace outdated entries with most recent

parse_games.py

BOOK = 'book3'

class GamesDict:
    """
    Class GamesDict takes a list of dictionaries and a database control object as variables.
    It calls on methods to parse the data pursuant to it's stat type (runline, props, etc)
    and then commits the data to it's corresponding table.
    
    """
    
    def __init__(self, listofdics, db_control):
        self._dic = listofdics[BOOK]['eventGroup']
        self.db_control = db_control
        
        self.events = self._dic.get('events',())
        self.categories = self._dic.get('offerCategories',())

        self.export_gamelist()
        self.export_runline()
        self.export_moneyline()
        self.export_total()
        self.export_propshomeruns()
        self.export_propshits()
        self.export_propsrbis()
        self.export_propsrunsscored()
        self.export_propsstolenbases()
        self.export_propssingles()
        self.export_propsdoubles()
        self.export_propstriples()
        self.export_propsstrikeouts()
        self.export_propsoutsrecorded()
        self.export_propswin()
        self.export_propshitsallowed()
        self.export_propswalks()
        self.export_propsearnedruns()
##        self.export_plateappearanceexact()

    def export_gamelist(self):
        self.gamelist = Gamelist().to_dict(self.events)
        GamelistControl(BOOK,self.db_control).commit_entry(self.gamelist)
        
    def export_runline(self):
        self.runline = Runline().to_dict(self.categories)
        RunlineControl(BOOK,self.db_control).commit_entry(self.runline)

    def export_moneyline(self):
        self.moneyline = Moneyline().to_dict(self.categories)
        MoneylineControl(BOOK,self.db_control).commit_entry(self.moneyline)
        
    def export_total(self):
        self.total = Total().to_dict(self.categories)
        TotalControl(BOOK,self.db_control).commit_entry(self.total)

    def export_propshomeruns(self):
        self.propshomeruns = PropsHomeruns().to_dict(self.categories)
        PropsControl(BOOK,self.db_control,'homeruns').commit_entry(self.propshomeruns)

    def export_propshits(self):
        self.propshits = PropsHits().to_dict(self.categories)
        PropsControl(BOOK,self.db_control,'hits').commit_entry(self.propshits)

    def export_propstotalbases(self):
        self.propstotalbases = PropsTotalBases().to_dict(self.categories)
        PropsControl(BOOK,self.db_control,'totalbases').commit_entry(self.propstotalbases)

    def export_propsrbis(self):
        self.propsrbis = PropsRBIs().to_dict(self.categories)
        PropsControl(BOOK,self.db_control,'rbis').commit_entry(self.propsrbis)

    def export_propsrunsscored(self):
        self.propsrunsscored = PropsRunsScored().to_dict(self.categories)
        PropsControl(BOOK,self.db_control,'runsscored').commit_entry(self.propsrunsscored)

    def export_propsstolenbases(self):
        self.propsstolenbases = PropsStolenBases().to_dict(self.categories)
        PropsControl(BOOK,self.db_control,'stolenbases').commit_entry(self.propsstolenbases)

    def export_propssingles(self):
        self.propssingles = PropsSingles().to_dict(self.categories)
        PropsControl(BOOK,self.db_control,'singles').commit_entry(self.propssingles)

    def export_propsdoubles(self):
        self.propsdoubles = PropsDoubles().to_dict(self.categories)
        PropsControl(BOOK,self.db_control,'doubles').commit_entry(self.propsdoubles)

    def export_propstriples(self):
        self.propstriples = PropsTriples().to_dict(self.categories)
        PropsControl(BOOK,self.db_control,'triples').commit_entry(self.propstriples)

    def export_propsstrikeouts(self):
        self.propsstrikeouts = PropsStrikeouts().to_dict(self.categories)
        PropsControl(BOOK,self.db_control,'strikeouts').commit_entry(self.propsstrikeouts)

    def export_propsoutsrecorded(self):
        self.propsoutsrecorded = PropsOutsRecorded().to_dict(self.categories)
        PropsControl(BOOK,self.db_control,'outsrecorded').commit_entry(self.propsoutsrecorded)

    def export_propswin(self):
        self.propswin = PropsWin().to_dict(self.categories)
        PropsControl(BOOK,self.db_control,'win').commit_entry(self.propswin)

    def export_propshitsallowed(self):
        self.propshitsallowed = PropsHitsAllowed().to_dict(self.categories)
        PropsControl(BOOK,self.db_control,'hitsallowed').commit_entry(self.propshitsallowed)

    def export_propswalks(self):
        self.propswalks = PropsWalks().to_dict(self.categories)
        PropsControl(BOOK,self.db_control,'walks').commit_entry(self.propswalks)

    def export_propsearnedruns(self):
        self.propsearnedruns = PropsEarnedRuns().to_dict(self.categories)
        PropsControl(BOOK,self.db_control,'earnedruns').commit_entry(self.propsearnedruns)
##
##    def export_plateappearanceexact(self):
##        self.plateappearanceexact = PAExact().to_dict(self.categories)
##        PAExact(BOOK,self.db_control).commit_entry(self.plateappearanceexact)
        
        
class Parser:
    CATEGORY: int
    SUBCATEGORY: int
    DISPLAY_NAME: str

    def to_dict(self, markets):
        return [market for market in self._get_records(markets)]

    @classmethod
    def _get_records(cls, data):
        raise NotImplementedError()


class Gamelist(Parser):
    """
    Class Gamelist is a dedicated parser for general game information
    such as event id and start time. 
    
    """
    
    CATEGORY = None
    SUBCATEGORY = None
    DISPLAY_NAME = 'Gamelist'

    @classmethod
    def _get_records(cls, events)
        for event in events:
            if "@" in event['name']:
                yield from cls._get_data(event)
    
    @classmethod
    def _get_data(cls, event)
        event_id = event['eventId']
        game_datetime = unix(event['startDate'])
        away_abbr = TEAMS.get(event["teamName1"])
        home_abbr = TEAMS.get(event["teamName2"])
        game_ref = (f'{away_abbr}@{home_abbr}')
        return ((event_id, game_ref, game_datetime, BOOK)),

    
class Category(Parser):
    """
    Class Category takes the larger dictionary and parses it based on the
    statistics specific category and subcategory. 
    
    """
    @classmethod
    def _get_records(cls, data)
        for item in data:
            if item['offerCategoryId'] == cls.CATEGORY:
                yield from cls._get_subcatdes(item['offerSubcategoryDescriptors'])

    @classmethod
    def _get_subcatdes(cls, data)
        for item in data:
            if item['subcategoryId'] == cls.SUBCATEGORY:
                yield from cls._get_subcat(item['offerSubcategory'].get('offers'))

    @classmethod
    def _get_subcat(cls, data)
        for item in data:
            for x in item:
                if cls.DISPLAY_NAME in x.get('label',()) and x.get('isOpen',()):
                    yield from cls._get_data(x)


class Runline(Category):
    """
    Class Runline is a dedicated parser for runline statistics. 
    """
    
    CATEGORY = 493
    SUBCATEGORY = 4519
    DISPLAY_NAME = 'Run Line'
            
    @classmethod
    def _get_data(cls, market):
        event_id = market['eventId']
        away_team, home_team = market['outcomes'][:2]
        away_abbr = TEAMS.get(away_team['label'])
        home_abbr = TEAMS.get(home_team['label'])

        away_odds = away_team['oddsDecimal']
        home_odds = home_team['oddsDecimal']
        away_line = away_team['line']
        home_line = home_team['line']
        return (
            (away_abbr, event_id, away_odds, away_line, BOOK),
            (home_abbr, event_id, home_odds, home_line, BOOK),
        )

class Moneyline(Category):
    """
    Class Moneyline is a dedicated parser for moneyline statistics. 
    """
    
    CATEGORY = 493
    SUBCATEGORY = 4519
    DISPLAY_NAME = 'Moneyline'
                
    @classmethod
    def _get_data(cls, market):
        event_id = market['eventId']
        away_team, home_team = market['outcomes'][:2]
        away_abbr = TEAMS.get(away_team['label'])
        home_abbr = TEAMS.get(home_team['label'])

        away_odds = away_team['oddsDecimal']
        home_odds = home_team['oddsDecimal']
        return (
            (away_abbr, event_id, away_odds, BOOK),
            (home_abbr, event_id, home_odds, BOOK),
        )

class Total(Category):
    """
    Class Total is a dedicated parser for total statistics. 
    """
    
    CATEGORY = 493
    SUBCATEGORY = 4519
    DISPLAY_NAME = 'Total'
                
    @classmethod
    def _get_data(cls, market):
        event_id = market['eventId']
        
        away_team, home_team = market['outcomes'][:2]
        line = away_team['line']
        over = away_team['oddsDecimal']
        under = home_team['oddsDecimal']
        return (
            (event_id, line, over, under, BOOK),
        )

class Props(Category):
    """
    Class Props is a dedicated parser for all props statistics.
    """
    
    @classmethod
    def _get_data(cls, market):
        event_id = market['eventId']

        cont1, cont2 = market['outcomes'][:2]
        player = cont1.get('participant','')
        line = cont1.get('line',0)

        over = cont1.get('oddsDecimal','')
        under = cont2.get('oddsDecimal','')
        return (
            (player, event_id, line, over, under, BOOK),
        )

class PropsHomeruns(Props):
    CATEGORY = 743
    SUBCATEGORY = 6606
    DISPLAY_NAME = 'Home Runs'


class PropsHits(Props):
    CATEGORY = 743
    SUBCATEGORY = 6719
    DISPLAY_NAME = 'Hits'
    
class PropsTotalBases(Props):
    CATEGORY = 743
    SUBCATEGORY = 6606
    DISPLAY_NAME = 'Total Bases'

class PropsRBIs(Props):
    CATEGORY = 743
    SUBCATEGORY = 8025
    DISPLAY_NAME = 'RBIs'

class PropsRunsScored(Props):
    CATEGORY = 743
    SUBCATEGORY = 7979
    DISPLAY_NAME = 'Runs'

class PropsStolenBases(Props):
    CATEGORY = 743
    SUBCATEGORY = 9872
    DISPLAY_NAME = 'Stolen Bases'

class PropsSingles(Props):
    CATEGORY = 743
    SUBCATEGORY = 11031
    DISPLAY_NAME = 'Singles'

class PropsDoubles(Props):
    CATEGORY = 743
    SUBCATEGORY = 11032
    DISPLAY_NAME = 'Doubles'

class PropsTriples(Props):
    CATEGORY = 743
    SUBCATEGORY = 11033
    DISPLAY_NAME = 'Triples'

class PropsStrikeouts(Props):
    CATEGORY = 1031
    SUBCATEGORY = 9885
    DISPLAY_NAME = 'Strikeouts'

class PropsOutsRecorded(Props):
    CATEGORY = 1031
    SUBCATEGORY = 9883
    DISPLAY_NAME = 'Outs'

class PropsWin(Props):
    CATEGORY = 1031
    SUBCATEGORY = 9884
    DISPLAY_NAME = 'Win?'

class PropsHitsAllowed(Props):
    CATEGORY = 1031
    SUBCATEGORY = 9886
    DISPLAY_NAME = 'Hits Allowed'

class PropsWalks(Props):
    CATEGORY = 1031
    SUBCATEGORY = 11035
    DISPLAY_NAME = 'Walks'

class PropsEarnedRuns(Props):
    CATEGORY = 1031
    SUBCATEGORY = 11064
    DISPLAY_NAME = 'Earned Runs'

db_control.py

class DBControl:
    def __init__(self):
        logging.info('[DBControl]: CONNECTING TO DATABASE')
        self.engine = create_engine('sqlite:///games.db', echo=False)
        self.inspector = inspect(self.engine)
        self.db_connection = self.engine.connect()
        self.create_session = sessionmaker(bind=self.engine)
        self.metadata = MetaData(bind=self.engine)


class GamelistMLBControl:
    def __init__(self,book,db_control):
        self.table_name = f'{book}_gamelist'
        self.db_control = db_control
        
        self.table = self.check_table()
            
    def commit_entry(self,site_data):
        write_session = scoped_session(self.db_control.create_session)
        insert_stmt = insert(self.table).values(site_data)
        write_session.execute(insert_stmt)
        write_session.commit()
        write_session.remove()

    def check_table(self):
        metadata = self.db_control.metadata
        table_name = self.table_name
        if table_name not in self.db_control.inspector.get_table_names():
            table_name = Table(
                str(table_name),
                metadata,
                Column("event_id", Integer, primary_key=True),
                Column("game_ref", String),
                Column("game_datetime", Integer),
                Column("book", String),
            )
    
            metadata.create_all(self.db_control.engine)
            logging.info(f'[DBControl]: {table_name} created')
        else:
            metadata.reflect(self.db_control.engine)
            logging.info(f'[DBControl]: {table_name} exists')

        return Table(table_name, metadata, autoload=True)

    
class GamelistControl:
    def __init__(self,book,db_control):
        self.table_name = f'{book}_gamelist'
        self.db_control = db_control
        self.table = self.check_table()
            
    def commit_entry(self,site_data):
        write_session = scoped_session(self.db_control.create_session)
        insert_stmt = insert(self.table).values(site_data)
        write_session.execute(insert_stmt)
        write_session.commit()
        write_session.remove()

    def check_table(self):
        metadata = self.db_control.metadata
        table_name = self.table_name
        if table_name not in self.db_control.inspector.get_table_names():
            table_name = Table(
                str(table_name),
                metadata,
                Column("event_id", Integer, primary_key=True),
                Column("game_ref", String, primary_key=True),
                Column("game_datetime", Integer, primary_key=True),
                Column("book", String),
                ForeignKeyConstraint(['game_ref', 'game_datetime'],
                                     ['mlb_gamelist.game_ref', 'mlb_gamelist.game_datetime'])
            )
            
            metadata.create_all(self.db_control.engine)
            logging.info(f'[DBControl]: {table_name} created')
        else:
            metadata.reflect(self.db_control.engine)
            logging.info(f'[DBControl]: {table_name} exists')

        return Table(table_name, metadata, autoload=True)

    
class RunlineControl:
    def __init__(self,book,db_control):
        self.table_name = f'{book}_runline'
        self.db_control = db_control

    def commit_entry(self,site_data):
        write_session = scoped_session(self.db_control.create_session)
        insert_stmt = insert(self.check_table()).values(site_data)
        write_session.execute(insert_stmt)
        write_session.commit()
        write_session.remove()

    def check_table(self):
        metadata = MetaData(bind=self.db_control.engine)
        if self.table_name not in self.db_control.inspector.get_table_names():
            table_name = Table(
                str(self.table_name),
                metadata,
                Column("team", String, primary_key=True),
                Column("event_id", Integer, primary_key=True),
                Column("odds", Integer),
                Column("line", Float),
                Column("book", String, primary_key=True),
                PrimaryKeyConstraint("event_id", "team", "book"),
            )
    
            metadata.create_all(self.db_control.db_connection)
            logging.info(f'[DBControl]: {table_name} created')
        else:
            metadata.reflect(self.db_control.engine)
            logging.info(f'[DBControl]: {table_name} exists')

        return Table(table_name, metadata, autoload=True)


class MoneylineControl:
    def __init__(self,book,db_control):
        self.table_name = f'{book}_moneyline'
        self.db_control = db_control

    def commit_entry(self,site_data):
        write_session = scoped_session(self.db_control.create_session)
        insert_stmt = insert(self.check_table()).values(site_data)
        write_session.execute(insert_stmt)
        write_session.commit()
        write_session.remove()

    def check_table(self):
        metadata = MetaData(bind=self.db_control.engine)
        if self.table_name not in self.db_control.inspector.get_table_names():
            table_name = Table(
                str(self.table_name),
                metadata,
                Column("team", String, primary_key=True),
                Column("event_id", Integer, primary_key=True),
                Column("odds", Integer),
                Column("book", String, primary_key=True),
                PrimaryKeyConstraint("event_id", "team", "book"),
            )
    
            metadata.create_all(self.db_control.db_connection)
            logging.info(f'[DBControl]: {table_name} created')
        else:
            metadata.reflect(self.db_control.engine)
            logging.info(f'[DBControl]: {table_name} exists')

        return Table(table_name, metadata, autoload=True)


class TotalControl:
    def __init__(self,book,db_control):
        self.table_name = f'{book}_total'
        self.db_control = db_control

    def commit_entry(self,site_data):
        write_session = scoped_session(self.db_control.create_session)
        insert_stmt = insert(self.check_table()).values(site_data)
        write_session.execute(insert_stmt)
        write_session.commit()
        write_session.remove()

    def check_table(self):
        metadata = MetaData(bind=self.db_control.engine)
        if self.table_name not in self.db_control.inspector.get_table_names():
            table_name = Table(
                str(self.table_name),
                metadata,
                Column("event_id", Integer, primary_key=True),
                Column("line", Float),
                Column("over", Integer),
                Column("under", Integer),
                Column("book", String, primary_key=True),
                PrimaryKeyConstraint("event_id", "book"),
            )
    
            metadata.create_all(self.db_control.db_connection)
            logging.info(f'[DBControl]: {table_name} created')
        else:
            metadata.reflect(self.db_control.engine)
            logging.info(f'[DBControl]: {table_name} exists')

        return Table(table_name, metadata, autoload=True)


class PropsControl:
    def __init__(self,book,db_control,prop):
        self.table_name = f'{book}_props{prop}'
        self.db_control = db_control

    def commit_entry(self,data):
        write_session = scoped_session(self.db_control.create_session)
        insert_stmt = insert(self.check_table()).values(data)
        write_session.execute(insert_stmt)
        write_session.commit()
        write_session.remove()

    def check_table(self):
        metadata = MetaData(bind=self.db_control.engine)
        if self.table_name not in self.db_control.inspector.get_table_names():
            table_name = Table(
                str(self.table_name),
                metadata,
                Column("player", String, primary_key=True),
                Column("event_id", Integer, primary_key=True),
                Column("line", Float),
                Column("over", Integer),
                Column("under", Integer),
                Column("book", String, primary_key=True),
                PrimaryKeyConstraint("event_id", "player", "book"),
            )
    
            metadata.create_all(self.db_control.db_connection)
            logging.info(f'[DBControl]: {table_name} created')
        else:
            metadata.reflect(self.db_control.engine)
            logging.info(f'[DBControl]: {table_name} exists')

        return Table(table_name, metadata, autoload=True)   

db_query.py

from sqlalchemy import (Table, MetaData)


class DBQuery:
    def __init__(self,books,db_control):
        self.books = books
        self.db_control = db_control
        self.session = self.db_control.create_session

       
    def gamelist(self,caesars_ids):
        meta = MetaData(self.db_control.engine)
        t1 = Table('mlb_gamelist', meta, autoload=True)
        t2 = Table('caesars_gamelist', meta, autoload=True)
        t3 = Table('fanduel_gamelist', meta, autoload=True)
        t4 = Table('draftkings_gamelist', meta, autoload=True)
        t5 = Table('pointsbet_gamelist', meta, autoload=True)
        t6 = Table('mgm_gamelist', meta, autoload=True)

        sess = self.session()
        data = sess.query(
            t1.c.event_id,
            t2.c.event_id,
            t3.c.event_id,
            t4.c.event_id,
            t5.c.event_id,
            t6.c.event_id
            ).join(
                t2
                ).join(
                    t3
                    ).join(
                        t4
                        ).join(
                            t5
                            ).join(
                                t6
                                )

        lst = []
        for x in data:
            dic = {'caesars': caesars_ids.get(x[1]),
                   'fanduel': x[2],
                   'draftkings': x[3],
                   'pointsbet': x[4],
                   'mgm': x[5]
                   }
            lst.append(dic)

        return lst

app.py

import logging
from time import sleep
from pprint import pprint

from config import configs
from extract.download import get_games, get_game
from db.db_control import DBControl
from db.db_query import DBQuery
from pymlb.parse_games import GamesDict as MLBGamesDict
from pycaesars.parse_games import GamesDict as CaesarsGamesDict
from pycaesars.parse_game import GameDict as CaesarsGameDict
from pyfanduel.parse_games import GamesDict as FanduelGamesDict
from pyfanduel.parse_game import GameDict as FanduelGameDict
from pydraftkings.parse_games import GamesDict as DraftkingsGamesDict
from pypointsbet.parse_games import GamesDict as PointsbetGamesDict
from pypointsbet.parse_game import GameDict as PointsbetGameDict
from pymgm.parse_games import GamesDict as MGMGamesDict
from pymgm.parse_game import GameDict as MGMGameDict
from utils import delete_file


if __name__ == "__main__":

    logging.basicConfig(level=logging.DEBUG)

    delete_file('games.db')

    booklist = [k for k,v in configs['books'].items() if v]
    logging.warning(f'[APP]: Retrieving GAMES data for {booklist}')

    games = get_games()

    db_control = DBControl()

    if configs['books']['mlb']:
        MLBGamesDict(games,db_control)
    if configs['books']['caesars']:
        caesars = CaesarsGamesDict(games,db_control)
        caesars_ids = caesars.id_dict
    if configs['books']['fanduel']:
        FanduelGamesDict(games,db_control)
    if configs['books']['draftkings']:
        DraftkingsGamesDict(games,db_control)
    if configs['books']['pointsbet']:
        PointsbetGamesDict(games,db_control)
    if configs['books']['mgm']:
        MGMGamesDict(games,db_control)

    logging.info(f'[APP]: Retrieving GAME data')
    game = DBQuery(booklist,db_control).gamelist(caesars_ids)

    data = []
    for event_ids in game:
        games = get_game(event_ids)
        data.append(games)
        sleep(1)

    if configs['books']['caesars']:
        CaesarsGameDict(data,db_control)
    if configs['books']['fanduel']:
        FanduelGameDict(data,db_control)
    if configs['books']['pointsbet']:
        PointsbetGameDict(data,db_control)
    if configs['books']['mgm']:
        MGMGameDict(data,db_control)

parse_games.py REVISED

class Parser:
    DISPLAY_NAME: str

    def __init__(self, dic):
        self._dic = dic[BOOK]['eventGroup']
        
        self.events = self._dic.get('events',())
        self.categories = self._dic.get('offerCategories',())


class Gamelist:
    DISPLAY_NAME = 'Gamelist'
    
    def __init__(self, events):
        self.events = events

    def to_dict(self):
        return [record for record in self.get_records()]

    def get_records(self):
        for event in self.events:
            if "@" in event['name']:
                yield from self.get_data(event)

    def get_data(self, event):
        book_id = endpoints.gamelist_book_id(event)
        away_abbr = endpoints.gamelist_away_abbr(event)
        home_abbr = endpoints.gamelist_home_abbr(event)
        game_ref = f'{away_abbr}@{home_abbr}'
        game_datetime = endpoints.gamelist_datetime(event)

        data = [book_id, game_ref, game_datetime]
        columns = ['book_id', 'game_ref', 'game_datetime']       
        return [dict(zip(columns,data))]


class Category:
    def __init__(self, dic, category, subcategory, display_name):
        self.dic = dic
        self.category = category
        self.subcategory = subcategory
        self.display_name = display_name

        self.data = tuple(self.get_category())

    def get_category(self):
        for item in self.dic:
            if item['offerCategoryId'] == self.category:
                yield from self.get_subcategory_description(item['offerSubcategoryDescriptors'])

    def get_subcategory_description(self, data):
        for item in data:
            if item['subcategoryId'] == self.subcategory:
                yield from self.get_subcategory(item['offerSubcategory'].get('offers'))

    def get_subcategory(self, data):
        for item in data:
            for x in item:
                if self.display_name in x.get('label',()) and x.get('isOpen',()):
                    yield x

                    
class Runline:
    CATEGORY = 493
    SUBCATEGORY = 4519
    DISPLAY_NAME = 'Run Line'

    def __init__(self,data):
        self.data = Category(data, self.CATEGORY, self.SUBCATEGORY, self.DISPLAY_NAME).data

    def to_dict(self):
        return [record for record in self.get_records()]
    
    def get_records(self):
        for market in self.data:
            yield from self.get_data(market)

    def get_data(self, market):
        book_id = endpoints.runline_book_id(market)
        away_abbr = endpoints.runline_away_abbr(market)
        home_abbr = endpoints.runline_home_abbr(market)
        away_odds = endpoints.runline_away_odds(market)
        home_odds = endpoints.runline_home_odds(market)
        home_line = endpoints.runline_home_line(market)

        away = [book_id, away_abbr, away_odds, -home_line, BOOK]
        home = [book_id, home_abbr, home_odds, home_line, BOOK]
        columns = ['book_id', 'team', 'odds', 'line', 'book']
        return [dict(zip(columns,away)), dict(zip(columns,home))]


class Moneyline:
    CATEGORY = 493
    SUBCATEGORY = 4519
    DISPLAY_NAME = 'Moneyline'

    def __init__(self,data):
        self.data = Category(data, self.CATEGORY, self.SUBCATEGORY, self.DISPLAY_NAME).data

    def to_dict(self):
        return [record for record in self.get_records()]
    
    def get_records(self):
        for market in self.data:
            yield from self.get_data(market)

    def get_data(self, market):
        book_id = endpoints.moneyline_book_id(market)
        away_abbr = endpoints.moneyline_away_abbr(market)
        home_abbr = endpoints.moneyline_home_abbr(market)
        away_odds = endpoints.moneyline_away_odds(market)
        home_odds = endpoints.moneyline_home_odds(market)

        away = [book_id, away_abbr, away_odds, BOOK]
        home = [book_id, home_abbr, home_odds, BOOK]
        columns = ['book_id', 'team', 'odds', 'book']
        return [dict(zip(columns,away)), dict(zip(columns,home))]


class Total:
    CATEGORY = 493
    SUBCATEGORY = 4519
    DISPLAY_NAME = 'Total'

    def __init__(self,data):
        self.data = Category(data, self.CATEGORY, self.SUBCATEGORY, self.DISPLAY_NAME).data

    def to_dict(self):
        return [record for record in self.get_records()]
    
    def get_records(self):
        for market in self.data:
            yield from self.get_data(market)

    def get_data(self, market):
        book_id = endpoints.total_book_id(market)
        line = endpoints.total_line(market)
        over = endpoints.total_over(market)
        under = endpoints.total_under(market)

        data = [book_id, line, over, under, BOOK]
        columns = ['book_id', 'line', 'over', 'under', 'book']
        return [dict(zip(columns,data))]

```
\$\endgroup\$
6
  • 1
    \$\begingroup\$ I believe you've asked this question a couple times before. Thanks for adding more code this time, but it's still unclear what all of this is supposed to do. The title should reflect the purpose of the app, not the programming practice you want reviewed. Reviews are going to be open-ended, so you should be prepared for feedback on all aspects of the application. \$\endgroup\$
    – ggorlen
    Aug 23, 2022 at 22:03
  • \$\begingroup\$ The purpose is to retrieve data from from multiple json links, parse it and insert to database tables in order to be evaluated. I could really use help on all aspects, so are you saying if I make another post with every script and use a proper title it will be acceptable? \$\endgroup\$
    – Nick
    Aug 24, 2022 at 1:53
  • \$\begingroup\$ No, you can edit this post. A one-sentence description still gives me very little to understand the app. What are "JSON links"? What is the input and result? What's being evaluated? To get good feedback on code, sufficient context is important. Imagine you were given a few hundred lines of code with that sentence. Would you be able to meaningfully review it if you had no prior context? How can I even run this? It's a bunch of classes but no client code that uses the classes, no main. \$\endgroup\$
    – ggorlen
    Aug 24, 2022 at 1:59
  • \$\begingroup\$ I completely understand. There is a reason why I've been vague but since I really need help I'll provide more context. Thanks for the explanation. \$\endgroup\$
    – Nick
    Aug 24, 2022 at 2:02
  • \$\begingroup\$ @ggorlen I provided further details and added more relevant portions of the project. Please let me know if there is anything else I need to do. \$\endgroup\$
    – Nick
    Aug 24, 2022 at 2:36

1 Answer 1

3
+50
\$\begingroup\$

Just... yikes, where to begin.

Add PEP484 typehints. If you don't know what they are or why they're beneficial, do some Googling; but in short - a signature like def to_dict(self, markets): tells us nothing about what it actually accepts and only a guess as to what it returns. Type hints don't fix Python's broken type system (they never will), but they are a great deal better than nothing.

Further to the above, a signature like def __init__(self, listofdics, db_control) accepts a listofdics (presumably : list[dict[???, ???]]). That tells us some - not enough - information about the type, and absolutely nothing about the contents. Is this games (a list of game dictionaries)?

GamesDict as a class is not well-named. It isn't a dict. Call it perhaps GamesRepo, if that's what it actually is.

Grammar - it's stat type -> its stat type; it's corresponding table -> its corresponding table.

The OOP strategy followed in GamesDict is inappropriate. Your export_* functions set members such as self.gamelist that should actually only be local variables. Also, the sequence of calls to these export functions should not be done in the constructor; it should be a separate function called from the outside.

Parser and its subclasses are also poor OOP: take _get_records_, _get_subcatdes and _get_subcat. These are all classmethods that accept the same parameter, data. It smells like data should be an instance member, and those should be parameter-less instance methods that refer to data through self.

There is significant code repetition between Runline._get_data and Moneyline._get_data. This suggests that the common code should be pulled out to a common ancestor of these classes.

A class like Total seems to have no actual instance data, and its _get_data returns an anonymous tuple. A more sensible pattern would be for _get_data to instantiate the class via cls(), with the anonymous tuple converted into a class instance with named parameters. By example,

@dataclass
class Total:
    """
    Class Total is a dedicated parser for total statistics. 
    """
    event_id: ???
    line: ???
    over: ???
    under: ???
    book: ???
    
    CATEGORY: ClassVar[int] = 493
    SUBCATEGORY: ClassVar[int] = 4519
    DISPLAY_NAME: ClassVar[str] = 'Total'
                
    @classmethod
    def _get_data(cls, market: dict[str, Any]) -> 'Total':
        event_id = market['eventId']
        
        away_team, home_team = market['outcomes'][:2]
        line = away_team['line']
        over = away_team['oddsDecimal']
        under = home_team['oddsDecimal']
        return cls(
            (event_id, line, over, under, BOOK),
        )

DBControl fails to close its connection and session. Convert it into a context manager whose __exit__ closes all resources that can be closed.

You fail to show where scoped_session is imported from. I assume that this is sqlalchemy.orm.scoping.scoped_session.

Your check_table pattern is fairly broken. SQLAlchemy is a complicated framework with a million and one ways for a beginner to shoot themselves in the foot. Your *Control classes should be converted into ORM classes that inherit from the declarative base. I strongly encourage you to do more reading and follow along with the official tutorial. When the dust settles, a query like this:

    meta = MetaData(self.db_control.engine)
    t1 = Table('mlb_gamelist', meta, autoload=True)
    t2 = Table('caesars_gamelist', meta, autoload=True)
    t3 = Table('fanduel_gamelist', meta, autoload=True)
    t4 = Table('draftkings_gamelist', meta, autoload=True)
    t5 = Table('pointsbet_gamelist', meta, autoload=True)
    t6 = Table('mgm_gamelist', meta, autoload=True)

    sess = self.session()
    data = sess.query(
        t1.c.event_id,
        t2.c.event_id,
        t3.c.event_id,
        t4.c.event_id,
        t5.c.event_id,
        t6.c.event_id
        ).join(
            t2
            ).join(
# ...

must not re-declare the referenced Tables, and instead reference your ORM classes.

Delete sleep(1).

\$\endgroup\$
4
  • \$\begingroup\$ Thanks for the response. This is my first attempt at writing something on a larger scale so I knew there would be plenty to critique. I have a few questions but my first would be pertaining to Parser being poor OOP. I understand the approach you suggest but am confused as to how I can pivot to instance methods while still being able to access the CATEGORY values in each subclass. Does that make sense? \$\endgroup\$
    – Nick
    Aug 28, 2022 at 23:00
  • \$\begingroup\$ Statics like CATEGORY are still accessible through self \$\endgroup\$
    – Reinderien
    Aug 29, 2022 at 15:24
  • \$\begingroup\$ So I ended up going to a method of parse_games without any inheritance and posted the revision. At this stage in the learning process the code makes more sense to me but obviously there is a lot more repetitive code. I had tried implementing your approach as far as _get_data instantiating the class but to be honest that left me even more confused. Could you elaborate on that or maybe provide a brief example? \$\endgroup\$
    – Nick
    Aug 30, 2022 at 3:35
  • \$\begingroup\$ Your example makes sense but I'm confused as to what I should do with the get_records function. Or maybe I better way if explaining is that I'm not sure where to implement a loop that returns the data for each individual game/record. \$\endgroup\$
    – Nick
    Aug 30, 2022 at 18:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.