Efficiently parsing JSON response data in python

Question

I'm using this API to get information on sports-betting odds. I am then processing it in Python to discard the irrelevant information.

Here's what the response looks like:

[
    {
        "id": "9eec062ad8ed334517ab3a4be8362739",
        "sport_key": "americanfootball_nfl",
        "sport_title": "NFL",
        "commence_time": "2021-10-01T00:20:00Z",
        "home_team": "Cincinnati Bengals",
        "away_team": "Jacksonville Jaguars",
        "bookmakers": [
            {
                "key": "unibet",
                "title": "Unibet",
                "last_update": "2021-09-29T16:41:30Z",
                "markets": [
                    {
                        "key": "spreads",
                        "outcomes": [
                            {
                                "name": "Cincinnati Bengals",
                                "price": -110,
                                "point": -7.5
                            },
                            {
                                "name": "Jacksonville Jaguars",
                                "price": -110,
                                "point": 7.5
                            }
                        ]
                    },
                    {
                        "key": "totals",
                        "outcomes": [
                            {
                                "name": "Over",
                                "price": -111,
                                "point": 46.0
                            },
                            {
                                "name": "Under",
                                "price": -109,
                                "point": 46.0
                            }
                        ]
                    }
                ]
            },
...

My code:

# filter http response
odds = []

for game in resp:
    for bookmaker in game["bookmakers"]:
        if bookmaker["key"] == "betmgm":
            for bet in bookmaker["markets"]:
                if bet["outcomes"][0]["name"] == "Over":
                    over_under_line = bet["outcomes"][0]["point"]
                elif bet["outcomes"][0]["name"] == game["home_team"]:
                    spread = bet["outcomes"][0]["point"]
    odds.append(
        {
            "home_team": game["home_team"],
            "away_team": game["away_team"],
            "spread": spread,
            "total": over_under_line,
        }
    )

I have a strong feeling that this could be made more pythonic. Any suggestions?

Answer 1

If this were for a job, I'd say you need to potentially translate the JSON soup into well-typed, well-defined models, but it's probably not. In that case, you should still break your logic up into functions with type hints.

Note that there's an opportunity for early-break since you probably only care about the first entries found for your over/under and spread values. The reusable way to represent this is a generator where only the first value is used, via next.

Try to factor out values like betmgm into parameters. It's also worth noting that your sample data are not a great representation of what your code is actually looking for, since you hard-code betmgm when only unibet is shown.

Your code is slightly fragile since it ignores bet['key'] when it should pay attention to that to filter for over/under and spread respectively.

Suggested

from typing import Dict, Any, Iterable, Tuple, Iterator

JsonDict = Dict[str, Any]


def get_over_under(markets: Iterable[JsonDict]) -> Iterator[float]:
    for bet in markets:
        if bet['key'] == 'totals':
            for outcome in bet['outcomes']:
                if outcome['name'] == 'Over':
                    yield outcome['point']


def get_spread(markets: Iterable[JsonDict], home_team: str) -> Iterator[float]:
    for bet in markets:
        if bet['key'] == 'spreads':
            for outcome in bet['outcomes']:
                if outcome['name'] == home_team:
                    yield outcome['point']


def get_bets(resp: Iterable[JsonDict], book_name: str) -> Iterator[Tuple[
    float,  # over/under
    float,  # spread
]]:
    for game in resp:
        for bookmaker in game['bookmakers']:
            if bookmaker['key'] == book_name:
                markets = bookmaker['markets']
                yield (
                    next(get_over_under(markets)),
                    next(get_spread(markets, home_team=game['home_team'])),
                )


def test() -> None:
    resp = [
        {
            'id': '9eec062ad8ed334517ab3a4be8362739',
            'sport_key': 'americanfootball_nfl',
            'sport_title': 'NFL',
            'commence_time': '2021-10-01T00:20:00Z',
            'home_team': 'Cincinnati Bengals',
            'away_team': 'Jacksonville Jaguars',
            'bookmakers': [
                {
                    'key': 'unibet',
                    'title': 'Unibet',
                    'last_update': '2021-09-29T16:41:30Z',
                    'markets': [
                        {
                            'key': 'spreads',
                            'outcomes': [
                                {
                                    'name': 'Cincinnati Bengals',
                                    'price': -110,
                                    'point': -7.5
                                },
                                {
                                    'name': 'Jacksonville Jaguars',
                                    'price': -110,
                                    'point': 7.5
                                },
                            ],
                        },
                        {
                            'key': 'totals',
                            'outcomes': [
                                {
                                    'name': 'Over',
                                    'price': -111,
                                    'point': 46.0
                                },
                                {
                                    'name': 'Under',
                                    'price': -109,
                                    'point': 46.0
                                },
                            ],
                        },
                    ],
                },
            ],
        },
    ]

    for over_under, spread in get_bets(
        resp,
        book_name='unibet'  # 'betmgm'
    ):
        print(f'Over/under: {over_under}  Spread: {spread}')


if __name__ == '__main__':
    test()

Answer 2

I don't think you win something by avoiding loops in that case...

Personally, I probably would just introduce a namedtuple, extract the processing of each game record to a function and use get(key) to access a value within a dict to avoid KeyErrors.

If you want it to be more pythonic, maybe you could consider using list comprehensions. I don't think it's more efficient but more readable. See Example 2.

(Example 1 does not follow EAFP (easier to ask for forgiveness than permission) and therefore not pythonic in that regard. However, when working with data from an API, I prefer to ask for permission instead of forgiveness.)

Note: I assume that [0] in bet["outcomes"][0]["name"] (in your question's code snippet) is purposely and therefore only the first outcome has to be considered.

from collections import namedtuple

Bet = namedtuple("Bet", ["home_team", "away_team", "spread", "total"])

def get_odds(game):
    
    home_team = game.get("home_team")
    away_team = game.get("away_team")
    spread = None
    over_under_line = None
    
    for bookmaker in game.get("bookmakers"):
        if bookmaker.get("key") == "betmgm":
            for bet in bookmaker.get("markets"):
                
                outcome = bet.get("outcomes")
                if not outcome:
                    continue  # or raise an exception
                             
                name = outcome[0].get("name")
                point = outcome[0].get("point")
                
                if name == "Over":
                    over_under_line = point
                elif name == home_team:
                    spread = point
    
    return Bet(home_team, away_team, spread, over_under_line)

odds = (get_odds(game) for game in data)  # generator comprehension

for record in odds:
    print(record)

---

Using List Comprehensions

In case you rather want to use list comprehensions:

from collections import namedtuple

Bet = namedtuple("Bet", ["home_team", "away_team", "spread", "total"])

def get_odds(game):
    
    home_team = game.get("home_team")
    away_team = game.get("away_team")
    spread = None
    over_under_line = None
    
    bets = [      
        bookmaker.get("markets") for bookmaker in game.get("bookmakers")
        if bookmaker.get("key") == "betmgm"
    ]
    outcomes = [  # possible KeyError
        bet.get("outcomes")[0] for bet_list in bets for bet in bet_list
    ]
    
    for outcome in outcomes:

        name = outcome.get("name")
        point = outcome.get("point")

        if name == "Over":
            over_under_line = point
        elif name == home_team:
            spread = point
    
    return Bet(home_team, away_team, spread, over_under_line)

odds = (get_odds(game) for game in data)  # generator comprehension

for record in odds:
    print(record)

---

Using JMESPath

I wouldn't call it more pythonic though. It's just another way to extract the data from the JSON string. Note: There are also alternatives to JMESPath: jsonpath-python and others.

from collections import namedtuple
import jmespath

Bet = namedtuple("Bet", ["home_team", "away_team", "spread", "total"])

def get_odds(game):
    
    home_team = game.get("home_team")
    away_team = game.get("away_team")
    
    search_outcomes = "bookmakers[?key=='betmgm'].markets[*].outcomes[0] | [0]"
    outcomes = {
        "outcomes": jmespath.search(search_outcomes, game)
    }
    
    over_under_line = jmespath.search("outcomes[?name=='Over'].point | [0]", outcomes)
    spread = jmespath.search(f"outcomes[?name=='{home_team}'].point | [0]", outcomes)
    
    return Bet(home_team, away_team, spread, over_under_line)

odds = (get_odds(game) for game in data)  # generator comprehension

for record in odds:
    print(record)

---

Note: your code example and all my example have magic numbers which is an anti-pattern.