1
\$\begingroup\$

I have been trying to figure out what is the best performance wise and also best practice to get data from a dict using Python. I currently have a data that looks like this:

data = {
    "cart": {
        "id": 123456,
        "currency": "GBP",
        "name": None,
        "rows": [
            {
                "id": 7079540,
                "unitPrice": 1089,
                "unitVat": 217.8,
                "quantity": 1,
                "pass": None,
                "extraData": {
                    "isLoggedIn": True,
                    "source": "product"
                },
                "product": {
                    "id": 279533,
                    "thumbnail": "\/images\/product\/279533?trim&h=80",
                    "name": "Seagate Game Drive f\u00f6r PS4 4TB",
                    "subTitle": "Extern H\u00e5rddisk 2,5\" \/ USB 3.0 \/ 4 TB",
                    "shippingClass": {
                        "id": 1,
                        "order": 1,
                        "prices": [
                            {
                                "price": 0,
                                "shippingMethodId": 1,
                                "isFixedPrice": False,
                                "maximumPackageSizeId": 2
                            },
                            {
                                "price": 0,
                                "shippingMethodId": 2,
                                "isFixedPrice": False,
                                "maximumPackageSizeId": 7
                            },
                            {
                                "price": 0,
                                "shippingMethodId": 3,
                                "isFixedPrice": False,
                                "maximumPackageSizeId": 7
                            },
                            {
                                "price": 0,
                                "shippingMethodId": 4,
                                "isFixedPrice": False,
                                "maximumPackageSizeId": 7
                            },
                            {
                                "price": 0,
                                "shippingMethodId": 19,
                                "isFixedPrice": True,
                                "maximumPackageSizeId": 2
                            },
                            {
                                "price": 0,
                                "shippingMethodId": 20,
                                "isFixedPrice": False,
                                "maximumPackageSizeId": 7
                            },
                            {
                                "price": 0,
                                "shippingMethodId": 21,
                                "isFixedPrice": False,
                                "maximumPackageSizeId": 7
                            },
                            {
                                "price": 0,
                                "shippingMethodId": 22,
                                "isFixedPrice": True,
                                "maximumPackageSizeId": None
                            },
                            {
                                "price": 0,
                                "shippingMethodId": 25,
                                "isFixedPrice": True,
                                "maximumPackageSizeId": 2
                            },
                            {
                                "price": 0,
                                "shippingMethodId": 26,
                                "isFixedPrice": True,
                                "maximumPackageSizeId": 7
                            },
                            {
                                "price": 0,
                                "shippingMethodId": 27,
                                "isFixedPrice": True,
                                "maximumPackageSizeId": 2
                            },
                            {
                                "price": 0,
                                "shippingMethodId": 28,
                                "isFixedPrice": True,
                                "maximumPackageSizeId": 2
                            }
                        ]
                    },
                    "packageSizeId": 1,
                    "isShippable": True,
                    "release": {
                        "timestamp": 1512450000,
                        "format": "Y-m-d"
                    },
                    "regularPrice": {
                        "price": "1199.00",
                        "currency": "SEK",
                        "vat": 239.8,
                        "type": None,
                        "endAt": "2022-03-12",
                        "maxQtyPerCustomer": None
                    },
                    "isRevenue": True,
                    "stock": {
                        "web": 51,
                        "supplier": None,
                        "displayCap": "50",
                        "1": 51,
                        "2": 0,
                        "5": 10,
                        "8": 4,
                        "9": 4,
                        "10": 10,
                        "11": 7,
                        "14": 2,
                        "15": 23,
                        "16": 0,
                        "19": 0,
                        "20": 0,
                        "21": 0,
                        "22": 4,
                        "23": 0,
                        "26": 4,
                        "27": 18,
                        "28": 1
                    },
                    "meta": [

                    ],
                    "statusCodes": [

                    ],
                    "packageVolume": 879750,
                    "packageWeight": 340,
                    "minimumRankLevel": 0,
                    "packageMeasurements": [
                        170,
                        115,
                        45
                    ]
                },
                "insurance": {
                    "id": 339702,
                    "name": "testing Care Pack (12 m\u00e5nader)",
                    "price": 289,
                    "provider": 1,
                    "length": 12
                }
            }
        ]
    },
    "corrections": [
        {
            "type": "PRICE_CHANGED",
            "rowId": None,
            "price": {
                "was": 0,
                "shouldBe": "1089.00"
            }
        }
    ],
    "errors": [

    ]
}

and I am currently trying to get specific data that I store into a dict and return the dict data. This is what I have done:

import pendulum

TESTING_STORES = {
    'web': 'Webblager',
    'supplier': 'Hos leverantör',
    '1': 'Hello World',
}

data = {
    "cart": {
        "id": 2656218,
        "currency": "SEK",
        "name": None,
        "rows": [
            {
                "id": 7079540,
                "unitPrice": 1089,
                "unitVat": 217.8,
                "quantity": 1,
                "pass": None,
                "extraData": {
                    "isLoggedIn": True,
                    "source": "product"
                },
                "product": {
                    "id": 279533,
                    "thumbnail": "\/images\/product\/279533?trim&h=80",
                    "name": "Seagate Game Drive f\u00f6r PS4 4TB",
                    "subTitle": "Extern H\u00e5rddisk 2,5\" \/ USB 3.0 \/ 4 TB",
                    "shippingClass": {
                        "id": 1,
                        "order": 1,
                        "prices": [
                            {
                                "price": 0,
                                "shippingMethodId": 1,
                                "isFixedPrice": False,
                                "maximumPackageSizeId": 2
                            },
                            {
                                "price": 0,
                                "shippingMethodId": 2,
                                "isFixedPrice": False,
                                "maximumPackageSizeId": 7
                            },
                            {
                                "price": 0,
                                "shippingMethodId": 3,
                                "isFixedPrice": False,
                                "maximumPackageSizeId": 7
                            },
                            {
                                "price": 0,
                                "shippingMethodId": 4,
                                "isFixedPrice": False,
                                "maximumPackageSizeId": 7
                            },
                            {
                                "price": 0,
                                "shippingMethodId": 19,
                                "isFixedPrice": True,
                                "maximumPackageSizeId": 2
                            },
                            {
                                "price": 0,
                                "shippingMethodId": 20,
                                "isFixedPrice": False,
                                "maximumPackageSizeId": 7
                            },
                            {
                                "price": 0,
                                "shippingMethodId": 21,
                                "isFixedPrice": False,
                                "maximumPackageSizeId": 7
                            },
                            {
                                "price": 0,
                                "shippingMethodId": 22,
                                "isFixedPrice": True,
                                "maximumPackageSizeId": None
                            },
                            {
                                "price": 0,
                                "shippingMethodId": 25,
                                "isFixedPrice": True,
                                "maximumPackageSizeId": 2
                            },
                            {
                                "price": 0,
                                "shippingMethodId": 26,
                                "isFixedPrice": True,
                                "maximumPackageSizeId": 7
                            },
                            {
                                "price": 0,
                                "shippingMethodId": 27,
                                "isFixedPrice": True,
                                "maximumPackageSizeId": 2
                            },
                            {
                                "price": 0,
                                "shippingMethodId": 28,
                                "isFixedPrice": True,
                                "maximumPackageSizeId": 2
                            }
                        ]
                    },
                    "packageSizeId": 1,
                    "isShippable": True,
                    "release": {
                        "timestamp": 1512450000,
                        "format": "Y-m-d"
                    },
                    "regularPrice": {
                        "price": "1199.00",
                        "currency": "SEK",
                        "vat": 239.8,
                        "type": None,
                        "endAt": "2022-03-12",
                        "maxQtyPerCustomer": None
                    },
                    "isRevenue": True,
                    "stock": {
                        "web": 51,
                        "supplier": None,
                        "displayCap": "50",
                        "1": 51,
                        "2": 0,
                        "5": 10,
                        "8": 4,
                        "9": 4,
                        "10": 10,
                        "11": 7,
                        "14": 2,
                        "15": 23,
                        "16": 0,
                        "19": 0,
                        "20": 0,
                        "21": 0,
                        "22": 4,
                        "23": 0,
                        "26": 4,
                        "27": 18,
                        "28": 1
                    },
                    "meta": [

                    ],
                    "statusCodes": [

                    ],
                    "packageVolume": 879750,
                    "packageWeight": 340,
                    "minimumRankLevel": 0,
                    "packageMeasurements": [
                        170,
                        115,
                        45
                    ]
                },
                "insurance": {
                    "id": 339702,
                    "name": "testing Care Pack (12 m\u00e5nader)",
                    "price": 289,
                    "provider": 1,
                    "length": 12
                }
            }
        ]
    },
    "corrections": [
        {
            "type": "PRICE_CHANGED",
            "rowId": None,
            "price": {
                "was": 0,
                "shouldBe": "1089.00"
            }
        }
    ],
    "errors": [

    ]
}

# --------------------------------------------------------------- #
# GET JSON DATA
# --------------------------------------------------------------- #
pageData = {
    'title': 'Unknown',
    'image': 'Not found',
    'price': 'Not found',
    'sizes': {}
}


def get_data():
    if doc := data.get('cart', {}).get('rows')[0].get('product', {}):
        # ---------------------------- ALL ---------------------------- #
        pageData.update({
            'title': doc.get('name'),
            'image': f"https://cdn.testing.com/images/product/{doc.get('id')}?trim&w=231",
            'price': doc.get('regularPrice', {}).get('price', {}),
        })

        # ---------------------------- Release date ---------------------------- #

        if release_date := doc.get('release', {}).get('timestamp', {}):
            if not pendulum.from_timestamp(release_date, tz='Europe/Stockholm').is_past():
                pageData['release_date'] = release_date

        # ---------------------------- Sizes ---------------------------- #

        if sizes := doc.get('stock'):
            pageData['sizes'] = {TESTING_STORES[store]: stock for store, stock in sizes.items() if
                                 store in TESTING_STORES and stock}

            if status := sizes.get('orders', {}).get('CL', {}):
                if not status.get('status', 0) == -1:
                    pageData['sizes']['Potential InStock'] = status.get('ordered')

        # ---------------------------- Campaign ---------------------------- #

        if doc.get('regularPrice', {}).get('type', '') == 'campaign':
            pageData['sizes']['InStock'] = None

        # ---------------------------- minimumRankLevel ---------------------------- #

        if minimumRankLevel := doc.get('minimumRankLevel', {}):
            pageData['minimumRankLevel'] = minimumRankLevel

    return pageData

if __name__ == '__main__':
    data = get_data()
    print(data)

The script itself works but I do also feel like I have done it incorrectly by not applying the best practice and I do know that using .get() is not the best performance compare to if in but also I do not know e.g. what if its a nested data I want to use.. how to apply it etc etc... my goal is to make the script to have the best performance wise and trying to keep it as simple as possible as well.

I wonder, if there is anything I could improve in my code?

\$\endgroup\$

3 Answers 3

Reset to default
3
\$\begingroup\$

Your code relies on dictionary soup, and if my memory serves correctly it's been like this for a long time. Concern about optimization should be secondary to concern about correctness; a.k.a. current concern about optimization is premature when this code has no strong types and has few to no guarantees about the structure of your data. As Akash suggests, you should move toward classes.

data should not be global and should be made a local of a test routine.

pageData should be called page_data, and should be made a local of your get_data() routine.

Add PEP484 type hints.

The words "unknown" and "not found" are presentation strings that are prematurely baked into your data. Use None instead.

In expressions like this:

if doc := data.get('cart', {}).get('rows')[0].get('product', {}):

I don't find the walrus to be helpful and you should probably just separate the assignment from the test. Also, your final get should not pass a default of {} and should just accept the default of None.

Your update() can use the new kwargs-style instead of the old dictionary literal style.

if not status.get('status', 0) == -1 should just be if status.get('status') != -1.

if minimumRankLevel := doc.get('minimumRankLevel', {}): does not save the minimum rank level if it's 0, since 0 is falsy. Don't pass {}, and use is not None.

Don't leave the price as a string; cast it to a Decimal.

You call from_timestamp (good!) but then throw it away (ungood) and for some reason forget the time entirely if it's in the past (also ungood). Save the deserialized datetime object regardless of whether it's in the past on not. Leave the past check for a separate logic-processing function.

Suggested

This implements some (not all) of the above.

from decimal import Decimal
from pprint import pprint
from typing import Dict, Any

TESTING_STORES = {
    'web': 'Webblager',
    'supplier': 'Hos leverantör',
    '1': 'Hello World',
}


def get_data(data: Dict[str, Any]) -> Dict[str, Any]:
    page_data = {
        'title': None,
        'image': None,
        'price': None,
        'sizes': {},
    }

    doc = data.get('cart', {}).get('rows')[0].get('product')
    if doc:
        page_data.update(
            title=doc.get('name'),
            image=f"https://cdn.testing.com/images/product/{doc.get('id')}?trim&w=231",
        )

        price = doc.get('regularPrice', {}).get('price')
        if price is not None:
            page_data['price'] = Decimal(price)

        release_date = doc.get('release', {}).get('timestamp')
        if release_date:
            if not pendulum.from_timestamp(release_date, tz='Europe/Stockholm').is_past():
                page_data['release_date'] = release_date

        sizes = doc.get('stock')
        if sizes:
            page_data['sizes'] = {
                TESTING_STORES[store]: stock
                for store, stock in sizes.items()
                if store in TESTING_STORES and stock
            }

            status = sizes.get('orders', {}).get('CL')
            if status and status.get('status') != -1:
                page_data['sizes']['Potential InStock'] = status.get('ordered')

        if doc.get('regularPrice', {}).get('type') == 'campaign':
            page_data['sizes']['InStock'] = None

        minimum_rank_level = doc.get('minimumRankLevel')
        if minimum_rank_level is not None:
            page_data['minimumRankLevel'] = minimum_rank_level

    return page_data


def test() -> None:
    data = {
        "cart": {
            "id": 2656218,
            "currency": "SEK",
            "name": None,
            "rows": [
                {
                    "id": 7079540,
                    "unitPrice": 1089,
                    "unitVat": 217.8,
                    "quantity": 1,
                    "pass": None,
                    "extraData": {
                        "isLoggedIn": True,
                        "source": "product"
                    },
                    "product": {
                        "id": 279533,
                        "thumbnail": "\/images\/product\/279533?trim&h=80",
                        "name": "Seagate Game Drive f\u00f6r PS4 4TB",
                        "subTitle": "Extern H\u00e5rddisk 2,5\" \/ USB 3.0 \/ 4 TB",
                        "shippingClass": {
                            "id": 1,
                            "order": 1,
                            "prices": [
                                {
                                    "price": 0,
                                    "shippingMethodId": 1,
                                    "isFixedPrice": False,
                                    "maximumPackageSizeId": 2
                                },
                                {
                                    "price": 0,
                                    "shippingMethodId": 2,
                                    "isFixedPrice": False,
                                    "maximumPackageSizeId": 7
                                },
                                {
                                    "price": 0,
                                    "shippingMethodId": 3,
                                    "isFixedPrice": False,
                                    "maximumPackageSizeId": 7
                                },
                                {
                                    "price": 0,
                                    "shippingMethodId": 4,
                                    "isFixedPrice": False,
                                    "maximumPackageSizeId": 7
                                },
                                {
                                    "price": 0,
                                    "shippingMethodId": 19,
                                    "isFixedPrice": True,
                                    "maximumPackageSizeId": 2
                                },
                                {
                                    "price": 0,
                                    "shippingMethodId": 20,
                                    "isFixedPrice": False,
                                    "maximumPackageSizeId": 7
                                },
                                {
                                    "price": 0,
                                    "shippingMethodId": 21,
                                    "isFixedPrice": False,
                                    "maximumPackageSizeId": 7
                                },
                                {
                                    "price": 0,
                                    "shippingMethodId": 22,
                                    "isFixedPrice": True,
                                    "maximumPackageSizeId": None
                                },
                                {
                                    "price": 0,
                                    "shippingMethodId": 25,
                                    "isFixedPrice": True,
                                    "maximumPackageSizeId": 2
                                },
                                {
                                    "price": 0,
                                    "shippingMethodId": 26,
                                    "isFixedPrice": True,
                                    "maximumPackageSizeId": 7
                                },
                                {
                                    "price": 0,
                                    "shippingMethodId": 27,
                                    "isFixedPrice": True,
                                    "maximumPackageSizeId": 2
                                },
                                {
                                    "price": 0,
                                    "shippingMethodId": 28,
                                    "isFixedPrice": True,
                                    "maximumPackageSizeId": 2
                                }
                            ]
                        },
                        "packageSizeId": 1,
                        "isShippable": True,
                        "release": {
                            "timestamp": 1512450000,
                            "format": "Y-m-d"
                        },
                        "regularPrice": {
                            "price": "1199.00",
                            "currency": "SEK",
                            "vat": 239.8,
                            "type": None,
                            "endAt": "2022-03-12",
                            "maxQtyPerCustomer": None
                        },
                        "isRevenue": True,
                        "stock": {
                            "web": 51,
                            "supplier": None,
                            "displayCap": "50",
                            "1": 51,
                            "2": 0,
                            "5": 10,
                            "8": 4,
                            "9": 4,
                            "10": 10,
                            "11": 7,
                            "14": 2,
                            "15": 23,
                            "16": 0,
                            "19": 0,
                            "20": 0,
                            "21": 0,
                            "22": 4,
                            "23": 0,
                            "26": 4,
                            "27": 18,
                            "28": 1
                        },
                        "meta": [

                        ],
                        "statusCodes": [

                        ],
                        "packageVolume": 879750,
                        "packageWeight": 340,
                        "minimumRankLevel": 0,
                        "packageMeasurements": [
                            170,
                            115,
                            45
                        ]
                    },
                    "insurance": {
                        "id": 339702,
                        "name": "testing Care Pack (12 m\u00e5nader)",
                        "price": 289,
                        "provider": 1,
                        "length": 12
                    }
                }
            ]
        },
        "corrections": [
            {
                "type": "PRICE_CHANGED",
                "rowId": None,
                "price": {
                    "was": 0,
                    "shouldBe": "1089.00"
                }
            }
        ],
        "errors": [

        ]
    }

    page_data = get_data(data)
    pprint(page_data)


if __name__ == '__main__':
    test()
\$\endgroup\$
1
  • \$\begingroup\$ Hello! Thank you for the review and for the suggestions! Alot of stuff make sense I do agree with you as well. The only reason I used walrus is to skip one extra "call/check" as you do with e.g. sizes where you declare the variable and then check if sizes. where u can use walrus that does the same thing but with 1 line :) Also good tips weith the from_timestamp - I should have another function for that. - But yes. I agree with you :) \$\endgroup\$
    – PythonNewbie
    Commented Dec 12, 2021 at 18:08
1
\$\begingroup\$

You already have some good advice, so I'll focus narrowly on two topics.

If the code you're writing is annoying, maybe you need a function. Suppose that you are unavoidably in a situation where the data is a deeply-nested structure and you find yourself frequently having to write code like this line:

data.get('cart', {}).get('rows')[0].get('product', {})

What could you do? Write a small utility function:

def dive(d, *keys):
    for k in keys:
        try:
            d = d[k]
        except (KeyError, IndexError):
            return None
    return d

# Usage illustration.
dive(data, 'cart', 'rows', 0, 'product')

Performance is often overrated. Granted, making the dive() function call and iterating over keys will come with some "performance costs". However -- and I cannot stress this enough -- fretting over micro-performance issues is usually wasted effort. The vast majority of programs and scripts are not performance-constrained in any meaningful sense. And even within a performance-constrained program, the vast majority of the lines of code in that program will not be on the directly relevant path affecting performance. Yes, learn the core algorithms and data structures. Use that knowledge to develop good coding habits. But don't waste effort micro-analyzing performance tradeoffs. Beyond being wasteful, such analysis is not infrequently misguided because your intuitions about micro-performance issues can be wildly incorrect (even counterintuitively opposed to the good habits you develop by learning algorithms and data structures), especially in languages like Python where some of the execution occurs at Python-speed and some of it occurs at C-speed.

\$\endgroup\$
-1
\$\begingroup\$

A dict might make the code CPU bound with so much nesting.

An alternative approach is to model the data as named tuples classes. A named tuple becomes the most nested data. An example of a named tuple class is extra data might be turned into a class. Another example of a named tuple is a price. The named tuples can use the id as an artificial key.

A faster way than looping through a whole list while avoiding a CPU bound code is to use a 2d list to state if a price id belongs to a product id. A product is a row and a price id is a col. Similarly, a cart id and product id is also stated as a 2d array of bool which has a col of cart id and a row of product id. Notably, the cart row has a cardinality of one so a 1d list is good.

\$\endgroup\$
1
  • 2
    \$\begingroup\$ Hello! Is it possible if you could do an example or based on something that I am trying to do? I cannot really see the visual of how it could be better really \$\endgroup\$
    – PythonNewbie
    Commented Dec 12, 2021 at 14:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.