Cached shortened urls

Question

Im currently writing my own cached shortened urls where I at the start of application read the database that has all the stored url:uuid from the database into a global dict value.

When a person enters a url. It checks if its already in the dict. If the url exists in the dict, then we re-use the uuid (instead of creating a new one). If it does not exists. Then we insert it to database and return the generated uuid.

My goal is to have a cached stored shortened urls so that it doesn't take any extra "hits" on the database and actually reuses the existed url:uuid.

from lib.database import Stores, Urls

SHORTENED_URLS: dict = {}
DOMAIN = 'https://helloworld.com/'

# add all uuid to url as a dict that are already stored in db
for i in Urls.get_all_by_store():
    SHORTENED_URLS[i.url] = i.uuid


def generate_url(url):
    # Check if the URL is in the dict
    if url in SHORTENED_URLS:
        # Return the uuid from the "cached" dict
        return f'{DOMAIN}{SHORTENED_URLS[url]}'
    # Else get the uuid from the database
    # Database will try to insert, if duplicated then get the uuid
    generated = Urls.get_uuid(url)
    # Add the url : uuid to the database
    SHORTENED_URLS[url] = generated
    return f'{DOMAIN}{generated}'

if __name__ == '__main__':
    get_url = generate_url('https://www.testing.com')
    print(get_url)

DATABASE

# ------------------------------------------------------------------------------- #
# Redirect urls
# ------------------------------------------------------------------------------- #
class Urls(Model):
    store_id = IntegerField(column_name='store_id')
    url = TextField(column_name='url')
    uuid = TextField(column_name='uuid')
    store = ForeignKeyField(Stores, backref='urls')

    class Meta:
        database = postgres_pool
        db_table = "urls"

    @classmethod
    def get_all_by_store(cls):
        try:
            return cls.select().where((cls.store_id == Stores.store_id))
        except peewee.IntegrityError as err:
            print(f"{type(err).__name__} at line {err.__traceback__.tb_lineno} of {__file__}, {url}: {err}")
            return False

    @classmethod
    def get_uuid(cls, url):
        try:
            return cls.select().where((cls.store_id == Stores.store_id) & (cls.url == url)).get().uuid
        except Urls.DoesNotExist:
            while True:
                try:
                    gen_uuid = ''.join(choices(string.ascii_letters + string.digits, k=8))
                    cls.insert(
                        store_id=Stores.store_id,
                        url=url,
                        uuid=gen_uuid
                    ).execute()
                    return gen_uuid
                except peewee.IntegrityError as err:
                    print(f"Duplicated key -> {err}")
                    postgres_pool.rollback()
                    sleep(1)

        except peewee.IntegrityError as err:
            print(f"{type(err).__name__} at line {err.__traceback__.tb_lineno} of {__file__}, {url}: {err}")
            return False

My question is, is there anything I can do to improve the shortened url cached?

Answer 1

Re. for i in Urls.get_all_by_store(), beware putting significant work like this in the global namespace. This bypasses your main check and will incur a delay whenever someone attempts to load your module.

As @FMc warns, this program - unless it's single-process - cannot scale. Caching is a very difficult and complicated thing to get right. As soon as there are multiple processes serving requests for your clients, how are you going to coordinate in-memory cache between them? There are off-the-shelf solutions for this, but broadly, I suspect that caching should not be your only concern when scaling. There's a whole constellation of decisions you need to make around service architecture that will influence which caching solution you need, or indeed if you need one at all.