Python REST API clients

Question

I am working on a project right now, where we need to search different APIs, then clean & persists data to database and finally pushing to Elasticsearch.

The high level code looks like this:

class BaseParser(object):

    def __init__(self, source):
        self.url = source.url
        self.api_key = source.api_key

    def _prepare_auth_header(self):
        headers = {
        "Authorization": "Basic {}".format(self.api_key)
        }
        return headers

    def get_response(self):
        raise NotImplementedError() 

class UnitedexchangeParser(BaseParser):

    def get_response(self):
        headers = self._prepare_auth_header()
        response = requests.get(self.url, headers=headers)
        return response.json()

class DubaiParser(BaseParser):

    def get_response(self):
        headers = self._prepare_auth_header()
        response = requests.get(self.url, headers=headers)
        return response.json()["data"]


class ParserFactory(object):

    def parse(self, source):
        if source.name == "Unitedexchange":
            parser = Unitedexchange(source)

        elif source.name == "Dubai":
            parser = DubaiParser(source)

        return parser.get_response()

The application is working fine, but we are adding more and more new sources which leads to change ParserFactory with same block of code.

How can I improve the layout and design such a way so we do minimal change for adding new parser? Performance is also an issue, and right now we are loading lots of individual classes which basically doing the same thing. It would be great if you can give some hint on that, too.

Answer 1

A few small comments:

1. When not supplying any argument to an exception, the () can be skipped: raise NotImplementedError.
2. I'm assuming ParseFactory is actually longer, otherwise there would be no need for it to be a class. A parse function on its own would be perfectly fine.
3. It would be better to make a dictionary of parsers, especially if the number of parsers grows:

class ParserFactory(object):
    parsers = {"Unitedexchange": UnitedexchangeParser, 
               "Dubai": DubaiParser}

    def parse(self, source):
        parser = ParserFactory.parsers[source.name]
        return parser.get_response()

This does not solve the problem of having to implement every parser.

If they are sufficiently similar, namely almost all have the same get_response, you could just put a standard implementation in BaseParser and use

class DubaiParser(BaseParser): 
    pass

Or, you could use type to create those classes that do not need any modifications dynamically and put them in a dictionary:

parsers = {"DubaiParser": None, "OtherParser": None, ...}
for name in parsers:
    parser = type(name, (BaseParser,), {})
    parser.__module__ = __name__
    parsers[name] = parser

There is even the possibility to override some methods, but this only makes sense if there are two different implementations of get_response

def get_response_json(self):
    headers = self._prepare_auth_header()
    response = requests.get(self.url, headers=headers)
    return response.json()["data"]

other_methods = {"get_response": get_response_json}

parsers = {"DubaiParser": None, "OtherParser": None, ...}
different_get_response = {"OtherParser", ...}
for name in parsers:
    if name in different_get_response:
        parser = type(name, (BaseParser,), {}, lambda ns: ns.update(other_methods)
    else:
        parser = type(name, (BaseParser,), {})
    parser.__module__ = __name__
    parsers[name] = parser

Answer 2

The code you've posted does not look like some real code example so I don't see much of reasons to comment it.

To answer your question how to register parsers automatically you can use either metaclasses or decorators for it. I would not go for metaclass while I find it interesting to use, this makes you code hard to understand so I will show you how to do it using decorators.

PARSERS_MAP = {}


def register_parser(inner):
    PARSERS_MAP[inner.__name__] = inner
    return inner


@register_parser
class FooParser:
    pass


class ParserFactory(object):

    def parse(self, source):
        parser = PARSERS_MAP[source.name]
        return parser(source).get_response()

As for your second question about performance, to be honest, I didn't get it. please show the code you are having problems with.