Skip to main content
edited tags
Link
200_success
  • 144.2k
  • 22
  • 188
  • 473
deleted 30 characters in body; edited title
Source Link
Jamal
  • 34.9k
  • 13
  • 133
  • 237

Python REST API clients improvement

I am working on a project right now, where we need to search different APIs, then clean & persists data to database and finally pushing to Elastic searchElasticsearch.

The high level code looks like belowthis:

class BaseParser(object):
    
    def __init__(self, source):
        self.url = source.url
        self.api_key = source.api_key
        
    def _prepare_auth_header(self):
        headers = {
        "Authorization": "Basic {}".format(self.api_key)
        }
        return headers
        
    def get_response(self):
        raise NotImplementedError() 
        
class UnitedexchangeParser(BaseParser):
    
    def get_response(self):
        headers = self._prepare_auth_header()
        response = requests.get(self.url, headers=headers)
        return response.json()
    
class DubaiParser(BaseParser):
    
    def get_response(self):
        headers = self._prepare_auth_header()
        response = requests.get(self.url, headers=headers)
        return response.json()["data"]
    
    
class ParserFactory(object):
    
    def parse(self, source):
        if source.name == "Unitedexchange":
            parser = Unitedexchange(source)
        
        elif source.name == "Dubai":
            parser = DubaiParser(source)
            
        return parser.get_response()

The application is working fine, but we are adding more and more new sources which leads to change ParserFactoryParserFactory with same block of code.

How I can I improve the lay outlayout and design such a way so we do minimal change for adding new parser? And also performancePerformance is also an issue, and right now we are loading lots of individual classes which basically doing the same thing, it. It would be great if you can give some hint on that, too.

Thanks in advance.

Python REST API clients improvement

I am working on a project right now, where we need to search different APIs, then clean & persists data to database and finally pushing to Elastic search.

The high level code looks like below:

class BaseParser(object):
    
    def __init__(self, source):
        self.url = source.url
        self.api_key = source.api_key
        
    def _prepare_auth_header(self):
        headers = {
        "Authorization": "Basic {}".format(self.api_key)
        }
        return headers
        
    def get_response(self):
        raise NotImplementedError() 
        
class UnitedexchangeParser(BaseParser):
    
    def get_response(self):
        headers = self._prepare_auth_header()
        response = requests.get(self.url, headers=headers)
        return response.json()
    
class DubaiParser(BaseParser):
    
    def get_response(self):
        headers = self._prepare_auth_header()
        response = requests.get(self.url, headers=headers)
        return response.json()["data"]
    
    
class ParserFactory(object):
    
    def parse(self, source):
        if source.name == "Unitedexchange":
            parser = Unitedexchange(source)
        
        elif source.name == "Dubai":
            parser = DubaiParser(source)
            
        return parser.get_response()

The application is working fine, but we are adding more and more new sources which leads to change ParserFactory with same block of code.

How I can improve the lay out and design such a way so we do minimal change for adding new parser? And also performance is also an issue, right now we are loading lots of individual classes which basically doing the same thing, it would be great if you can give some hint on that too.

Thanks in advance.

Python REST API clients

I am working on a project right now, where we need to search different APIs, then clean & persists data to database and finally pushing to Elasticsearch.

The high level code looks like this:

class BaseParser(object):
    
    def __init__(self, source):
        self.url = source.url
        self.api_key = source.api_key
        
    def _prepare_auth_header(self):
        headers = {
        "Authorization": "Basic {}".format(self.api_key)
        }
        return headers
        
    def get_response(self):
        raise NotImplementedError() 
        
class UnitedexchangeParser(BaseParser):
    
    def get_response(self):
        headers = self._prepare_auth_header()
        response = requests.get(self.url, headers=headers)
        return response.json()
    
class DubaiParser(BaseParser):
    
    def get_response(self):
        headers = self._prepare_auth_header()
        response = requests.get(self.url, headers=headers)
        return response.json()["data"]
    
    
class ParserFactory(object):
    
    def parse(self, source):
        if source.name == "Unitedexchange":
            parser = Unitedexchange(source)
        
        elif source.name == "Dubai":
            parser = DubaiParser(source)
            
        return parser.get_response()

The application is working fine, but we are adding more and more new sources which leads to change ParserFactory with same block of code.

How can I improve the layout and design such a way so we do minimal change for adding new parser? Performance is also an issue, and right now we are loading lots of individual classes which basically doing the same thing. It would be great if you can give some hint on that, too.

This lacks the context to improve performance.
Link
Peilonrayz
  • 43.5k
  • 7
  • 76
  • 155
fixed headers missing and self missing
Source Link
Alex benz
  • 113
  • 1
  • 6
Loading
added header to other parser method
Source Link
Graipher
  • 41.1k
  • 7
  • 69
  • 133
Loading
fixed headers missing and self missing
Source Link
Alex benz
  • 113
  • 1
  • 6
Loading
fixed typo in class name
Source Link
Alex benz
  • 113
  • 1
  • 6
Loading
Source Link
Alex benz
  • 113
  • 1
  • 6
Loading