Algorithm that receives a dictionary, converts it to a GET string, and is optimized for big data

Question

I found this question online as an example from a technical interview and it seems to be a flawed question in many ways. It made me curious how I would answer it. So, If you were on a technical Python interview and asked to do the following:

Write an algorithm that receives a dictionary, converts it to a GET string, and is optimized for big data.

Which option would you consider the best answer? Any other code related comments are welcome.

Common:

import requests
base_url = "https://api.github.com"
data = {'per_page': 10}
node = 'users/arctelix/repos'

Option 1:

My first thought was just answer the question in the simplest form and use pagination to control the size of the data returned.

def get_query_str(node, data=None):
    # base query
    query_str = "%s/%s" % (base_url, node)
    # build query params dict
    query_params = "&".join(["%s=%s" % (k,str(v)) 
                             for k, v in data.items()])
    if query_params:
        query_str += "?%s" % query_params
    return query_str

print("\n--Option 1--\n")
url = get_query_str(node, data)
print("url = %s" % url)

Option 2:

Well, that's not really optimized for big data and the requests library will convert a dict to params for me. Secondly, a generator would be a great way to keep memory in check with very large data sets.

def get_resource(node, data=None):
    url = "%s/%s" % (base_url, node)
    print("geting resource : %s %s" % (url, data))
    resp = requests.get(url, params=data)
    json = resp.json()
    yield json

print("\n--Option 2--\n")
results = get_resource(node, data)
for r in results:
    print(r)

Option 3:

Just in case the interviewer was really looking to see if I knew how join() and a list comprehension could be used to convert a dictionary to a string of query parameters. Let's put it all together and use a generator for not only the pages, but the objects as well. get_query_str is totally unnecessary, but again the task was to write something that returned a "GET string"..

class Github:
    base_url = "https://api.github.com"

    def get_query_str(self, node, data=None):
        # base query
        query_str = "%s/%s" % (self.base_url, node)
        # build query params dict
        query_params = "&".join(["%s=%s" % (k,str(v)) 
                                 for k, v in data.items()])
        if query_params:
            query_str += "?%s" % query_params
        return query_str

    def get(self, node, data=None):
        data = data or {}
        data['per_page'] = data.get('per_page', 50)
        page = range(0,data['per_page'])
        p=0
        while len(page) == data['per_page']:
            data['page'] = p
            query = self.get_query_str(node, data)
            page = list(self.req_resource(query))
            p += 1
            yield page

    def req_resource(self, query):
        print("geting resource : %s" % query)
        r = requests.get(query)
        j = r.json()
        yield j


gh = Github()
pages = gh.get(node, data)
print("\n--Option 3--\n")
for page in pages:
    for repo in page:
        print("repo=%s" % repo)

Answer 1

There are a bunch of things that are not said or rendered implicit by the question so I’m going to assume that the optimized for big data part is about the GitHub API response. So I’d go with the third version. But first, some general advices:

1. Document your code. Docstrings are missing all around your code. You should describe what each part of your API is doing or no-one will make the effort to figure it out and use it.
2. Don't use %, sprintf-like formatting. These are things of the past and have been superseeded by the str.format function. You may also want to try and push newest features such as formatted string litterals (or f-strings) of Python 3.6: query_str = f'{self.base_url}/{node}'.
3. You should use a generator expression rather than a list-comprehension in your '&'.joins as you will discard the list anyway. It will save you some memory management. Just remove the brakets and you’re good to go.
4. You shouldn't use f"{k}={v}" for k, v in data.items(): what if a key or a value contains a '&' or an '='? You should encode the values in your dictionnary before joining them. urllib.parse.urlencode (which is called by requests for you) is your friend.

Now about handling the response:

1. page = list(self.req_resource(query)) defeats the very purpose of having a generator in the first place. Consider using yield from self.req_resource(query) instead.
2. Pagination of the Github API should be handled using the Link header instead of manually incrementing the page number. Use the request's headers dictionnary on your response to easily get them.
3. Consider using the threading module to fetch the next page of data while you are processing the current one.

Answer 2

Considered as an interview question, my response would be, "what do you mean by 'optimized for big data'?" Wikipedia says,

Big data is a term for data sets that are so large or complex that traditional data processing applications are inadequate to deal with them

and it's not clear how this is related to the problem of converting a dictionary to a query string (which would normally be solved by calling urllib.parse.urlencode). So there seems to be a hidden requirement here. Or possibly the interviewer is confused. Either way, you need to go through the requirements capture and analysis process before starting to write code.

Answer 3

I probably have more questions than answers, but it's easier to give them here rather than via a comment:

• GET string optimized for big data is a bit of an oxymoron. GET strings usually do not go beyond 4/8/16 KB due to server limitations. (https://stackoverflow.com/questions/812925/what-is-the-maximum-possible-length-of-a-query-string). POST would be a different matter (and a better interview question).
• Your option 1 and option 2 seem to solve different problems, so there's not much of a comparison. Option 1 indeed generates a GET request from a dictionary (obviously not optimized for big data), while option 2 generates dictionaries of JSON elements from the response.
• In option 3, don't see much reason for using a generator in req_resource, as you end up anyway storing it all in memory under page list. Speaking of which page is not the best name for a list of JSON items.

Now, if we take the original question as is, I'd ask the interviewer about the use case. If they envision a scenario with a massive GET string that they don't want to store in entirety in memory, then we could have a generator that creates fragments of the GET parameters. The consumer of this generator would send said fragments directly to the networking stack, and finalize the requests with `\r\n\r\n' when the generator finishes. Again, I'd consider the question itself artificial.