# Algorithm that receives a dictionary, converts it to a GET string, and is optimized for big data

I found this question online as an example from a technical interview and it seems to be a flawed question in many ways. It made me curious how I would answer it. So, If you were on a technical Python interview and asked to do the following:

Write an algorithm that receives a dictionary, converts it to a GET string, and is optimized for big data.

Which option would you consider the best answer? Any other code related comments are welcome.

Common:

import requests
base_url = "https://api.github.com"
data = {'per_page': 10}
node = 'users/arctelix/repos'


Option 1:

My first thought was just answer the question in the simplest form and use pagination to control the size of the data returned.

def get_query_str(node, data=None):
# base query
query_str = "%s/%s" % (base_url, node)
# build query params dict
query_params = "&".join(["%s=%s" % (k,str(v))
for k, v in data.items()])
if query_params:
query_str += "?%s" % query_params
return query_str

print("\n--Option 1--\n")
url = get_query_str(node, data)
print("url = %s" % url)


Option 2:

Well, that's not really optimized for big data and the requests library will convert a dict to params for me. Secondly, a generator would be a great way to keep memory in check with very large data sets.

def get_resource(node, data=None):
url = "%s/%s" % (base_url, node)
print("geting resource : %s %s" % (url, data))
resp = requests.get(url, params=data)
json = resp.json()
yield json

print("\n--Option 2--\n")
results = get_resource(node, data)
for r in results:
print(r)


Option 3:

Just in case the interviewer was really looking to see if I knew how join() and a list comprehension could be used to convert a dictionary to a string of query parameters. Let's put it all together and use a generator for not only the pages, but the objects as well. get_query_str is totally unnecessary, but again the task was to write something that returned a "GET string"..

class Github:
base_url = "https://api.github.com"

def get_query_str(self, node, data=None):
# base query
query_str = "%s/%s" % (self.base_url, node)
# build query params dict
query_params = "&".join(["%s=%s" % (k,str(v))
for k, v in data.items()])
if query_params:
query_str += "?%s" % query_params
return query_str

def get(self, node, data=None):
data = data or {}
data['per_page'] = data.get('per_page', 50)
page = range(0,data['per_page'])
p=0
while len(page) == data['per_page']:
data['page'] = p
query = self.get_query_str(node, data)
page = list(self.req_resource(query))
p += 1
yield page

def req_resource(self, query):
print("geting resource : %s" % query)
r = requests.get(query)
j = r.json()
yield j

gh = Github()
pages = gh.get(node, data)
print("\n--Option 3--\n")
for page in pages:
for repo in page:
print("repo=%s" % repo)


There are a bunch of things that are not said or rendered implicit by the question so I’m going to assume that the optimized for big data part is about the GitHub API response. So I’d go with the third version. But first, some general advices:

1. Document your code. Docstrings are missing all around your code. You should describe what each part of your API is doing or no-one will make the effort to figure it out and use it.
2. Don't use %, sprintf-like formatting. These are things of the past and have been superseeded by the str.format function. You may also want to try and push newest features such as formatted string litterals (or f-strings) of Python 3.6: query_str = f'{self.base_url}/{node}'.
3. You should use a generator expression rather than a list-comprehension in your '&'.joins as you will discard the list anyway. It will save you some memory management. Just remove the brakets and you’re good to go.
4. You shouldn't use f"{k}={v}" for k, v in data.items(): what if a key or a value contains a '&' or an '='? You should encode the values in your dictionnary before joining them. urllib.parse.urlencode (which is called by requests for you) is your friend.

1. page = list(self.req_resource(query)) defeats the very purpose of having a generator in the first place. Consider using yield from self.req_resource(query) instead.
2. Pagination of the Github API should be handled using the Link header instead of manually incrementing the page number. Use the request's headers dictionnary on your response to easily get them.
3. Consider using the threading module to fetch the next page of data while you are processing the current one.
• These are all GREAT tips and damn, getting the last page from the link header would solve my generator to list issue. – arctelix Oct 19 '16 at 15:37
• I have to say again, your advice was awesome! Thank you for not focusing on what we all know is a flawed question and for addressing the provided solutions with constructive criticism! You pointed out so many things that will serve me well moving forward. – arctelix Oct 20 '16 at 4:43
• @arctelix Glad that helped. Don't hesitate to ask a new question as a followup when you rewrite your code. – 301_Moved_Permanently Oct 20 '16 at 6:52

Considered as an interview question, my response would be, "what do you mean by 'optimized for big data'?" Wikipedia says,

Big data is a term for data sets that are so large or complex that traditional data processing applications are inadequate to deal with them

and it's not clear how this is related to the problem of converting a dictionary to a query string (which would normally be solved by calling urllib.parse.urlencode). So there seems to be a hidden requirement here. Or possibly the interviewer is confused. Either way, you need to go through the requirements capture and analysis process before starting to write code.

• Thanks, I did completely miss the url encoding part which is certainly a good idea. – arctelix Oct 19 '16 at 15:27

I probably have more questions than answers, but it's easier to give them here rather than via a comment:

• GET string optimized for big data is a bit of an oxymoron. GET strings usually do not go beyond 4/8/16 KB due to server limitations. (https://stackoverflow.com/questions/812925/what-is-the-maximum-possible-length-of-a-query-string). POST would be a different matter (and a better interview question).
• Your option 1 and option 2 seem to solve different problems, so there's not much of a comparison. Option 1 indeed generates a GET request from a dictionary (obviously not optimized for big data), while option 2 generates dictionaries of JSON elements from the response.
• In option 3, don't see much reason for using a generator in req_resource, as you end up anyway storing it all in memory under page list. Speaking of which page is not the best name for a list of JSON items.

Now, if we take the original question as is, I'd ask the interviewer about the use case. If they envision a scenario with a massive GET string that they don't want to store in entirety in memory, then we could have a generator that creates fragments of the GET parameters. The consumer of this generator would send said fragments directly to the networking stack, and finalize the requests with `\r\n\r\n' when the generator finishes. Again, I'd consider the question itself artificial.

• Hey, @RomanK, thanks for your response. I agree with your comments. The question is flawed in too many ways to count and i certainly doubt the size of a get req string could be an issue. This is why i felt that data needed to be returned to justify a big data optimization. The page = list / memory issue is dificult. Needed a way to exit the request loop on an unknown number of pages.. Since the objects are generators as well, were only talking about a fraction of the memory compared to the objects themselves. The page variable refers to the objects from one pagination as apposed to all. – arctelix Oct 19 '16 at 6:06
• Understood. However, you paginate the JSON-parsed response, not the request, so there is still a mismatch between the question and the response. You might want to update the question text. – RomanK Oct 19 '16 at 15:18
• The request is paginated by the page and per_page params in the request string. Perhaps im mis understanding you? – arctelix Oct 19 '16 at 15:32