Collect a list of unique "Origin" headers from a repeated URL request [closed]

Question

I want to create a list of the different response headers I get when querying an URL, I succeeded in doing this this way:

import requests
l= []
for _ in(5):
    a = requests.get("https://google.com", allow_redirects=True).headers['Origin']

    if a not in l:
        l.append(a)

This is readable but I feel like this is very basic, how would an advanced Pythonista handle this?

Answer 1

As mentioned in Thomas Weller's post, your for loop only executes a single time. You want to loop over a range(5).

---

You may wish to use a set to collect your unique results. Test for containment using a not in l is an \$O(N)\$ operation when l is a list, but is \$O(1)\$ when l is a set, so can be significantly faster.

Moreover, since a set naturally cannot contain duplicate entries, the test for containment before appending the item can be entirely removed and replace with a simple .add(a) call.

---

The PEP-8 -- Style Guide for Python Code recommends a single space on both sides of the = assignment operator. So l= [] should be written as l = [].

---

The variable names l and a are not at all descriptive. You need to heavily comment the code to explain what these variables are, and what they are for, which you have not done.

Using descriptive names significantly reduces the need to comment the code, as great naming will make the code self documenting. l could be list_of_origin_headers and a could be an_origin_header. While those names are very self documenting, they may be too long; brevity is also important. origin_headers and origin might be sufficient.

---

First reworking, using above points:

import requests

origin_headers = set()

for _ in range(5):
    origin = requests.get("https://google.com", allow_redirects=True).headers['Origin']
    origin_headers.add(origin)

# Convert set back into a list, to match the original design type
origin_headers = list(origin_headers)

The only tricky point in this code is origin_headers starts off as a set, but becomes a list at the end of the code, so a comment was necessary.

---

The code does something interesting enough to warrant its own function, which improves organization, and provides additional opportunity for documentation.

import requests

def fetch_origin_headers(url: str, repeat: int) -> list:
    """
    Fetch the given URL several times, and return a list containing the unique
    'Origin' headers.
    """

    if repeat < 1:
        raise ValueError("Invalid repeat.  Must be at least 1")

    origin_headers = set()

    for _ in range(repeat):
        origin = requests.get(url, allow_redirects=True).headers['Origin']
        origin_headers.add(origin)

    # Return a list instead of the temporary set
    return list(origin_headers)

if __name__ == '__main__':

    google_origins = fetch_origin_headers("https://google.com", 5)

In addition to the new function, I've added some basic type-hints and the beginnings of a """docstring""" to the function. I've also added a main-guard, so this "test code"(?) fetching the results of https://google.com is not always executed.

---

Now that we've raise the bar a little, your Pythonista would look at the inner loop and think "set comprehension":

def fetch_origin_headers(url: str, repeat: int) -> list:
    """
    Fetch the given URL several times, and return a list containing the unique
    'Origin' headers.
    """

    if repeat < 1:
        raise ValueError("Invalid repeat.  Must be at least 1")

    # Accumulate results in a set, to eliminate duplicates
    origin_headers = { requests.get(url, allow_redirects=True).headers['Origin']
                       for _ in range(repeat) }

    # Return a list instead of the temporary set
    return list(origin_headers)

Answer 2

Your code does not run as it is shown there.

The problem, as mentioned in the comments, is that int 5 is not iterable. range(5) is, which would perform the action five times.

Looking for

if a not in l:
    l.append(a)

in each loop requires looking through that list each time.

Using sets instead, this operation is still required somewhere under the hood, but likely faster (implemented on a lower level). We also no longer have to care for it. In the case here, a set can be used like the list, with the added benefit that sets can only hold unique entries. Since you care for the different values, and filter duplicates out manually with the code above, this is a step in the right direction.

Generally, comprehensions are faster than manual looping and appending in Python. There are also set comprehensions.

One last issue I had was a KeyError for "Origin". However, "Date" was available in the response, so I used that for demonstration purposes:

def get_google():
    return requests.get("https://google.com", allow_redirects=True)

dates = {get_google().headers["Date"] for _ in range(5)}

print(dates)

My machine took more than one second for the five tries, so the output was:

{'Wed, 29 Apr 2020 15:43:12 GMT', 'Wed, 29 Apr 2020 15:43:13 GMT'}

Two unique/different date entries (str) from five requests.

If you want to build onto that, you can for example check if the response was ok (200) with the ok attribute of the response.get() object.