Web crawler class

Question

I'm using my crawler class in the following manner and I'm beginning to think it's bad practice:

crawler.py

import requests

class Crawler():
    def __init__(self, url):
        self.url = url

    def web_crawler(self):
        requests.get(self.url)
        return requests.text

main.py

for url in urls:
    crawler = Crawler(url)
    results = crawler.web_crawler()

Would it be better to move the url parameter outside of Crawler's __init__ and move it into the web_crawler function? That way the class won't have to be reinitialized multiple times in main.py.

Answer 1

As the Crawler class just has one method along with __init__, you can avoid a class altogether and write:

def web_crawler(url):
    requests.get(url)
    return requests.text

You now have to initialize exactly 0 times, thus removing the problem from the root:

for url in urls:
    results = web_crawler(url)

The code is also simplified, both in definition and usage.

Answer 2

You can also create a field name url, and use getter and setter to obtain/change the value outside of the class.