Skip to main content
deleted 27 characters in body
Source Link
Reinderien
  • 65.3k
  • 5
  • 69
  • 187

Data safety

Your current HTML_search_attributes is type-unsafe - it's closer to a serialized format than an in-memory format.

Consider moving those data to a .json file. Deserializing it will give you exactly what you have now, though I recommend going one step further. Make a class or at least a named tuple to represent a scraped domain, with attributes of price, name, and has_javascript. This will go farther to validate your data and increase the confidence in correctness of your code.

URL parsing

Don't do it by hand. This:

    d = url.replace("https://", "").replace("www.", "")

will explode for sites such as

https://foo.com/www.section/

At the least, you should regex-match to ^, the beginning of the string. More likely, you should use https://docs.python.org/3/library/urllib.parse.html .

The next problem is your class representation of these URL parts. After a class is initialized, one should be able to assume within reason that its properties are accessible, but yours are not until url is run. The solution to this is to move this block:

    d = url.replace("https://", "").replace("www.", "")
    r = requests.get(url, headers=headers)
    self._domain = d.split("/", maxsplit=1)[0]
    self._soup = BeautifulSoup(r.text, parser)
    self._url = url

into the constructor.

There probably shouldn't even be a public url setter. It only makes sense for it to be initialized once, in the constructor.

The real problem is that the scraping occurs immediately when the class is instantiated. Don't do lengthier operations such as soup calls in __init__; do them in a separate method.

Invert your logic

Rather than

    if js is False:
        # ...
    else:
        # ...

do

if js:
    # ...
else:
    # ...

into your __init__.

Data safety

Your current HTML_search_attributes is type-unsafe - it's closer to a serialized format than an in-memory format.

Consider moving those data to a .json file. Deserializing it will give you exactly what you have now, though I recommend going one step further. Make a class or at least a named tuple to represent a scraped domain, with attributes of price, name, and has_javascript. This will go farther to validate your data and increase the confidence in correctness of your code.

URL parsing

Don't do it by hand. This:

    d = url.replace("https://", "").replace("www.", "")

will explode for sites such as

https://foo.com/www.section/

At the least, you should regex-match to ^, the beginning of the string. More likely, you should use https://docs.python.org/3/library/urllib.parse.html .

The next problem is your class representation of these URL parts. After a class is initialized, one should be able to assume within reason that its properties are accessible, but yours are not until url is run. The solution to this is to move this block:

    d = url.replace("https://", "").replace("www.", "")
    r = requests.get(url, headers=headers)
    self._domain = d.split("/", maxsplit=1)[0]
    self._soup = BeautifulSoup(r.text, parser)
    self._url = url

into the constructor.

There probably shouldn't even be a public url setter. It only makes sense for it to be initialized once, in the constructor.

The real problem is that the scraping occurs immediately when the class is instantiated. Don't do lengthier operations such as soup calls in __init__; do them in a separate method.

Invert your logic

Rather than

    if js is False:
        # ...
    else:
        # ...

do

if js:
    # ...
else:
    # ...

into your __init__.

Data safety

Your current HTML_search_attributes is type-unsafe - it's closer to a serialized format than an in-memory format.

Consider moving those data to a .json file. Deserializing it will give you exactly what you have now, though I recommend going one step further. Make a class or at least a named tuple to represent a scraped domain, with attributes of price, name, and has_javascript. This will go farther to validate your data and increase the confidence in correctness of your code.

URL parsing

Don't do it by hand. This:

    d = url.replace("https://", "").replace("www.", "")

will explode for sites such as

https://foo.com/www.section/

At the least, you should regex-match to ^, the beginning of the string. More likely, you should use https://docs.python.org/3/library/urllib.parse.html .

The next problem is your class representation of these URL parts. After a class is initialized, one should be able to assume within reason that its properties are accessible, but yours are not until url is run. The solution to this is to move this block:

    d = url.replace("https://", "").replace("www.", "")
    r = requests.get(url, headers=headers)
    self._domain = d.split("/", maxsplit=1)[0]
    self._soup = BeautifulSoup(r.text, parser)
    self._url = url

into the constructor.

There probably shouldn't even be a public url setter. It only makes sense for it to be initialized once, in the constructor.

The real problem is that the scraping occurs immediately when the class is instantiated. Don't do lengthier operations such as soup calls in __init__; do them in a separate method.

Invert your logic

Rather than

    if js is False:
        # ...
    else:
        # ...

do

if js:
    # ...
else:
    # ...
added 25 characters in body
Source Link
Reinderien
  • 65.3k
  • 5
  • 69
  • 187

Data safety

Your current HTML_search_attributes is type-unsafe - it's closer to a serialized format than an in-memory format.

Consider moving those data to a .json file. Deserializing it will give you exactly what you have now, though I recommend going one step further. Make a class or at least a named tuple to represent a scraped domain, with attributes of price, name, and has_javascript. This will go farther to validate your data and increase the confidence in correctness of your code.

URL parsing

Don't do it by hand. This:

    d = url.replace("https://", "").replace("www.", "")

will explode for sites such as

https://foo.com/www.section/

At the least, you should regex-match to ^, the beginning of the string. More likely, you should use https://docs.python.org/3/library/urllib.parse.html .

The next problem is your class representation of these URL parts. After a class is initialized, one should be able to assume within reason that its properties are accessible, but yours are not until url is run. The solution to this is to move this block:

    d = url.replace("https://", "").replace("www.", "")
    r = requests.get(url, headers=headers)
    self._domain = d.split("/", maxsplit=1)[0]
    self._soup = BeautifulSoup(r.text, parser)
    self._url = url

into the constructor.

There probably shouldn't even be a public url setter. It only makes sense for it to be initialized once, in the constructor.

The real problem is that the scraping occurs immediately when the class is instantiated. Don't do lengthier operations such as soup calls in __init__; do them in a separate method.

Invert your logic

Rather than

    if js is False:
        # ...
    else:
        # ...

do

if js:
    # ...
else:
    # ...

into your __init__.

Data safety

Your current HTML_search_attributes is type-unsafe - it's closer to a serialized format than an in-memory format.

Consider moving those data to a .json file. Deserializing it will give you exactly what you have now, though I recommend going one step further. Make a class or at least a named tuple to represent a scraped domain, with attributes of price, name, and has_javascript. This will go farther to validate your data and increase the confidence in correctness of your code.

URL parsing

Don't do it by hand. This:

    d = url.replace("https://", "").replace("www.", "")

will explode for sites such as

https://foo.com/www.section/

At the least, you should regex-match to ^, the beginning of the string. More likely, you should use https://docs.python.org/3/library/urllib.parse.html .

The next problem is your class representation of these URL parts. After a class is initialized, one should be able to assume within reason that its properties are accessible, but yours are not until url is run. The solution to this is to move this block:

    d = url.replace("https://", "").replace("www.", "")
    r = requests.get(url, headers=headers)
    self._domain = d.split("/", maxsplit=1)[0]
    self._soup = BeautifulSoup(r.text, parser)
    self._url = url

There probably shouldn't even be a public url setter. It only makes sense for it to be initialized once, in the constructor.

The real problem is that the scraping occurs immediately when the class is instantiated. Don't do lengthier operations such as soup calls in __init__; do them in a separate method.

Invert your logic

Rather than

    if js is False:
        # ...
    else:
        # ...

do

if js:
    # ...
else:
    # ...

into your __init__.

Data safety

Your current HTML_search_attributes is type-unsafe - it's closer to a serialized format than an in-memory format.

Consider moving those data to a .json file. Deserializing it will give you exactly what you have now, though I recommend going one step further. Make a class or at least a named tuple to represent a scraped domain, with attributes of price, name, and has_javascript. This will go farther to validate your data and increase the confidence in correctness of your code.

URL parsing

Don't do it by hand. This:

    d = url.replace("https://", "").replace("www.", "")

will explode for sites such as

https://foo.com/www.section/

At the least, you should regex-match to ^, the beginning of the string. More likely, you should use https://docs.python.org/3/library/urllib.parse.html .

The next problem is your class representation of these URL parts. After a class is initialized, one should be able to assume within reason that its properties are accessible, but yours are not until url is run. The solution to this is to move this block:

    d = url.replace("https://", "").replace("www.", "")
    r = requests.get(url, headers=headers)
    self._domain = d.split("/", maxsplit=1)[0]
    self._soup = BeautifulSoup(r.text, parser)
    self._url = url

into the constructor.

There probably shouldn't even be a public url setter. It only makes sense for it to be initialized once, in the constructor.

The real problem is that the scraping occurs immediately when the class is instantiated. Don't do lengthier operations such as soup calls in __init__; do them in a separate method.

Invert your logic

Rather than

    if js is False:
        # ...
    else:
        # ...

do

if js:
    # ...
else:
    # ...

into your __init__.

Source Link
Reinderien
  • 65.3k
  • 5
  • 69
  • 187

Data safety

Your current HTML_search_attributes is type-unsafe - it's closer to a serialized format than an in-memory format.

Consider moving those data to a .json file. Deserializing it will give you exactly what you have now, though I recommend going one step further. Make a class or at least a named tuple to represent a scraped domain, with attributes of price, name, and has_javascript. This will go farther to validate your data and increase the confidence in correctness of your code.

URL parsing

Don't do it by hand. This:

    d = url.replace("https://", "").replace("www.", "")

will explode for sites such as

https://foo.com/www.section/

At the least, you should regex-match to ^, the beginning of the string. More likely, you should use https://docs.python.org/3/library/urllib.parse.html .

The next problem is your class representation of these URL parts. After a class is initialized, one should be able to assume within reason that its properties are accessible, but yours are not until url is run. The solution to this is to move this block:

    d = url.replace("https://", "").replace("www.", "")
    r = requests.get(url, headers=headers)
    self._domain = d.split("/", maxsplit=1)[0]
    self._soup = BeautifulSoup(r.text, parser)
    self._url = url

There probably shouldn't even be a public url setter. It only makes sense for it to be initialized once, in the constructor.

The real problem is that the scraping occurs immediately when the class is instantiated. Don't do lengthier operations such as soup calls in __init__; do them in a separate method.

Invert your logic

Rather than

    if js is False:
        # ...
    else:
        # ...

do

if js:
    # ...
else:
    # ...

into your __init__.