3
\$\begingroup\$

I have a code that keeps checking a webpage with a price on it and when the price meets a number I set it to, the thing is purchased. I am looking for ways to increase the speed of it.

I have added multiprocessing and its performance may have improved, but I don't know if its the best-case scenario for this code and there probably are better methods for speed and efficiency.

from multiprocessing.dummy import Pool
from functools import partial

session = requests.session()
session.cookies["cookie"] = ""

login = session.get("https://www.example.com")
if login.status_code == 200:
    print("logged in")
else:
    raise ValueError("Invalid Cookie")

crsf_token = ""

def token():
    while True:
        global crsf_token
        crsf_token = re.search(r"<script>XsrfToken.setToken\('(.*?)'\);</script>", session.get('https://www.example.com').text).group(1)
        time.sleep(5)

def _cthread():
    with Pool() as pool:
        while True:
            try:
                req = session.get(f"https://www.example.com")
                if req.status_code == 429:
                    time.sleep(5)
                    continue

                allposts = [f'https://www.example.com&Price={i["Price"]}'
                        for i in req.json()["data"]["Sellers"] if i["Price"] <= 10]
                if allposts:
                    pool.map(partial(session.post, headers={"X-CSRF-TOKEN": crsf_token}), allposts)
            except requests.urllib3.exceptions.ConnectTimeoutError as E:
                pass


threading.Thread(target=_cthread).start()
threading.Thread(target=token).start()
\$\endgroup\$
3
\$\begingroup\$

Constant reuse

Store your base URL, "https://www.example.com", in a constant for reuse.

HTTP codes

Some status codes, such as 200, are common and obvious, while others such as 429 are less so. Fundamentally these are all magic numbers, and requests.codes contains symbols - including too_many_requests.

URL construction

           allposts = [f'https://www.example.com&Price={i["Price"]}'
                    for i in req.json()["data"]["Sellers"] if i["Price"] <= 10]

Firstly - you don't need to be adding the URL in here. You can move the query parameter to the params dict to be passed to session.post. Then the URL can be passed to session.post directly without any baked-in query parameters.

Variable case

ConnectTimeoutError as E:

Don't capitalize E since it's a variable. Past that, you shouldn't even write as E, since you aren't using the exception.

Import long-form symbols

requests.urllib3.exceptions.ConnectTimeoutError

would benefit from an import of this class so that you can use it without package qualification.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.