Beating the queue

Question

Inspired by this question on CodeReview, and because I'm never able to beat the queue.

I'm just too slow I guess.. :(

I made a little python utility that reads the queue's information every 30 seconds and opens the link in your browser if we can review something.

from bs4 import BeautifulSoup
from urllib.request import urlopen
import webbrowser
import time

if __name__ == '__main__':
    url = 'https://codereview.stackexchange.com/review'

    html_doc = urlopen(url)    
    soup = BeautifulSoup(html_doc, 'html.parser')
    excluded = ['Close Votes', 'Reopen Votes']

    try:
        while True:        
            for item in soup.find_all(class_='dashboard-item'):
                try:
                    name = item.find('div', class_='dashboard-title').text.strip()
                    count = item.find('div', class_='dashboard-num').text.strip()

                    if not name in excluded:
                        print("{} with open count: {}".format(name, count))

                        if int(count) > 0:
                            print('You can review, opening in the browser: \n')
                            new_url = '{}/{}'.format(url, '-'.join(name.lower().split()))
                            webbrowser.open(new_url)                

                except AttributeError:
                    pass

            # Wait for 10 seconds
            print('\nWaiting for 30 seconds\n')
            time.sleep(30)

    except KeyboardInterrupt:
        print('\nStopped searching')

Mostly this is just for fun, but I'm still not that good in webscraping.

Therefore I post it here for a proper review.

Answer 1

Web-scraping related improvements

• I would replace the "by class" searches with more concise CSS selectors
• .text.strip() could be replaced with .get_text(strip=True)
• you can switch to lxml from html.parser to speed HTML parsing up
• another performance improvement would be to use SoupStrainer to parse only the desired part of the document

Other improvements

• I would probably replace if not name in excluded: with a more natural if name not in excluded:
• re-organize imports per PEP8
• you have the # Wait for 10 seconds comment but actually wait for 30 seconds

Modified code with some of the improvements applied:

from urllib.request import urlopen
import time
import webbrowser

from bs4 import BeautifulSoup


if __name__ == '__main__':
    url = 'https://codereview.stackexchange.com/review'

    html_doc = urlopen(url)
    soup = BeautifulSoup(html_doc, 'lxml')
    excluded = ['Close Votes', 'Reopen Votes']

    try:
        while True:
            for item in soup.select('.dashboard-item'):
                try:
                    name = item.select_one('.dashboard-title').get_text(strip=True)
                    count = item.select_one('.dashboard-num').get_text(strip=True)

                    if name not in excluded:
                        print("{} with open count: {}".format(name, count))

                        if int(count) > 0:
                            print('You can review, opening in the browser: \n')
                            new_url = '{}/{}'.format(url, '-'.join(name.lower().split()))
                            webbrowser.open(new_url)
                except AttributeError:
                    pass

            print('\nWaiting for 30 seconds\n')
            time.sleep(30)
    except KeyboardInterrupt:
        print('\nStopped searching')