3
\$\begingroup\$

Inspired by this question on CodeReview, and because I'm never able to beat the queue.

I'm just too slow I guess.. :(

I made a little python utility that reads the queue's information every 30 seconds and opens the link in your browser if we can review something.

from bs4 import BeautifulSoup
from urllib.request import urlopen
import webbrowser
import time

if __name__ == '__main__':
    url = 'https://codereview.stackexchange.com/review'

    html_doc = urlopen(url)    
    soup = BeautifulSoup(html_doc, 'html.parser')
    excluded = ['Close Votes', 'Reopen Votes']

    try:
        while True:        
            for item in soup.find_all(class_='dashboard-item'):
                try:
                    name = item.find('div', class_='dashboard-title').text.strip()
                    count = item.find('div', class_='dashboard-num').text.strip()

                    if not name in excluded:
                        print("{} with open count: {}".format(name, count))

                        if int(count) > 0:
                            print('You can review, opening in the browser: \n')
                            new_url = '{}/{}'.format(url, '-'.join(name.lower().split()))
                            webbrowser.open(new_url)                

                except AttributeError:
                    pass

            # Wait for 10 seconds
            print('\nWaiting for 30 seconds\n')
            time.sleep(30)

    except KeyboardInterrupt:
        print('\nStopped searching')

Mostly this is just for fun, but I'm still not that good in webscraping.

Therefore I post it here for a proper review.

\$\endgroup\$
2
\$\begingroup\$

Web-scraping related improvements

  • I would replace the "by class" searches with more concise CSS selectors
  • .text.strip() could be replaced with .get_text(strip=True)
  • you can switch to lxml from html.parser to speed HTML parsing up
  • another performance improvement would be to use SoupStrainer to parse only the desired part of the document

Other improvements

  • I would probably replace if not name in excluded: with a more natural if name not in excluded:
  • re-organize imports per PEP8
  • you have the # Wait for 10 seconds comment but actually wait for 30 seconds

Modified code with some of the improvements applied:

from urllib.request import urlopen
import time
import webbrowser

from bs4 import BeautifulSoup


if __name__ == '__main__':
    url = 'https://codereview.stackexchange.com/review'

    html_doc = urlopen(url)
    soup = BeautifulSoup(html_doc, 'lxml')
    excluded = ['Close Votes', 'Reopen Votes']

    try:
        while True:
            for item in soup.select('.dashboard-item'):
                try:
                    name = item.select_one('.dashboard-title').get_text(strip=True)
                    count = item.select_one('.dashboard-num').get_text(strip=True)

                    if name not in excluded:
                        print("{} with open count: {}".format(name, count))

                        if int(count) > 0:
                            print('You can review, opening in the browser: \n')
                            new_url = '{}/{}'.format(url, '-'.join(name.lower().split()))
                            webbrowser.open(new_url)
                except AttributeError:
                    pass

            print('\nWaiting for 30 seconds\n')
            time.sleep(30)
    except KeyboardInterrupt:
        print('\nStopped searching')
\$\endgroup\$
  • \$\begingroup\$ Maybe I'm missing something, but what makes CSS-Selector searches better than class searches? \$\endgroup\$ – Ludisposed Nov 6 '17 at 14:07
  • \$\begingroup\$ @Ludisposed there is no big difference in this particular case and can be considered as an individual preference, but overall CSS selectors is a powerful tool. Thanks. \$\endgroup\$ – alecxe Nov 6 '17 at 14:09
  • \$\begingroup\$ @Ludisposed just one simple example from the top of the head. Imagine you would need to partial match a class: if you would do it using find() or find_all(), it would involve either a regular expression or a filter function. But, in css selectors it would be just [class*=partial].. \$\endgroup\$ – alecxe Nov 6 '17 at 14:11
  • \$\begingroup\$ @Ludisposed there is also this subtle point that CSS selectors are more universally used that might help to, say, when you would need to migrate that code to something like selenium - as opposed to beautifulsoup-specific location techniques..just a thought. \$\endgroup\$ – alecxe Nov 6 '17 at 14:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.