Community moderator elections on the Stack Exchange network are really exciting. Alas, on the page of the primaries, I find it mildly annoying that candidates are randomly reordered on every page load, even if it's surely for a good reason.

So, I cooked up (pun totally intended) this script using Beautiful Soup 4 in Python to list the candidates in an election by their score. To use it, install beautifulsoup4 (for example with pip install --user beautifulsoup4), and then to list the candidates running for Code Review:

$ python primaries.py so -n 6  # the 6th election on Stack Overflow
10905   Martijn Pieters
7310    meagar
5923    Jon Clements
5461    Matt
5211    Madara Uchiha
4133    deceze
3301    Raghav Sood
3180    Paresh Mayani
3126    Jeremy Banks
2651    Ed Cottrell

Since all the Stack Exchange sites seem to use the same format, it can work with all Stack Exchange sites. For the sake of a POC I included support for a few other sites (use "so" for Stack Overflow, "sf" for Server Fault), obviously it should be extended for others. Another feature I plan to add soon is to find the latest election by default, rather than using the first.

I mainly suspect some improvement opportunities concerning my use of Beautiful Soup, but I'm open to any kind of improvement ideas in general.

#!/usr/bin/env python

from argparse import ArgumentParser
from urllib import request

from bs4 import BeautifulSoup

STANDARD_URL_FORMAT = 'http://{}.stackexchange.com/election/{}?tab=primary'
COM_URL_FORMAT = 'http://{}.com/election/{}?tab=primary'

    (('cr', 'codereview'), STANDARD_URL_FORMAT, 'codereview'),
    (('sf', 'serverfault'), COM_URL_FORMAT, 'serverfault'),
    (('so', 'stackoverflow'), COM_URL_FORMAT, 'stackoverflow'),

def build_sites_info():
    sites_info = {}
    for info in SITES_INFO_HELPER:
        names, url_format, url_component = info
        for name in names:
            sites_info[name] = url_format, url_component
    return sites_info

def load_html_doc(url):
    return request.urlopen(url).read()

def get_soup(url):
    html_doc = load_html_doc(url)
    soup = BeautifulSoup(html_doc, 'html.parser')
    # print(soup.prettify())  # for debugging
    return soup

def main():
    sites_info = build_sites_info()

    parser = ArgumentParser(description='Show candidates of an SE election in sorted order')
    parser.add_argument('site', choices=sites_info.keys(),
                        help='Site short name or abbreviation')
    parser.add_argument('-n', type=int, default=1,
                        help='Election number')
    args = parser.parse_args()

    site_info = sites_info[args.site]
    url_format, url_component = site_info
    url = url_format.format(url_component, args.n)

    soup = get_soup(url)
    scores = []
    for tr in soup.find_all('tr'):
        if tr.find('td', attrs={'class': 'votecell'}):
            username = tr.find(attrs={'class': 'user-details'}).find('a').text
            votes = int(tr.find(attrs={'class': 'vote-count-post'}).text)
            scores.append((username, votes))

    for name, score in sorted(scores, key=lambda x: x[1], reverse=True):
        print('{}\t{}'.format(score, name))

if __name__ == '__main__':
  • 1
    \$\begingroup\$ Nice script! But I get this warning when running it: /home/jacwah/.local/lib/python3.4/site-packages/bs4/__init__.py:166: UserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system ("lxml"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently. To get rid of this warning, change this: BeautifulSoup([your markup]) to this: BeautifulSoup([your markup], "lxml") markup_type=markup_type)) \$\endgroup\$
    – jacwah
    Jul 9 '15 at 11:30
  • \$\begingroup\$ Oh I see. I didn't get such warnings, but I see in the docs that the constructor of BeautifulSoup needs a second argument, to tell the parser to use. I just tried now html.parser, it works well, so I updated the post. I hope your warning will go away too! \$\endgroup\$
    – janos
    Jul 9 '15 at 13:44
  • 1
    \$\begingroup\$ I’d modify the print() statement so that the scores are right-aligned, which will make them look nicer vertically. Also, there’s some JavaScript on Meta.se which you could probably crib to automate the site list. \$\endgroup\$
    – alexwlchan
    Jul 9 '15 at 13:57

Since your list SITES_INFO_HELPER remains constant throughout the entire program, that list should be a tuple instead

Tuples are faster than lists and are write-protected, which doesn't make a difference to you because you are writing to them.

Read more about when you should use a tuple and when you should use a list here.

Good job using ArgumentParser in your code; that is often something that you don't see used in python code and is actually an extremely useful tool.

You could make your code more object oriented by turning the information inside SITES_INFO_HELPER into instances of a class, probably called Site.

It doesn't have to be anything too complicated; it's just supposed to be a way to store some values:

class Site:
    def __init__(self, name, initials, format=STANDARD_URL_FORMAT):
        self.name = name
        self.initials = initials
        self.format = format

Note: I set the format property to be STANDARD_URL_FORMAT by default because most stack exchange sites follow this format.

Then, when storing the sites, you'd store them like this:

SITES_INFO_HELPER = ( Site("codereview", "cr"), Site("stackoverflow", "so", COM_URL_FORMAT) ...)

I'm not entirely sure, but storing site information like this may rid of the need of build_sites_info.

I think you made a good choice using urllib and BeautifulSoup together to read the election webpage. And, from what I know about BeautifulSoup, what you are doing looks just fine.

Also, looking over the election page layout, how you are scanning the webpage seems to be the only/best way.

  • \$\begingroup\$ Is there an advantage to using urllib for this purpose rather than requests? \$\endgroup\$
    – Alex A.
    Mar 31 '16 at 23:12
  • \$\begingroup\$ @AlexA. Not exactly. However, the use of urllib here is so minimal that a change to requests would be unnecessary. \$\endgroup\$
    – SirPython
    Apr 1 '16 at 1:20
  • \$\begingroup\$ Oh, okay. I don't actually know much about the differences between the two modules so I didn't know if there were any advantages to using one over the other. Thanks! \$\endgroup\$
    – Alex A.
    Apr 1 '16 at 1:28
  • \$\begingroup\$ @AlexA. No problem! I don't know much about them either, but I do know this: requests was created to simplify urllib. I guess if there was more interaction with internet connections, then it'd probably be cleaner to go with requests. \$\endgroup\$
    – SirPython
    Apr 1 '16 at 1:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.