Get a notification when questions get posted

Question

I like to help people on StackOverflow, matter of fact nowadays it depends on how quick you are when answering question. With that being said, I can't always keep a tab and an eye to refresh the browser to view new questions.

I made a very basic script to send me a notification (OSX) whenever there's a new question.

import requests, os, threading, time
from bs4 import BeautifulSoup
from timeloop import Timeloop
from datetime import timedelta

tl = Timeloop()

SE_REALTIME = "https://stackexchange.com/questions?tab=realtime"
WATCHED_TAG = "swift"
QUE_LI = []



# The notifier function
def notify(title, subtitle, message, link):
    t = '-title {!r}'.format(title)
    s = '-subtitle {!r}'.format(subtitle)
    m = '-message {!r}'.format(message)
    l = '-open {!r}'.format(link)
    os.system('terminal-notifier {}'.format(' '.join([m, t, s, l])))

def fetch():
    # The SOUP
    soup = BeautifulSoup(requests.get(SE_REALTIME).content, "html.parser")

    # List of Questions with the sid: Stackoverflow.com
    li = soup.find_all('div', {'data-sid': 'stackoverflow.com'})

    # Loop Through
    for child in li:
        id = child['class'][2]
        all_tags = child.find("span", attrs={'class': 'realtime-tags'}).text.strip()
        tags = all_tags.split()
        if WATCHED_TAG in tags:
            if id not in QUE_LI: 
                question = child.find("a", attrs={'class': 'realtime-question-url realtime-title'})
                QUE_LI.append(id)
                notify(title = 'New Question {}'.format(' '.join([WATCHED_TAG])),
                subtitle = 'Stackoverflow.com',
                message  = question.text.strip(),
                link = question['href'])



@tl.job(interval=timedelta(seconds=2))
def fetch_questions_every_2s():
    fetch()

if __name__ == "__main__":
    tl.start(block=True)

Ok, This code works well. But I'm looking for:

1. A way to listen for changes on the div tree (without making a request every 2 second)
2. Less code.

Dependencies:

Terminal-Notifier
BeautifulSoup
TimeLoop
Requests

Answer 1

Apart from using the SE API (for example via some module like py-stackexchange), there are some other optimizations possible.

In your notify function you could use the new f-strings to simplify building the command:

def notify(title, subtitle, message, link):
    os.system(f"terminal-notifier -title {title!r}"
              f" -subtitle {subtitle!r} -message {message!r} -open {link!r}")

Note that strings on multiple lines like here are automatically joined.

Alternatively you could use the subprocess module, which has a lot more advanced features, which are not needed here, though:

from subprocess import run

def notify(title, subtitle, message, link):
    run(["terminal-notifier", "-title", repr(title), "-subtitle", repr(subtitle),
         "-message", repr(message), "-open", repr(link)])

In your fetch function you have a possible slow-down. Since QUE_LI (not the most informative name, btw, should also probably be lowercase since it is not a global constant) is a list, checking in is \$\mathcal{O}(n)\$. Just make it a set instead to get \$\mathcal{O}(1)\$ in. I would also and the two if conditions. This saves one level of indentation.

Instead of using "html.parser", consider using "lxml" (you might have to install it first). It is usually faster.

Also note that you are shadowing the built-in function id here and the way you call notify does not conform to Python's official style-guide, PEP8. It took me a while to realize that the following lines are all part of the call and its keyword arguments, and not just new lines. It is arguable whether the keywords are needed here at all.

It is also unclear if WATCHED_TAG is supposed to be one tag or multiple tags. If it is just one, as the name suggests, then ' '.join([WATCHED_TAG]) is quite pointless, since it is just WATCHED_TAG. If it is multiple, then that command won't join them properly and if WATCHED_TAG in tags will only return true if the order of the tags is the same.

seen_ids = set()

def fetch():
    r = requests.get(SE_REALTIME)
    # r.raise_for_status()  # to enable failing if the request fails
    soup = BeautifulSoup(r.content, "lxml")
    for child in soup.find_all('div', {'data-sid': 'stackoverflow.com'}):
        question_id = child['class'][2]
        tags = child.find("span#", attrs={'class': 'realtime-tags'}).text.strip()
        if WATCHED_TAG in tags.split() and question_id not in seen_ids:
            seen_ids.add(question_id)
            question = child.find("a", attrs={'class': 'realtime-question-url realtime-title'})
            notify(f'New Question {WATCHED_TAG}', 'Stackoverflow.com',
                   question.text.strip(), question['href'])

Answer 2

Why would you use stackexchange.com and only take ones with the title of stackoverflow.com? That's a huge waste.

You could change your url from https://stackexchange.com/questions?tab=realtime to https://stackoverflow.com/questions?tab=realtime.

However, actually visiting the website the way you are is hacky and overkill. Instead of web scraping, use the Stackexchange API.

There is no way to get a notification when a new question is posted. You will have to make a call every time you want the latest results.

Edit: there may be a way to get notifications, see vogel612's comment:

there is websockets to receive notifications when a new question is posted. To my knowledge these are very much undocumented and easier to get access to with a userscript (which would imply changing to javascript), but it's not like there's no way to receive notifications from SE directly.