10
\$\begingroup\$

Since I haven't really used Python's new async features yet, I took some older code of mine, which took all of my answers here on Code Review and generated a word cloud from them, and updated it to fetch the pages in an asynchronous way.

This script uses the py-stackexchange package for the API (don't forget to get your API key in order to increase the number of requests you can make to 10k). There are other packages for the API out there, but this one is easy to use IMO, especially for getting all questions/answers of one particular user. However, unfortunately (or luckily for me) it does not support getting the body of an answer (only of a question). So that part is done with aiohttp and BeautifulSoup, which is where the asynchronous part comes in. The text is split into words using nltk and the word cloud is generated via wordcloud.

To install everything:

$ pip install aiohttp bs4 lxml matplotlib nltk py-stackexchange wordcloud
$ python
>>> import nltk
>>> nltk.download('punkt')

Any and all feedback, especially on the use of the async stuff, is welcome. Maybe I should've split up fetching the page and processing it more? Maybe I missed some important performance trick?

import aiohttp
import asyncio
from bs4 import BeautifulSoup, SoupStrainer
from itertools import chain
import matplotlib.pyplot as plt
from nltk.tokenize import word_tokenize

import stackexchange
from wordcloud import WordCloud

API_KEY = '**redacted**'    # https://meta.stackexchange.com/q/261829/342577
CR = stackexchange.Site("CodeReview", API_KEY)

STRAINER = SoupStrainer(
    'div', attrs={'class': ['answer', 'answer accepted-answer']})


async def fetch(session, url, answer_id):
    async with session.get(url) as response:
        page = await response.text()
    soup = BeautifulSoup(page, "lxml", parse_only=STRAINER)
    try:
        answer_text = soup.select_one(
            f'div#answer-{answer_id} div.post-text').text
    except AttributeError:
        print("Failure:", url)
        return []
    else:
        print("Success:", url)
        return word_tokenize(answer_text)


async def fetch_all(urls, answer_ids):
    async with aiohttp.ClientSession() as session:
        jobs = [fetch(session, url, answer_id)
                for url, answer_id in zip(urls, answer_ids)]
        results = await asyncio.gather(*jobs)
    return results


if __name__ == "__main__":  
    user = CR.user(98493)   # that's me
    user.answers.fetch()  # needed to initialize it...

    urls = (answer.url.replace(CR.domain, "codereview.stackexchange.com")
            for answer in user.answers)
    answer_ids = (answer.id for answer in user.answers)

    loop = asyncio.get_event_loop()
    words = list(chain.from_iterable(
        loop.run_until_complete(fetch_all(urls, answer_ids))))

    plt.figure()
    wordcloud = WordCloud(width=480, height=480, colormap="Blues")
    wordcloud = wordcloud.generate(" ".join(words))
    plt.imshow(wordcloud, interpolation="bilinear")
    plt.axis("off")
    plt.margins(x=0, y=0)
    plt.show()

The image produced by this code looks something like this. Seems like I define, and talk about, a lot of functions...

enter image description here

\$\endgroup\$
  • 2
    \$\begingroup\$ Nice image. Unfortunately I don't speak python. \$\endgroup\$ – Heslacher Feb 13 at 17:32
6
\$\begingroup\$

Quick bits

You have some issues that some linters would pick up:

  • I would suggest moving your main code into a function. So that it doesn't pollute the global namespace.
  • You've got some trailing whitespace.
  • Add some docstrings to your code. Even something basic like "Fetch words in answers."
  • Your imports are kinda all over the place. I can't make any sense of them, and so I think they're just randomly placed there as and when you needed them.
  • I don't think print is the best tool for logging. I would suggest using logging.

Async

I'm not a fan of your current fetch and fetch_all functions. I would prefer it if fetch only called session.get. This may seem strange, but it means that you can change your code to allow for caching of objects or easier logging.

Given that I've not done any of this I've left it returning just a plain RequestContextManager. However if I were to expand on this I would change it to my own custom class. This is because then you can keep the data you want / need such as the page body in a cache. Using your own class also means that can guarantee values will exist, and hide ones you can't guarantee.

Moving the content of the old fetch into a fetch_all_words allows almost the exact same code, and allows us to build the word list without the use of a convoluted itertools and asyncio one-liner.

Interestingly since the majority of the content of the fetch_all_words function is not async code, there is little to no performance difference between using asyncio.as_completed and asyncio.gather. In a small test function I found that asyncio.as_completed performs as well or better than asyncio.gather.

Finally I make main an async function, as calling asyncio.run(main()) is simpler than building a loop and running until completion.

"""Stack Exchange word cloud generator."""
import asyncio
import logging
import itertools
import operator

import aiohttp
import bs4
import nltk.tokenize
import matplotlib.pyplot as plt
import stackexchange
import wordcloud

API_KEY = '**redacted**'
CR = stackexchange.Site("CodeReview", API_KEY)

STRAINER = bs4.SoupStrainer(
    'div',
    attrs={'class': ['answer', 'answer accepted-answer']}
)


async def fetch(session, url):
    return url, await session.get(url)


async def fetch_all(urls):
    async with aiohttp.ClientSession() as session:
        tasks = [fetch(session, url) for url in urls]
        for task in asyncio.as_completed(tasks):
            yield await task


async def fetch_all_words(answers):
    words = []
    async for url, resp in fetch_all(answers):
        answer_id = answers[url]

        async with resp as response:
            page = await response.text()
        soup = bs4.BeautifulSoup(page, "lxml", parse_only=STRAINER)
        answer = soup.select_one(f'div#answer-{answer_id} div.post-text')
        try:
            answer_text = answer.text
        except AttributeError:
            logging.error(url)
            answer_words = []
        else:
            logging.info(url)
            answer_words = nltk.tokenize.word_tokenize(answer_text)
        words.extend(answer_words)
    return words


async def main():
    """Main code."""
    logging.getLogger().setLevel(logging.INFO)

    user = CR.user(42401)
    user.answers.fetch()

    answers = {
        answer.url.replace(CR.domain, "codereview.stackexchange.com"): answer.id
        for answer in user.answers
    }
    words = await fetch_all_words(answers)

    plt.figure()
    wc = wordcloud.WordCloud(width=480, height=480, colormap="Blues")
    wc = wc.generate(" ".join(words))
    plt.imshow(wc, interpolation="bilinear")
    plt.axis("off")
    plt.margins(x=0, y=0)
    plt.show()


if __name__ == "__main__":
    asyncio.run(main())

Additional comments

  • The code heavily violates the SRP principle. Given that this is, I assume, an untested one-off script this doesn't matter much.

    However in the future I think the changes to fetch_all makes fetch_all_words easier to split up to achieve this.

  • I have butchered your style.
    This may be hard to believe but I rewrote the code around three times. I've not changed much, but I don't think much needs to be changed. I mostly focused on trying to get fetch_all and fetch a way I like.

    Whilst I like my style more, it's not intended to be some subtle hint yours is bad.

  • You have a bug apparently "n't", "ll", "n't use" and "ca n't" are words I commonly say.
    Also, "n't" appears on your image too.

  • Thanks for posting this, it was a fun little puzzle. First time I'd really looked into asyncio too!

Peilonrayz' wordmap

| improve this answer | |
\$\endgroup\$
  • \$\begingroup\$ I've been a bit nosy and looked at some other users posts. And found some funny ones like "li li" and "td td" without "li" or "td" being listed. Interestingly I've yet to find "wouldn't" or "don't" only lots of "n't" \$\endgroup\$ – Peilonrayz Feb 15 at 0:30
  • \$\begingroup\$ To be honest I didn't investigate the word splitter a lot. I just wanted to use something slightly more sophisticated than str.split so that at least punctuation is treated separately, especially to combine PEP8 and PEP8,. Regarding the imports, they are just alphabetically sorted, and the second group are the non-standard ones (for me, although I agree that that distinction is not very clean). \$\endgroup\$ – Graipher Feb 15 at 16:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.