7
\$\begingroup\$

I'm trying to write a script that would (a) fetch StackOverflow questions and (b) update a markdown table with new entries. With github actions I automated this script to run daily.

Idea borrowed from simonw

Code:

import re
import requests
from pathlib import Path
from datetime import datetime


URL = "https://api.stackexchange.com/2.2/questions"
DATE = datetime.utcnow().date()
ROOT = Path(__file__).parent.resolve()


def get_epochs(date):
    """ Get epoch dates for the start and end of the (current) day. """

    start = datetime(
        year=date.year, month=date.month, day=date.day,
        hour=0, minute=0, second=0
    )
    end = datetime(
        year=date.year, month=date.month, day=date.day,
        hour=23, minute=59, second=59
    )
    return int(start.timestamp()), int(end.timestamp())


def fetch_questions(start, end, tag, site="stackoverflow"):
    """ Fetch questions from stackoverflowAPI. """

    _params = {
        "fromdate": start,
        "todate": end,
        "order": "desc",
        "sort": "votes",
        "tagged": tag,
        "site": site,
    }
    return requests.get(URL, params=_params).json()


def build_table(*args, **kwargs):
    """ Build a markdown table from a list of entries. """

    columns = [
        "\n".join(
            "* [{title}]({url}) - {score} votes".format(
                # to prevent code from breaking if special characters are present
                title=re.sub(r'[^\w\s]', '', item["title"]),
                url=item["link"],
                score=item["score"]
            )
            for item in chunk["items"][:8]
        )
        for chunk in args
    ]
    return columns


def replace_chunk(content, marker, chunk, inline=False):
    """ Replace chunks of README.md """

    r = re.compile(
        r"<!\-\- {} starts \-\->.*<!\-\- {} ends \-\->".format(marker, marker),
        re.DOTALL,
    )
    if not inline:
        chunk = "\n{}\n".format(chunk)
    chunk = "<!-- {} starts -->{}<!-- {} ends -->".format(marker, chunk, marker)
    return r.sub(chunk, content)


if __name__ == "__main__":
    
    readme = ROOT / "README.md"
    start, end = get_epochs(DATE)

    pandas, beautifulsoup, code_review = build_table(
        fetch_questions(start, end, tag="pandas"),
        fetch_questions(start, end, tag="beautifulsoup"),
        fetch_questions(start, end, tag="python", site="codereview")
    )

    readme_contents = readme.open().read()
    rewritten = replace_chunk(readme_contents, "date", DATE.strftime("%Y-%m-%d"), inline=True)
    rewritten = replace_chunk(rewritten, "pandas", pandas)
    rewritten = replace_chunk(rewritten, "bs", beautifulsoup)
    rewritten = replace_chunk(rewritten, "code_review", code_review)

    with open(readme, "w") as output:
        output.write(rewritten)

Markdown file:

# Stackoverflow daily top questions: <!-- date starts --> date <!-- date ends -->

<table><tr><td valign="top" width="33%">

### Pandas
<!-- pandas starts -->
pandas content
<!-- pandas ends -->
</td><td valign="top" width="34%">


### BeautifulSoup
<!-- bs starts -->
bs4 content
<!-- bs ends -->
</td><td valign="top" width="34%">


### Python code review submissions
<!-- code_review starts -->
code review content
<!-- code_review ends -->
</td><td valign="top" width="34%">

Sample output:

enter image description here

\$\endgroup\$
2
  • \$\begingroup\$ oh, and how it all works with a github workflows is here \$\endgroup\$ Aug 3, 2020 at 8:22
  • 1
    \$\begingroup\$ If this is python 3+, you can tag as python-3.x \$\endgroup\$
    – hjpotter92
    Aug 5, 2020 at 8:22

2 Answers 2

2
\$\begingroup\$

Things look good and well organised at first glance. However, there are a few things you can make use of.

  1. Use f-string over using str.format. The former is more performant.
  2. You can group the columns and iterate over them, instead of having multiple lines for each section. This way, you can add/remove to this iterable without having to change much of the code itself.
  3. When using raw strings, you do not need to escape the - character, unless they are being used inside a character set (for regex).
  4. The build_table can be split to another separate function.
  5. Use type hinting.
  6. If you're not using kwargs, no need to declare them.

Rewritten snippets (you might need to fit them in your code accordingly):

sections = (
    {'tag': "pandas", "marker": "pandas"},
    {'tag': "beautifulsoup", "marker": "bs"},
    {'tag': "python", "site": "codereview", "marker": "code_review"},
)
for section in sections:
    questions_list = fetch_questions(start, end, **section)
    # Using `**section` might throw an error for the unknown kwarg: `marker`.
    # But it is trivial to handle that.
    .
    ...

def replace_chunk(content: str, marker: str, chunk: str, inline: bool = False):
    """ Replace chunks of README.md """
    r = re.compile(
        rf"<!-- {marker} starts -->.*<!-- {marker} ends -->",
        re.DOTALL,
    )
    if not inline:
        chunk = "\n{}\n".format(chunk)
    chunk = f"<!-- {marker} starts -->{chunk}<!-- {marker} ends -->"
    return r.sub(chunk, content)

def get_item_string(item):
    # to prevent code from breaking if special characters are present
    title = re.sub(r'[^\w\s]', '', item["title"])
    return f"* [{title}]({item['link']}) - {item['score']} votes"

def build_table(*args):
    """ Build a markdown table from a list of entries. """
    columns = [
        "\n".join(
            map(get_item_string, chunk["items"][:8])
        )
        for chunk in args
    ]
    return columns
\$\endgroup\$
1
  • \$\begingroup\$ This is great, thank you very much! \$\endgroup\$ Aug 5, 2020 at 13:08
1
\$\begingroup\$

The code looks really good, so I'll ask some questions on the design decisions that have been made.

Path(__file__).parent.resolve()

I haven't used this before, so I may be wrong, but this looks a little fragile.

The Python docs for pathlib include a very useful note under parent

enter image description here

Namely, it may do something unexpected if the path is not an absolute path. From this stack overflow answer, the result returned from __file__ may be relative, if the directory is in sys.path. In conclusion, you probably want to resolve before getting the parent, to avoid some unexpected paths.

ROOT = Path(__file__).resolve().parent

Also, why set the root based on the file location, rather than __main__ or a fixed location? Another consideration is __file__ is not always set, do you care about the use-case of running the code in an interpreter?


def get_epochs(date):
    """ Get epoch dates for the start and end of the (current) day. """
    ...

This code all looks good. I would maybe include a note specifying that the API is inclusive of the end time, and is quantised in seconds. A change to either of these assumptions (which is unlikely) would warrant changing the code here.


def fetch_questions(start, end, tag, site="stackoverflow"):
    """ Fetch questions from stackoverflowAPI. """

    _params = {
        ...
    }
    requests.get(URL, params=_params).json()

Some very minor points

  • The API is for stack exchange, not stack overflow.
  • You don't need the underscore in front of params. It is a local name to the function, so it doesn't need to be marked private. Keyword args can share their name with the parameter name. params=params works.
  • I would split up the request, and extracting result as a certain file format. If this function starts returning bad results, it will be easier to debug if the fetch and parsing of results are separated.
  • I would change the parameter tag to tags since you can pass more than one tag by delimiting with ';'. A note on how to use tag would be a good addition to the docstring.

if __name__ == "__main__":
    
    readme = ROOT / "README.md"
    start, end = get_epochs(DATE)

    pandas, beautifulsoup, code_review = build_table(
        fetch_questions(start, end, tag="pandas"),
        fetch_questions(start, end, tag="beautifulsoup"),
        fetch_questions(start, end, tag="python", site="codereview")
    )
    
    readme_contents = readme.open().read()
    rewritten = replace_chunk(readme_contents, "date", DATE.strftime("%Y-%m-%d"), inline=True)
    rewritten = replace_chunk(rewritten, "pandas", pandas)
    rewritten = replace_chunk(rewritten, "bs", beautifulsoup)
    rewritten = replace_chunk(rewritten, "code_review", code_review)
    
    with open(readme, "w") as output:
        output.write(rewritten)

I would move this code to a function, since it will be a little easier to test (and therefore update if needed). What happens if you make a typo in a tag between fetch_questions and replace_chunk?

\$\endgroup\$
1
  • \$\begingroup\$ Thank you for such a detailed answer! Those are very relevant questions. I'll reconsider everything once I have more free time. \$\endgroup\$ Aug 6, 2020 at 6:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.