2
\$\begingroup\$

I was trying to fetch some data (daily new cases and daily new deaths) from Worldometers and I came up with this:

import requests
from bs4 import BeautifulSoup as bs
import numpy as np 
import matplotlib.pyplot as plt 

# Plotting Function
def plot(data , country):
    fontsize = 10
    csfont = {'fontname':'Times New Roman'}
    plt.plot(data)
    plt.xlabel(f'Days since the beginning of the COVID-19 Pandemic in {country}',fontsize=fontsize, fontweight='bold',**csfont)
    plt.ylabel('Daily new Cases',fontsize=fontsize, fontweight='bold',**csfont)
    plt.tight_layout()
    plt.show()

# Daily New Cases
def DNC(country,Plot = False):
    url = f"https://www.worldometers.info/coronavirus/country/{country}/"
    r = requests.get(url)
    htmlcontent = r.content
    soup = str(bs(htmlcontent, "html.parser"))
    n = soup.find("name: 'Daily Cases',")
    n2 = soup[n:].find("data:")
    m = soup[n:].find(']')
    data = np.array(soup[n+n2+7:n+m].replace('null','0').split(','),dtype=int)
    if Plot == True:
        plot(data,country)
    return data

# Daily New Deaths
def DND(country,Plot = False):
    url = f"https://www.worldometers.info/coronavirus/country/{country}/"
    r = requests.get(url)
    htmlcontent = r.content
    soup = str(bs(htmlcontent, "html.parser"))
    n = soup.find("name: 'Daily Deaths',")
    n2 = soup[n:].find("data:")
    m = soup[n:].find(']')
    data = np.array(soup[n+n2+7:n+m].replace('null','0').split(','),dtype=int)
    if Plot == True:
        plot(data,country)
    return data

if __name__ == '__main__':
    DNC('us',Plot=True)
    DND('us',Plot=True)

Which technically works, but I'm not happy with how it finds the data in the HTML (I think I have used the dumbest idea, to convert the soup to string and then find it there with counting letters and so on). Is there a better way to do this?

\$\endgroup\$
1
  • \$\begingroup\$ You should make your link match the URLs in your code. \$\endgroup\$ Aug 7, 2021 at 1:41

2 Answers 2

3
\$\begingroup\$

Overall it's an interesting (if a little strange) idea. Why are you re-plotting data locally when the plots have already been rendered to downloadable SVG elements on the website in question? Effectively you're writing a scraper and converter from JavaScript Highcharts library calls to Matplotlib calls.

Assuming that this is a good idea at all (I'm not convinced that this is true), your methods - whereas they're a good start, for learning purposes - are somewhat inefficient, non-generalized, fragile and buggy.

From the top:

  • plot() is buggy because it claims Daily new cases regardless of whether that's true for the current call.
  • plot() does not properly use the matplotlib date formatting support on the x-axis. "Days since" is a far less intuitive axis than simply showing dates.
  • DNC should not be abbreviated and should instead read daily_new_cases
  • Plot = False should not be capitalized and should be plot
  • Do not bake plotting into your scraping functions
  • Check your requests response for failure, which you don't do currently
  • You load into BeautifulSoup (good) but then re-flatten it to a single string, which... what? Why?
  • Do not pass r.content to BeautifulSoup; pass r.text which is encoding-aware
  • Look at script tags only, rather than the entire response text
  • Use a JavaScript parsing library like slimit to navigate to the actual chart data in the syntax tree, rather than attempting to guess your way through the string
  • You download the page all over again for every chart type; don't do this - only download it once
  • Don't use integers for your series: there are float elements in the site's markup
  • Use a SoupStrainer to pre-select only the elements you care about
  • You can represent each chart as a class instance
  • Consider a generic method that parses out all of the charts on the page, not just hard-coded for two

Suggested

import re
from dataclasses import dataclass
from datetime import datetime, timedelta
from typing import Iterable, Optional, Tuple, Dict

import requests
import numpy as np
import matplotlib.pyplot as plt
from bs4 import BeautifulSoup, SoupStrainer
from matplotlib.axes import Axes
from matplotlib.dates import num2date
from matplotlib.figure import Figure
from slimit.ast import ExprStatement, Node, Assign, Object, Number, UnaryOp, Null
from slimit.parser import Parser


SCRIPTS = SoupStrainer(
    name='script',
    type='text/javascript',
)

CHART_PAT = re.compile(r'Highcharts\.chart'),

# Using this as a naive, assumed-local-timezone date is dubious, but
# Worldometers has left ambiguous their date boundaries
EPOCH = datetime.fromtimestamp(0)
DAY = timedelta(days=1)


def get_chart_props(parser: Parser, script: str) -> Iterable[
    Tuple[
        str,  # chart name
        Dict[
            str,   # property name
            Node,  # property node value - an Object or an Array
        ],
    ]
]:
    tree = parser.parse(script)

    for node in tree.children():
        if (
            isinstance(node, ExprStatement)
            and node.expr.identifier.node.value == 'Highcharts'
            and node.expr.identifier.identifier.value == 'chart'
        ):
            name_arg, chart_arg, *_ = node.expr.args
            name = name_arg.value.strip("'")
            props = {
                prop.left.value: prop.right
                for prop in chart_arg.properties
            }
            yield name, props


def get_prop(obj: Object, key: str) -> Node:
    for prop in obj.properties:
        if isinstance(prop, Assign) and prop.left.value == key:
            return prop.right
    raise ValueError(f'Key {key} not found')


def to_float(x: Node) -> float:
    if isinstance(x, Number):
        return float(x.value)
    if isinstance(x, UnaryOp) and x.op == '-':
        return -float(x.value.value)
    if isinstance(x, Null):
        return float('NaN')
    raise ValueError(f'Not a float: {x}')


@dataclass
class Chart:
    country: str
    name: str
    title: str
    subtitle: Optional[str]
    y_title: str

    # Epoch days as floats; see https://matplotlib.org/stable/api/dates_api.html
    x_data: np.ndarray
    # floats
    y_data: Dict[str, np.ndarray]

    @classmethod
    def from_script(cls, country: str, name: str, props: Dict[str, Node]) -> 'Chart':
        title = props['title'].properties[0].right.value.strip("'")

        subtitle_props = props.get('subtitle')
        subtitle = subtitle_props and get_prop(
            subtitle_props, 'text',
        ).value.strip("'")

        y_title = get_prop(
            get_prop(props['yAxis'], 'title'), 'text'
        ).value.strip("'")

        x_array = get_prop(props['xAxis'], 'categories').items
        x_data = [
            (
                datetime.strptime(item.value, '"%b %d, %Y"')
                - EPOCH
            ) / DAY
            for item in x_array
        ]

        series = {
            get_prop(obj, 'name').value.strip("'"):
            np.array([
                to_float(x) for x in get_prop(obj, 'data').items
            ])
            for obj in props['series'].items
        }

        chart = cls(
            country=country,
            name=name,
            title=title,
            subtitle=subtitle,
            x_data=np.array(x_data),
            y_data=series,
            y_title=y_title,
        )
        return chart

    def plot(self) -> Figure:
        fig: Figure
        ax: Axes
        fig, ax = plt.subplots()
        fig.suptitle(self.title)
        if self.subtitle is not None:
            ax.set_title(self.subtitle)
        ax.set_ylabel(self.y_title)

        for name, series in self.y_data.items():
            ax.plot(num2date(self.x_data), series, label=name)

        fig.autofmt_xdate()
        fig.legend()
        return fig


def download_data(country: str) -> BeautifulSoup:
    with requests.get(
        url=f"https://www.worldometers.info/coronavirus/country/{country}/"
    ) as resp:
        resp.raise_for_status()
        return BeautifulSoup(resp.text, "html.parser", parse_only=SCRIPTS)


def load_all(soup: BeautifulSoup, country: str) -> Iterable[Chart]:
    scripts = soup.find_all(text=CHART_PAT)
    parser = Parser()
    for script in scripts:
        for name, props in get_chart_props(parser, script):
            yield Chart.from_script(country=country, name=name, props=props)


def main() -> None:
    country = 'canada'
    doc = download_data(country)
    charts = tuple(load_all(doc, country))

    for chart in charts:
        chart.plot()
    plt.show()


if __name__ == '__main__':
    main()

Output

Here are two of the example charts produced by the above:

example plots

\$\endgroup\$
0
\$\begingroup\$

Sure. One way would be to use another source. Another might be to separate out the "script" tag (for example, using a bs selector) and at least only use that as the string. Unless you want to run javascript, there's not a much better way to read the data from inside the script.

I do recommend replacing your string parsing of the array with json.loads though

\$\endgroup\$
2
  • 1
    \$\begingroup\$ Unless you want to run javascript - You don't need to run it; you just need to parse it. You can get an AST from something like slimit. \$\endgroup\$
    – Reinderien
    Aug 8, 2021 at 21:36
  • \$\begingroup\$ That's a neat suggestion. I actually didn't know it was possible. That said, I think the current regex method is probably more stable against future changes, than either running or parsing the javascript \$\endgroup\$ Aug 9, 2021 at 20:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.