I am writing a Python script that will be used to generate line protocol for inserting metrics into an InfluxDB. I'm parsing the status page of my cable modem. That part works. I receive the data and can extract the table of information I need.

The HTML for the table looks like this (yes, it has the commented out lines)

The I have below works, but it feels inefficient. I feel like I have a lot of loops and these feel unnecessary. I'm not sure how to be more efficient though.

The only change between this and my real code is that I've moved the HTML to pastebin to cut down on the length of this post. That pastebin is here and the HTML is not written or editable by me. It's generated on the cable modem. Since you aren't able to see the results of my cable modem by running the code I use to extract it from the modem, I hope this is good enough.

from bs4 import BeautifulSoup
import requests

results_url = "https://pastebin.com/raw/bLZLFzy6"
content = requests.get(results_url).text

measurement = "modem"
hostname = "home-modem"

def strip_uom(data):
    """Strip Unit of Measurement

    Some of our fields come with unit of measure. Rip that out and keep only 
    the value of the field.
    uom = ["dB", "Hz", "dBmV", "Ksym/sec"]
    dataset = []
    for d in data:
        if d.split(" ")[-1] in uom:
            dataset.append(d.split(" ")[0])
    return dataset

soup = BeautifulSoup(content, 'html.parser')
dstable = soup.find('table', {'id': 'dsTable'})

# Pull the headers from the downstream table 
# We want to make these tag friendly, so make them lowercase and remove spaces
dstable_tags = [td.get_text().lower().replace(" ", "_") for td in dstable.tr.find_all('td')]

# Pull out the rest of the data for each row in the table; associate it with the correct tag; Strip UoM 
downstream_data = []
for row in dstable.find_all('tr')[1:]:
    column_values = [col.get_text() for col in row.find_all('td')]
    downstream_data.append(dict(zip(dstable_tags, strip_uom(column_values))))

# Print line protocol lines for telegraf's inputs.exec plugin to handle
for data in downstream_data:
    line_protocol_line = f"{measurement},hostname={hostname}"
    fields = []
    for key, value in data.items():
    # Check if our value is a number. If it's not, surround it in quotes. 
    # Don't actually use the float() value, as some numbers are returned as 
    # valid integers
            _ = float(value)
        except ValueError:
    fieldset = ",".join(fields)
    line_protocol_line = line_protocol_line + f",channel={data['channel']} {fieldset}"

Finally, the output this script generates is this:

modem,hostname=netgear-cm400,channel=1 channel=1,lock_status="Locked",modulation="QAM 256",channel_id=121,frequency=585000000,power=5.6,snr=37.0,correctables=19443,uncorrectables=11263
modem,hostname=netgear-cm400,channel=2 channel=2,lock_status="Locked",modulation="QAM 256",channel_id=6,frequency=591000000,power=5.7,snr=37.0,correctables=19531,uncorrectables=9512
modem,hostname=netgear-cm400,channel=3 channel=3,lock_status="Locked",modulation="QAM 256",channel_id=7,frequency=597000000,power=5.7,snr=36.8,correctables=17457,uncorrectables=9736
modem,hostname=netgear-cm400,channel=4 channel=4,lock_status="Locked",modulation="QAM 256",channel_id=8,frequency=603000000,power=5.9,snr=37.0,correctables=12750,uncorrectables=11156
modem,hostname=netgear-cm400,channel=5 channel=5,lock_status="Locked",modulation="QAM 256",channel_id=122,frequency=609000000,power=2.6,snr=36.3,correctables=1855538,uncorrectables=18388
modem,hostname=netgear-cm400,channel=6 channel=6,lock_status="Locked",modulation="QAM 256",channel_id=10,frequency=615000000,power=2.6,snr=37.0,correctables=846194,uncorrectables=14615
modem,hostname=netgear-cm400,channel=7 channel=7,lock_status="Locked",modulation="QAM 256",channel_id=11,frequency=621000000,power=2.6,snr=37.6,correctables=281431,uncorrectables=13998
modem,hostname=netgear-cm400,channel=8 channel=8,lock_status="Locked",modulation="QAM 256",channel_id=12,frequency=627000000,power=2.4,snr=36.1,correctables=78059,uncorrectables=13695

I have loops and list comprehensions to get the tags, associate data with tags for each row, and to generate the line protocol lines. I have two additional comprehensions in those and a loop to see if I need to strip out an included unit of measure. Can I eliminate some of these to be more efficient?

I am running all of this code on Python 3.6.7.

Example Table to show field values (this is NOT the HTML table)

Channel Lock Status Modulation  Channel ID  Frequency       Power       SNR         Correctables    UnCorrectables
1       Locked      QAM 256     121         585000000 Hz    4.9 dBmV    37.0 dB     20513           11263
2       Locked      QAM 256     6           591000000 Hz    5.0 dBmV    37.0 dB     20571           9512
3       Locked      QAM 256     7           597000000 Hz    4.9 dBmV    36.8 dB     18347           9736
4       Locked      QAM 256     8           603000000 Hz    5.1 dBmV    37.0 dB     13391           11156
5       Locked      QAM 256     122         609000000 Hz    1.9 dBmV    36.3 dB     1936410         18388
6       Locked      QAM 256     10          615000000 Hz    1.9 dBmV    37.0 dB     882543          14615
7       Locked      QAM 256     11          621000000 Hz    1.8 dBmV    37.6 dB     293494          13998
8       Locked      QAM 256     12          627000000 Hz    1.7 dBmV    35.9 dB     81559           13695
  • \$\begingroup\$ About your unit stripping. What does the stuff before it look like? Is it always a floating-point number? Regexes are good at this, and the regex can be simplified by targeting the number instead of the unit. \$\endgroup\$
    – Reinderien
    Apr 6, 2019 at 17:09
  • \$\begingroup\$ And what's consuming the output of this script? Because that output format is a strange mix of JSON-like key-value pairs and CSV. If you can change it, you should. \$\endgroup\$
    – Reinderien
    Apr 6, 2019 at 17:12
  • \$\begingroup\$ Oh, I guess it's to feed this - github.com/influxdata/telegraf/tree/master/plugins/inputs/exec \$\endgroup\$
    – Reinderien
    Apr 6, 2019 at 17:14
  • \$\begingroup\$ @Reinderien Yes, the output is consumed by Telegraf's exec plugin. I've also attached the table in tabular format so that you can see the values I receive and how it looks with (and without) the UoM on the fields. The real table is in the HTML, this is just to give you an idea of what the lines looks like a little more easily. \$\endgroup\$
    – NewGuy
    Apr 6, 2019 at 20:22

1 Answer 1


Use regexes

...and a generator. This matches against an unknown prefix, and an optional unit suffix, and drops the suffix.

def strip_uom(data):
    rex = re.compile(r'^(.+?) *(dB|Hz|dBmV|Ksym/sec)?$')
    return (rex.match(d)[1] for d in data)

However, this is more easily done by matching on the number itself:

def strip_uom(data):
    rex = re.compile(r'^([0-9.]+)')
    return (rex.match(d)[1] for d in data)

I haven't tested this, so you should.

Make a main function

To house your global code.

Avoid a list for fields

Instead, you can use a generator function here too:

def pairs_from_dict(data):
    for key, value in data.items():
        except ValueError:
            value = f'"{value}"'
        yield f'{key}={value}'

Potential loop reduction

downstream_data = []
for row in dstable.find_all('tr')[1:]:
    column_values = [col.get_text() for col in row.find_all('td')]
    downstream_data.append(dict(zip(dstable_tags, strip_uom(column_values))))

can be

downstream_data = [
    dict(zip(dstable_tags, strip_uom(
         col.get_text() for col in row.find_all('td')
    for row in dstable.find_all('tr')[1:]

though I don't particularly think that's an improvement.

  • \$\begingroup\$ I tested the regular expression. It doesn't work against the text columns (Lock Status and Modulation). Since those don't start with a numeral, they return a NoneType (TypeError: 'NoneType' object is not subscriptable). Thank you for the idea in pairs_from_dict. \$\endgroup\$
    – NewGuy
    Apr 7, 2019 at 3:29
  • \$\begingroup\$ You can either choose to apply the regex only to columns you expect to have units; swallow the exception and take the whole column; or use the first regex shown that drops the suffixes listed explicitly. \$\endgroup\$
    – Reinderien
    Apr 7, 2019 at 3:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.