Eliminate unnecessary looping while parsing cable modem status table

Question

I am writing a Python script that will be used to generate line protocol for inserting metrics into an InfluxDB. I'm parsing the status page of my cable modem. That part works. I receive the data and can extract the table of information I need.

The HTML for the table looks like this (yes, it has the commented out lines)

The I have below works, but it feels inefficient. I feel like I have a lot of loops and these feel unnecessary. I'm not sure how to be more efficient though.

The only change between this and my real code is that I've moved the HTML to pastebin to cut down on the length of this post. That pastebin is here and the HTML is not written or editable by me. It's generated on the cable modem. Since you aren't able to see the results of my cable modem by running the code I use to extract it from the modem, I hope this is good enough.

from bs4 import BeautifulSoup
import requests

results_url = "https://pastebin.com/raw/bLZLFzy6"
content = requests.get(results_url).text

measurement = "modem"
hostname = "home-modem"


def strip_uom(data):
    """Strip Unit of Measurement

    Some of our fields come with unit of measure. Rip that out and keep only 
    the value of the field.
    """
    uom = ["dB", "Hz", "dBmV", "Ksym/sec"]
    dataset = []
    for d in data:
        if d.split(" ")[-1] in uom:
            dataset.append(d.split(" ")[0])
        else:
            dataset.append(d)
    return dataset


soup = BeautifulSoup(content, 'html.parser')
dstable = soup.find('table', {'id': 'dsTable'})

# Pull the headers from the downstream table 
# We want to make these tag friendly, so make them lowercase and remove spaces
dstable_tags = [td.get_text().lower().replace(" ", "_") for td in dstable.tr.find_all('td')]

# Pull out the rest of the data for each row in the table; associate it with the correct tag; Strip UoM 
downstream_data = []
for row in dstable.find_all('tr')[1:]:
    column_values = [col.get_text() for col in row.find_all('td')]
    downstream_data.append(dict(zip(dstable_tags, strip_uom(column_values))))

# Print line protocol lines for telegraf's inputs.exec plugin to handle
for data in downstream_data:
    line_protocol_line = f"{measurement},hostname={hostname}"
    fields = []
    for key, value in data.items():
    # Check if our value is a number. If it's not, surround it in quotes. 
    # Don't actually use the float() value, as some numbers are returned as 
    # valid integers
        try:
            _ = float(value)
            fields.append(f'{key}={value}')
        except ValueError:
            fields.append(f'{key}="{value}"')
    fieldset = ",".join(fields)
    line_protocol_line = line_protocol_line + f",channel={data['channel']} {fieldset}"
    print(line_protocol_line)

Finally, the output this script generates is this:

modem,hostname=netgear-cm400,channel=1 channel=1,lock_status="Locked",modulation="QAM 256",channel_id=121,frequency=585000000,power=5.6,snr=37.0,correctables=19443,uncorrectables=11263
modem,hostname=netgear-cm400,channel=2 channel=2,lock_status="Locked",modulation="QAM 256",channel_id=6,frequency=591000000,power=5.7,snr=37.0,correctables=19531,uncorrectables=9512
modem,hostname=netgear-cm400,channel=3 channel=3,lock_status="Locked",modulation="QAM 256",channel_id=7,frequency=597000000,power=5.7,snr=36.8,correctables=17457,uncorrectables=9736
modem,hostname=netgear-cm400,channel=4 channel=4,lock_status="Locked",modulation="QAM 256",channel_id=8,frequency=603000000,power=5.9,snr=37.0,correctables=12750,uncorrectables=11156
modem,hostname=netgear-cm400,channel=5 channel=5,lock_status="Locked",modulation="QAM 256",channel_id=122,frequency=609000000,power=2.6,snr=36.3,correctables=1855538,uncorrectables=18388
modem,hostname=netgear-cm400,channel=6 channel=6,lock_status="Locked",modulation="QAM 256",channel_id=10,frequency=615000000,power=2.6,snr=37.0,correctables=846194,uncorrectables=14615
modem,hostname=netgear-cm400,channel=7 channel=7,lock_status="Locked",modulation="QAM 256",channel_id=11,frequency=621000000,power=2.6,snr=37.6,correctables=281431,uncorrectables=13998
modem,hostname=netgear-cm400,channel=8 channel=8,lock_status="Locked",modulation="QAM 256",channel_id=12,frequency=627000000,power=2.4,snr=36.1,correctables=78059,uncorrectables=13695

I have loops and list comprehensions to get the tags, associate data with tags for each row, and to generate the line protocol lines. I have two additional comprehensions in those and a loop to see if I need to strip out an included unit of measure. Can I eliminate some of these to be more efficient?

I am running all of this code on Python 3.6.7.

Example Table to show field values (this is NOT the HTML table)

Channel Lock Status Modulation  Channel ID  Frequency       Power       SNR         Correctables    UnCorrectables
1       Locked      QAM 256     121         585000000 Hz    4.9 dBmV    37.0 dB     20513           11263
2       Locked      QAM 256     6           591000000 Hz    5.0 dBmV    37.0 dB     20571           9512
3       Locked      QAM 256     7           597000000 Hz    4.9 dBmV    36.8 dB     18347           9736
4       Locked      QAM 256     8           603000000 Hz    5.1 dBmV    37.0 dB     13391           11156
5       Locked      QAM 256     122         609000000 Hz    1.9 dBmV    36.3 dB     1936410         18388
6       Locked      QAM 256     10          615000000 Hz    1.9 dBmV    37.0 dB     882543          14615
7       Locked      QAM 256     11          621000000 Hz    1.8 dBmV    37.6 dB     293494          13998
8       Locked      QAM 256     12          627000000 Hz    1.7 dBmV    35.9 dB     81559           13695

Answer 1

Use regexes

...and a generator. This matches against an unknown prefix, and an optional unit suffix, and drops the suffix.

def strip_uom(data):
    rex = re.compile(r'^(.+?) *(dB|Hz|dBmV|Ksym/sec)?$')
    return (rex.match(d)[1] for d in data)

However, this is more easily done by matching on the number itself:

def strip_uom(data):
    rex = re.compile(r'^([0-9.]+)')
    return (rex.match(d)[1] for d in data)

I haven't tested this, so you should.

Make a main function

To house your global code.

Avoid a list for fields

Instead, you can use a generator function here too:

def pairs_from_dict(data):
    for key, value in data.items():
        try:
            float(value)
        except ValueError:
            value = f'"{value}"'
        yield f'{key}={value}'

Potential loop reduction

downstream_data = []
for row in dstable.find_all('tr')[1:]:
    column_values = [col.get_text() for col in row.find_all('td')]
    downstream_data.append(dict(zip(dstable_tags, strip_uom(column_values))))

can be

downstream_data = [
    dict(zip(dstable_tags, strip_uom(
         col.get_text() for col in row.find_all('td')
    )))
    for row in dstable.find_all('tr')[1:]
]

though I don't particularly think that's an improvement.