I am writing a Python script that will be used to generate line protocol for inserting metrics into an InfluxDB. I'm parsing the status page of my cable modem. That part works. I receive the data and can extract the table of information I need.
The HTML for the table looks like this (yes, it has the commented out lines)
The I have below works, but it feels inefficient. I feel like I have a lot of loops and these feel unnecessary. I'm not sure how to be more efficient though.
The only change between this and my real code is that I've moved the HTML to pastebin to cut down on the length of this post. That pastebin is here and the HTML is not written or editable by me. It's generated on the cable modem. Since you aren't able to see the results of my cable modem by running the code I use to extract it from the modem, I hope this is good enough.
from bs4 import BeautifulSoup
import requests
results_url = "https://pastebin.com/raw/bLZLFzy6"
content = requests.get(results_url).text
measurement = "modem"
hostname = "home-modem"
def strip_uom(data):
"""Strip Unit of Measurement
Some of our fields come with unit of measure. Rip that out and keep only
the value of the field.
"""
uom = ["dB", "Hz", "dBmV", "Ksym/sec"]
dataset = []
for d in data:
if d.split(" ")[-1] in uom:
dataset.append(d.split(" ")[0])
else:
dataset.append(d)
return dataset
soup = BeautifulSoup(content, 'html.parser')
dstable = soup.find('table', {'id': 'dsTable'})
# Pull the headers from the downstream table
# We want to make these tag friendly, so make them lowercase and remove spaces
dstable_tags = [td.get_text().lower().replace(" ", "_") for td in dstable.tr.find_all('td')]
# Pull out the rest of the data for each row in the table; associate it with the correct tag; Strip UoM
downstream_data = []
for row in dstable.find_all('tr')[1:]:
column_values = [col.get_text() for col in row.find_all('td')]
downstream_data.append(dict(zip(dstable_tags, strip_uom(column_values))))
# Print line protocol lines for telegraf's inputs.exec plugin to handle
for data in downstream_data:
line_protocol_line = f"{measurement},hostname={hostname}"
fields = []
for key, value in data.items():
# Check if our value is a number. If it's not, surround it in quotes.
# Don't actually use the float() value, as some numbers are returned as
# valid integers
try:
_ = float(value)
fields.append(f'{key}={value}')
except ValueError:
fields.append(f'{key}="{value}"')
fieldset = ",".join(fields)
line_protocol_line = line_protocol_line + f",channel={data['channel']} {fieldset}"
print(line_protocol_line)
Finally, the output this script generates is this:
modem,hostname=netgear-cm400,channel=1 channel=1,lock_status="Locked",modulation="QAM 256",channel_id=121,frequency=585000000,power=5.6,snr=37.0,correctables=19443,uncorrectables=11263
modem,hostname=netgear-cm400,channel=2 channel=2,lock_status="Locked",modulation="QAM 256",channel_id=6,frequency=591000000,power=5.7,snr=37.0,correctables=19531,uncorrectables=9512
modem,hostname=netgear-cm400,channel=3 channel=3,lock_status="Locked",modulation="QAM 256",channel_id=7,frequency=597000000,power=5.7,snr=36.8,correctables=17457,uncorrectables=9736
modem,hostname=netgear-cm400,channel=4 channel=4,lock_status="Locked",modulation="QAM 256",channel_id=8,frequency=603000000,power=5.9,snr=37.0,correctables=12750,uncorrectables=11156
modem,hostname=netgear-cm400,channel=5 channel=5,lock_status="Locked",modulation="QAM 256",channel_id=122,frequency=609000000,power=2.6,snr=36.3,correctables=1855538,uncorrectables=18388
modem,hostname=netgear-cm400,channel=6 channel=6,lock_status="Locked",modulation="QAM 256",channel_id=10,frequency=615000000,power=2.6,snr=37.0,correctables=846194,uncorrectables=14615
modem,hostname=netgear-cm400,channel=7 channel=7,lock_status="Locked",modulation="QAM 256",channel_id=11,frequency=621000000,power=2.6,snr=37.6,correctables=281431,uncorrectables=13998
modem,hostname=netgear-cm400,channel=8 channel=8,lock_status="Locked",modulation="QAM 256",channel_id=12,frequency=627000000,power=2.4,snr=36.1,correctables=78059,uncorrectables=13695
I have loops and list comprehensions to get the tags, associate data with tags for each row, and to generate the line protocol lines. I have two additional comprehensions in those and a loop to see if I need to strip out an included unit of measure. Can I eliminate some of these to be more efficient?
I am running all of this code on Python 3.6.7.
Example Table to show field values (this is NOT the HTML table)
Channel Lock Status Modulation Channel ID Frequency Power SNR Correctables UnCorrectables
1 Locked QAM 256 121 585000000 Hz 4.9 dBmV 37.0 dB 20513 11263
2 Locked QAM 256 6 591000000 Hz 5.0 dBmV 37.0 dB 20571 9512
3 Locked QAM 256 7 597000000 Hz 4.9 dBmV 36.8 dB 18347 9736
4 Locked QAM 256 8 603000000 Hz 5.1 dBmV 37.0 dB 13391 11156
5 Locked QAM 256 122 609000000 Hz 1.9 dBmV 36.3 dB 1936410 18388
6 Locked QAM 256 10 615000000 Hz 1.9 dBmV 37.0 dB 882543 14615
7 Locked QAM 256 11 621000000 Hz 1.8 dBmV 37.6 dB 293494 13998
8 Locked QAM 256 12 627000000 Hz 1.7 dBmV 35.9 dB 81559 13695