14
\$\begingroup\$

Disclaimer: not a Pythonista, first actual Python programme written right here.


I'm reading a DHT11 sensor from GPIO pin 4 on my Raspberry Pi (thanks Thomas Ward for sending the RPi to me), and I want to log it to a file which will later be parsed into a web-page.

So I wrote the following Python script which is called by a Cron job every minute, and it reads the sensor 5 times then logs the average, min and max to a file for humidity and temperature.

Without further ado, here's my crap:

#!/usr/bin/python

import sys
import Adafruit_DHT
import time

def getLine(array):
    return str(sum(array) / float(len(array))) + "," + str(min(array)) + "," + str(max(array))

sensor = Adafruit_DHT.DHT11
gpio = 4

h = []
t = []

count = 5
date = time.strftime("%Y-%m-%d %H:%M:%S", time.gmtime())

for num in range(count):
    th, tt = Adafruit_DHT.read_retry(sensor, gpio)
    h.append(th)
    t.append(tt)

with open("MainLog.txt", "a") as file:
    file.write(date + ": ")
    # Write Humidity data
    file.write(getLine(h))
    file.write(" | ")
    file.write(getLine(t))
    # Write line break
    file.write("\n")

I have no idea how "good" or "bad" it is, if it were C# I would know but it's not. All suggestions welcome.

When I get my new hardware (looks like two-three days from now) I'm going to be adding a BMP180 sensor (barometric pressure/altitude).

The Adafruit_DHT library is on GitHub: https://github.com/adafruit/Adafruit_Python_DHT


I'm going to be replacing this with a DHT22, but that's one LoC change:

sensor = Adafruit_DHT.DHT11

To:

sensor = Adafruit_DHT.DHT22
\$\endgroup\$
7
\$\begingroup\$

When I get my new hardware (looks like two-three days from now) I'm going to be adding a BMP180 sensor (barometric pressure/altitude).

I'm going to be replacing this with a DHT22, but that's one LoC change

I’m going to address these bits, but will build upon @alecxe and @ThomasWard answers, so check them out first.

The thing to note is that both the building of your stats and their formatting is not easily extensible to add further hardware. You should try to come up with a more generic approach, mostly using iterator capabilities of the language.

The first thing you’ll need to learn is that, any construct of the form:

def spam():
    egg = []
    for bacon in stuff:
        egg.append(process(bacon))
    return egg

can (and must) be replaced by:

  1. a list-comprehension:

    def spam():
        return [process(bacon) for bacon in stuff]
    
  2. or a generator:

    def spam():
        for bacon in stuff:
            yield process(bacon)
    

The generator version does nothing when the function is called but will generate results as needed when the generator (the result of the function call) is iterated over. I’ll use this approach.

A second tool to learn is zip and tuple unpacking. zip, when fed with N lists (or iterables, for what its worth) will return a list of N-tuples of the size of the smallest of the input. i.e.:

>>> a = [1, 2, 3, 4]
>>> b = range(10, 5555)
>>> c = ['spam', 'egg', 'bacon']
>>> zip(a, b, c)
[(1, 10, 'spam'), (2, 11, 'egg'), (3, 12, 'bacon')]

This is useful if you have N 2-stats values, to retrieve 2 N-values per stat:

>>> read_1 = [21.42, 56.23]
>>> read_2 = [21.41, 56.12]
>>> read_3 = [21.45, 56.24]
>>> temperatures, humidities = zip(read_1, read_2, read_3)
>>> temperatures
(21.42, 21.41, 21.45)
>>> humidities
(56.23, 56.12, 56.24)

Here temperatures, humidities = zip(...) is called unpacking. zip returned a list of 2 elements, each one is being assigned to a different variable. This work because the length of the list is equal to the number of variables to unpack into. It is also possible to use the same approach (with a sligth change of syntax) to use a list of N values as N differents parameters to a function call:

>>> read = [[21.42, 56.23], [21.41, 56.12], [21.45, 56.24]]
>>> temperatures, humidities = zip(*read)
>>> temperatures
(21.42, 21.41, 21.45)
>>> humidities
(56.23, 56.12, 56.24)

Recall that list-comprehension and generator thing? The *param_name syntax works with both. So it's easy to use a function that build an N×Y iterable of Y sensor values measured N times and turn it into a Y×N list to compute statistics on each N values for each of the Y sensors.

Using the named tuple approach presented by alecxe, we can extend it further (read inherit) to separate even more the concerns of computation vs. representation:

class Stats(namedtuple('Stats', 'average min max')):
    def __str__(self):
        return '{}, {}, {}'.format(self.average, self.min, self.max)

This way, using str on a Stats instance or including it in a format call will return the proper format for each stat.

And, last but not least, the way to make your script more extensible: functools.partial. This allows you to "preconfigure" a function call with some arguments. So, for instance, Adafruit_DHT.read_retry(Adafruit_DHT.DHT11, 4) can be called in a two-way process using partial with: reader = partial(Adafruit_DHT.read_retry, Adafruit_DHT.DHT11, 4) and then reader(). How is that an improvement? You can pass a list of partial objects to a main function, each partial being responsible to call a different hardware. And the function will just blindly iterate over each reader to generate statistics:

#!/usr/bin/python

import datetime
from collections import namedtuple
from functools import partial

import Adafruit_DHT


class Stats(namedtuple('Stats', 'average min max')):
    def __str__(self):
        return '{}, {}, {}'.format(self.average, self.min, self.max)


def read_sensor(reader, measurements_count):
    for _ in range(measurements_count):
        yield reader()


def compute_statistics(readers, measurements_count):
    for reader in readers:
        measurements = read_sensor(reader, measurements_count)
        for sensor_values in zip(*measurements):
            average = sum(sensor_values) / len(sensor_values)
            yield Stats(average, min(sensor_values), max(sensor_values))


def main(readers, measurements_count=5, log_filename='MainLog.txt'):
    date = datetime.utcnow()
    statistics = compute_statistics(readers, measurements_count)
    formated_stats = ' | '.join(str(stat) for stat in statistics)

    with open(log_filename, 'a') as log_file:
        log_file.write('{:%Y-%m-%d %H:%M:%S}: {}\n'.format(date, formated_stats))


if __name__ == '__main__':
    dht = partial(Adafruit_DHT.read_retry, Adafruit_DHT.DHT11, 4)
    # bmp = partial(Adafruit_DHT.read_retry, Adafruit_DHT.XXX, XX)
    main([dht])
    # main([dht, bmp])

Note that I turned the strftime call into a format string specifier.

You can even turn the building of readers into its own function to name things and make it clearer what the call is supposed to do:

def equipment_reader(equipment_kind, gpio_pin):
    return partial(Adafruit_DHT.read_retry, equipment_kind, gpio_pin)


if __name__ == '__main__':
    dht = equipment_reader(Adafruit_DHT.DHT11, gpio_pin=4)
    main([dht])

Now, I’m not a huge fan of using Python 2 anymore; and since Adafruit got some Python 3 support, you can improve things a bit by turning the list that main requires into a variable length argument:

def main(*readers, measurements_count=5, log_filename='MainLog.txt'):
    ...


if __name__ == '__main__':
    dht = equipment_reader(Adafruit_DHT.DHT11, gpio_pin=4)
    bmp = equipment_reader(XXX, gpio_pin=XX)
    main(dht, bmp)

This feels cleaner to me, but it is invalid syntax before Python 3.5.

\$\endgroup\$
12
\$\begingroup\$

Here is a list of things I would improve:

  • define constants in upper-case (PEP8 reference)
  • put the main program logic into the if __name__ == '__main__': to avoid it being executed on import
  • naming according to PEP8 - getLine() to get_line() (or even better name would be something like get_stats())
  • remove unused sys import
  • follow the PEP8 import organization guidelines
  • use _ as a throw-away variable name (instead of num)
  • use str.format() instead of string concatenation
  • use more meaningful variable names

The code after the suggested fixes applied:

#!/usr/bin/python
import time

import Adafruit_DHT


SENSOR = Adafruit_DHT.DHT11
GPIO = 4
MEASUREMENTS_COUNT = 5


def get_stats(array):
    average = sum(array) / float(len(array))
    return "{average},{min},{max}".format(average=average, min=min(array), max=max(array))


if __name__ == '__main__':
    date = time.strftime("%Y-%m-%d %H:%M:%S", time.gmtime())

    humidities = []
    temperatures = []
    for _ in range(MEASUREMENTS_COUNT):
        humidity, temperature = Adafruit_DHT.read_retry(SENSOR, GPIO)
        humidities.append(humidity)
        temperatures.append(temperature)

    with open("MainLog.txt", "a") as file:
        file.write("{date}: {th} | {tt}\n".format(date=date, th=get_stats(humidities), tt=get_stats(temperatures)))

By the way, I've done something similar, but using the Particle Photon and both DHT11 and BMP180, getting the average values from both sensors - I was surprised how different the values from these sensors were (may be I did something wrong there).


I was also thinking about improving the way stats are formed and using a namedtuple for it:

from collections import namedtuple

Stats = namedtuple("Stats", ["average", "min", "max"])

Then, the get_stats could be:

def get_stats(measurements):
    return Stats(average=sum(measurements) / float(len(measurements)),
                 min=min(measurements),
                 max=max(measurements))

which will change the way we output to the file:

message_template = "{date}: {th.average}, {th.min}, {th.max} | {tt.average}, {tt.min}, {tt.max}\n"
with open("MainLog.txt", "a") as file:
    file.write(message_template.format(date=date, th=get_stats(humidities), tt=get_stats(temperatures)))

This looks a little bit more pythonic - I like the way we construct a message in a single place and keep the stats in a more cleaner manner.

\$\endgroup\$
  • 2
    \$\begingroup\$ This is basically all my suggestions, however I have one potential improvement: from datetime import datetime and datetime.utcnow().strftime("%Y-%m-%d %H:%M:%S"). time.gmtime() is... not exactly clear to everyone what it does, but you can easily understand what the datetime command I listed does., \$\endgroup\$ – Thomas Ward Feb 27 '17 at 1:11
10
\$\begingroup\$

Since this is my idea, I'll share it. (Though, alecxe beat me to the rest of my suggestions)

Use datetime.utcnow().strftime from datetime instead of time.gmtime

As I stated in my comment, I would use a more understandable command than time.gmtime.

(1) Replace import time with from datetime import datetime.

(2) Your date assignment will be like this instead:

date = datetime.utcnow().strftime("%Y-%m-%d %H:%M:%S")

To anyone who reads this, they'll understand you're working with a UTC time zone. (When testing your code, to understand parts, I noted that time.gmtime() returned the same time string as datetime.datetime.utcnow(), but the second here is more understandable than gmtime without looking at an API or documentation guide.)

Other than that, I have no other suggestions since they were taken from my mind by alecxe.

\$\endgroup\$
5
\$\begingroup\$

Not bad for a first program.

You do formatting work in a couple of places, most notably in getLine(). What does "getLine" even mean? In this context, it only gives you half a line! I recommend deferring formatting until as late as possible, since stringifying data degrades it and makes it less reusable. Returning a tuple would be preferable.

I suggest from __future__ import division for better compatibility with Python 3, and to avoid the awkward float() conversion.

Stylistically, I prefer to use list comprehensions to define lists "all at once" instead of building them in a loop.

#!/usr/bin/python

from __future__ import division
import Adafruit_DHT
import time

def avg_min_max(data):
    """Given a list, return a tuple of the mean, min, and max."""
    return sum(data) / len(data), min(data), max(data)

# Configuration
sensor = Adafruit_DHT.DHT11
gpio = 4
count = 5

date = time.gmtime()
humid_temp_data = [Adafruit_DHT.read_retry(sensor, gpio) for _ in range(count)]
humid_stats = avg_min_max([humid for humid, _ in humid_temp_data])
temp_stats = avg_min_max([temp for _, temp in humid_temp_data])

with open("MainLog.txt", "a") as file:
    file.write(time.strftime("%Y-%m-%d %H:%M:%S: ", date))
    file.write("{0},{1},{2} | {3},{4},{5}\n".format(*(humid_stats + temp_stats)))

Reconsider the output format. You are going to parse MainLog.txt at some point. Why not just use a CSV format instead of your weird collection of delimiters? Alternatively, design the format so that each record has a fixed length, so that you will be able to perform a binary search. Better yet, write to a database such as , which handles issues like locking and efficient searching for you.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.