# Sensor logger for Raspberry Pi in a stratospheric probe

I'm writing a Python Script for a Raspberry Pi to measure different sensors. We are planning to send the Pi with that Script running to the stratosphere, so the power usage for the Pi is limited.

I excuse myself in advance for the code, I had no prior experience with Python. Are there any ways I can make this code more battery friendly? Would it be beneficial to write 10 rows at once instead of writing one row at a time?

#!/usr/bin/env python3

from sense_hat import SenseHat
import time
import csv
import datetime

sense = SenseHat()
sense.clear()
sense.set_imu_config(True, True, True)
sense.low_light = True

with open('data.csv', mode='w') as file:
writer = csv.writer(file, delimiter=';', quotechar='"',     quoting=csv.QUOTE_MINIMAL)
writer.writerow(['Zeit','Temperatur1', 'Temperatur2', 'Temperatur3',     'Luftdruck', 'Luftfeuchtigkeit', 'Yaw', 'Pitch', 'Roll', 'Compass X', 'Compass     Y', 'Compass Z', 'Gyro X', 'Gyro Y', 'Gyro Z'])

with open('acc.csv', mode='w') as file:
writer = csv.writer(file, delimiter=';', quotechar='"',     quoting=csv.QUOTE_MINIMAL)
writer.writerow(['Zeit','Acc_X','Acc_Y','Acc_Z'])

with open('log.csv', mode='w') as file:
writer = csv.writer(file, delimiter=';', quotechar='"',     quoting=csv.QUOTE_MINIMAL)
writer.writerow(['Zeit','Fehler'])

# Farben definieren
red = (255, 0, 0)
green = (0, 255, 0)
black = (0,0,0)

def writeDataToCsv(temperature, temperature2, temperature3, pressure, humidty,     yaw, pitch, roll, mag_x, mag_y, mag_z, gyro_x, gyro_y, gyro_z):
with open('data.csv', mode='a') as file:
writer = csv.writer(file, delimiter=';', quotechar='"',     quoting=csv.QUOTE_MINIMAL)
writer.writerow([datetime.datetime.now(),temperature, temperature2,     temperature3, pressure, humidty, yaw, pitch, roll, mag_x, mag_y, mag_z,     gyro_x, gyro_y, gyro_z])

def writeAccelerationToCsv(x,y,z):
with open('acc.csv', mode='a') as file:
writer = csv.writer(file, delimiter=';', quotechar='"',     quoting=csv.QUOTE_MINIMAL)
writer.writerow([datetime.datetime.now(),x,y,z])
sense.set_pixel(0, 0, green)
time.sleep(.05)
sense.set_pixel(0, 0, black)

def main():
sense.set_pixel(0, 0, black)
counter = 0
try:
while True:
#Region Acceleration
acceleration = sense.get_accelerometer_raw()
acc_x = acceleration['x']
acc_y = acceleration['y']
acc_z = acceleration['z']
writeAccelerationToCsv(acc_x,acc_y,acc_z)
time.sleep(.250)
counter+=1

#Region Data
if(counter == 4):
temperature = sense.get_temperature()
temperature2 = sense.get_temperature_from_humidity()
temperature3 = sense.get_temperature_from_pressure()

pressure = sense.get_pressure()
humidty = sense.get_humidity()

orientation = sense.get_orientation()
yaw = orientation["yaw"]
pitch = orientation["pitch"]
roll = orientation["roll"]

mag = sense.get_compass_raw()
mag_x = mag["x"]
mag_y = mag["y"]
mag_z = mag["z"]

gyro = sense.get_gyroscope_raw()
gyro_x = gyro["x"]
gyro_y = gyro["y"]
gyro_z = gyro["z"]

writeDataToCsv(temperature, temperature2, temperature3,     pressure, humidty, yaw, pitch, roll, mag_x, mag_y, mag_z, gyro_x, gyro_y,     gyro_z)

counter = 0;
except Exception as e:
with open('log.csv', mode='a') as file:
writer = csv.writer(file, delimiter=';', quotechar='"',     quoting=csv.QUOTE_MINIMAL)
writer.writerow([datetime.datetime.now(),str(e)])
sense.set_pixel(1, 0, red)
finally:
pass
main()

if __name__ == '__main__':
main()

• The Pi uses quite a lot of power, with only a little of it going to the CPU, so you can't make much of a difference in the code there. – Simon Richter Feb 24 '19 at 5:31
• Is there a reason you can't use a lower power device, equip a bigger battery, or use solar panels? – forest Feb 24 '19 at 8:27
• @forest I havent thought about the solar panels yet, but the weight is probably the biggest limitation for us. We only have a limited size for the balloon and this limits the weight. We decided to use the raspberry as its a cheap and easy to use device. Noone of our team has the knowledge to do this with for example an ardiuno. – Lexu Feb 25 '19 at 15:05
• Pick a tool suitable for the task. Pi and Arduino are hobbyist boards, unsuitable for pretty much anything beyond tinkering in your free time. Where the former draws quite a bit of current, 100-200mA at least, not counting LEDs etc. This doesn't look like something that needs tons of processing power, so a Cortex M0 would be ideal. I would advise to hire professionals for this job, as no amount of optimization in Python will change the hardware you have picked. If none of your team has any knowledge of embedded systems, then the project is pretty much doomed to fail. – Lundin Feb 27 '19 at 9:08
• @Lundin Well it's a school project, so we are quite limited by time and money. It's more a proof of concept to see if that would work. – Lexu Feb 27 '19 at 10:38

Have you already executed the code to see how it performs and if the battery will last? There is that famous Donald Knuth quote saying premature optimization is the root of all evil (or at least most of it) in programming.

I never had to think about the energy consumption of a program, so I cannot tell you about the power efficieny. But as vnp already did, I can also share my opinion about the code structure to help you to identify bottlenecks more easily. Also, a different structure should help you to still log some data even in case of exceptions.

Here is what struck me on first read:

• most of the code is defined in the main method
• you overwrite the complete data files at the beginning of the program
• repetition of the csv write (violates the zen of python - not dry - dont repeat yourself)

I tried to resolve some of the issues and refactored the structure of the code:

#!/usr/bin/env python3

from sense_hat import SenseHat
import time
import csv
import datetime

# defined constants on moduel level and capitalized the names (pep8: https://www.python.org/dev/peps/pep-0008/#constants)
RED = (255, 0, 0)
GREEN = (0, 255, 0)
BLACK = (0,0,0)

class DataLogger(object):

def __init__(self, init_csv_files=False):
# initalize the commonly ued sensor
self.sense = SenseHat()
self.sense.clear()
self.sense.set_imu_config(True, True, True)
self.sense.low_light = True

# only initialize the csv files, if intended
# I would suggest not to init them in the same program though.
# If - for some reasons - the python interpreter crashes and the script is restarted,
# the init of the csv_files will overwrite all the data which was logged so far.
if init_csv_files:
self.init_csv_files()

def write_data_to_file(self, data, file_name, mode='a', delimiter=';', quotechar='"', quoting=csv.QUOTE_MINIMAL):
"""
Helper method to write the given data to a csv file. Using 'append' as default mode to avoid accidental overwrites.
"""
with open(file_name, mode=mode) as file:
writer = csv.writer(file, delimiter=delimiter, quotechar=quotechar, quoting=quoting)
writer.writerow(data)

def init_csv_files(self):
# see comment in init method
data_headings = ['Zeit','Temperatur1', 'Temperatur2', 'Temperatur3', 'Luftdruck', 'Luftfeuchtigkeit', 'Yaw', 'Pitch', 'Roll', 'Compass X', 'Compass Y', 'Compass Z', 'Gyro X', 'Gyro Y', 'Gyro Z']

def start_logging(self):
# actual execution
sense.set_pixel(0, 0, BLACK)
counter = 0

while True:
# moved the accelleration logging to a different method
# and catched possible exceptions there, so the counter will still be increase
# and the rest of the data may still be logged even if the accelleration data
# always raises exceptions
self.log_accelleration()
time.sleep(.250)
counter += 1

# using counter % 4 == 0 instead of counter == 4
# this will evaluate to true for every number divisible by 4
# If you do the strict comparision, you could find yourself in the scenario
# where the data logging is never executed, if the counter is larger than 4
# (in this case this is very unlikely, but in threaded scenarios it would be possible,
# so doing modulo 4 is more defensive)
if(counter % 4 == 0):
self.log_data()
counter = 0

def log_accelleration(self):
acceleration_data = get_accelleration()
if acceleration_data:
try:
self.write_data_to_file(acceleration_data, 'acc.csv')
except Exception as e:
self.log_exception(e)
pass
else:
# no exception occurred
self.sense.set_pixel(0, 0, green)
time.sleep(.05)
finally:
self.sense.set_pixel(0, 0, black)

def log_data(self):
# saving datetime first, before reading all the sensor data
data = [datetime.datetime.now()]

# moved each of the calls to sense in a separate method
# exceptions will lead to empty entries being logged but
# if e.g. get_pressure raises an exceptions, the other data may still get logged
data += self.get_temperature()
data += self.get_pressure()
data += self.get_humidity()
data += self.get_orientation()
data += self.get_mag()
data += self.get_gyro()

self.write_data_to_file(data, 'data.csv')

def log_exception(self, exception):
sense.set_pixel(1, 0, red)
self.write_data_to_file([datetime.datetime.now(), str(exception)], 'log.csv')
sense.set_pixel(0, 0, black)

def get_accelleration(self):
try:
acceleration = self.sense.get_accelerometer_raw()
except Exception as e:
self.log_exception(e)
return

acc_x = acceleration['x']
acc_y = acceleration['y']
acc_z = acceleration['z']

return[datetime.datetime.now(), acc_x, acc_y, acc_z]

def get_temperature(self):
try:
temperature1 = sense.get_temperature()
temperature2 = sense.get_temperature_from_humidity()
temperature3 = sense.get_temperature_from_pressure()
except Exception as e:
return [None, None, None]
return [temperature1, temperature2, temperature3]

def get_pressure(self):
try:
pressure = sense.get_pressure()
except Exception as e:
return [None]
return [pressure]

def get_humidity(self):
try:
humidty = sense.get_humidity()
except Exception as e:
return [None]
return [humidty]

def get_orientation(self):
try:
orientation = sense.get_orientation()
except Exception as e:
return [None, None, None]
return [orientation["yaw"], orientation["pitch"], orientation["roll"]]

def get_mag(self):
try:
mag = sense.get_compass_raw()
except Exception as e:
return [None, None, None]
return [mag["x"], mag["y"], mag["z"]]

def get_gyro(self):
try:
gyro = sense.get_gyroscope_raw()
except Exception as e:
return [None, None, None]
return [gyro["x"], gyro["y"], gyro["z"]]

if __name__ == '__main__':
data_logger = DataLogger(init_csv_files=True)
try:
data_logger.start_logging()
except Exception as e:
data_logger.log_exception(e)


Further steps for improvements:

• Catch specific exceptions (e.g. IOErrors in the write csv, or SenseHat specific exceptions
• Log exceptions (where needed) and return different defaults in cases of error
• Refactor the write to - as you suggested - log the data in memory and only write every 10th entry to the csv. Attention: If you only log every 10th or even every 100th data entry and the python interpreter crashes, the recently logged data will be lost
• Don't write the csv headers in code, but manually prepare the csv files and put them next to the script
• Use a sqlite database and log the data here instead of in CSVs

In order to figure out where to start with the optimizations, you can now profile the helper methods (write_data_to_file, get_temperature and the other get_... methods) and derive appropriate measurements to take.

PS. Fair warning: I never executed the code in a python shell, so it may not be free from bugs :see_no_evil:.

• In particular, batteries tend to perform poorly when cold, and need to be tested specifically for that. – 200_success Feb 24 '19 at 20:01
• @200_success I haven't thought about that but we will put the battery in a styrofoam box to insulate it from the outside. – Lexu Feb 25 '19 at 15:09
• @Kim Thank you very much for your detailed answer. I would have done something like this if I knew how to do it. You certainly helped a lot (For our project and for my python skills.) – Lexu Feb 25 '19 at 15:11
• Do not call main recursively. You are setting yourself up for stack overflow. Consider instead

def main():
while True:
try:
your_logic_here
except Exception as e:
your_logging_here

• Testing for counter == 4 is better done in a loop:

    for _ in range(4):
handle_acceleration
handle_the_rest

• An unattended controller should handle exceptions more diligently. For sure, you want to act differently for the exceptions raised by sense (if any) vs exceptions raised by writing to the file.

• Regarding battery, avoid binary-to-text conversions. Store your data as binary, and convert them to CSV offline, after the Pi safely returns.

• > Testing for counter == 4 is better done in a loop Why exactly is this true? It'll save 4 check statements? Wouldn't that equate to a while inside a while now? – insidesin Feb 25 '19 at 7:34
• @insidesin Few reasons. 1) Looks better 2) Less total indentation 3) Expresses intention better 4) One failing test versus 3 per four iterations – vnp Feb 25 '19 at 7:40
• Those are all subjective, it also increases indentation if I can see correctly. If the testing is improved then it would make sense. – insidesin Feb 25 '19 at 8:00
• @vnp I also agree with 3), plus it also assures the execution of handle_the_reast, after 4 acceleration logs (not regarding possible exceptions). Can you explain what you mean with 4)? Are you talking about unit tests or do you mean the equality checks? – Kim Feb 25 '19 at 8:25

Opening and closing files takes resources:

with open('babar.txt', 'a') as f: f.write('a'*10000)


takes 300 micro-seconds while:

for _ in range(10000):
with open('babar.txt', 'a') as f: f.write('a')


takes 648000 micro-seconds

So to answer your question Would it be beneficial to write 10 rows at once instead of writing one row at a time?. The answer, as always is YES, but...

You shouldn't implement a buffer yourself instead use the third argument of open:

f = open('babar.txt', 'a', 500)
for _ in range(10000):
f.write('a')
f.close()
# takes 2200 micro-seconds for a 500 buffer
# and 3660 micro-seconds for a 50 buffer


It is the buffer-size (4096 chars by default I think). Put the close() in a finally block to avoid corruption of your files.

I think less opening and closing would take a lot less resources but implementing a buffer yourself is less safe then letting the built-in function handle it for you. Beware of the risks you take, not writing data mean your data is lost if power goes down, and as you can see dividing the buffer by 10 doesn't necessarily mean it takes 10x more resources.

note: battery consumption is hard to measure and is not directly related to cpu time.

For a system that will be in stratosphere, you dont need any color or light mechanism. Get rid of all code about visualisation, e.g. Setting colors or setting light value. That way light also wont consume your battery. Once your code is cleared, apply other answers.

• Absolutely. And do the same with the hardware. Go for the bare minimum, there are stripped versions of the Raspberry Pi available. – Mast Feb 24 '19 at 20:34
• Thanks for the input. I'll reduce it but probably still keep some small visualisation, as the last thing I want to happen is that the balloon takes off with the raspberry not meassuring anything and us not realizing this. – Lexu Feb 25 '19 at 15:07