# Averaging a signal to remove noise with Python

I am working on a small project in the lab with an Arduino Mega 2560 board. I want to average the signal (voltage) of the positive-slope portion (rise) of a triangle wave to try to remove as much noise as possible. My frequency is 20Hz and I am working with a data rate of 115200 bits/second (fastest recommended by Arduino for data transfer to a computer).

The raw signal looks like this:

My data is stored in a text file, with each line corresponding to a data point. Since I do have thousands of data points, I expect that some averaging would smooth the way my signal looks and make a close-to-perfect straight line in this case. However, other experimental conditions might lead to a signal where I could have features along the positive-slope portion of the triangle wave, such as a negative peak, and I absolutely do need to be able to see this feature on my averaged signal.

I am a Python beginner so I might not have the ideal approach to do so and my code might look bad for most of you. I would still like to get your hints / ideas on how I can improve my signal processing code to achieve a better noise removal by averaging the signal.

#!/usr/bin/python
import matplotlib.pyplot as plt
import math

# *** OPEN AND PLOT THE RAW DATA ***
data_filename = "My_File_Name"
filepath = "My_File_Path" + data_filename + ".txt"

# Open the Raw Data
with open(filepath, "r") as f:

# Remove the \n
rawdata = map(lambda s: s.strip(), rawdata)

# Plot the Raw Data
plt.plot(rawdata, 'r-')
plt.ylabel('Lightpower (V)')
plt.show()

# *** FIND THE LOCAL MAXIMUM AND MINIMUM
# Number of data points for each range
datarange = 15 # This number can be changed for better processing
max_i_range = int(math.floor(len(rawdata)/datarange))-3

#Declare an empty lists for the max and min
min_list = []
max_list = []

min_list_index = []
max_list_index = []

i=0

for i in range(0, max_i_range):
delimiter0 = i * datarange
delimiter1 = (i+1) * datarange
delimiter2 = (i+2) * datarange
delimiter3 = (i+3) * datarange

sumrange1 = sum(float(rawdata[i]) for i in range(delimiter0, delimiter1 + 1))
averagerange1 = sumrange1 / len(rawdata[delimiter0:delimiter1])
sumrange2 = sum(float(rawdata[i]) for i in range(delimiter1, delimiter2 + 1))
averagerange2 = sumrange2 / len(rawdata[delimiter1:delimiter2])
sumrange3 = sum(float(rawdata[i]) for i in range(delimiter2, delimiter3 + 1))
averagerange3 = sumrange3 / len(rawdata[delimiter2:delimiter3])

# Find if there is a minimum in range 2
if ((averagerange1 > averagerange2) and (averagerange2 < averagerange3)):
min_list.append(min(rawdata[delimiter1:delimiter2])) # Find the value of all the minimum

#Find the index of the minimum
min_index = delimiter1 + [k for k, j in enumerate(rawdata[delimiter1:delimiter2]) if j == min(rawdata[delimiter1:delimiter2])][0] # [0] To use the first index out of the possible values
min_list_index.append(min_index)

# Find if there is a maximum in range 2
if ((averagerange1 < averagerange2) and (averagerange2 > averagerange3)):
max_list.append(max(rawdata[delimiter1:delimiter2])) # Find the value of all the maximum

#Find the index of the maximum
max_index = delimiter1 + [k for k, j in enumerate(rawdata[delimiter1:delimiter2]) if j == max(rawdata[delimiter1:delimiter2])][0] # [0] To use the first index out of the possible values
max_list_index.append(max_index)

# *** PROCESS EACH RISE PATTERN ***

# One rise pattern goes from a min to a max
numb_of_rise_pattern = 50 # This number can be increased or lowered. This will average 50 rise patterns

max_min_diff_total = 0

for i in range(0, numb_of_rise_pattern):
max_min_diff_total = max_min_diff_total + (max_list_index[i]-min_list_index[i])

# Find the average number of points for each rise pattern
max_min_diff_avg = abs(max_min_diff_total / numb_of_rise_pattern)

# Find the average values for each of the rise pattern
avg_position_value_list = []

for i in range(0, max_min_diff_avg):

sum_position_value = 0
for j in range(0, numb_of_rise_pattern):

sum_position_value = sum_position_value + float( rawdata[ min_list_index[j] + i ] )
avg_position_value = sum_position_value / numb_of_rise_pattern

avg_position_value_list.append(avg_position_value)

#Plot the Processed Signal
plt.plot(avg_position_value_list, 'r-')
plt.title(data_filename)
plt.ylabel('Lightpower (V)')
plt.show()


At the end, the processed signal looks like this:

I would expect a straighter line, but I could be wrong. I believe that there are probably a lot of flaws in my code and there would certainly be better ways to achieve what I want.

I have included a link to a text file with some raw data if any of you want to have fun with it.

• Welcome to Code Review! Great job on your first question! Feb 23 '16 at 22:03
• Wait, why should the processed signal look like a straight line with a positive slope? I'm not sure what this algorithm is trying to do I guess. Feb 23 '16 at 22:18
• That file-sharing site's popups are obnoxious. Feb 23 '16 at 22:22
• The signal sent is a triangle wave. I am trying to extract only one of the two portion of the triangle wave (the one with the positive slope) and average it to remove as much noise as possible. In the above code, I take 50 different triangle profiles and average the portion with the positive slope to obtain what should look like a line with a positive slope with the least noise as possible. Sorry if it's not clear, I didn't mean a line with 0 slope (straight line). Feb 23 '16 at 22:23
• Can you add some more context about the use-case for this? It isn't clear to me whether there's some contextual reason for you to average multiple waveforms rather than low-pass filtering your samples. Feb 23 '16 at 22:42

As David Morris indicates, it might be simpler to use a filtering/smoothing function, such as a moving window average. This is pretty simple to implement using the rolling function from pandas.Series. (Only 501 points are shown.) Tweak the numerical argument (window size) to get different amounts of smoothing.

import pandas as pd
import matplotlib.pyplot as plt

# Plot the Raw Data
ts = rawdata[0:500]
plt.plot(ts, 'r-')
plt.ylabel('Lightpower (V)')

# original version
# smooth_data = pd.rolling_mean(ts,5).plot(style='k')
# pandas has changed; so, here's the new version:
smooth_data = pd.Series(ts).rolling(window=7).mean().plot(style='k')
plt.show()


(Moving average)

A moving average is, basically, a low-pass filter. So, we could also implement a low-pass filter with functions from SciPy as follows:

import scipy.signal as signal

# First, design the Butterworth filter
N  = 3    # Filter order
Wn = 0.1 # Cutoff frequency
B, A = signal.butter(N, Wn, output='ba')
smooth_data = signal.filtfilt(B,A, rawdata[0:500])
plt.plot(rawdata[0:500],'r-')
plt.plot(smooth_data[0:500],'b-')
plt.show()


(Low-Pass Filter)

This filter method is from OceanPython.org BTW.

• thank you for this answer, I was going to try the code, but what is the ts in the first code ? Feb 13 '20 at 9:05
• Thanks for the note. I intended ts to be a subset of the raw data. I've modified my answer to make this clear. BTW, as the comments show, the pandas module has changed from rolling_mean to rolling as an attribute of pandas.Series. I've fixed this. Feb 18 '20 at 1:27
• Convert data to float once and forever:

with open(filepath, "r") as f:
rawdata = [float(line.strip()) for line in f.readlines()]

• len(rawdata[delimiter0:delimiter1]) equals to delimiter1 - delimiter0. I suppose the latter is somewhat cleaner.

• Detecting slopes is questionable. It seems that it introduces a phase shift. I recommend to add a second step: since you know it must be a straight line, min-square interpolate what you found at the first step.

In any case it should be factored out into a function.

• Averaging slopes is very questionable. Each slope may have different number of samples, and you account for that by computing an average number. This means that shorter slopes will contribute data which actually belong to the falling slope, and longer slopes do not contribute their trailing parts. You may want to resample them.

I also suspect that the slopes not being aligned correctly (see above) contributes a lot.