9
\$\begingroup\$

I am working on a small project in the lab with an Arduino Mega 2560 board. I want to average the signal (voltage) of the positive-slope portion (rise) of a triangle wave to try to remove as much noise as possible. My frequency is 20Hz and I am working with a data rate of 115200 bits/second (fastest recommended by Arduino for data transfer to a computer).

The raw signal looks like this:

Raw Signal

Raw Signal Zoom

Raw Signal Zoom Plus

My data is stored in a text file, with each line corresponding to a data point. Since I do have thousands of data points, I expect that some averaging would smooth the way my signal looks and make a close-to-perfect straight line in this case. However, other experimental conditions might lead to a signal where I could have features along the positive-slope portion of the triangle wave, such as a negative peak, and I absolutely do need to be able to see this feature on my averaged signal.

I am a Python beginner so I might not have the ideal approach to do so and my code might look bad for most of you. I would still like to get your hints / ideas on how I can improve my signal processing code to achieve a better noise removal by averaging the signal.

#!/usr/bin/python
import matplotlib.pyplot as plt
import math

# *** OPEN AND PLOT THE RAW DATA ***
data_filename = "My_File_Name"
filepath = "My_File_Path" + data_filename + ".txt"

# Open the Raw Data
with open(filepath, "r") as f:
    rawdata = f.readlines()

# Remove the \n     
rawdata = map(lambda s: s.strip(), rawdata)

# Plot the Raw Data
plt.plot(rawdata, 'r-')
plt.ylabel('Lightpower (V)')
plt.show()

# *** FIND THE LOCAL MAXIMUM AND MINIMUM
# Number of data points for each range 
datarange = 15 # This number can be changed for better processing
max_i_range = int(math.floor(len(rawdata)/datarange))-3

#Declare an empty lists for the max and min
min_list = []
max_list = []

min_list_index = []
max_list_index = []

i=0

for i in range(0, max_i_range):
    delimiter0 = i * datarange
    delimiter1 = (i+1) * datarange
    delimiter2 = (i+2) * datarange
    delimiter3 = (i+3) * datarange

    sumrange1 = sum(float(rawdata[i]) for i in range(delimiter0, delimiter1 + 1))
    averagerange1 = sumrange1 / len(rawdata[delimiter0:delimiter1])
    sumrange2 = sum(float(rawdata[i]) for i in range(delimiter1, delimiter2 + 1))
    averagerange2 = sumrange2 / len(rawdata[delimiter1:delimiter2])
    sumrange3 = sum(float(rawdata[i]) for i in range(delimiter2, delimiter3 + 1))
    averagerange3 = sumrange3 / len(rawdata[delimiter2:delimiter3])

    # Find if there is a minimum in range 2
    if ((averagerange1 > averagerange2) and (averagerange2 < averagerange3)):
        min_list.append(min(rawdata[delimiter1:delimiter2])) # Find the value of all the minimum 

        #Find the index of the minimum
        min_index = delimiter1 + [k for k, j in enumerate(rawdata[delimiter1:delimiter2]) if j == min(rawdata[delimiter1:delimiter2])][0] # [0] To use the first index out of the possible values
        min_list_index.append(min_index)


    # Find if there is a maximum in range 2
    if ((averagerange1 < averagerange2) and (averagerange2 > averagerange3)):
        max_list.append(max(rawdata[delimiter1:delimiter2])) # Find the value of all the maximum

        #Find the index of the maximum
        max_index = delimiter1 + [k for k, j in enumerate(rawdata[delimiter1:delimiter2]) if j == max(rawdata[delimiter1:delimiter2])][0] # [0] To use the first index out of the possible values
        max_list_index.append(max_index)



# *** PROCESS EACH RISE PATTERN ***

# One rise pattern goes from a min to a max
numb_of_rise_pattern = 50 # This number can be increased or lowered. This will average 50 rise patterns

max_min_diff_total = 0

for i in range(0, numb_of_rise_pattern):
    max_min_diff_total = max_min_diff_total + (max_list_index[i]-min_list_index[i])

# Find the average number of points for each rise pattern
max_min_diff_avg = abs(max_min_diff_total / numb_of_rise_pattern)

# Find the average values for each of the rise pattern
avg_position_value_list = []


for i in range(0, max_min_diff_avg):

    sum_position_value = 0
    for j in range(0, numb_of_rise_pattern):

        sum_position_value = sum_position_value + float( rawdata[ min_list_index[j] + i ] )
    avg_position_value = sum_position_value / numb_of_rise_pattern

    avg_position_value_list.append(avg_position_value)


#Plot the Processed Signal
plt.plot(avg_position_value_list, 'r-')
plt.title(data_filename)
plt.ylabel('Lightpower (V)')
plt.show()

At the end, the processed signal looks like this:

Averaged Signal

I would expect a straighter line, but I could be wrong. I believe that there are probably a lot of flaws in my code and there would certainly be better ways to achieve what I want.

I have included a link to a text file with some raw data if any of you want to have fun with it.

\$\endgroup\$
  • \$\begingroup\$ Welcome to Code Review! Great job on your first question! \$\endgroup\$ – SirPython Feb 23 '16 at 22:03
  • 1
    \$\begingroup\$ Wait, why should the processed signal look like a straight line with a positive slope? I'm not sure what this algorithm is trying to do I guess. \$\endgroup\$ – Jared Goguen Feb 23 '16 at 22:18
  • \$\begingroup\$ That file-sharing site's popups are obnoxious. \$\endgroup\$ – 200_success Feb 23 '16 at 22:22
  • \$\begingroup\$ The signal sent is a triangle wave. I am trying to extract only one of the two portion of the triangle wave (the one with the positive slope) and average it to remove as much noise as possible. In the above code, I take 50 different triangle profiles and average the portion with the positive slope to obtain what should look like a line with a positive slope with the least noise as possible. Sorry if it's not clear, I didn't mean a line with 0 slope (straight line). \$\endgroup\$ – LaGuille Feb 23 '16 at 22:23
  • \$\begingroup\$ Can you add some more context about the use-case for this? It isn't clear to me whether there's some contextual reason for you to average multiple waveforms rather than low-pass filtering your samples. \$\endgroup\$ – David Morris Feb 23 '16 at 22:42
5
\$\begingroup\$

As David Morris indicates, it might be simpler to use a filtering/smoothing function, such as a moving window average. This is pretty simple to implement using the rolling_mean function from pandas. (Only 501 points are shown.) Tweak the numerical argument (window size) to get different amounts of smoothing.

import pandas as pd
import matplotlib.pyplot as plt

# Plot the Raw Data
plt.plot(rawdata[0:500], 'r-')
plt.ylabel('Lightpower (V)')

smooth_data = pd.rolling_mean(ts,5).plot(style='k')
plt.show()

enter image description here (Moving average)

A moving average is, basically, a low-pass filter. So, we could also implement a low-pass filter with functions from SciPy as follows:

import scipy.signal as signal

# First, design the Buterworth filter
N  = 3    # Filter order
Wn = 0.1 # Cutoff frequency
B, A = signal.butter(N, Wn, output='ba')
smooth_data = signal.filtfilt(B,A, rawdata[0:500])
plt.plot(rawdata[0:500],'r-')
plt.plot(smooth_data[0:500],'b-')
plt.show()

enter image description here (Low-Pass Filter)

This filter method is from OceanPython.org BTW.

\$\endgroup\$
3
\$\begingroup\$
  • Convert data to float once and forever:

    with open(filepath, "r") as f:
        rawdata = [float(line.strip()) for line in f.readlines()]
    
  • len(rawdata[delimiter0:delimiter1]) equals to delimiter1 - delimiter0. I suppose the latter is somewhat cleaner.

  • Detecting slopes is questionable. It seems that it introduces a phase shift. I recommend to add a second step: since you know it must be a straight line, min-square interpolate what you found at the first step.

    In any case it should be factored out into a function.

  • Averaging slopes is very questionable. Each slope may have different number of samples, and you account for that by computing an average number. This means that shorter slopes will contribute data which actually belong to the falling slope, and longer slopes do not contribute their trailing parts. You may want to resample them.

    I also suspect that the slopes not being aligned correctly (see above) contributes a lot.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.