Python program to calculate statistics for laboratory data

Question

I am new in Python, but I have laboratory work with some routine computing, so I have decided do these calculations in Python.

Program input is physical characteristic, measured several times (i.e list of floats). The program calculates average value and square average deviation for the list. After this, it makes table of intermediate calculations (I need to record this table in the laboratory work) and prints it. Then, using square average deviations, the program checks for data that are suspect for a measurement error. If wrong data exist, the programm removes it and recalculate and reprint average value, square average deviation and table. After this, the program calculates and prints random and relative measurement error and also prints answer.

The program works, but I am worrying about its design and structure.

• Is it a common practice for Python programmers to first place import statements, then some functions and then main part of program? Is there a more readable program code structure?
• Are multi-line comments (i.e. """ """ ) a standard of de facto Python programming or are there are common standards for multi-line commenting?
• Is there a common code style for Python community for function's comments?

Code:

#!/usr/bin/env python3
import sys
import math
import decimal
from prettytable import PrettyTable
from functools import reduce
from collections import Counter

ctx = decimal.Context()
ctx.prec = 20
def toFixedStr(f):
    d1 = ctx.create_decimal(repr(f))
    return format(d1, 'f')

# Round only significant digits
def roundSig(x, sig=2):
    if x != 0.0:
        return round(x, sig-int(math.floor(math.log10(abs(x))))-1)
    else:
        return x


# Round, enough for laboratory work
def enoughRound(x):
    return roundSig(x,3)

def studentCoefficient(data):
    # Student coefficients for alpha=0.9 (from 2 to 11 experiments)
    coefs = [2.92, 2.35, 2.13, 2.02, 1.94, 1.89, 1.86, 1.83, 1.81, 1.80]
    infinityCoeff = 1.60
    length = len(data)
    if length >= 3:
        if length < 13:
            return coefs[length-3]
        else:
            return infinityCoeff
    else:
        raise RuntimeWarning("There aren't Student coefficient for {} experiments!".format(length))
    return -1

# -------------------------------------------------------------
# Average value for experiments's data
# -------------------------------------------------------------
def averangeValueOf(data):
    result = 0.0
    for elem in data:
        result += elem
    result = result / len(data)
    return enoughRound(result)

#-------------------------------------------------------------
# square average deviation value for experiments's data
#-------------------------------------------------------------
def meanSquareDeviationOf(data):
    averange = averangeValueOf(data)
    result = reduce(lambda acc, x: acc+((averange-x)**2), data, 0.0)
    result = math.sqrt(result / len(data) / (len(data)-1))
    return enoughRound(result)

#-------------------------------------------------------------
# Create calculation table
#-------------------------------------------------------------
def makeCalculationTable(data):
    averange = averangeValueOf(data)

    table = PrettyTable(['N', 'x', '|<x> - x_i|','(<x> - x_i)^2'])
    for i,x in enumerate(data):
        table.add_row([
            i,
            toFixedStr(x),
            toFixedStr(abs(enoughRound(x-averange))),
            toFixedStr(enoughRound((x-averange)**2))
            ])

    squareDeviation = meanSquareDeviationOf(calcs)
    return averange, squareDeviation, table

#-------------------------------------------------------------
# Check data for 'missings' and return all 'missing' value
#-------------------------------------------------------------
def missCheck(data):
    averange = averangeValueOf(data)
    squareDeviation = meanSquareDeviationOf(data)
    # If |<x> - x_i| > 3 * S_<x> * sqrt(N) then x_i is 'missing'
    missCheckFunc = lambda x: abs(averange - x) > 3 * squareDeviation * math.sqrt(len(data))
    result = []
    for i, x in enumerate(data):
        if missCheckFunc(x):
            result.append(i)
    return result

#-------------------------------------------------------------
# Calculate random Error and relative Error
#-------------------------------------------------------------
def randomError(data):
    return enoughRound(meanSquareDeviationOf(data)*studentCoefficient(data))

def relativeError(data):
    # in percents
    return round(randomError(data) / averangeValueOf(data) * 100, 1)

#-------------------------------------------------------------
# Main program body
#-------------------------------------------------------------
try:
    calcs = list(map(float, sys.argv[1:]))
except ValueError as err:
    print("One of input values isn't number. Correct a mistake")
    print(err)
    exit()

print("Input data")
print(calcs)
print("")

averange, deviation, table = makeCalculationTable(calcs)
print("<x> = {} у.е".format(averange))
print("S_<x> = {} у.е".format(deviation))
print("")

print("Calculation table")
print(table)
print("")

# Checking for 'missings'
missing = missCheck(calcs)
if len(missing) != 0:
    print("'Missings' found")
    table = PrettyTable(['№', 'x'])
    for index in missing:
        table.add_row([index+1, calcs[index]])
    print(table)
    print("")

    # Remove found 'missings'
    for index in missing:
        del calcs[index]

    averange, deviation, table = makeCalculationTable(calcs)
    print("New <x> = {} у.е ".format(averange))
    print("New S_<x> = {} у.е".format(deviation))
    print("")

    print("New calculation table")
    print(table)
else:
    print("Without 'missings'")
print("")


randErr = randomError(calcs)
print("Random error = {}".format(randErr))
print("")

relErr = relativeError(calcs)
if relErr < 10:
    print("Relative error = {}%".format(relErr))
else:
    print("Too big relative error ({}%)! Check your input data!".format(relErr))
print("")

averange = averangeValueOf(calcs)

print("Answer")
print("x = {} ± {} у.е".format(averange,randErr))
print("")

print("Confidence interval")
print("[{0}-{1}; {0}+{1}]".format(averange,randErr))

Answer 1

Your questions are all answered in the PEP-8 style guide.

• Yes, imports come first.
• You are doing "multi-line comments" correctly
• PEP-0257 outlines function docstring conventions - in short you should use a """multi-line comment styled docstring""" inside your function declaration.

After taking a quick pass over your code I have some additional thoughts (and personal opinions):

1. I feel you are under-utilizing built-in functions like avg, map, and reduce. (missCheck would be a good place to use map).
2. I like camelCase (aka mixedCase), but PEP-8 says function names and variables should be lower_case_with_underscores.
3. consider using the logging module rather than print.
4. rather than printing an error and then exiting, consider raising an exception (you can use one of the built-in exceptions if you don't want to implement a custom one).

Answer 2

will try to answer some of your concerns regarding the code

Function comments

First will start for the function comments. Usually, you would like to have a docstring (this is the comment section of a function) which describes what the function does, how it does it (if necessary) and some info about parameters and return values

Applying this to one of your functions:

# Round only significant digits
def roundSig(x, sig=2):

The result will be the following

def roundSig(x, sig=2):
    """
    Round only significant digits
    :param x: number to round
    :param sig: significant digits to use
    :return: number after performing rounding
    """

This will be of great help when someone is approaching your code and has to understand what your function is doing. Or when you do the same in 3 months :)

Improving readability

Something that should be a concern, when your program starts to grow, is to keep it readable. For that, apart from some conventions which @7yl4r has mentioned, you can use small tricks to help make code more understandable

One of the easiest ways to improve readability is with accurate naming. If you approach a function called roundSig(x, sig=2) it will be hard to understand what it does. Also, in Python you try to write functions using underscore to separate words on its definition, so first step will be to change naming to round_sig.

But we are just warming up

Let's try to think what the function is doing: rounding a number. It accepts two parameters: the number, and how many significant digits we want to take into account. That should be enough tools to rewrite the function naming into something more legible

def round_number(number, significant_digits=2):
    """
    Rounds number using only significant_digits
    ...

Magic numbers

In programming, sometimes a function makes some calculations and not others, based on a condition. If this condition evaluates a number which can seem arbitrary if you have no knowledge of the application, we can say this number is magic

In your code, it appears in some functions there are some magic numbers, and decisions are taken that are hard to understand just reading the code

Take this example in studentCoefficient(data):

length = len(data)
if length >= 3:
    if length < 13:
        ...

Seems the numbers 3 and 13 are there but there is no insight as why they appear, and not 20, or 100 or whatever else

So would be good to introduce this numbers into a variable, and then use it for the conditional checks. min_experiments = 3 and max_experiments = 13 could work, though you may choose better naming

Logging

As it was mentioned in the previous comment, would be very interesting to use some logging tool for your program, since you can not only display the results in console, but at the same time store them in a file for future checks

Code organisation

If you plan to run the program as a script, you may want to include a main clause in the body of the file, where you will group the execution steps together. Which in this case is all that happens below the Main program body comment.

Bit of more info about it here https://stackoverflow.com/questions/419163/what-does-if-name-main-do

Once you are done with it, you will discover the amount of things happening in the body is huge, so will be interesting to extract parts of it into smaller functions, and better separate the functionality you are maintaining.

For example, everything that happens below # Checking for 'missings' can be extracted into a function called check_for_missings(calcs) or something with better naming you can think of

In general there is many room for improvements, so I encourage you to follow up on the comments and try to work on your code a bit more

Enjoy coding! :)

Answer 3

If you are doing many numeric calculations in python (and doing statistics definitely counts as this), you should probably learn about numpy. it is a Python module that implements arrays and fast vectorized operations on them.

You can convert a list (or actually, any iterable) to a numpy array with:

import numpy as np
data = np.array([1,2,0.5])

Your two statistics functions are way easier to implement:

def average(data):
    return data.mean()

def mean_square_deviation(data):
    mu = data.mean()
    result = ((data - mu)**2).sum()
    return enough_round(np.sqrt(result) / len(data))

Note that you normally use either \$1/N\$ or \$1/(N-1)\$, not a mix between the two. Note also that this is not actually the mean square (since you take the square-root afterwards). This is actually just the standard deviation, which you can get with:

def std(data):
    return data.std()