3
\$\begingroup\$

I went through a series of lessons in data science using pandas and numpy.

I have attempted to replicate some of the more common algorithms based on maths forumlas and psuedocode, except for the ECDF method which was the first one in the course.

As far as i can tell they are correct and I have put some up against numpy functions in an effort to assert my calculations are correct.

EDIT:

This is not an application I am writing to reinvent the wheel.. It is merely an exercise in coding and exploration of what is under the hood

This will maybe explain the use of words like sum in the methods..

I put it up to review as confirmation that I have:

a. coded the algorithms effectively and pythonic. b. that they are providing the correct results.

x = [41, 19, 23, 40, 55, 57, 33]
y = [60, 61, 71, 78, 76, 82, 945]

import numpy as np


class MafStats:

    # ecdf
    def ecdf(self, data):
        """Compute ECDF for a one-dimensional array of measurements."""
        # Number of data points: n
        n = len(data)
        # x-data for the ECDF: x
        x = np.sort(data)
        # y-data for the ECDF: y
        y = np.arange(1, n + 1) / n
        return x, y

    # simple square root function
    def square_root(self, data):
        """ A function to calculate the Square root of a number"""
        return data ** (.5)

    # calculate the number of items int he array
    def sum(self, list):
        """ function that calculates the sum of an array"""
        total_sum = 0
        for i in list:
            total_sum += i
        return float(total_sum)

    # calculate the mean average
    def mean(self, array):
        """ a function to calculate the mean average of an array"""
        n = float(len(array))
        return self.sum(array) / n

    # for fun calculate hte medain average
    def median(self, array):
        """ function that finds the median average of an array"""
        # https://www.mathsisfun.com/median.html
        n = len(array)
        if n < 1:
            return None
        if n % 2 == 1:
            # for an odd array, return the middle number
            return sorted(array)[n // 2]
        else:
            return float(sum(sorted(array)[self.mean(array)]) / 2.0)

    # calulate the variance
    def variance(self, array):
        """ Function to calculate the Variance of data 
        by calculating the average of the squared differnces from the mean  """
        n = float(len(array))
        total_sum = 0
        for i in array:
            total_sum += ((self.mean(array) - i) ** 2) / n
        return total_sum

    # calculate standard deviation
    def std_deviation(self, array):
        """ A function to calculate the Standard Deviation of Data 
        by caclulating the Square Root of The Variance"""
        return self.square_root(self.variance(array))

    # calculate the Covariance of 2 arrays
    def covariance(self, array1, array2):
        """ function that calculates the Covariance  -
        Covariance measures how two variables move together. 
        It measures whether the two move in the same direction (a positive covariance) or 
        in opposite directions (a negative covariance).
        """
        assert float(len(array1)) == float(len(array2))
        n = float(len(array1))
        sum_array = 0

        for (x, y) in zip(array1, array2):
            sum_array += (x - mean_result_x) * (y - mean_result_y)

        return sum_array / (n - 1)

    # correlation co-efficient
    def correlation(self, array1, array2):
        """ a function to determ,ine the correlation co-efficient between 2 sets of data """
        return  covar_result / (std_dev_x * std_dev_y)

    # Pearson Coefficient
    def pearson_correlation(self, array1, array2):
        """Calculcates the Pearson Coefficient """
        n = len(array1)
        x_times_y = 0
        x_sq = 0
        y_sq = 0
        for i in range(n):
            x_min_mean = array1[i] - self.mean(array1)
            y_min_mean = array2[i] - self.mean(array2)
            x_times_y += x_min_mean * y_min_mean
            x_sq += x_min_mean ** 2
            y_sq += y_min_mean ** 2

        return x_times_y / self.square_root(x_sq * y_sq)

    # Least square regression
    def least_square_regression(self, x, y, var):
        """ 
        Least square regression is a method for finding a line that summarizes the relationship between the two variables, at least within the domain of the explanatory variable x.
        calculate the least square regression line equation with the given x and y values. 
        """
        # Count the number of given x values.
        n = len(x)

        # Find XY, X2 for the given values.
        def xy_x2(x, y):

            xx = [i ** 2 for i, j in zip(x, y)]
            xy = [i * j for i, j in zip(x, y)]
            return xx, xy

        xx, xy = xy_x2(x, y)

        # Find ∑X, ∑Y, ∑XY, ∑X2 for the values
        ex = sum(x)
        ey = sum(y)
        exy = sum(xy)
        ex2 = sum(xx)

        # Slope Formula
        # Slope(b) = (N∑XY - (∑X)(∑Y)) / (N∑X2 - (∑X)2)
        b = ((n) * (exy) - (ex) * (ey)) / ((n) * (ex2) - (ex) ** 2)

        # intercept formula
        # Intercept(a) = (∑Y - b(∑X)) / N
        a = (ey - b * ex) / n

        # Regression Equation(y) = a + b
        return a + (b * var)

    # factorial
    def factorial(self, n):
        if n == 0:
            return 1
        else:
            return n * self.factorial(n - 1)

    def poisson(self, events, interval):
        """
        In probability theory, the Poisson distribution is a very common discrete probability distribution.  A Poisson distribution helps in describing the chances of occurrence of a number of events in some given time interval or given space conditionally that the value of average number of occurrence of the event is known. This is a major and only condition of Poisson distribution.

        1. The experiment results in outcomes that can be classified as successes or failures.
        2. The average number of successes (μ) that occurs in a specified region is known.
        3. The probability that a success will occur is proportional to the size of the region.
        4. The probability that a success will occur in an extremely small region is virtually zero.

        """
        # base value of the system of natural logarithm
        e = 2.71828459

        # The mean number of successes - Average Rate of Success.
        u = events

        # The actual number of successes that occur - Poisson Random Variable
        x = interval

        x1 = self.factorial(x)

        # The Poisson probability that exactly x successes occur in a Poisson experiment, when the mean number of successes is μ.
        return ((e ** -u) * (u ** x)) / x1

    def cul_poisson(self, events, list):
        """
        In probability theory, the Poisson distribution is a very common discrete probability distribution.  A Poisson distribution helps in describing the chances of occurrence of a number of events in some given time interval or given space conditionally that the value of average number of occurrence of the event is known. This is a major and only condition of Poisson distribution.

        1. The experiment results in outcomes that can be classified as successes or failures.
        2. The average number of successes (μ) that occurs in a specified region is known.
        3. The probability that a success will occur is proportional to the size of the region.
        4. The probability that a success will occur in an extremely small region is virtually zero.

        """
        # base value of the system of natural logarithm
        e = 2.71828459

        # The mean number of successes that occur in a specified region.
        uc = events

        # A list of the actual number of successes that occur in a specified region
        xl = range(list)

        cul = 0

        for i in xl:
            r = self.poisson(uc, i)
            cul += r

        return cul


ms = MafStats()

# ECDF
print("The ECDF of X is: \n", ms.ecdf(x), "\n")
print("The ECDF of Y is:\n", ms.ecdf(y), "\n")

# SQUAER ROOT
num_in = 25
print("The Square root of {} is:".format(num_in), ms.square_root(num_in))

# SUM OF ARRAY
return_sum_x = ms.sum(x)
print("\n\nThe Sum of X is:", return_sum_x)
return_sum_y = ms.sum(y)
print("The Sum of Y is:", return_sum_y)

# MEAN AVERAGE
mean_result_x = ms.mean(x)
print("The Mean Average of  X is:", mean_result_x)
print(np.mean(x))
mean_result_y = ms.mean(y)
print("The Mean Average of Y is:", mean_result_y)
print(np.mean(y))


# MEDIAN AVERAGE
median_res = ms.median(x)
print("The Median of  X is:", median_res)
print(np.median(x))
median_res1 = ms.median(y)
print("The Median of Y is:", median_res1)
print(np.median(y))

# VARIANCE
var_result_x = ms.variance(x)
print("The Variance of  X is:", var_result_x)
var_result_y = ms.variance(y)
print("The Variance of Y is:", var_result_y)

# Standard Deviation
std_dev_x = ms.std_deviation(x)
print("The Standard Deviation Of Data  X is:", std_dev_x)
std_dev_y = ms.std_deviation(y)
print("The Standard Deviation Of Data Y is:", std_dev_y, "\n")

# calculate the Covariance of 2 arrays
covar_result = ms.covariance(x, y)

print("The Covariance of X and Y is:", covar_result, "\n")

# standard correlation
# standard correlation

corr_result = ms.correlation(x, y)
print("The Correlation Coefficient between  X and Y is:", corr_result, "\n\n")

# Pearson Coefficient
print("The Pearson Correlation Coefficient is:", ms.pearson_correlation(x, y))

# Least Squares Regression
print(
    "THe Least Square Regression Line Equation of X and Y is: ",
    ms.least_square_regression(x, y, 64),
)


# factorial
print(ms.factorial(2))

# Poisson distributoin
result = ms.poisson(5, 2)
print(result)


r_cul = ms.cul_poisson(5, 4)
print("Poisson Distribution is: ", r_cul)

# Test With Numpy functions..
import numpy as np
print("\nnumpy .cov function: \n", np.cov(x, y))
print("\nnumpy .corrcoef function: \n", np.corrcoef(x, y))
\$\endgroup\$
1
\$\begingroup\$

There is no state being kept in the MafStats object, so it is nothing more than a namespace with an awkward usage. These should all just be functions in a mafstats module.

You're obviously deliberately reinventing the wheel here, since numpy.var() and other statistical functions already exist in NumPy and SciPy. The question is, how far do you go with writing code the hard way? sum() is a built-in function; ** .5 is already quite compact and efficient; math.sqrt() and math.factorial() already exist. Clearly, the built-in sum() function is the best way to calculate a sum in Python, so I'm not sure what we're reviewing here. I can say, though, that float(total_sum) has a pointless cast to float and a self-redundant variable name.

\$\endgroup\$
  • \$\begingroup\$ I started writing a class but now I see it is not useful. It is an exercise in understanding how the code works, not reinventing anything. I would still use sum, numpy et al in everything other than this..I will update my question to make this more clear. \$\endgroup\$ – johnashu Sep 19 '18 at 10:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.