T-test Independence for a Pandas DataFrame

Question

I want to calculate the scipy.stats.ttest_ind() for numeric columns in a pandas DataFrame with the binary target variable.

import pandas as pd
from scipy import stats

def calculate_tStatistic(df, target, numeric_cols):

    """
    Calculate the t-test on TWO RELATED samples of scores, a and b.

    This is a two-sided test for the null hypothesis that 2 related or 
    repeated samples have identical average (expected) values.

    Params
    =====================================
    df               : dataframe
    target           : target column name(str)
    numeric_cols     : list of numeric columns

    Return
    =======================================
    results_df       : A dataframe with features and t-statistic p-val
    """

    # create an empty dictionary
    t_test_results = {}
    # loop over column_list and execute code 
    for column in numeric_cols:
        group1 = df.where(df[target] == 0).dropna()[column]
        group2 = df.where(df[target] == 1).dropna()[column]
        # add the output to the dictionary 
        t_test_results[column] = stats.ttest_ind(group1,group2)
    results_df = pd.DataFrame.from_dict(t_test_results,orient='Index')
    results_df.columns = ['t-statistic','t_stat_pval']

    results_df.reset_index(inplace=True)
    results_df.rename(columns = {"index":"x"}, inplace=True)
    return results_df

Sample

import seaborn as sns
titanic = sns.load_dataset("titanic")

calculate_tStatistic(titanic, "survived", ["age", "fare"])

Results

      x       t-statistic     t_stat_pval
0    age        3.479558       0.000630
1    fare      -1.767759       0.078795

Can someone confirm that I'm actually doing it the right way?

Is there a better / more elegant / more accurate way to do this??

Answer 1

There may be a bug in your code. Look at these two lines:

        group1 = df.where(df[target] == 0).dropna()[column]
        group2 = df.where(df[target] == 1).dropna()[column]

What is happening here? First, the where method sets all rows in which your condition (df[target] == 0) is not true to NaN. Then, you drop the NaN values by using dropna(). The problem: The dropna() method removes all rows where at least one element is NaN, but this not only applies to the rows not meeting your condition, but also to other rows. For example, check out the deck column in the dataset, it has lots of rows with NaN values and all of them get completely removed (that is the entire row). This is not what you want, I guess.

My suggestion:

        group1 = df.loc[df[target]==0, column].dropna()
        group2 = df.loc[df[target]==1, column].dropna()

This code first extracts the column of interest and then removes the NaN values. Not the other way around. In this way, you avoid removing too many lines. In addition, I use loc to select the values, as this is the preferred pandas method, when you need to index rows and columns simultaneously, as opposed to chained indexing like df[idx1][idx2]. Fore more details see here: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy

---

A thing I just learned: As an alternative to using the dropna() method, you can set the nan_policy to 'omit' when calling the stats.ttest_ind function:

stats.ttest_ind(group1, group2, nan_policy='omit')