Algorithm

Let 𝑀0 denote the null model which contains no predictors. This model simply predicts the sample mean of each observation. 

For 𝑘=1,2,....,𝑛:
         Fit all (𝑛 \choose 𝑘) models that contain exactly 𝑘 predictors.

         Pick the best among these (𝑛 choose 𝑘) models, and call it 𝑀_𝑘.
         Here the best is defined as having the smallest 𝑅𝑆𝑆 or equivalent measure.

   Select the single best model among 𝑀0,𝑀1,...,𝑀𝑛 using cross validated prediction error, 𝐶𝑝, BIC, 𝑅2𝑎𝑑𝑗 or any other method.

# This function takes in a subset of a dataframe representing independent  
# variables (X) and a column for dependent variable (Y). This function fits 
# separate models for each possible combination of the k predictors (which is 
# based on the column length of X) and then select the best subset. The 
# resulting output is a dataframe.

def BestSubsetSelection(X,Y):
    # number of predictors
    k = len(X.columns)
    # Store the RSS from a linear regression model
    RSS_list = []
    # Store the R-square from a linear regression model
    R_squared_list = []
    # Store the features for a given iteration. 
    feature_list = []
    # Store the number of features used for a given iteration. This corresponds with the feature_list. 
    numb_features = []
    
    # Loop over all possible combinations of k features
    for k in range(1, len(X.columns) + 1):
            # Looping over all possible combinations: from 11 choose k
            for combo in itertools.combinations(X.columns,k):
                # Store temporary results
                temp_results = fit_linear_reg(X[list(combo)],Y)

                # Append RSS to RSS Lists
                RSS_list.append(temp_results[0])
            
                # Append R-Squared TO R-Squared list
                R_squared_list.append(temp_results[1])
            
                # Append Feature/s to Feature list
                feature_list.append(combo)
            
                # Append the number of features to the number of features list
                numb_features.append(len(combo))
            
    df = pd.DataFrame({
        'No_of_Features': numb_features,
        'RSS' : RSS_list,
        'R-Squared' : R_squared_list,
        'Features' : feature_list
    })
    
    # Finding the Best Subsets for each number of features
    
    # The smallest RSS
    df_min = df[df.groupby('No_of_Features')['RSS'].transform(min) == df['RSS']]
    # The Largest R-Squared Value
    df_max = df[df.groupby('No_of_Features')['R-Squared'].transform(min) == df['R-Squared']]
    display(df_min)
    display(df_max)
    
    # Adding columns to the dataframe with RSS and R-Squared values of the best subset
    df['min_RSS'] = df.groupby('No_of_Features')['RSS'].transform(min)
    df['max_R_Squared'] = df.groupby('No_of_Features')['R-Squared'].transform(max)
    
```

Algorithm

Let \$M_0\$ denote the null model which contains no predictors. This model simply predicts the sample mean of each observation.

For \$k=1,2,\ldots,n\$:

• Fit all \$n \choose k\$ models that contain exactly \$k\$ predictors.
• Pick the best among these \$n \choose 𝑘\$ models, and call it \$M_k\$. Here the best is defined as having the smallest RSS or equivalent measure.

Select the single best model among \$M_0,M_1,\ldots,M_n\$ using cross validated prediction error, \$C_p\$, BIC, \$R^2_{\mathit{adj}}\$ or any other method.

# This function takes in a subset of a dataframe representing independent  
# variables (X) and a column for dependent variable (Y). This function fits 
# separate models for each possible combination of the k predictors (which is 
# based on the column length of X) and then select the best subset. The 
# resulting output is a dataframe.

def BestSubsetSelection(X,Y):
    # number of predictors
    k = len(X.columns)
    # Store the RSS from a linear regression model
    RSS_list = []
    # Store the R-square from a linear regression model
    R_squared_list = []
    # Store the features for a given iteration. 
    feature_list = []
    # Store the number of features used for a given iteration. This corresponds with the feature_list. 
    numb_features = []
    
    # Loop over all possible combinations of k features
    for k in range(1, len(X.columns) + 1):
            # Looping over all possible combinations: from 11 choose k
            for combo in itertools.combinations(X.columns,k):
                # Store temporary results
                temp_results = fit_linear_reg(X[list(combo)],Y)

                # Append RSS to RSS Lists
                RSS_list.append(temp_results[0])
            
                # Append R-Squared TO R-Squared list
                R_squared_list.append(temp_results[1])
            
                # Append Feature/s to Feature list
                feature_list.append(combo)
            
                # Append the number of features to the number of features list
                numb_features.append(len(combo))
            
    df = pd.DataFrame({
        'No_of_Features': numb_features,
        'RSS' : RSS_list,
        'R-Squared' : R_squared_list,
        'Features' : feature_list
    })
    
    # Finding the Best Subsets for each number of features
    
    # The smallest RSS
    df_min = df[df.groupby('No_of_Features')['RSS'].transform(min) == df['RSS']]
    # The Largest R-Squared Value
    df_max = df[df.groupby('No_of_Features')['R-Squared'].transform(min) == df['R-Squared']]
    display(df_min)
    display(df_max)
    
    # Adding columns to the dataframe with RSS and R-Squared values of the best subset
    df['min_RSS'] = df.groupby('No_of_Features')['RSS'].transform(min)
    df['max_R_Squared'] = df.groupby('No_of_Features')['R-Squared'].transform(max)
    

This code is taken from my IPython notebook.

I looked up a tutorial and created a function based upon its script. It's essentially used so I can select dependent variables that's a subset of a data frame. It runs but it is very very slow.

I tried implementing an enumerate version but it did not work. I'm not sure, how I can flatten the nested for loop, such that I can append the results of a function to a list.

Algorithm Let 𝑀_{0} denote the null model which contains no predictors. This model simply predicts the sample mean of each observation.

For 𝑘=1,2,....,𝑛:
         Fit all (𝑛 \choose 𝑘) models that contain exactly 𝑘 predictors.

         Pick the best among these (𝑛 choose 𝑘) models, and call it 𝑀_𝑘.
         Here the best is defined as having the smallest 𝑅𝑆𝑆 or equivalent measure.

   Select the single best model among 𝑀0,𝑀1,...,𝑀𝑛 using cross validated prediction error, 𝐶𝑝, BIC, 𝑅2𝑎𝑑𝑗 or any other method.

Algorithm

Let 𝑀0 denote the null model which contains no predictors. This model simply predicts the sample mean of each observation. 

For 𝑘=1,2,....,𝑛:
         Fit all (𝑛 \choose 𝑘) models that contain exactly 𝑘 predictors.

         Pick the best among these (𝑛 choose 𝑘) models, and call it 𝑀_𝑘.
         Here the best is defined as having the smallest 𝑅𝑆𝑆 or equivalent measure.

   Select the single best model among 𝑀0,𝑀1,...,𝑀𝑛 using cross validated prediction error, 𝐶𝑝, BIC, 𝑅2𝑎𝑑𝑗 or any other method.

I looked up a tutorial and created a function based upon its script. It's essentially used so I can select dependent variables that's a subset of a data frame. It runs but it is very very slow.

I tried implementing an enumerate version but it did not work. I'm not sure, how I can flatten the nested for loop, such that I can append the results of a function to a list.

I tried implementing an enumerate version but it did not work. Ideally I'd like the complexity to be linear, currently, it's at 2^n. I'm not sure, how I can flatten the nested for loop, such that I can append the results of a function to a list.

Algorithm

Let 𝑀_{0} denote the null model which contains no predictors. This model simply predicts the sample mean of each observation. 

For 𝑘=1,2,....,𝑛:
         Fit all (𝑛 \choose 𝑘) models that contain exactly 𝑘 predictors.

         Pick the best among these (𝑛 choose 𝑘) models, and call it 𝑀_𝑘.
         Here the best is defined as having the smallest 𝑅𝑆𝑆 or equivalent measure.

   Select the single best model among 𝑀0,𝑀1,...,𝑀𝑛 using cross validated prediction error, 𝐶𝑝, BIC, 𝑅2𝑎𝑑𝑗 or any other method.

# This function takes in a subset of a dataframe representing independent  
# variables (X) and a column for dependent variable (Y). This function fits 
# separate models for each possible combination of the k predictors (which is 
# based on the column length of X) and then select the best subset. The 
# resulting output is a dataframe.

def BestSubsetSelection(X,Y):
    # number of predictors
    k = len(X.columns)
    # Store the RSS from a linear regression model
    RSS_list = []
    # Store the R-square from a linear regression model
    R_squared_list = []
    # Store the features for a given iteration. 
    feature_list = []
    # Store the number of features used for a given iteration. This corresponds with the feature_list. 
    numb_features = []
    
    # Loop over all possible combinations of k features
    for k in range(1, len(X.columns) + 1):
            # Looping over all possible combinations: from 11 choose k
            for combo in itertools.combinations(X.columns,k):
                # Store temporary results
                temp_results = fit_linear_reg(X[list(combo)],Y)

                # Append RSS to RSS Lists
                RSS_list.append(temp_results[0])
            
                # Append R-Squared TO R-Squared list
                R_squared_list.append(temp_results[1])
            
                # Append Feature/s to Feature list
                feature_list.append(combo)
            
                # Append the number of features to the number of features list
                numb_features.append(len(combo))
            
    df = pd.DataFrame({
        'No_of_Features': numb_features,
        'RSS' : RSS_list,
        'R-Squared' : R_squared_list,
        'Features' : feature_list
    })
    
    # Finding the Best Subsets for each number of features
    
    # The smallest RSS
    df_min = df[df.groupby('No_of_Features')['RSS'].transform(min) == df['RSS']]
    # The Largest R-Squared Value
    df_max = df[df.groupby('No_of_Features')['R-Squared'].transform(min) == df['R-Squared']]
    display(df_min)
    display(df_max)
    
    # Adding columns to the dataframe with RSS and R-Squared values of the best subset
    df['min_RSS'] = df.groupby('No_of_Features')['RSS'].transform(min)
    df['max_R_Squared'] = df.groupby('No_of_Features')['R-Squared'].transform(max)
    
    return df

A copy of the full implementation can be found here: https://github.com/melmaniwan/Elections-Analysis/blob/master/Implementing%20Subset%20Selections.ipynb

# Loop over all possible combinations of k features
for k in range(1, len(X.columns) + 1):
       # Looping over all possible combinations: from 11 choose k
       for combo in itertools.combinations(X.columns,k):
           # Store temporary results
           temp_results = fit_linear_reg(X[list(combo)],Y)

           # Append RSS to RSS Lists
           RSS_list.append(temp_results[0])

I tried implementing an enumerate version but it did not work. I'm not sure, how I can flatten the nested for loop, such that I can append the results of a function to a list.

Algorithm Let 𝑀_{0} denote the null model which contains no predictors. This model simply predicts the sample mean of each observation.

For 𝑘=1,2,....,𝑛:
         Fit all (𝑛 \choose 𝑘) models that contain exactly 𝑘 predictors.

         Pick the best among these (𝑛 choose 𝑘) models, and call it 𝑀_𝑘.
         Here the best is defined as having the smallest 𝑅𝑆𝑆 or equivalent measure.

   Select the single best model among 𝑀0,𝑀1,...,𝑀𝑛 using cross validated prediction error, 𝐶𝑝, BIC, 𝑅2𝑎𝑑𝑗 or any other method.

# This function takes in a subset of a dataframe representing independent  
# variables (X) and a column for dependent variable (Y). This function fits 
# separate models for each possible combination of the k predictors (which is 
# based on the column length of X) and then select the best subset. The 
# resulting output is a dataframe.

def BestSubsetSelection(X,Y):
    # number of predictors
    k = len(X.columns)
    # Store the RSS from a linear regression model
    RSS_list = []
    # Store the R-square from a linear regression model
    R_squared_list = []
    # Store the features for a given iteration. 
    feature_list = []
    # Store the number of features used for a given iteration. This corresponds with the feature_list. 
    numb_features = []
    
    # Loop over all possible combinations of k features
    for k in range(1, len(X.columns) + 1):
            # Looping over all possible combinations: from 11 choose k
            for combo in itertools.combinations(X.columns,k):
                # Store temporary results
                temp_results = fit_linear_reg(X[list(combo)],Y)

                # Append RSS to RSS Lists
                RSS_list.append(temp_results[0])
            
                # Append R-Squared TO R-Squared list
                R_squared_list.append(temp_results[1])
            
                # Append Feature/s to Feature list
                feature_list.append(combo)
            
                # Append the number of features to the number of features list
                numb_features.append(len(combo))
            
    df = pd.DataFrame({
        'No_of_Features': numb_features,
        'RSS' : RSS_list,
        'R-Squared' : R_squared_list,
        'Features' : feature_list
    })
    
    # Finding the Best Subsets for each number of features
    
    # The smallest RSS
    df_min = df[df.groupby('No_of_Features')['RSS'].transform(min) == df['RSS']]
    # The Largest R-Squared Value
    df_max = df[df.groupby('No_of_Features')['R-Squared'].transform(min) == df['R-Squared']]
    display(df_min)
    display(df_max)
    
    # Adding columns to the dataframe with RSS and R-Squared values of the best subset
    df['min_RSS'] = df.groupby('No_of_Features')['RSS'].transform(min)
    df['max_R_Squared'] = df.groupby('No_of_Features')['R-Squared'].transform(max)
    
```