I have some survey data that represents individual's responses to multiple survey questions. There are about 10,000 people in my actual dataset, and each person answered 35 questions. From these 35 survey questions, I create 1 composite score for each individual by taking the average of the values of all the questions that person answered. I am looking to see if I can find a subset of less than all 35 variables that produces a composite score that is highly correlated with the scores individuals receive if I use all of the questions.

Essentially, I want to be able identify which questions you should ask, if you can only ask X questions, and still end up with a composite score that is similar to if you asked all the questions.

I have created an example dataset, but with only 1,000 individuals and 10 questions. I have then written the code below to identify every subset of variables and compare the correlation of individual's scores when using just that subset to individual's scores when using all of the variables.

While my method below works when there are 1,000 individuals and 10 questions, it does not scale to my reality of 10,000 individuals and 35 questions, because of all the possible combinations of variables (2^35 = 34,359,738,368).

(Note: I have simplified my question and the sample data below for the purpose of this question.)

Code on GitHub: https://github.com/CurtLH/variable_subset/blob/master/powerset%20correlations.ipynb

import random
import numpy as np
import pandas as pd
from itertools import chain, combinations

# initial parameters
num_ids = 1000
num_vars = 10

# create repeating IDs
ids = sorted(list(range(0, num_ids)) * num_vars)

# create repeating variables
variables = list(range(0, num_vars)) * num_ids

# create random integers
values = np.random.randint(1, 5, size = num_ids * num_vars)

# create a dataframe with these values
df = pd.DataFrame({"id": ids, "variable": variables, "values": values})

# sort the dataframe
df.sort_values(['id', 'variable'], inplace=True)

def powerset(iterable):
    Thanks to https://stackoverflow.com/questions/1482308
    s = list(iterable)
    return chain.from_iterable(combinations(s, r) for r in range(len(s)+1))

# create every subset of variables
subsets = list(powerset(set(variables)))

# calculate the average value by ID when including all variables
actual = df.groupby('id')['values'].mean()

def calculate(df, subsets):

    1. Iterate over each permutation of variables
    2. Subset the dataframe to only include those variables
    3. Group by ID and recalculate the mean values per ID
    4. Measure correlation with complete set of variables

    # create a dictionary to hold the results
    results = {}

    # iterate over each subset
    for s in subsets:

        # make sure there is at least 1 variable...
        if len(s) > 0:

            # filter the dataframe to only the variables in the subset
            sub = df[df['variable'].isin(s)]

            # group by ID and calculate average value
            scores = sub.groupby('id')['values'].mean()

            # calculate correlation compared to complete set of variables
            corr = actual.corr(scores)

            # add results to dictionary
            results[s] = {'num_items': len(s),
                          'correlation': corr}

    return results

# time how long it takes to run
%%timeit results = calculate(df, subsets)

Timing results:

  • 2.23 s ± 3.91 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
  • \$\begingroup\$ This... smells. Like a few lines of R. Calculate your composite score, call cor, and chop out anything below a chosen threshold. \$\endgroup\$ – Reinderien Apr 6 at 17:01
  • \$\begingroup\$ I suppose the bigger question is - why try combinations at all? If a certain variable is uncorrelated to your composite, what's the harm in cutting it out wholesale instead of retrying it with other combinations of variables? \$\endgroup\$ – Reinderien Apr 6 at 17:06
  • 1
    \$\begingroup\$ Reduce dimensions with PCA etc. \$\endgroup\$ – E.Coms Apr 7 at 1:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.