I have written a k-means function in Python to understand the methodology. I am trying to use this on a more complex dataset with a larger value for k, but it is running super slow. Does anyone have any recommendations for how I can improve this? I have provided code below, along with loading in an example dataset and applying the algorithm.
def Euc(x,y): return math.sqrt(sum([(a - b) ** 2 for a,b in zip(x,y)])) def K_means(TE,k): Iteration = 0 R =  O_a =  Epoch = 0 Tol = 1 Old_Tol = 2 Tol_r =  start_time = time.time() mean_cl = [[random.uniform(TE.iloc[i].min(),TE.iloc[i].max()) for i in range(len(TE.columns))] for c in range(0,k)] for n in range(len(TE)): D = [Euc(TE.iloc[n].tolist(),mean_cl[c]) for c in range(0,k)] O_a.append(D.index(min(D))) while(abs(Old_Tol - Tol) > 0.005): Old_Tol = Tol Epoch = Epoch + 1 mean_cl = [TE.iloc[[j for j, x in enumerate(O_a) if x == i]].mean() for i in range(0,k)] N_a =  for n in range(len(TE)): Iteration = Iteration + 1 D = [Euc(TE.iloc[n].tolist(),mean_cl[c]) for c in range(0,k)] N_a.append(D.index(min(D))) Tol = np.mean([x != y for x,y in zip(O_a,N_a)]) Tol_r.append(Tol) O_a = N_a R.append(time.time() - start_time) R.append(Tol_r) R.append(N_a) R.append(Iteration) return R def load_Pima(): url = "http://www.stats.ox.ac.uk/pub/PRNN/pima.tr" Pima_training = pd.read_csv(url,sep = '\s+') url = "http://www.stats.ox.ac.uk/pub/PRNN/pima.te" Pima_testing = pd.read_csv(url,sep = '\s+') Pima_training = Pima_training.iloc[1:] Pima_testing = Pima_testing.iloc[1:] Pima_training.loc[:,"type"] = Pima_training.loc[:,"type"].apply(lambda x : 0 if x == 'Yes' else 1) Pima_testing.loc[:,"type"] = Pima_testing.loc[:,"type"].apply(lambda x : 0 if x == 'Yes' else 1) Features = Pima_training.loc[:,Pima_training.columns != "type"] Means = Features.mean() SDs = Features.std() for name in Features.columns: Pima_training[name] = (Pima_training[name]-Means[name])/SDs[name] Pima_testing[name] = (Pima_testing[name]-Means[name])/SDs[name] return Pima_training, Pima_testing Pima_training, Pima_testing = load_Pima() class_var = "type" random.seed(2031) k = 2 TE = Pima_testing TE = TE.loc[:,TE.columns != class_var] km = K_means(TE,k)
The function returns the runtime of the algorithm, the tolerance at each epoch (% of changes in cluster assignment), the final cluster assignments, and the total number of iterations. I have already removed four for loops, which has sped it up quite a bit. But I fear my lack of Python programming is holding me back from making this more efficient. Any help is appreciated!
NameError: name 'Euc' is not defined. Can you include that function too? \$\endgroup\$
Eucfunction? One efficiency step would be to vectorize this and apply it to the whole dataframe
TEat once instead of row by row. \$\endgroup\$