I have a dataset composed of ~700k samples. Each sample is composed of n_features features. Each feature is a word. Each feature has his own vocabulary. The size of the vocabularies range from 18 to 32000.
np.shape(x) -> (n_samples, n_features)
Instead of having lists of features I would like to have lists of indexes corresponding to the indexes in vocabularies. This is my code:
vocs = [np.array(list(set(x[:,i]))) for i in range(np.shape(x))] x_new = [[np.argwhere(vocs[j]==x[i,j]) for j,feature in enumerate(features)] for i,features in enumerate(x)]
This code works but I wonder if there is a way to improve the performances. These 2 lines take 10min to run on my i7-7700HQ @ 2.8GHz.
For more context, what I'm working on is natural language processing. I want to train a classifier to predict relations between words in a sentence. For this I have a conllu file which give sentences and for each word of each sentence a list of features and with which word it's related and how. These features may be the word itself, its lemma, its position in the sentence etc... I'm trying a different set of features and type of embedding and I want to test the embedding described above.