I have a code to get most similar element in an array of categories of jobs to another job using Google's Universal Sentence Encoder.

#@title Setup common imports and functions
import numpy as np
import pandas as pd
import tensorflow.compat.v2 as tf
import tensorflow_hub as hub
from tensorflow_text import SentencepieceTokenizer
import sklearn.metrics.pairwise

from simpleneighbors import SimpleNeighbors
from tqdm import tqdm
from tqdm import trange

def most_similar(embeddings_1, embeddings_2, labels_1, labels_2):

  assert (len(embeddings_1) == len(labels_1) and len(embeddings_2) == len(labels_2))

  # arccos based text similarity (Yang et al. 2019; Cer et al. 2019)
  sim = 1 - np.arccos(sklearn.metrics.pairwise.cosine_similarity(embeddings_1, embeddings_2))/np.pi

  embeddings_1_col, embeddings_2_col, sim_col = [], [], []
  for i in range(len(embeddings_1)):
    for j in range(len(embeddings_2)):
  df = pd.DataFrame(zip(embeddings_1_col, embeddings_2_col, sim_col),
                    columns=['embeddings_1', 'embeddings_2', 'sim'])

  # return the higest similarity one
  category = df['embeddings_1'].iloc[df['sim'].argmax()]
  return category

def main():

    X = pd.read_csv('X.csv')
    y = pd.read_csv('y.csv')
    df_rni = pd.read_csv('df.csv').head()

    # The 16-language multilingual module is the default but feel free
    # to pick others from the list and compare the results.
    module_url = 'https://tfhub.dev/google/universal-sentence-encoder-multilingual/3' #@param ['https://tfhub.dev/google/universal-sentence-encoder-multilingual/3', 'https://tfhub.dev/google/universal-sentence-encoder-multilingual-large/3']

    model = hub.load(module_url)

    def embed_text(input):
        return model(input)

    # get unique job categories and job of people
    job_categories = X.S02Q11_Professional_field.unique()
    # turn them to list
    job_categories = job_categories.tolist()
    # emebedding job categories 
    references_result = embed_text(job_categories[1:])

    for _, row in df_rni.iterrows():
        actual_job = row['new_professionactuelle']
        # check for nan that can't be embedded
        if str(actual_job) != 'nan':
            # embedding actual job
            target_result = embed_text(actual_job)
            # visualize similarity
            category = most_similar(references_result, target_result, job_categories[1:], [actual_job])
            row['category'] = category
        else: row['category'] = category

if __name__ == "__main__":

Within most_similar(), in a given loop it returns the element with the highest similarity, whatever the languages. For instance with Chef d'Entreprise:

                            embeddings_1        embeddings_2       sim
0              Property and construction  Chef D'entreprise   0.543505
1                                  Trade  Chef D'entreprise   0.578675
2             Leisure, sport and tourism  Chef D'entreprise   0.499804
3       Accountancy, banking and finance  Chef D'entreprise   0.529382
4               Creative arts and design  Chef D'entreprise   0.537169
5             Charity and voluntary work  Chef D'entreprise   0.558755
6                                  Sales  Chef D'entreprise   0.598833
7                             Healthcare  Chef D'entreprise   0.563151
8          Engineering and manufacturing  Chef D'entreprise   0.560561
9                            Social care  Chef D'entreprise   0.558178
10  Agriculture, farming and environment  Chef D'entreprise   0.547525
11                                   Law  Chef D'entreprise   0.545288
12                Other. Please specify:  Chef D'entreprise   0.511661
13        Teacher training and education  Chef D'entreprise   0.563531
14     Hospitality and events management  Chef D'entreprise   0.566105
15               Transport and logistics  Chef D'entreprise   0.503070
16                  Energy and utilities  Chef D'entreprise   0.523951
17    Public services and administration  Chef D'entreprise   0.541838
18                Information technology  Chef D'entreprise   0.535399
19   Business, consulting and management  Chef D'entreprise   0.605267
20                    Recruitment and HR  Chef D'entreprise   0.621728
21          Law enforcement and security  Chef D'entreprise   0.526561

Inputs and outputs

  • Inputs:

    1. X.csv a csv/dataframe of actual jobs that look like this one:
    0   Entrepreneur    
    1   طالبة   
  1. df.csv job categories that must include all the following categories:

    ['Agriculture, farming and environment',
       'Accountancy, banking and finance',
       'Teacher training and education', 'Leisure, sport and tourism',
       'Transport and logistics', 'Information technology',
       'Hospitality and events management',
       'Business, consulting and management', 'Creative arts and design',
       'Trade', 'Law enforcement and security',
       'Property and construction', 'Law',
       'Engineering and manufacturing', 'Social care',
       'Charity and voluntary work', 'Sales',
       'Public services and administration', 'Other. Please specify:',
       'Healthcare', 'Energy and utilities',
       'Marketing, advertising and PR', 'Media and internet',
       'Recruitment and HR', 'Science and pharmaceuticals']
    • Output would be the column in X.csv plus a new column, the most similar job.
        new_professionactuelle  job category
    0   Entrepreneur            Accountancy, banking and finance
    1   طالبة                   Law


it returns Recruitment and HR. I want to reduce this code to its simplest and to make it performant (as I will probably test 90000 jobs). Not considering readability I am worried I can get rid of functions that are actually useful, like the assert one that makes me sure that I have the labels corresponding to the embeddings. However I am sure I can do it simpler.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.