Code to calculate similarity between strings - R

Question

I have a large datafile with names and I want to create a similarity distance matrix. With this matrix I want to get similar names that could be the same person (or not) and that I could compare these rows and check whether more variables match or not.

However the code I have is quite slow. The dataframe has 58797 rows and some of them are repeated names. I was wondering for other options or a better way to get the information I'm looking for.

This is the code I have so far:

similar <- list()
for (i in 1:dim(data)[1]) {
    ids <- which(levenshteinSim(data$nomeAlt[i], data$nomeAlt) != 1 & 
                 levenshteinSim(data$nomeAlt[i], data$nomeAlt) > 0.85)
    # ifelse only returns first element of list, instead use separate if else
    similar[[i]] <- if (length(ids) == 0) NA else ids
    print(i) # to get an update of the progress
} 

Basically, the output returns rownames which I can get the names. In a working exemple I got names such as "ABEL MACEDO ALVES" and "ABEL MACHADO ALVES".

Any suggestion would be appreciated. Thank you!

Answer 1

Here is an implementation of the ideas I had suggested in the comments: to store the output of levenshteinSim so it is only called once, and to limit the expensive name comparisons to individuals that share the same initials. I hope it helps.

names_vec <- data$nomeAlt
initials  <- gsub("\\b(.).*?\\b", "\\1", x)

similar <- list()
for (i in 1:length(names_vec)) {
   ini <- initials == initials[i]
   sim <- levenshteinSim(names_vec[i], names_vec[ini])
   idx <- which(sim > 0.85 & sim != 1)
   similar[[i]] <- if (length(idx) == 0) NA else ini[idx]
   print(i) # to get an update of the progress
}