Remove duplicates from a Pandas dataframe taking into account lowercase letters and accents

Question

I have the following DataFrame in pandas:

code	town	district	suburb
02	Benalmádena	Málaga	Arroyo de la Miel
03	Alicante	Jacarilla	Jacarilla, Correntias Bajas (Jacarilla)
04	Cabrera d'Anoia	Barcelona	Cabrera D'Anoia
07	Lanjarón	Granada	Lanjaron
08	Santa Cruz de Tenerife	Santa Cruz de Tenerife	Centro-Ifara
09	Córdoba	Córdoba	Cordoba

For each row in the suburb column, if the value it contains is equal (in lower case and without accents) to district or town columns, it becomes NaN.

This is the code I am using:

df['suburb'] = np.where(
    ((df['suburb'].str.normalize('NFKD').str.encode('ascii', errors='ignore').str.decode('utf-8').str.lower() == df['town'].str.normalize('NFKD').str.encode('ascii', errors='ignore').str.decode('utf-8').str.lower())
    |  (df['suburb'].str.normalize('NFKD').str.encode('ascii', errors='ignore').str.decode('utf-8').str.lower() == df['district'].str.normalize('NFKD').str.encode('ascii', errors='ignore').str.decode('utf-8').str.lower())
    ),
    np.nan, df['suburb']) 
df

Example result:

code	town	district	suburb
02	Benalmádena	Málaga	Arroyo de la Miel
03	Alicante	Jacarilla	Jacarilla, Correntias Bajas (Jacarilla)
04	Cabrera d'Anoia	Barcelona	NaN
07	Lanjarón	Granada	NaN
08	Santa Cruz de Tenerife	Santa Cruz de Tenerife	Centro-Ifara
09	Córdoba	Córdoba	NaN

I would like to reduce the amount of code, as I am sure it can be made shorter with the same performance.

Answer 1

Looks like you could use a function:

def accent_free(s: str):
    return unicodedata.normalize('NFKD', s).encode('ascii', errors='ignore').decode('utf-8').lower()

Of course, you want this vectorized for numpy, so:

accent_free = np.vectorize(accent_free)

Now, you just need to use this function:

df['suburb'] = np.where(
    ((accent_free(df['suburb']) == accent_free(df['town'])) |
     (accent_free(df['suburb']) == accent_free(df['district']))
    ),
    np.nan, df['suburb']) 

Complete working example:

import unicodedata
import numpy as np
import pandas as pd

def accent_free(s: str):
    return unicodedata.normalize('NFKD', s).encode('ascii', errors='ignore').decode('utf-8').lower()

accent_free = np.vectorize(accent_free)

data = [
    ["02", "Benalmádena", "Málaga", "Arroyo de la Miel"],
    ["03", "Alicante", "Jacarilla", "Jacarilla, Correntias Bajas (Jacarilla)"],
    ["04", "Cabrera d'Anoia", "Barcelona", "Cabrera D'Anoia"],
    ["07", "Lanjarón", "Granada", "Lanjaron"],
    ["08", "Santa Cruz de Tenerife", "Santa Cruz de Tenerife", "Centro-Ifara"],
    ["09", "Córdoba", "Córdoba", "Cordoba"],
    ]

df = pd.DataFrame(data, columns=["code", "town", "district", "suburb"])
df['suburb'] = np.where(
    ((accent_free(df['suburb']) == accent_free(df['town'])) |
     (accent_free(df['suburb']) == accent_free(df['district']))
    ),
    np.nan, df['suburb']) 

print(df.to_string())