2
\$\begingroup\$

I have the following DataFrame in pandas:

code town district suburb
02 Benalmádena Málaga Arroyo de la Miel
03 Alicante Jacarilla Jacarilla, Correntias Bajas (Jacarilla)
04 Cabrera d'Anoia Barcelona Cabrera D'Anoia
07 Lanjarón Granada Lanjaron
08 Santa Cruz de Tenerife Santa Cruz de Tenerife Centro-Ifara
09 Córdoba Córdoba Cordoba

For each row in the suburb column, if the value it contains is equal (in lower case and without accents) to district or town columns, it becomes NaN.

This is the code I am using:

df['suburb'] = np.where(
    ((df['suburb'].str.normalize('NFKD').str.encode('ascii', errors='ignore').str.decode('utf-8').str.lower() == df['town'].str.normalize('NFKD').str.encode('ascii', errors='ignore').str.decode('utf-8').str.lower())
    |  (df['suburb'].str.normalize('NFKD').str.encode('ascii', errors='ignore').str.decode('utf-8').str.lower() == df['district'].str.normalize('NFKD').str.encode('ascii', errors='ignore').str.decode('utf-8').str.lower())
    ),
    np.nan, df['suburb']) 
df

Example result:

code town district suburb
02 Benalmádena Málaga Arroyo de la Miel
03 Alicante Jacarilla Jacarilla, Correntias Bajas (Jacarilla)
04 Cabrera d'Anoia Barcelona NaN
07 Lanjarón Granada NaN
08 Santa Cruz de Tenerife Santa Cruz de Tenerife Centro-Ifara
09 Córdoba Córdoba NaN

I would like to reduce the amount of code, as I am sure it can be made shorter with the same performance.

\$\endgroup\$
1
  • \$\begingroup\$ can't you factor out the repeated operations? The str processing on that column is repeated several times. \$\endgroup\$
    – 2e0byo
    Jul 13, 2022 at 14:28

1 Answer 1

2
\$\begingroup\$

Looks like you could use a function:

def accent_free(s: str):
    return unicodedata.normalize('NFKD', s).encode('ascii', errors='ignore').decode('utf-8').lower()

Of course, you want this vectorized for numpy, so:

accent_free = np.vectorize(accent_free)

Now, you just need to use this function:

df['suburb'] = np.where(
    ((accent_free(df['suburb']) == accent_free(df['town'])) |
     (accent_free(df['suburb']) == accent_free(df['district']))
    ),
    np.nan, df['suburb']) 

Complete working example:

import unicodedata
import numpy as np
import pandas as pd

def accent_free(s: str):
    return unicodedata.normalize('NFKD', s).encode('ascii', errors='ignore').decode('utf-8').lower()

accent_free = np.vectorize(accent_free)

data = [
    ["02", "Benalmádena", "Málaga", "Arroyo de la Miel"],
    ["03", "Alicante", "Jacarilla", "Jacarilla, Correntias Bajas (Jacarilla)"],
    ["04", "Cabrera d'Anoia", "Barcelona", "Cabrera D'Anoia"],
    ["07", "Lanjarón", "Granada", "Lanjaron"],
    ["08", "Santa Cruz de Tenerife", "Santa Cruz de Tenerife", "Centro-Ifara"],
    ["09", "Córdoba", "Córdoba", "Cordoba"],
    ]

df = pd.DataFrame(data, columns=["code", "town", "district", "suburb"])
df['suburb'] = np.where(
    ((accent_free(df['suburb']) == accent_free(df['town'])) |
     (accent_free(df['suburb']) == accent_free(df['district']))
    ),
    np.nan, df['suburb']) 

print(df.to_string())
\$\endgroup\$
6
  • \$\begingroup\$ I get this error: UnboundLocalError: local variable 'accent_free' referenced before assignment \$\endgroup\$
    – Carola
    Jul 15, 2022 at 9:01
  • \$\begingroup\$ If I remove the command accent_free = np.vectorize(accent_free), I get this other error: TypeError: normalize() argument 2 must be str, not Series \$\endgroup\$
    – Carola
    Jul 15, 2022 at 9:43
  • \$\begingroup\$ I've added a MCVE. Hope that helps. \$\endgroup\$
    – AJNeufeld
    Jul 15, 2022 at 16:28
  • \$\begingroup\$ continues the same error: accent_free = np.vectorize(accent_free) UnboundLocalError: local variable 'accent_free' referenced before assignment \$\endgroup\$
    – Carola
    Jul 18, 2022 at 10:09
  • \$\begingroup\$ Sounds like you are trying to put that statement inside another function, instead of at the global context where accent_free is defined. The statement is redefining the function, and must only be done once. You don’t want to revectorize an already vectorized function, so you do not want the statement executed every function call! Note: I’m using CPython, not IronPython or some other snake. Instead of using the same name, you could assign to a different name, and use the new name in your query. \$\endgroup\$
    – AJNeufeld
    Jul 18, 2022 at 15:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.