1
\$\begingroup\$

From a public dataset available on film rating I created a contingency table as follow.

Honestly I don't like all these "for-loops" I think the quality of the code can be definitely improved in a more pythonic way.

import pandas as pd
import numpy as np

movies_df = pd.read_csv("https://raw.githubusercontent.com/uomodellamansarda/GentleIntroduction2MLandDataScience/main/L10/movies%20(1).csv")

# Getting series of lists by applying split operation.
movies_df.genres = movies_df.genres.str.split('|')

# Getting distinct genre types for generating columns of genre type.
genre_columns = list(set([j for i in movies_df['genres'].tolist() for j in i]))

# Iterating over every list to create and fill values into columns.
for j in genre_columns:
    movies_df[j] = 0

for i in range(movies_df.shape[0]):
    for j in genre_columns:
        if(j in movies_df['genres'].iloc[i]):
            movies_df.loc[i,j] = 1

print(movies_df)

And this is the output after running my code enter image description here

\$\endgroup\$
2
  • 1
    \$\begingroup\$ Is there a reason you're working with0, 1 and NaN instead of True, False and None? \$\endgroup\$
    – Mast
    Jan 10 at 19:15
  • \$\begingroup\$ No reason in particular, generally as for .get_dummies() method they transform in 0,1 pandas.pydata.org/pandas-docs/stable/reference/api/… \$\endgroup\$ Jan 10 at 20:06
1
\$\begingroup\$

Pandas isn't very good at this - it's not very vectorisable. But you can still avoid some of your loops.

"Pythonic" somewhat secondary here, and instead you're looking for "Pandas-idiomatic".

You can replace your nested list/set/list comprehension with a call to reduce(set.union) if you've cast your genres column to set using apply. I tried agg but it didn't work.

You can avoid loc and iloc if you use another apply per column value using set membership.

Suggested

from functools import reduce
import pandas as pd

movies_df = pd.read_csv(
    "https://raw.githubusercontent.com/uomodellamansarda"
    "/GentleIntroduction2MLandDataScience/main/L10/movies%20(1).csv")

movies_df.genres = movies_df.genres.str.split('|').apply(set)
genre_columns = reduce(set.union, movies_df.genres)

for genre in genre_columns:
    movies_df[genre] = movies_df.genres.apply(lambda s: genre in s)

print(movies_df)
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.