1
\$\begingroup\$

I found out the hard way, when utilizing pandas is better to use pandas.Series than lists data type to manipulate data. Even if I understood how pandas stores different data types, I did not realize the ramification of using lists of strings instead of pandas.Series of objects.

Pandas dtype

Pandas dtype

Like the pic above shows, pandas store strings as objects, it does not discriminate between strings, lists, dictionaries and etc, python iterable.
I made my own function that returns a pandas.DataFrame column names, columns' pandas dtypes and columns' python data types, see pic below.

enter image description here

Most pandas.DataFrames that I encountered have a NaN values representing any values that are undefined.
A pandas.DataFrame column of string objects, first_names for example, can contain NaN values, NaN is a float data type.
The main reason that the NaN value is commonly utilize, it is due to its usefulness, when combine with a function like DataFrame.dropna(), it becomes a well recognize and powerful tool for data manipulation.

Most pandas teaching materials utilize lists and pandas.DataFrames to manipulate data, so I did the same, see code below:

    # Removes Html code and \n
    for name in essay_names:
        # Initializes essay text list 
        essay_txts = []
        for essay in profiles[name]:
            # Checks if the essay is empty, NaN
            if type(essay) == type(.0):
                # Adds NaN text to the essay texts Series, empty text
                essay_txts.append(np.NaN)
            else:
                essay_clean = re.sub('', '', essay).replace('\n', ' ').replace('  ', ' ')    
                essay_txts.append(essay_clean)
        # Stores cleaned essay text 
        profiles[f'{name}_text'] = essay_txts

The code lines essay_txts = [] and essay_txts.append(np.NaN) create issues when trying to manipulating the DataFrames data with DataFrame.dropna(), the essay_txts is a list of strings, the np.nan value was not saved in the list as a float but as string, 'nan', and by consequence it was also saved in the DataFrame as a string.
The DataFrame.dropna() function will not recognize the value as NaN float and it will not drop the DataFrame row where the values was inputted.

Replacing the code lines
essay_txts = []
essay_txts.append(np.NaN)
with
essay_txts = pd.Series(dtype='object')
essay_txts = essay_txts.append(pd.Series(np.NaN), ignore_index=True)
is my solution to the issues created by using lists and trying to utilize the NaN value to manipulate data with pandas.

See full code below

    # Removes Html code and \n
    for name in essay_names:
        # I use pandas Series, a Series will accept NaN float and object (string) values
        essay_txts = pd.Series(dtype='object')
        for essay in profiles[name]:
            # Checks if the essay is empty, NaN
            if type(essay) == type(.0):
                # Adds NaN text to the essay texts Series, empty text
                essay_txts = essay_txts.append(pd.Series(np.NaN), ignore_index=True)
            else:
                essay_clean = re.sub('', '', essay).replace('\n', ' ').replace('  ', ' ')
                essay_txts = essay_txts.append(pd.Series(essay_clean), ignore_index=True)
        # Stores cleaned essay text 
        profiles[f'{name}_text'] = essay_txts

Thank you for feedback, or for sharing a better way to address the issues created by using lists and trying to utilize the NaN value to manipulate data with pandas.

\$\endgroup\$
1
  • \$\begingroup\$ Welcome to Code Review, from your question it seems me you are proposing two distinct solutions to the same problem asking for comparison. If this is the scenario, you can add the comparative-review tag to your question. \$\endgroup\$ Feb 28 at 9:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.