1
\$\begingroup\$

I recently updated Pandas and noticed that it is no longer possible to merge DataFrames with mismatched dtypes. I have a script which relied on merging and seemed to work in the past despite having mismatched dtypes. I need to display to the user which columns in two dataframes are causing problems, so the user can then adjust the types accordingly. (Specifically, one dataframe is read in from a database and represents what is currently in the DB, while the second dataframe includes any changes/new data to be applied to the database. Once the the user finds the problem columns they can determine if the DataFrame that contains the changes was meant to change the database's type or whether there is an error in the changes). The following code appears to work, but I feel like pandas must have a built in better way to deal with this problem.

def get_mismatched_dtypes(self, df1, df2):
    mismatch = {}
    for key, val in df1.dtypes.iteritems():
        if key in df2.dtypes and df2.dtypes[key] != val:
            mismatch[key] = (f"df1:{val}, df2: {df2.dtypes[key]}")
    print(mismatch)
    return mismatch
\$\endgroup\$

1 Answer 1

1
\$\begingroup\$

self

Why put this method on a class? the lack of use of self in the method should act as a flag

string result

you format the mismatch as a string (f"df1:{val}, df2: {df2.dtypes[key]}"). This way you can not do anything about it any more. Better would be to use a tuple or dict here

generator

generally, when I see

def method():
    result = {}
    for key in ...:
        result[key] = ...
    return key

it is more clear to work with a generator

def method():
    for key in ...:
        yield key, ...

If the user of the method needs a dict, he can do dict(method()). If he wants to iterate over the items, there is no need to instantiate the dict

alternative approach

you can use sets to see which types are different:

def get_mismatched_dtypes(df1, df2):
    df1_types = set(df1.dtypes.items())
    df2_types = set(df2.dtypes.items())
    for column_name, df1_type in df1_types - df2_types:
        yield column_name, (df1_type, df2.dtypes[column_name])



df1 = pd.DataFrame({"col1": [1, 2, 3], "col2": [4, 5, 6]}, index=list(range(3)))
df2 = pd.DataFrame({"col1": list("abc"), "col2": [4, 5, 6]}, index=list(range(3,6)))
dict(get_mismatched_dtypes(df1, df2))
{'col1': (dtype('int64'), dtype('O'))}

for clarification: df1_types - df2_types is {('col1', dtype('int64'))}

If you want it formatted like in your code:

{
    column_name: f"df1: {df1_type}, df2: {df2_type}"
    for column_name, (df1_type, df2_type) in get_mismatched_dtypes(df1, df2)
}
{'col1': 'df1: int64, df2: object'}
\$\endgroup\$
3
  • \$\begingroup\$ thanks! great answer. One follow-up question: If this function is only used by one class, would you leave it as a function in the same module outside of the class or just declare it as a static method, using the @static flag? Is this just a matter of preference/how you choose to organize things or are there other considerations? \$\endgroup\$ Oct 23, 2018 at 20:24
  • \$\begingroup\$ why would you attach the function to a class, instead of at module level? \$\endgroup\$ Oct 24, 2018 at 6:56
  • \$\begingroup\$ To keep it close to the code that calls it. So far it is only used once. If it starts getting used more or by other modules then I see the need to factor it out, but as it is, it makes some sense to me to keep in the class to make it easy to scroll through the code. I guess I am wondering if there are other reasons to avoid adding a method to a class besides keeping things organized...if that's the only reason then it seems like preference, since you could argue it's more organized to keep the code close to the code that calls it. \$\endgroup\$ Oct 24, 2018 at 23:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.