I have dataframe like this:

date        tableName                attributeName
2019-03-29  [order as o, costumer]   [costumer.customerID, o.orderID]
2019-03-30  [customer c, payment]    [payment.paymentID, c.firstName]

I have a function that map the tableName to attribute, like this:

import pandas as pd
import re
def countTableAttribute(dataFrame, tableName, attributeName):
    a       = (dataFrame[tableName].values.tolist())
    r       = re.compile(r'\b(as|\s)\b',re.IGNORECASE)
    alias   = list((x,n) for x in range(len(a)) for n in a[x] if bool(r.search(n)))
    df3     = (pd.DataFrame(dataFrame[tableName].values.tolist(), index=dataFrame.index).stack().str.split(' as | ', expand=True)).dropna()
    if alias != []:
        d   = dict(zip(df3[1], df3[0]))
        d   = dict(zip(df3[0], df3[0]))
    dfs     = pd.DataFrame(columns=[-1,0,1])
    for i in range(len(dataFrame)):
        for x in dataFrame[attributeName][i]:
            if not '.' in x:
                ser = pd.Series([dataFrame['date'][i],dataFrame[tableName][i][0], x], index=dfs.columns)
                dfs = dfs.append(ser, ignore_index=True)
                ser = pd.Series([dataFrame['date'][i],x.split('.')[0], x.split('.')[1]], index=dfs.columns)
                dfs = dfs.append(ser, ignore_index=True)
    dfs.columns         = ['Date','tableName','attributeName']
    dfs['tableName']    = dfs['tableName'].replace(d)
    dfs                 = dfs.groupby(['Date','tableName','attributeName'], sort=False).size().reset_index(name='Count')
    return dfs

so the output is like this:

   date        tableName   attributeName  count
2019-03-29     order       orderID          1
2019-03-29     costumer    customerID       1
2019-03-30     customer    firstName        1
2019-03-30     payment     paymentID        1

But as this is my first try, I need an opinion about what I've tried, because my code runs slow.

Thank you


Some suggestions:

  • black can format your code to be more idiomatic.
  • flake8 can check your code for remaining non-idiomatic code.
  • mypy can enforce typing to make it more obvious what the code is meant to do.
  • Names like a, r, d and dfs are unhelpful. Naming is a hard skill to learn, but is enormously helpful in making code more readable and therefore maintainable.
  • if not '.' in x: etc looks like it might be a good candidate for a separate function - it doesnt't take too many inputs, returns only one thing (dfs), and is short.
  • It looks like you are doing a lot of conversions: dataFrame[tableName].values.tolist(), list(…), dict(). I'm not familiar with Pandas, but I expect you can speed things up significantly by not converting values.
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.