5
\$\begingroup\$

I have a dataframe in pandas like this

Date         cell         tumor_size (assume it is three dimensional)
25/10/2015    113           [51, 52, 55]
22/10/2015    222           [50, 68, 22]
22/10/2015    883           [45, 23, 67]
20/10/2015    334           [35, 23, 76]

What I want to do is compare the size of the tumors detected on the different days. Let's consider the cell 222 as an example; I want to compare its size to different cells but detected on earlier days e.g. I will not compare its size with cell 883, because they were detected on the same day. Or I will not compare it with cell 113, because it was detected later on. As my dataset is too large, I have iterate over the rows. Here is my sample code:

# These will be our lists of pairs and size differences.
pairs = []
diffs = []

# Loop over all unique dates
for date in df.Date.unique():
    # Only take dates earlier then current date.
    compare_df =df.loc[df.Date < pd.Timestamp(date).to_pydatetime()].copy()
    # Loop over each cell for this date and find the minimum
    for row in df.loc[df.Date ==  pd.Timestamp(date).to_pydatetime()].itertuples():
        # If no cells earlier are available use nans.
        if compare_df.empty:
            pairs.append(float('nan'))
            diffs.append(float('nan'))
        # Take lowest absolute value and fill in otherwise
        else:
            compare_df['distance'] = compare_df['tumor_size'].map(lambda x: np.linalg.norm(x - row.tumor_size))
            row_of_interest = compare_df.loc[compare_df.distance == compare_df.distance.min()]
            cell = row.cell
            Date = row.Date
            most_similar_to = row_of_interest.cell.values[0]
            similarity = row_of_interest.distance.values[0]
            with open('final_csv', "a", newline="") as f:
                writer = csv.writer(f, dialect="excel-tab")
                writer.writerow([cell, Date, most_similar_to, similarity])

How could I improve the speed of the code? I have over one million cells.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.