# 3D metrics for segmentation evaluation

I am new to programming and I am trying to develop a machine learning approach for image segmentation (https://en.wikipedia.org/wiki/Image_segmentation) and here I want to evaluate the predicted (output) segmentation. In order to do this, I am using multiple (common) metrics (dice score, jacquard index, surface distance Hausdorff distance, diameter error and volume error).

The code works, but it really looks bad! I heard that one of the best practices is to avoid repetition in the code. For my piece I have to create multiple arrays which will all have different usage, but generating a lot of repetitions, and I am sure there might be better way to get better performance for this piece of code.

Do you have an idea of how to optimize this for speed and readability?

The details of the code:

folder = glab() : gets the name of the different folders containing the arrays that I want to open (glab() = glob() with natural sorting)

prediction/ground_truth : loads the predicted segmentation and ground_truth to be compared

scores() : calculates the different metrics (dice score, jacquard index, sensibility and specificity) between the ground truth and prediction

surfd() : calculates the surface distance between ground truth and prediction

diam_vol : calculates the diameter/volume and error between ground truth and prediction

Since I am evaluating multiple cases I want to know the mean() metrics to assess my model that's why I call for np.mean() for all the values at the end.

def metrics(savedir):

"""3D metrics for segmentation evaluation:

dice score : https://en.wikipedia.org/wiki/S%C3%B8rensen%E2%80%93Dice_coefficient
jacquard index : https://en.wikipedia.org/wiki/Jaccard_index
sensibility & specificity : https://en.wikipedia.org/wiki/Sensitivity_and_specificity
diameter and volume error
Surface distance : https://www.cs.ox.ac.uk/files/7732/CS-RR-15-08.pdf
Hausdorff distance : https://en.wikipedia.org/wiki/Hausdorff_distance
"""

folder = glab(savedir+"/OutputData/*/")

# array initialization
np_dice = np.zeros([len(folder)])
np_jacq = np.zeros([len(folder)])
np_sens = np.zeros([len(folder)])
np_spec = np.zeros([len(folder)])
np_surf = np.zeros([len(folder)])
np_haus = np.zeros([len(folder)])

np_diam_GT = np.zeros([len(folder)])
np_diam_pred = np.zeros([len(folder)])
np_diam_err = np.zeros([len(folder)])

np_vol_GT = np.zeros([len(folder)])
np_vol_pred = np.zeros([len(folder)])
np_vol_err = np.zeros([len(folder)])

for number, folder_name in enumerate(folder):

scores = scoring(ground_truth, prediction)
surface_distance = surfd(prediction, ground_truth, [0.625, 0.625, 0.625], 1)
diam_vol = diam(prediction, ground_truth)

np_dice[number] = scores
np_jacq[number] = scores
np_sens[number] = scores
np_spec[number] = scores
np_surf[number] = surface_distance.mean()
np_haus[number] = surface_distance.max()

np_diam_GT[number] = diam_vol
np_diam_pred[number] = diam_vol
np_diam_err[number] = diam_vol

np_vol_GT[number] = diam_vol
np_vol_pred[number] = diam_vol
np_vol_err[number] = diam_vol

mean_dice = np.mean(np_dice)
mean_jacq = np.mean(np_jacq)
mean_sens = np.mean(np_sens)
mean_spec = np.mean(np_spec)
mean_surf = np.mean(np_surf)
mean_haus = np.mean(np_haus)
mean_diam_GT = np.mean(np_diam_GT)
mean_diam_pred = np.mean(np_diam_pred)
mean_diam_err = np.mean(np_diam_err)
mean_vol_GT = np.mean(np_vol_GT)
mean_vol_pred = np.mean(np_vol_pred)
mean_vol_err = np.mean(np_vol_err)

return ([mean_dice, mean_jacq, mean_sens, mean_spec, mean_surf, mean_haus, mean_diam_GT, mean_diam_pred, mean_diam_err, mean_vol_GT, mean_vol_pred, mean_vol_err])

• Welcome to Code Review. Please read How to Ask. Tell us what this code accomplishes. By glab(), do you mean glob()? Is this real working code? What do scoring_baby(), surfd() and diam() do? – 200_success May 23 '19 at 3:11
• Welcome to Code Review. I have rolled back your last edit. Please do not update the code in your question to incorporate feedback from answers, doing so goes against the Question + Answer style of Code Review. This is not a forum where you should keep the most updated version in your question. Please see what you may and may not do after receiving answers. – Heslacher May 23 '19 at 4:20
• Hi, and thank you very much for your time. glab() is glob() with natural sorting (so I can find myself more easily in my data folder); scoring_baby() calculates the dice score, jacquard index, sensibility and specificity between a ground truth label (binary mask) and a predicted label (binary mask as well), surfd() calculates the surface distance between the two lables, and diam() calculates the different diameter and volume of the 3D segmentation. All the code is working, I just wanted to improve and understand more how to write proper code. – Unic0 May 23 '19 at 4:24
• It would be nice if you provide a description or link to explain segmentation, for those who know Python but don't have the same domain knowledge of the problem space. – Toby Speight May 24 '19 at 8:28

Realy, dud its childish!

only ones:

lenf = len(folder)


just one 2d array:

len_types = 12
arr = np.zeros([lenf ,  12])


learn how to work with numpy arrays, you will avoid the loop as well. And for example the mean can be done in one line for 2d array:

means = np.mean(arr, axis = 1)


https://docs.scipy.org/doc/numpy/reference/generated/numpy.mean.html

• Thank you very much, sorry I am new to this and clearly need improvement on the manipulation of matrices with numpy. Using you advice I was able to correct my code [see EDIT first post]. But I feel more can be done, do you have other suggestions ? (sorry for the noobism here) – Unic0 May 23 '19 at 2:59
• wait a bit, i can create good numpy tutorial from this example and ll post here later – user8426627 May 23 '19 at 14:52

# return lists

your functions seem to return lists or tuples. For 2 or 3 values, this is ok, but diam returns 6 values, metrics 12. This is very unclear, and unstable when you want to add, reorder or remove a metric. Better would be then to either return a dict or a namedtuple

# spacing

pep-8 recommends around operators (+,...) I use a code formatter (black) to format my code, so I don't have to worry about this sort of details any more

# variable naming

longer variable names don't make your program slower. And if you use a capable IDE, it doesn't slow the coding either. abbreviated variable names on the other hand make code more unclear. There is no need to shorten jacquard to jack

# magical values

[0.625, 0.625, 0.625], 1 as arguments to surface_distance are magical values, and very unclear what they mean. Easier would be to replace them with a variable with a clear name, or use keyword arguments.

# containers

If you have a lot of metrics, it is generally cleaner to keep them in 1 data structure (dict for example) instead of 12 different variables, and 12 other variables for the means.

# pandas

Instead of using numpy directly, why not use a pandas DataFrame as data structure? This will allow you to work with labeled data.

import pandas as pd

def metrics(savedir):

"""3D metrics for segmentation evaluation:

dice score : https://en.wikipedia.org/wiki/S%C3%B8rensen%E2%80%93Dice_coefficient
jacquard index : https://en.wikipedia.org/wiki/Jaccard_index
sensibility & specificity : https://en.wikipedia.org/wiki/Sensitivity_and_specificity
diameter and volume error
Surface distance : https://www.cs.ox.ac.uk/files/7732/CS-RR-15-08.pdf
Hausdorff distance : https://en.wikipedia.org/wiki/Hausdorff_distance
"""

folder = glab(savedir + "/OutputData/*/")
results = pd.DataFrame(
index=folder,
columns=[
"dice",
"jacquard",
"sensibility",
"specificity",
"surface",
"hausdorff",
"diam_GT",
"diam_pred",
"diam_err",
"vol_GT",
"vol_pred",
"vol_err",
],
dtype=float,
)
for folder_name in folder:

scores = scoring(ground_truth, prediction)
surface_distance = surfd(
prediction, ground_truth, [0.625, 0.625, 0.625], 1
)
diam_vol = diam(prediction, ground_truth)
results.loc[folder_name] = pd.Series(
{
"dice": scores,
"jacquard": scores,
"sensibility": scores,
"specificity": scores,
"surface": surface_distance.mean(),
"hausdorff": surface_distance.max(),
"diam_GT": diam_vol,
"diam_pred": diam_vol,
"diam_err": diam_vol,
"vol_GT": diam_vol,
"vol_pred": diam_vol,
"vol_err": diam_vol,
}
)

return results.mean().to_dict()

• Thank you very much for you answer. Your solution is clean and very nice but I would try to use numpy as much as I can because I am learning and I feel like I am missing some of the basic mechanics of numpy which could help me in the future. But I'll look more into panda because I know that the dataframe class in panda can be interesting to work with. – Unic0 May 25 '19 at 3:12
• Pandas uses numpy under the hood. For the thing you want to do, with the support of labelled data, it's way better suited then raw numpy – Maarten Fabré May 25 '19 at 5:39
• Noted, I look more into it. But I noticed that when I use your method I have different results that doesn't seem correct (nothing else was changed but the piece of code you provided). And I don't know why. I'll keep investigating. Thank you very much – Unic0 May 26 '19 at 1:36
• Possibly, you need to add axis=1 as argument to mean – Maarten Fabré May 26 '19 at 6:10