Plotting in Python

Question

This is a Part2 to this question where the task was to plot the data that I got from my time measurements.

I've never worked with Jupyter Notebooks (ipynb) since most people I talked to were critiquing (in a bad way) them.

A lot of repetition is noticeable, that's because I haven't gotten the hang of how the code blocks work (I thought they work separately -> lifetime of variables).

Edit: .ipynb files are a nightmare to format in here, I used:

###
# Plot [n] placeholder (indicated where the plot is)
###

###
# new cell (indicates new cell)
###

Code:

lists = [[]]
sizes = [10, 100, 1000, 10000, 100000]
current_list = 0

with open('data.txt', 'r') as file:
    lines = file.readlines()

for line in lines:
    line = line.strip()
    if line == "":
        lists.append([])
        current_list += 1
    elif line.startswith('[time]'):
        time = int(line.split(':')[1].strip())
        lists[current_list].append(time)

ms1_times = lists[0]
qs1_times = lists[1]
hs1_times = lists[2]
ms2_times = lists[3]
qs2_times = lists[4]
hs2_times = lists[5]
ms3_times = lists[6]
qs3_times = lists[7]
hs3_times = lists[8]

###
# new cell
###

import matplotlib.pyplot as plt

sizes = [10, 100, 1000, 10000, 100000]
ms1_times = lists[0]
ms2_times = lists[3]
ms3_times = lists[6]

plt.figure(figsize=(10, 6))

plt.plot(sizes, ms1_times, marker='o', label='MS #1')
plt.plot(sizes, ms2_times, marker='s', label='MS #2')
plt.plot(sizes, ms3_times, marker='^', label='MS #3')

plt.xlabel('Sizes')
plt.ylabel('Times [ms]')
plt.title('Comparison of Measure Sort Execution')
plt.legend()

plt.show()

###
# Plot 1 placeholder
###

###
# new cell
###

import matplotlib.pyplot as plt

sizes = [10, 100, 1000, 10000, 100000]
ms1_times = lists[1]
ms2_times = lists[4]
ms3_times = lists[7]

plt.figure(figsize=(10, 6)) 

plt.plot(sizes, ms1_times, marker='o', label='QS #1')
plt.plot(sizes, ms2_times, marker='s', label='QS #2')
plt.plot(sizes, ms3_times, marker='^', label='QS #3')

plt.xlabel('Sizes')
plt.ylabel('Times [ms]')
plt.title('Comparison of Quick Sort Execution')
plt.legend()
plt.show()

###
# Plot 2 placeholder
###

###
# new cell
###

import matplotlib.pyplot as plt

sizes = [10, 100, 1000, 10000, 100000]
ms1_times = lists[2]
ms2_times = lists[5]
ms3_times = lists[8]

plt.figure(figsize=(10, 6))

plt.plot(sizes, ms1_times, marker='o', label='HS #1')
plt.plot(sizes, ms2_times, marker='s', label='HS #2')
plt.plot(sizes, ms3_times, marker='^', label='HS #3')

plt.xlabel('Sizes')
plt.ylabel('Time [ms]')
plt.title('Comparison of Heap Sort Execution')
plt.legend()
plt.show()

###
# Plot 3 placeholder
###

###
# new cell
###

import matplotlib.pyplot as plt

ms_10 = ms1_times[0]
qs_10 = qs1_times[0]
hs_10 = hs1_times[0]

ms_100 = ms1_times[1]
qs_100 = qs1_times[1]
hs_100 = hs1_times[1]

ms_1000 = ms1_times[2]
qs_1000 = qs1_times[2]
hs_1000 = hs1_times[2]

x = ['10', '100', '1000']
y_ms = [ms_10, ms_100, ms_1000]
y_qs = [qs_10, qs_100, qs_1000]
y_hs = [hs_10, hs_100, hs_1000]

plt.figure(figsize=(10, 6))  # Width: 8 inches, Height: 6 inches

plt.scatter(x, y_hs, c='red', label='Heap Sort')
plt.plot(x, y_hs, c='red', linestyle='-')

plt.scatter(x, y_ms, c='blue', label='Merge Sort')
plt.plot(x, y_ms, c='blue', linestyle='--')

plt.scatter(x, y_qs, c='green', label='Quick Sort')
plt.plot(x, y_qs, c='green', linestyle='-')

plt.xlabel('Size')
plt.ylabel('Time')
plt.title('Smaller Sizes [1 - 100]')
plt.legend()
plt.show()

###
# Blue plot and the Red plot are overlapping due to the speed of the sorting
# algorithm with small arrays (size: 10, 100, 1000)
###

###
# Plot 4 placeholder
###

###
# new cell
###

import numpy as np
import matplotlib.pyplot as plt

ms1_times = lists[0]
ms2_times = lists[3]
ms3_times = lists[6]

ms_times = [ms1_times, ms2_times, ms3_times]
qs_times = [qs1_times, qs2_times, qs3_times]
hs_times = [hs1_times, hs2_times, hs3_times]

ms_avg = np.mean(ms_times, axis=0)
qs_avg = np.mean(qs_times, axis=0)
hs_avg = np.mean(hs_times, axis=0)

sizes = [1, 10, 100, 1000, 10000]
plt.figure(figsize=(10, 6))  # Width: 8 inches, Height: 6 inches

plt.plot(sizes, ms_avg, marker='o', markersize=4, alpha=1, label='Merge Sort')
plt.plot(sizes, qs_avg, marker='o', markersize=4, alpha=1, label='Quick Sort')
plt.plot(sizes, hs_avg, marker='o', markersize=4, alpha=1, label='Heap Sort')

plt.xlabel('Sizes')
plt.ylabel('Time')
plt.title('Averages of MS, QS, and HS')
plt.legend()
plt.show()

###
# Plot 5 placeholder
###

###
# new cell
###

import matplotlib.pyplot as plt

ms_1000 = ms1_times[2]
qs_1000 = qs1_times[2]
hs_1000 = hs1_times[2]

ms_10000 = ms1_times[3]
qs_10000 = qs1_times[3]
hs_10000 = hs1_times[3]

ms_100000 = ms1_times[4]
qs_100000 = qs1_times[4]
hs_100000 = hs1_times[4]

x = ['1000','10000', '1000000']
y_ms = [ms_1000, ms_10000, ms_100000] 
y_qs = [qs_1000, qs_10000, qs_100000]  
y_hs = [hs_1000, hs_10000, hs_100000]  

plt.figure(figsize=(10, 6))

plt.scatter(x, y_hs, c='red', label='Heap Sort')
plt.plot(x, y_hs, c='red', linestyle='-')

plt.scatter(x, y_ms, c='blue', label='Merge Sort')
plt.plot(x, y_ms, c='blue', linestyle='-')

plt.scatter(x, y_qs, c='green', label='Quick Sort')
plt.plot(x, y_qs, c='green', linestyle='-')

plt.xlabel('Size')
plt.ylabel('Time')
plt.title('Bigger Sizes [1000 - 100000]')
plt.legend()
plt.show()

###
# Plot 6 placeholder
###

Data

• Note: If anyone is going to review this, please copy the following data into a data.txt since that is the file my script is looking for in the directory.

[time]:2
[time]:31
[time]:168
[time]:2015
[time]:22956

[time]: 2
[time]: 13
[time]: 186
[time]: 2125
[time]: 24799

[time]:1
[time]:15
[time]:246
[time]:3192
[time]:41653

[time]: 2
[time]: 14
[time]: 169
[time]: 2061
[time]: 23578

[time]: 2
[time]: 12
[time]: 160
[time]: 2218
[time]: 26630

[time]: 1
[time]: 17
[time]: 253
[time]: 3367
[time]: 42713

[time]: 2
[time]: 14
[time]: 163
[time]: 1980
[time]: 22682

[time]: 2
[time]: 12
[time]: 164
[time]: 2092
[time]: 25826

[time]: 1
[time]: 15
[time]: 245
[time]: 3253
[time]: 40700

I would love some feedback on potential improvements and design choice recommendations. I am going into Data Science next year where the use of Jupyter Notebooks and Python is very common.

Answer 1

Your code needs a lot of improvements.

Splitting data

for line in lines:
    line = line.strip()
    if line == "":
        lists.append([])
        current_list += 1
    elif line.startswith('[time]'):
        time = int(line.split(':')[1].strip())
        lists[current_list].append(time)

In the above code, you are turning your example into this:

[[2, 31, 168, 2015, 22956],
 [2, 13, 186, 2125, 24799],
 [1, 15, 246, 3192, 41653],
 [2, 14, 169, 2061, 23578],
 [2, 12, 160, 2218, 26630],
 [1, 17, 253, 3367, 42713],
 [2, 14, 163, 1980, 22682],
 [2, 12, 164, 2092, 25826],
 [1, 15, 245, 3253, 40700]]

This is a rather complicated and inefficient way to do it.

Observe each group of lines has five items, and each number is preceded by a common prefix [time]:.

So you can just use a list comprehension to do it.

First split by two newlines '\n\n' to get each groups of five, then split the groups of five lines into individual lines and remove prefix then cast to int:

times = np.array([
    [
        int(row.removeprefix('[time]:')) 
        for row 
        in group.splitlines()
    ] 
    for group 
    in lines.split('\n\n')
])

Reorganizing data

ms1_times = lists[0]
qs1_times = lists[1]
hs1_times = lists[2]
ms2_times = lists[3]
qs2_times = lists[4]
hs2_times = lists[5]
ms3_times = lists[6]
qs3_times = lists[7]
hs3_times = lists[8]

What purpose do the above lines serve? They are absolutely unnecessary and completely useless.

If you want to access anything in the list, just use the index, like you already do.

And then in the below lines you immediately reassign the names. So the above lines really are purposeless, even in your script.

But judging from the variable names, you likely want this:

array([[    2,    31,   168,  2015, 22956],
       [    2,    13,   186,  2125, 24799],
       [    1,    15,   246,  3192, 41653],
       [    2,    14,   169,  2061, 23578],
       [    2,    12,   160,  2218, 26630],
       [    1,    17,   253,  3367, 42713],
       [    2,    14,   163,  1980, 22682],
       [    2,    12,   164,  2092, 25826],
       [    1,    15,   245,  3253, 40700]])

To become this:

array([[[    2,    31,   168,  2015, 22956],
        [    2,    13,   186,  2125, 24799],
        [    1,    15,   246,  3192, 41653]],

       [[    2,    14,   169,  2061, 23578],
        [    2,    12,   160,  2218, 26630],
        [    1,    17,   253,  3367, 42713]],

       [[    2,    14,   163,  1980, 22682],
        [    2,    12,   164,  2092, 25826],
        [    1,    15,   245,  3253, 40700]]])

It is easy to do.

Using a list comprehension based approach:

times = [times[i:i+3] for i in range(0, len(times), 3)]

But that is inefficient. You can use np.reshape to do this with better effifiency:

times = times.reshape((len(times)//15, 3, 5))

Repeated global imports

import matplotlib.pyplot as plt

The above line appeared multiple times in the global scope. Don't do that. Once you import a name, it is visible to all subsequent code within that scope, so there is absolutely no need to reimport the same name. And you need to put imported names to the top of the script to conform to PEP8.

Repeated code

sizes = [10, 100, 1000, 10000, 100000]
ms1_times = lists[0]
ms2_times = lists[3]
ms3_times = lists[6]

plt.figure(figsize=(10, 6))

plt.plot(sizes, ms1_times, marker='o', label='MS #1')
plt.plot(sizes, ms2_times, marker='s', label='MS #2')
plt.plot(sizes, ms3_times, marker='^', label='MS #3')

plt.xlabel('Sizes')
plt.ylabel('Times [ms]')
plt.title('Comparison of Measure Sort Execution')
plt.legend()

plt.show()

You used the above structure three times, with very little variation. You should make it a function. sizes is already defined and you never changed it, so there is absolutely no need to redefine it whatsoever. And since it isn't mutated, you need to make it a global immutable tuple.

Refactored:

SIZES = (10, 100, 1000, 10000, 100000)
MARKERS = ('o', 's', '^')

def plot(rows, algorithm):
    plt.figure(figsize=(10, 6))
    letter = algorithm[0].upper()
    for i, row in enumerate(rows):
        plt.plot(SIZES, row, marker=MARKERS[i], label=f'{letter}S #{i+1}')

    plt.xlabel('Sizes')
    plt.ylabel('Times [ms]')
    plt.title(f'Comparison of {algorithm} Sort Execution')
    plt.legend()

    plt.show()

Use like this:

plot(arr[:, 0], 'Measure')

Then your second and third plots:

plot(arr[:, 1], 'Quick')
plot(arr[:, 2], 'Heap')

You can do all three in one loop:

algorithms = ('Measure', 'Quick', 'Heap')
for i, algorithm in enumerate(algorithms):
    plot(arr[:, i], algorithm)

Repetition

Your next three code blocks also contain a huge amount of repetition, and also need to be refactored into functions. But I have things to do, and if I do it all for you you won't learn anything. I will leave it as an assignment for you.

Some tips for you:

ms1_times, qs1_times, hs1_times is times[0] if you follow my instructions..

ms_times = times[:, 0]
qs_times = times[:, 1]
hs_times = times[:, 2]

times[..., 2]
times[..., 3]
times[..., 4]

Do you know what the above three examples mean?

Answer 2

Use the most appropriate tool for the job

Jupyter Notebook can be a great tool for some use cases. It is designed to allow combining code, rich text, and presenting outputs. It is indeed fairly widely used in academia (in many fields, including data science) as it can be a great teaching and presentational tool.

Your example code, however, doesn't take advantage of these features. It runs just fine (although it can be improved, see Ξένη-Γήινος's answer) as a standard Python script, as there are no text/markdown cells.

Considering that, there are two paths you can take:

• Use vanilla Python: this is the most boring path, as you can simply run your current code as a Python script
• Use Jupyter Notebook's features: your code processes some unlabeled data and presents it as a handful of graphs. You can add value to that data by explaining where it comes from, why each graph presents an interesting result, what conclusions you can draw from those results... This is what this tool is about.