0
\$\begingroup\$

This script is part of the Cameleonica project, which aims at creating a versatile filesystem. This script measures performance characteristics of a disk: how long are seeks on average, and so on.

I would appreciate if reviewers would also upload their output to gist, if convenient. I only have access to 2 hard disks for testing.

#!/usr/bin/python3
import sys, os, time, timeit, random, ctypes


#--------------------------------------------------------------------------------------------------

def BytesString(n):
    suffixes = ['B','KB','MB','GB','TB','PB','EB','ZB','YB']
    suffix = 0
    while n % 1024 == 0 and suffix+1 < len(suffixes):
        suffix += 1
        n //= 1024
    return '{0}{1}'.format(n, suffixes[suffix])

def BytesInt(s):
    if all(c in '0123456789' for c in s):
        return int(s)
    suffixes = ['B','KB','MB','GB','TB','PB','EB','ZB','YB']
    for power,suffix in reversed(list(enumerate(suffixes))):
        if s.endswith(suffix):
            return int(s.rstrip(suffix))*1024**power
    raise ValueError('BytesInt requires proper suffix ('+' '.join(suffixes)+').')

def BytesStringFloat(n):
    x = float(n)
    suffixes = ['B','KB','MB','GB','TB','PB','EB','ZB','YB']
    suffix = 0
    while x > 1024.0 and suffix+1 < len(suffixes):
        suffix += 1
        x /= 1024.0
    return '{0:0.2f}{1}'.format(x, suffixes[suffix])


#--------------------------------------------------------------------------------------------------

if len(sys.argv) < 2:
    print('Syntax: program /dev/sda > log')
    print('Path can also use /dev/disk/by-id/  by-label/  by-path/  by-uuid/')
    print('Redirect to a log file is optional.')
    sys.exit()

dev = os.path.realpath(sys.argv[1]).split('/')[-1]
disk = open('/dev/%s' % dev, 'rb')
disksize = disk.seek(0, 2)
os.system('echo noop | sudo tee /sys/block/%s/queue/scheduler > /dev/null' % dev)

print('Disk name: {0}  Disk size: {1}  Scheduler disabled.'.format(
    disk.name, BytesStringFloat(disksize)))


#--------------------------------------------------------------------------------------------------

libc = ctypes.CDLL(None, use_errno=True)

def readahead(fileno, offset, count):
    libc.readahead(ctypes.c_int(fileno), ctypes.c_longlong(offset), ctypes.c_size_t(count))

bufsize = 512
bufcount = 100

print()
print('Measuring: Concurrent random seek time using readahead.')
print('Samples: {0}   Sample size: {1}'.format(
    bufcount, bufsize))

for area in [BytesInt('1MB')*2**i for i in range(0,64)]+[disksize]:
    if area > disksize:
        continue

    os.system('echo 3 | sudo tee /proc/sys/vm/drop_caches > /dev/null')

    offsets = [random.randint(0, area-bufsize) for i in range(bufcount)]

    for i in offsets:
        readahead(disk.fileno(), i, bufsize)

    times = [timeit.timeit(lambda: os.pread(disk.fileno(), bufsize, i), number=1) for i in offsets]

    print('Area tested: {0:6}   Average: {1:5.2f} ms   Max: {2:5.2f} ms   Total: {3:0.2f} sec'.format(
        BytesString(area) if area < disksize else BytesStringFloat(area), 
        sum(times)/len(times)*1000, max(times)*1000, sum(times)))


#--------------------------------------------------------------------------------------------------

bufsize = 512
bufcount = 100

print()
print('Measuring: Random seek time using beginning of disk.')
print('Samples: {0}   Sample size: {1}'.format(
    bufcount, bufsize))

for area in [BytesInt('1MB')*2**i for i in range(0,64)]+[disksize]:
    if area > disksize:
        continue

    os.system('echo 3 | sudo tee /proc/sys/vm/drop_caches > /dev/null')

    offsets = [random.randint(0, area-bufsize) for i in range(bufcount)]

    os.pread(disk.fileno(), bufsize, 0)
    times = [timeit.timeit(lambda: os.pread(disk.fileno(), bufsize, i), number=1) for i in offsets]

    print('Area tested: {0:6}   Average: {1:5.2f} ms   Max: {2:5.2f} ms   Total: {3:0.2f} sec'.format(
        BytesString(area) if area < disksize else BytesStringFloat(area), 
        sum(times)/len(times)*1000, max(times)*1000, sum(times)))


#--------------------------------------------------------------------------------------------------

print()
print('Measuring: Random read throughput with various sizes.')

for i in range(8):
    bufsize = BytesInt('1MB')*2**i
    bufcount = int(128/((4/3)**i))

    os.system('echo 3 | sudo tee /proc/sys/vm/drop_caches > /dev/null')

    offsets = [random.randint(0, disksize-bufsize) for i in range(bufcount)]

    times = [timeit.timeit(lambda: os.pread(disk.fileno(), bufsize, i), number=1) for i in offsets]

    avg = bufsize/(sum(times)/len(times))
    print('Buffer: {0:4}   Average: {1:8}/sec   Samples: {2:3}   Total: {3:0.2f} sec'.format(
        BytesString(bufsize), BytesStringFloat(avg), bufcount, sum(times)))


#--------------------------------------------------------------------------------------------------

print()
print('Measuring: Sequential read throughput using beginning of disk.')

bufsize = BytesInt('10MB')
bufcount = 100

os.system('echo 3 | sudo tee /proc/sys/vm/drop_caches > /dev/null')

disk.seek(0)
times = [timeit.timeit(lambda: disk.read(bufsize), number=1) for i in offsets]

avg = bufsize/(sum(times)/len(times))
print('Buffer: {0:4}   Average: {1:8}/sec   Samples: {2:3}   Total: {3:0.2f} sec'.format(
    BytesString(bufsize), BytesStringFloat(avg), bufcount, sum(times)))


#--------------------------------------------------------------------------------------------------

os.system('echo cfq | sudo tee /sys/block/%s/queue/scheduler > /dev/null' % dev)

print()
print('Returned disk scheduler to CFQ.')
\$\endgroup\$
2
\$\begingroup\$

Python's official style guide is PEP-8 and you have quite a few areas of noncompliance with its guidance, including:

  • Multiple imports on one line (and not in alphabetical order);
  • Missing whitespace;
  • CamelCase names where lowercase_with_underscores should be used;
  • Lines in excess of 80 characters (the #--- comments imply you've adopted a 100 character limit, but you haven't kept to that, either...); and
  • Single-character identifiers (e.g. n, i), including reusing the same identifier (i) within the same block.

It would also be good to include docstrings for the functions you define, and standardise on either % or str.format for string interpolation.


You repeatedly define:

suffixes = ['B','KB','MB','GB','TB','PB','EB','ZB','YB']

in functions, rather than defining it globally once at the top of the file:

SUFFIXES = ['B', 'KB', 'MB', 'GB', 'TB', 'PB', 'EB', 'ZB', 'YB']

(note naming convention, whitespace).


Rather than printing multiple lines:

print('Syntax: program /dev/sda > log')
print('Path can also use /dev/disk/by-id/  by-label/  by-path/  by-uuid/')
print('Redirect to a log file is optional.')

it would be neater to print a single multiline block (see e.g. Avoiding Python multiline string indentation):

print(dedent('''
    Syntax: program /dev/sda > log
    Path can also use /dev/disk/by-id/  by-label/  by-path/  by-uuid/
    Redirect to a log file is optional.
'''))

Better yet, use e.g. argparse to provide a more conventional command line interface to users of your program.

You can also start a string with \n to include a line break, rather than preceding it with an empty print().


You have a lot of code running at the top level of your script, which makes it more difficult to import and reuse elsewhere. You should make a single entry point, then call it from within an if __name__ == '__main__': guard clause (see e.g. What does if __name__ == “__main__”: do?)

| improve this answer | |
\$\endgroup\$
  • \$\begingroup\$ For the record, I do believe in the idea of word wrap so sticking to 80 characters per line is something I see as a funny thing of the long forgotten past. But then a lot of people use PEP8, so... I will make use of your suggestions as much as possible. \$\endgroup\$ – ArekBulski Aug 9 '16 at 21:00
  • \$\begingroup\$ @ArekBulski In part, the character limit is about preventing you from letting lines get too long and complex to read easily. If you can't fit a line in 80 characters, you might be trying to do too much in one go. \$\endgroup\$ – SuperBiasedMan Aug 10 '16 at 9:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.