5
\$\begingroup\$

I'm doing a side project at work and I consider it as a learning opportunity more than work itself, but it does hold a purpose if I can complete it. Anyway, I'll post the code, what it actually does and hopefully people can critique/improve what I've done.

os.system("bjobs -u all| awk ' NF>1' > file")

lines = open('file', 'r').readlines()
del lines[0]
open('file', 'w').writelines(lines)

source = open("file")                           
out = open("file1", "w")                              
for line in source:                             

        out.write(line.split(" ")[0] + "\n")            

out.close()
source.close()

This code does 2 parts. Firstly the bjobs -u all| awk ' NF>1' > file will print this to a file (plus however many jobs are running):

JOBID   USER    STAT  QUEUE      FROM_HOST   EXEC_HOST   JOB_NAME   SUBMIT_TIME
111111  ######   ###   ###        ######     comp113     ########   Mar 27 12:50
                                             comp113
                                             comp113
                                             comp113
222222  ######   ###   ###        ######     comp114     ########   Mar 27 12:50
                                             comp114
                                             comp114
                                             comp114

Then remove the first line of the file ("JOBID USER STAT" etc) as this isn't needed.

Then strip the additional "comp113" lines away (this is done using the awk command to remove any lines that don't contain more than 1 field - maybe it's a bad idea mixing shell commands to format text within Python) leaving just:

111111  ######   ###   ###        ######     comp113     ########   Mar 27 12:50
222222  ######   ###   ###        ######     comp114     ########   Mar 27 12:50

From this, extract just the JOBID and store it within a separate file for later use (this is done by deleting any characters after the first whitespace in the line).

Mainly I'm just looking at maybe how I can tidy the code up and improve on anything I've done. I do plan on commenting the code when I get around to it. Anything that I've left unexplained please do ask and I'll do my best to explain. There is a lot more to this project which I plan on asking questions about in the future! Just didn't want to get ahead of myself.

\$\endgroup\$
7
\$\begingroup\$

I'm having a major doubt: should I give you the same advice I gave you the last time here? Would you follow them this time? Or there was something not clear?

Well, I'll try my best.

Don't use os.system

Use the subprocess module. Quoting from the doc:

This module intends to replace several other, older modules and functions, such as:

os.system
os.spawn*

Use next() to skip the header

file ibjects are iterable, so to skip the header usually one does:

with open('example.txt') as f:
    next(f)
    for line in f:
        # start to work on the lines here

But I'd say that you won't need that one if you don't really need to have the file file on your hard disk.

Check the output

It seems that you don't really need the file file to exist, so you could just check the output of your command with subprocess.check_output().

Notice that subprocess.check_output() will Run command with arguments and return its output as a byte string. So you'll have to call the .decode() method on it. If you don't have any idea of what I'm talking about take a look at the doc here.

Use the with statement

with open("file1", "w") as output:
    # ...

.split() vs. .split(" ")

Those two have different behaviours:

>>> s = 'This is        a test\tstring'
>>> s.split()
['This', 'is', 'a', 'test', 'string']
>>> s.split(" ")
['This', 'is', '', '', '', '', '', '', '', 'a', 'test\tstring']

.split() will split multiple spaces and tabs too.

You don't need awk

After checking the output just skip the line that you don't want:

cmd_output = subprocess.check_call(...).decode()
with open('file1.txt') as output_file:
    for line in cmd_outpu.split('\n')[1:]:    # [1:] will skip the header
        splitted_line = line.split()
        if len(splitted_line) > 1:
            output_file.write(splitted_line[0] + '\n')

You could obviously do it without decoding and working with bytes, but this way it's more clear to start with.

| improve this answer | |
\$\endgroup\$
  • \$\begingroup\$ I literally only used your previous answer to finish the problem I was having (which you did thank you). Now I'm looking to tidy my code up (funnily I found this website through your profile) and was already applying some comments in your previous answer and now you've gave me some more help so I do appreciate your effort! I can post the rest of the code if it'll help make sense of what I'm trying to do. Thanks again. \$\endgroup\$ – ashleh Mar 27 '12 at 15:38
  • \$\begingroup\$ @AshleyJohnKent: Happy to help :) \$\endgroup\$ – Rik Poggi Mar 27 '12 at 20:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.