# Read group of keywords from file, modify value, store new group to variable

I would like to use the following routine in my submission script for GAMESS calculations. I am not entirely sure if this is the optimum way to go.

This function would need one of the messaging routines, message, to function properly, but I decided to put echo in front of it, since this is basically what it does any way. It is taken from an earlier review of me. I am not posting the whole script due to its length, and I am not yet finished. But in order to avoid rewriting it I hope to get some insight here. I need to program similar routines for different keywords of the program, so I'd like to use this one as a template.

The function parses an input file for the GAMESS program and substitutes part of it if present with values supplied through the script in order to match the request, that is later given to the queueing system. I'll include one input file in this post.

What I am completely unsure about is that I am calling tr in more than one occasion and if there is a batter way to extract the command group(s) in the first place. Note that there could be multiple occurrences throughout the whole input.

#!/bin/sh

#Values obtained from main script
file=$1 mem=$2

#
# Parsing the specified inputfile
#

modifySystemGroup ()
{
# $1 is the used inputfile #$2 is the requested memory in mwords
local truncSystemGroup
# Extracting the SYSTEM command group, delete newlines
readSystemGroup=$(sed -n "/[[:space:]]\+$[sS][yY][sS][tT][eE][mM][ ]*/,/$[eE][nN][dD]/p"$1\
| tr -d "\r\n" )
if [ -z "$readSystemGroup" ]; then echo message "No $SYSTEM group detected. Default will be added." else echo read : \'"readSystemGroup"\' #Delete any SYSTEM an END words truncSystemGroup={readSystemGroup//$[sS][yY][sS][tT][eE][mM]/} truncSystemGroup=${truncSystemGroup//$[eE][nN][dD]/} #Delete any MWORDS statement since it will be replaced by script values if [[ truncSystemGroup =~ [mM][wW][oO][rR][dD][sS]=[[:digit:]]*[[:space:]] ]]; then truncSystemGroup={truncSystemGroup//{BASH_REMATCH[0]}/} fi fi writeSystemGroup=(echo "$SYSTEM MWORDS=$2$truncSystemGroup $END" | tr -s [:space:] ) echo message "Applied 'MWORDS=2' to the input file." } modifySystemGroup file mem echo write: \'"writeSystemGroup"\' Sample input file (It is very important, that the keywords ... are indented by at least on space. GAMESS would ignore them otherwise.) CONTRL EXETYP=CHECK END SYSTEM MWORDS=100 END CONTRL SCFTYP=RHF RUNTYP=OPTIMIZE END SYSTEM Parall=.t. END GUESS GUESS=HUCKEL END DATA Water C1 HYDROGEN 1.0 -0.754909 0.000000 0.563845 S 3 1 13.0107010 0.19682158E-01 2 1.9622572 0.13796524 3 0.44453796 0.47831935 S 1 1 0.12194962 1.0000000 P 1 1 0.8000000 1.0000000 HYDROGEN 1.0 0.754909 0.000000 0.563845 S 3 1 13.0107010 0.19682158E-01 2 1.9622572 0.13796524 3 0.44453796 0.47831935 S 1 1 0.12194962 1.0000000 P 1 1 0.8000000 1.0000000 OXYGEN 8.0 0.000000 0.000000 -0.087201 S 5 1 2266.1767785 -0.53431809926E-02 2 340.87010191 -0.39890039230E-01 3 77.363135167 -0.17853911985 4 21.479644940 -0.46427684959 5 6.6589433124 -0.44309745172 S 1 1 0.80975975668 1.0000000 S 1 1 0.25530772234 1.0000000 P 3 1 17.721504317 0.43394573193E-01 2 3.8635505440 0.23094120765 3 1.0480920883 0.51375311064 P 1 1 0.27641544411 1.0000000 D 1 1 1.2000000 1.0000000 END Here are alternative headers (part until the empty line), that all need to result in the same line returned from the function: CONTRL EXETYP=CHECK END SYSTEM MWORDS=100 END CONTRL SCFTYP=RHF RUNTYP=OPTIMIZE END CONTRL EXETYP=CHECK END SYSTEM MWORDS=100 Parall=.t. END CONTRL SCFTYP=RHF RUNTYP=OPTIMIZE END CONTRL EXETYP=CHECK END SYSTEM MWORDS=100 Parall=.t. END CONTRL SCFTYP=RHF RUNTYP=OPTIMIZE END CONTRL EXETYP=CHECK END SYSTEM MWORDS=100 Parall=.t. END CONTRL SCFTYP=RHF RUNTYP=OPTIMIZE END CONTRL EXETYP=CHECK END SYSTEM MWORDS=100 Parall=.t. END CONTRL SCFTYP=RHF RUNTYP=OPTIMIZE END There could be many, many more cases, that split the command group in different statements, adding other keywords, spanning multiple lines, and more that I cannot think of. For my purposes, even when the MWORDS=... statement is missing, it needs to be inserted, hence the following must also produce the same result as the other examples: CONTRL EXETYP=CHECK END CONTRL SCFTYP=RHF RUNTYP=OPTIMIZE END SYSTEM Parall=.t. END Expected output The output of the function itself must produce be the same. The examples differ obviously in what is read from the file. Hence echo read : \'"readSystemGroup"\' produces different things, but this is just a control statement. That can be ignored - I will probably delete it in the final version anyway, so the log file is a bit cleaner. The important information is in any case what is stored in writeSystemGroup, which must be in any case the same. Currently the above routine produces this, given mem=500: read : ' ... ' message Applied 'MWORDS=500' to the input file. write: ' SYSTEM MWORDS=500 Parall=.t. END' And that is the behaviour I desire. ## 1 Answer ### Consider replacing with Perl It looks to me that essentially you are converting input of this format: SYSTEM MWORDS=100 ... SYSTEM Parall=.t. ... To output like this: SYSTEM MWORDS=NNNN Parall=.t. END Where NNNN is a parameter of the script. The logic seems to be: • Keep the values that appear after SYSTEM • ... except if it's MWORDS=... • Form the output line by joining the values found The scrips achieves in a fairly complicated way, using a series of sed, awk, Bash pattern substitution, and conditional statements. Here's a Perl that compresses all that into a single line: values=(perl -ne '/$system (.*)/i && do { print "$1 " if$1 !~ /^mwords/i; }' "$1") From which you can create the final writeSystemGroup value like this: writeSystemGroup=" $SYSTEM MWORDS=2 values$END" How does this work and why is it better? • Simpler logic, focused on the$SYSTEM line that seems to carry all the information we need
• Simpler regex, thanks to the /.../i flag of Perl
• Simpler extraction of the values using the capture group (.*), and the conditional to exclude the MWORDS value
• A single process instead of multiple processes
• This Perl works in osx too, unlike the original sed

Update

After you added more variations of inputs, the Perl needs a bit more work. Here's the complete rewritten modifySystemGroup function:

modifySystemGroup() {
local values
values=$( perl -e ' chomp(@_ = <>);$_ = join(" ", @_);
s/ {2,}/ /g;
@matches = / $system (.*?)$end/ig;
for (@matches) {
$value = ""; for (split(/ /)) {$value .= "$_ " if !/^mwords=/i; } print$value if $value; }' "$1"
)
writeSystemGroup=" $SYSTEM MWORDS=2 values$END"
echo message "Applied 'MWORDS=$2' to the input file." } ### Other issues and improvements The script starts with #!/bin/sh, but actually it's using several Bash features that not all implementations of /bin/sh will have. So it's better to make that clear by making the first line #!/bin/bash. Instead of this: echo read : \'"$readSystemGroup"\'

This is exactly the same, just simpler to write and read:

echo read : "'$readSystemGroup'" Remember to enclose path variables within double-quotes. You failed to do that for$1 here:

readSystemGroup=$(sed -n "/[[:space:]]\+$[sS][yY][sS][tT][eE][mM][ ]*/,/$[eE][nN][dD]/p"$1 | tr -d "\r\n" )
• I have never used Perl, that would be completely new for me. How does that Perl command know, it needs to read until $END? The commands don't have to follow the initial key word, there could be a line break there, or the two groups could have been combined in one four line group. Would it still work then? (I'm at home now, on mobile, so I can't update more example input files.) Is Perl readily available in a standard Linux distribution or does it need to be installed? – Martin - マーチン Apr 2 '16 at 3:03 • @Martin-マーチン Perl is readily available in Linux, OSX, *BSD, Solaris, and even in Git Bash. My rewrite using Perl is based on the input you showed, and may not work with other input. Even so, it can certainly be adapted to make it work, and still be far simpler than a mixture of sed+tr+bash. – janos Apr 2 '16 at 8:30 • I have looked into perl and am more confused than ever. While I appreciate the tools it provides, I am too busy to learn the language in order to extend the command you proposed. I included more examples in the original post, so that it is more obvious, how big the variety of inputs is. – Martin - マーチン Apr 4 '16 at 11:25 • @Martin-マーチン please also add the expected output for each example – janos Apr 4 '16 at 11:28 • I have extended the part of the output. The crucial part of those examples is that they all need to produce the same value within the variable$writeSystemGroup. – Martin - マーチン Apr 4 '16 at 11:46