0
\$\begingroup\$

I try to create compare script using awk and another command and running successful.

But I think the script I create it's to long.

Is there anyone can shorten my script below ?

After plan A shorten the code than, plan B is :

 1. I want eliminated a lot temp file (.txt), only need `lengkap.txt`
 2. Put command in variable if can

It's my pleasure if anyone can help me

Below is the code that I create based searching and trying.
#!/bin/bash

### Path Folder who will be compare ###
path1=/home/rio/apps1
path2=/home/rio/apps2

### Find all filename and convert to MD5 ###
find $path1 -type f | xargs md5sum > checksums.md5
find $path2 -type f | xargs md5sum > checksums2.md5

### Compare to find different folder ###
awk 'NR==FNR{c[$1]++;next};c[$1] == 0' checksums.md5 checksums2.md5 > hasil1.txt
awk 'NR==FNR{c[$1]++;next};c[$1] == 0' checksums2.md5 checksums.md5 > hasil2.txt

### Merge result of compare ###
awk '{print $0}' hasil1.txt hasil2.txt > perbedaan.txt

### Filter Just Filename Difference ###
cat perbedaan.txt | awk '{print $2}' > hasilperbedaan.txt

### File about result compare (just filename) ###
cekhasil=/home/rio/hasilperbedaan.txt

### Check if File result compare empty or not ###
if [ -s "$cekhasil" ]
then 
   echo " file exists and is not empty "
    ### Find All filename and date, after that put as we want ###
    find $path1 -type f -ls | awk '{print $11" "$8" "$9" "$10 }' > filedate1.txt
    find $path2 -type f -ls | awk '{print $11" "$8" "$9" "$10 }' > filedate2.txt

    ### Compare to get the date of filename ###
    awk 'A[$1]++' hasilperbedaan.txt filedate1.txt > pre_hasil1.txt
    awk 'A[$1]++' hasilperbedaan.txt filedate2.txt > pre_hasil2.txt
    ### Merge result of compare with date ###
    awk '{print $0}' pre_hasil1.txt pre_hasil2.txt > lengkap.txt
else
   echo " file does not exist, or is empty "
fi
\$\endgroup\$
6
  • 4
    \$\begingroup\$ Welcome to Code Review Stack Exchange! You might find useful the advice on choosing a good title for a question. You can edit yours. codereview.stackexchange.com/help/how-to-ask \$\endgroup\$ Oct 14, 2020 at 12:43
  • 3
    \$\begingroup\$ Please tell us what the code is supposed to do. \$\endgroup\$
    – pacmaninbw
    Oct 14, 2020 at 12:50
  • \$\begingroup\$ one thing that seems fishy about the script is that it writes a lot of files in cwd, which are all not cleaned after the job is done. \$\endgroup\$
    – hjpotter92
    Oct 15, 2020 at 19:58
  • \$\begingroup\$ @pacmaninbw it's for compare folder with different server but same folder (example : apps). \$\endgroup\$
    – Ario
    Oct 16, 2020 at 7:42
  • \$\begingroup\$ @hjpotter92, yes, I think so too. That's why need advice to shorten the script. \$\endgroup\$
    – Ario
    Oct 16, 2020 at 7:43

1 Answer 1

3
\$\begingroup\$

Well, it took time to figure out what your program is doing. So I did shorten it in two phases.

First phase: I removed all duplicate temporary files, and used a pipe when a temp file was used once.

STEP1: you just make a checksum of all files in $path1 and $path2

### Find all filename and convert to MD5 ###
find $path1 -type f | xargs md5sum > checksums.md5
find $path2 -type f | xargs md5sum > checksums2.md5

You don't need 2 temp files, as path is included in filename. So you can replace this with a single find. I use sort to be able to use uniq afterwards:

find "$path1" "$path2" -type f | xargs md5sum  | sort > cksum.md5

STEP2: you find unique checksums in both files, and get the corresponding filenames.

### Compare to find different folder ###
awk 'NR==FNR{c[$1]++;next};c[$1] == 0' checksums.md5 checksums2.md5 > hasil1.txt
awk 'NR==FNR{c[$1]++;next};c[$1] == 0' checksums2.md5 checksums.md5 > hasil2.txt

### Merge result of compare ###
awk '{print $0}' hasil1.txt hasil2.txt > perbedaan.txt

### Filter Just Filename Difference ###
cat perbedaan.txt | awk '{print $2}' > hasilperbedaan.txt

As we have a checksum-sorted file, we just filter with uniq, and get filenames.

Note: As we use a MD5 checksum (128 bits), length is 128bits / 8bits * 2hex=32

uniq -u -w32 cksum.md5 | awk '{print $2}' > "$cekhasil"

If you prefer awk to uniq, just compare previous record's $1 with current's.

STEP 3: you match found entries with a new search (find) in the two source paths.

if [ -s "$cekhasil" ]
then
    echo " file exists and is not empty "
    ### Find All filename and date, after that put as we want ###
    find $path1 -type f -ls | awk '{print $11" "$8" "$9" "$10 }' > filedate1.txt
    find $path2 -type f -ls | awk '{print $11" "$8" "$9" "$10 }' > filedate2.txt

    ### Compare to get the date of filename ###
    awk 'A[$1]++' hasilperbedaan.txt filedate1.txt > pre_hasil1.txt
    awk 'A[$1]++' hasilperbedaan.txt filedate2.txt > pre_hasil2.txt
    ### Merge result of compare with date ###
    awk '{print $0}' pre_hasil1.txt pre_hasil2.txt > lengkap.txt
else
    echo " file does not exist, or is empty "
fi

Here, it is better to avoid new full find, and just loop over filenames.

if [ -s "$cekhasil" ]
then
    echo " file exists and is not empty "
    while read -r fn ; do
        ls -dils "$fn" | awk '{print $11" "$8" "$9" "$10 }'
    done < "$cekhasil" > lengkap2.txt

else
    echo " file does not exist, or is empty "
fi

Alltogether, the script becomes:

#!/bin/bash

path1=/home/br/dev/tools/bash
path2=/home/br/dev/tools/bash2
outfile=hasilperbedaan.txt

find "$path1" "$path2" -type f | xargs md5sum  | sort > cksum.md5

uniq -u -w32 cksum.md5 | awk '{print $2}' > "$outfile"

if [ -s "$outfile" ]; then
    echo " file exists and is not empty "
    while read -r fn ; do
        ls -dils "$fn" | awk '{print $11" "$8" "$9" "$10 }'
    done < "$output" > lengkap2.txt

else
    echo " file does not exist, or is empty "
fi

At this point, we can notice the 2 remaining temp files are used only once. We just get rid of them, and just pipe all together to get the output file.

    #!/bin/bash
    
    path1=/home/br/dev/tools/bash
    path2=/home/br/dev/tools/bash2
    outputfile="lengkap-v2.txt"
    
    find "$path1" "$path2" -type f | xargs md5sum  | sort |
        uniq -u -w32 | awk '{print $2}' |
        while read -r fn ; do
            ls -dils "$fn" | awk '{print $11" "$8" "$9" "$10 }'
        done > "$outputfile"
    [[ -s "$outputfile" ]] && echo " file exists and is not empty " ||
            echo " file does not exist, or is empty "

Notes:

  • I don't understand your second question: "Put command in variable if can", so I skip that one.
  • You should never use the output of find (except with -print0 options) or ls as input of any command. But it is out of scope for this answer.
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.