Extending this answer solution, I've wrote the following bash script to get the html of my StackExchange accounts page and parse it to a YAML file. The objective was list for each community I participate its url and my user id.



NETWORK_PROFILE_URL=$1 # https://stackexchange.com/users/3581081/artu-hnrq
read -r ID USER <<< \
    $( cut -d'/' -f5,6 <<< $NETWORK_PROFILE_URL \
        | sed 's/\// /g' \


echo \
"user: $USER
  url: http://stackexchange.com
  id: $ID" \

    local HTTPS="https:\/\/"
    local COMMUNITY="[^\.]*"
    local DOMAIN="[^\/]*"
    local PROFILE_ID="\d*"

    local MODEL="\1:\n  url: $HTTPS\1\2\n  id: \3"


curl "https://stackexchange.com/users/$ID/$USER?tab=accounts" \
    | grep "account-container" -A1 \
    | grep "http" \
    | cut -d"/" -f3,5 \ # stackoverflow.com/3581081
    | parse \

I would like to receive some opinions about the parse process (usage of grep, cut and sed commands) and about control flow management could be improved to the script

output example:

user: artu-hnrq
  url: https://stackexchange.com
  id: 3581081
  url: https://askubuntu.com
  id: 689894
  url: https://stackoverflow.com
  id: 2989289
  url: https://codereview.stackexchange.com
  id: 213097

Thanks a lot!


1 Answer 1


I see 2 errors and have some more advice.

The Shebang is wrong.

should be

Do not put comment after the \ escape for a new line

| cut -d"/" -f3,5 \ # stackoverflow.com/3581081
# should be
| cut -d"/" -f3,5 \

I would move all functions (in your case only parse()) to the top of the file.

Use lowercase variables (uppercase is reserved for system variables like PATH).

# change to

Combine cut and sed

read -r ID USER <<< \
    $( cut -d'/' -f5,6 <<< $NETWORK_PROFILE_URL \
        | sed 's/\// /g' \
# can be replaced by
read -r ID USER < <(awk -F "/" '{print $(NF-1) " " $NF }' <<< $NETWORK_PROFILE_URL)
# which is easier to read then
source <(sed -r 's#.*/(.*)/(.*)#ID=\1; USER=\2#' <<< $NETWORK_PROFILE_URL)

When you end a line with a pipe, you do not need to escape the newline with \.

curl "https://stackexchange.com/users/$ID/$USER?tab=accounts" \
    | grep "account-container" -A1 \
    | grep "http" \
    | cut -d"/" -f3,5 \
    | parse 
# can be changed into
curl "https://stackexchange.com/users/$ID/$USER?tab=accounts" |
    grep "account-container" -A1 |
    grep "http" |
    cut -d"/" -f3,5 |

You had a hard job parsing the curl output with different tools. In this situation you won't need to parse large files, the performancy penalty is very small. It is worth the effort for other cases to look for a solution with awk.
The grep -A1 is replaced by setting a flag for parsing the next line.
Fields are found with the delimiter '/'.
Combining substr is only slightly more complex:

curl -s "https://stackexchange.com/users/$ID/$USER?tab=accounts" |
   awk -F "[/]" '
      parse==1 {
         print substr($3,1,index($3,".")-1) ":";
         printf("  url: https://%s\n", $3);
         printf("  id: %s\n", $5);
      /account-container/ {parse=1}
  • 1
    \$\begingroup\$ Hi, Walter! Thanks for your attention here. I'll improve the script with your suggestions \$\endgroup\$
    – artu-hnrq
    Mar 21, 2021 at 23:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.