Script to loop through a list of YouTube channels, and output metadata to a Markdown file

Question

Overview

I have created a bash script (triggered via GitHub Actions) that does the following:

1. Parse a list of YouTube channel IDs and nicknames.
2. Fetch their metadata via YouTube's Channel API.
3. Build up Markdown tables using this metadata.
4. Load a Markdown template, and replace a placeholder with the generated Markdown.

Additional functionality implemented is:

1. Display optional arbitrary emoji next to specific channel names (${ARRAY_LINE[2]}).
2. Format numbers to be human readable (e.g. 1200 -> 1.2K).
3. Log channels being processed.

Whilst I've run the code through ShellCheck and made other improvements, I suspect there are weaknesses around:

• Parsing output.json 5x, fetching a different field each time.
• Replacing the placeholder text.

Code

The script itself youtube-update.sh:

#!/bin/bash

HEADER_PREFIX="#### "
PLACEHOLDER_TEXT="dynamic-channel-data"
OUTPUT=""

# Convert list of channels into Markdown tables
while read -r LINE; do
    if [[ ${LINE} == ${HEADER_PREFIX}* ]]; then
        echo "Adding header ${LINE}"
        OUTPUT="${OUTPUT}\n${LINE}\n\n"
        OUTPUT="${OUTPUT}| Channel | # Videos | Subscribers | Views |\n| --- | --- | --- | --- |\n"
    else
        IFS=';' read -r -a ARRAY_LINE <<< "${LINE}" # Split line by semi-colon
        echo "Adding channel ${ARRAY_LINE[1]} (${ARRAY_LINE[0]})"
        curl "https://youtube.googleapis.com/youtube/v3/channels?part=statistics,snippet&id=${ARRAY_LINE[0]}&key=${API_KEY}" \
            --header 'Accept: application/json' \
            -fsSL -o output.json

        # Pull channel data out of response if possible
        if [[ $(jq -r '.pageInfo.totalResults' output.json) == 1 ]]; then
            TITLE=$(jq -r '.items[0].snippet.title' output.json)
            URL=$(jq -r '.items[0].snippet.customUrl' output.json)
            VIDEO_COUNT=$(jq -r '.items[0].statistics.videoCount' output.json | numfmt --to=si)
            SUBSCRIBER_COUNT=$(jq -r '.items[0].statistics.subscriberCount' output.json | numfmt --to=si)
            VIEW_COUNT=$(jq -r '.items[0].statistics.viewCount' output.json | numfmt --to=si)
            echo "Added ${TITLE}: ${VIDEO_COUNT} videos (${VIEW_COUNT} views)"
            OUTPUT="${OUTPUT}| ${ARRAY_LINE[2]}[${TITLE}](https://youtube.com/${URL}) | ${VIDEO_COUNT} | ${SUBSCRIBER_COUNT} | ${VIEW_COUNT} |\n"
        else
            echo "Failed! Bad response received: $(<output.json)"
            exit 1
        fi
    fi
done < "${WORKSPACE}/automation/channels.txt"

# Replace placeholder in template with output, updating the README
TEMPLATE_CONTENTS=$(<"${WORKSPACE}/automation/template.md")
echo -e "${TEMPLATE_CONTENTS//${PLACEHOLDER_TEXT}/${OUTPUT}}" > "${WORKSPACE}/README.md"

# Debug
cat "${WORKSPACE}/README.md"

For additional context, this script is triggered via a GitHub actions workflow (metadata-update.yml):

name: Update YouTube stats

on:
  schedule:
    - cron: '0 8 * * *'
  workflow_dispatch:

jobs:
  metadata-update:
    runs-on: ubuntu-latest
    permissions:
      contents: write
    steps:
    - name: Checkout channel config file
      uses: actions/[email protected]
      with: 
        sparse-checkout: |
          automation/*
          README.md
        sparse-checkout-cone-mode: false
    - name: Update YouTube data
      run: |
        chmod +x ./automation/youtube-update.sh
        ./automation/youtube-update.sh
      env:
          API_KEY: ${{ secrets.API_KEY }}
          WORKSPACE: ${{ github.workspace }}
    - name: Save changes
      uses: stefanzweifel/git-auto-commit-action@v4
      with:
        commit_message: Updated YouTube statistics
        commit_author: GitHub Actions <[email protected]>
        file_pattern: 'README.md'

The YouTube API response (truncated to relevant fields) looks like:

{
  "pageInfo": {
    "totalResults": 1
  },
  "items": [
    {
      "snippet": {
        "title": "Google for Developers",
        "customUrl": "@googledevelopers"
      },
      "statistics": {
        "viewCount": "234466180",
        "subscriberCount": "2300000",
        "videoCount": "5807"
      }
    }
  ]
}

Examples

• Console log of the script executing
• Example channel list
• Example template
• Example output file.

A typical run might convert:

#### Stream Archives
UC2oWuUSd3t3t5O3Vxp4lgAA;2018-streams;🐶
UC4ik7iSQI1DZVqL18t-Tffw;2016-2018streams
UCjyrSUk-1AGjALTcWneRaeA;2016-2017streams

into:

#### Stream Archives

| Channel | # Videos | Subscribers | Views |
| --- | --- | --- | --- |
| 🐶[Jerma Stream Archive](https://youtube.com/@jermastreamarchive) | 770 | 274K | 88M |
| [Ster/Jerma Stream Archive](https://youtube.com/@sterjermastreamarchive) | 972 | 47K | 20M |
| [starkiller201096x](https://youtube.com/@starkiller201096x) | 79 | 2.9K | 1.5M |

Answer 1

Nice script!

Read multiple values by name rather than into an array

Instead of:

IFS=';' read -r -a ARRAY_LINE <<< "${LINE}" # Split line by semi-colon

You could get the values directly into variables with descriptive names:

IFS=';' read -r channel_id channel_name emoji <<< "${line}"

Read multiple values from a single jq call

You could read multiple values with a single call by making jq print all the relevant fields, and using read, for example:

{
  read -r title
  read -r url
} < <(jq -r '.items[0].snippet.title, .items[0].snippet.customUrl' < output.json)

To avoid the line becoming too long, I would put the fields into an array like this:

jq_fields=(
  '.items[0].snippet.title'
  '.items[0].snippet.customUrl'
  '.items[0].statistics.videoCount'
  '.items[0].statistics.subscriberCount'
  '.items[0].statistics.viewCount'
)
{
  read -r title
  read -r url
  read -r video_count
  read -r subscriber_count
  read -r view_count
} < <(IFS=','; jq -r "${jq_fields[*]}" < output.json)

Accumulate lines in an array

The way you accumulated the lines in the string value OUTPUT is ok.

I prefer to use arrays in situations like this, it would look something like this:

output=()

# ...

output+=("${line}")
output+=("")
output+=("| Channel | # Videos | Subscribers | Views |\n| --- | --- | --- | --- |")

# ...

output+=("| ${emoji}[${title}](https://youtube.com/${url}) | ${video_count} | ${subscriber_count} | ${view_count} |")

# ...

(
  IFS=$'\n'
  echo "${template_content//${placeholder_text}/${output[*]}}"
) >"${WORKSPACE}/README.md"

Do not use ALL_CAPS names for your variables

To avoid conflict and confusion with system environment variables, it's recommended to not use ALL_CAPS names in script variables.

By making the script's own variables lowercase, it becomes clear what is expected to be present in the environment and what belongs to the script, which improves readability.

Put repeatedly used constant values into variables

output.json is referenced in multiple places. To leave open the option to use a different name or at a different path, I would put this into a variable. This way code editors could also help you avoid typos.

Define important constants in variables early in the file

The paths "${WORKSPACE}/automation/channels.txt" and "${WORKSPACE}/README.md" are very important key pieces in the behavior of the script. To make them easy to see (and adjust), I would put these values in variables, defined near the top of the file, right alongside header_prefix and placeholder_text.