2
\$\begingroup\$

I use zypper to download and install for openSUSE Tumbleweed (much like everyone else of course). Unfortunately it downloads packages serially, and having about a thousand packages to update a week, it gets quite boring. Also, while zypper can download packages one-by-one in advance, it can't be called concurrently. I found the libzypp-bindings project but it is discontinued. I set myself to improve the situation.

Goals:

  • Download all repositories in parallel (most often different servers);
  • Download up to MAX_PROC (=6) packages from each repository in parallel;
  • Save packages where zypper picks them up during system update: /var/cache/zypp/packages;
  • Alternatively, download to $HOME/.cache/zypp/packages;
  • Avoid external dependencies, unless necessary.

Outline:

  1. Find the list of packages to update;
  2. Find the list of repositories;
  3. For each repository:
    1. Keep up to $MAX_PROC curl processes downloading packages.
  4. Copy files to default package cache;
#!/bin/bash

MAX_PROC=6

function repos_to_update () {
    zypper list-updates | grep '^v ' | awk -F '|' '{ print $2 }' | sort --unique | tr -d ' '
}

function packages_from_repo () {
    local repo=$1

    zypper list-updates | grep " | $repo " | awk -F '|' '{ print $6, "#", $3, "-", $5, ".", $6, ".rpm" }' | tr -d ' '
}

function repo_uri () {
    local repo=$1

    zypper repos --uri | grep " | $repo " | awk -F '|' '{ print $7 }' | tr -d ' '
}

function repo_alias () {
    local repo=$1

    zypper repos | grep " | $repo " | awk -F '|' '{ print $2 }' | tr -d ' '
}

function download_package () {
    local alias=$1
    local uri=$2
    local line=$3
    IFS=# read arch package_name <<< "$line"

    local package_uri="$uri/$arch/$package_name"
    local local_dir="$HOME/.cache/zypp/packages/$alias/$arch"
    local local_path="$local_dir/$package_name"
    printf -v y %-30s "$repo"
    printf "Repository: $y Package: $package_name\n"
    if [ ! -f "$local_path" ]; then
        mkdir -p $local_dir
        curl --silent --fail -L -o $local_path $package_uri
    fi
}

function download_repo () {
    local repo=$1

    local uri=$(repo_uri $repo)
    local alias=$(repo_alias $repo)
    local pkgs=$(packages_from_repo $repo)
    local max_proc=$MAX_PROC
    while IFS= read -r line; do
        if [ $max_proc -eq 0 ]; then
            wait -n
            ((max_proc++))
        fi
        download_package "$alias" "$uri" "$line" &
        ((max_proc--))
    done <<< "$pkgs"
    wait
}

function download_all () {
    local repos=$(repos_to_update)
    while IFS= read -r line; do
        download_repo $line &
    done <<< "$repos"
    wait
}

download_all
sudo cp -r ~/.cache/zypp/packages/* /var/cache/zypp/packages/

There's 2 or 3 places where grep/tr are subject to issues, but the nature of the data doesn't require much more than that.

\$\endgroup\$
1
\$\begingroup\$

I completely changed the approach which resulted in faster downloads, and improved functionality. The overall approach of the initial version works well for a generic solution looking for adding concurrency to independent jobs.

  1. Removed zypper list-updates. This format is easier for machine consumption, but it's not intended for Tumbleweed. Replaced with zypper dup/inr/in --details;
  2. Removed background jobs and waits. Replaced with aria2 which handles the maximum number of concurrent connections;
  3. It pays off learning more about awk capabilities, in order to replace sequences of grep/awk/tr with a single awk;
  4. The main job of the script became building a plain text file with URIs and target directory for each .rpm file

aria2 is really great tool. Superb quality. curl is not reliable in its native concurrent download capabilities.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.