6
\$\begingroup\$

I have a MediaWiki 1.33.0 website with only one external addon installed (external extension), which is ContactPage.
I have created the following script with that logic:

if there is an upgrade to either MediaWiki core OR ContactPage these will be upgraded respectively;
else, the script keeps the original state (that is, the state before the upgrade attempt).
Successful upgrading might require further manual changes although I didn't have a case of needing to make manual changes.

I tested the script and it seems to work as expected; if no new version was released, the current ones are kept.

The script:

# Declare MediaWiki download variables (ensure latest versions are downloaded because as of 08/10/19 there aren't version-agnostic download links):

latest_mediawiki_core="https://releases.wikimedia.org/mediawiki/1.33/mediawiki-1.33.0.tar.gz"
latest_contactpage_extension="https://extdist.wmflabs.org/dist/extensions/ContactPage-REL1_33-abdcab9.tar.gz"

# Declare other important variables:

current_date="$(date +%Y-%m-%d-%H-%M-%S)"
war="$HOME/public_html" # Web Application Root # Change to your Web Application Root if needed
domain="example.com" # Change to relevant domain
db-username_and_db-name="DB_CREDENTIALS" # Change to relevant DB credentials

# Create backup directories:

mkdir -p "${war}/mediawiki_general_backups"
mkdir -p "${war}/mediawiki_specific_backups"

# General backups:

zip -r "${war}/mediawiki_general_backups/${domain}-directory-backup-${current_date}.zip" "${war}/${domain}"
mysqldump \
-u "${db-username_and_db-name}" \
-p "${db-username_and_db-name}" \
> "${war}/mediawiki_general_backups/${db-username_and_db-name}-${current_date}.sql"

# Specific backups:

rm "${war}"/mediawiki_specific_backups/*
rm "${war}"/mediawiki_specific_backups/.* # If I won't run this, a specific backup of .htaccess, in that directory, won't get deleted;

cp "${war}/${domain}"/.htaccess*                  "${war}/mediawiki_specific_backups"/.htaccess*
cp "${war}/${domain}"/LocalSettings.php           "${war}/mediawiki_specific_backups"/LocalSettings.php
cp "${war}/${domain}"/robots.txt                  "${war}/mediawiki_specific_backups"/robots.txt
cp "${war}/${domain}"/${domain}.png               "${war}/mediawiki_specific_backups"/${domain}.png
cp "${war}/${domain}"/googlec69e044fede13fdc.html "${war}/mediawiki_specific_backups"/googlec69e044fede13fdc.html

# Downloads and configurations:

rm -rf "${war}/${domain}"
mkdir "${war}/${domain}"
wget "${latest_mediawiki_core}" -O - | tar -xzv --strip-components 1 -C "${war}/${domain}"
wget "${latest_contactpage_extension}" -O - | tar -xzv -C "${war}/${domain}"/extensions/
cp -a "${war}/mediawiki_specific_backups"/* "${war}/${domain}"

# Create a new sitemap:

mkdir -p "${war}/${domain}/sitemap"
php "${war}/${domain}"/maintenance/generateSitemap.php \
--memory-limit=50M \
--fspath=/"${war}/${domain}/sitemap" \
--identifier="${domain}" \
--urlpath=/sitemap/  \
--server=https://"${domain}" \
--compress=yes

# Update DB (One might need to change LocalSettings.php before doing so):
php "${war}/${domain}"/maintenance/update.php
\$\endgroup\$
4
+200
\$\begingroup\$

The reading could be clearer using meaningful variable names. The concatenation of very long variable names does not improve the reading either. That is to say, the shell syntax is not suited to define several variables in a row: the code becomes more difficult to read.

Variables

Some variables seem necessary but you may remove or at least ignore some variables. The shell commands (or utilities) should stand out in the shell script. The command line should looks like the following one.

wget -q -O - "http://wordpress.org/latest.tar.gz" | tar -xzf - -C /var/www

Eventually, you may use some variables in replacement to avoid hard-coding values as in your snippet.

wget "${latest_mediawiki_core}" -O - | tar -xzv --strip-components 1 -C "${war}/${domain}"

Personally, I would have used macros instead of shell variables to improve the readability.

wget MEDIAWIKI -O - | tar -xzv --strip-components 1 -C DESTDIR

Code factorization

You may replace several command invocations by only one or shorten the code.

cp "${war}/${domain}"/.htaccess* "${war}/mediawiki_specific_backups"/.htaccess*
cp "${war}/${domain}"/LocalSettings.php "${war}/mediawiki_specific_backups"/LocalSettings.php
cp "${war}/${domain}"/robots.txt "${war}/mediawiki_specific_backups"/robots.txt
cp "${war}/${domain}"/${domain}.png "${war}/mediawiki_specific_backups"/${domain}.png
cp "${war}/${domain}"/googlec69e044fede13fdc.html "${war}/mediawiki_specific_backups"/googlec69e044fede13fdc.html

In the above code snippet, you can see a lot of duplicated information.

# move in the target directory
pushd DESTDIR >/dev/null
# copy some configuration files
cp -v .htaccess* LocalSettings.php robots.txt home.png googlec69e044fede13fdc.html NEWDIR
# come back to the previous directory
popd >/dev/null

Eventually, you may use a subshell to perform the directory change.

(cd DESTDIR || exit; 
 cp -v .htaccess* LocalSettings.php robots.txt home.png googlec69e044fede13fdc.html NEWDIR) 

Note: pushd and popd are Bash builtins. The comments made in the above code snippet are not particularly appropriate, it depends on your scripting knowledge.

Conclusion

Using a shell script to update your MediaWiki installation may be good enough by making the suited changes.


M4 macros

define(`MEDIAWIKI_SITE', `https://releases.wikimedia.org/mediawiki')dnl
define(`MEDIAWIKI_VERSION', `1.33')dnl
define(`MEDIAWIKI_RELEASE', `MEDIAWIKI_VERSION.1')dnl
define(`MEDIAWIKI_SOURCE', `"MEDIAWIKI_SITE/MEDIAWIKI_VERSION/mediawiki-core-MEDIAWIKI_RELEASE.tar.gz"')dnl
define(`MEDIAWIKI', defn(`MEDIAWIKI_SOURCE'))dnl
define(`DESTDIR', `/tmp/some_directory')dnl

We may define some m4 macros (it is just an example).

#!/bin/bash

echo 'wget MEDIAWIKI -O - | tar -xzv --strip-components 1 -C DESTDIR'

M4 is then used to substitute the macros in the shell script.

cat macros.m4 mediawiki.bash | m4 | bash
|improve this answer|||||
\$\endgroup\$
  • 2
    \$\begingroup\$ Don't use pushd and popd in scripts. They are meant for interactive use. Prefer to use a subshell instead (and check that the cd succeeds): ( cd "$SRCDIR" && cp -v "${FILES[@]}" "$DESTDIR" ) \$\endgroup\$ – Toby Speight Nov 27 '19 at 17:07
  • \$\begingroup\$ Or avoid the directory change by using brace-expansion: cp -t "${war}/mediawiki_specific_backups/" "${war}/${domain}"/.htaccess "${war}/${domain}"/{LocalSettings.php,robots.txt,${domain}.png,googlec69e044fede13fdc.html} \$\endgroup\$ – Toby Speight Nov 27 '19 at 17:16
  • 2
    \$\begingroup\$ They're not portable, and they always produce output, which then needs to be redirected (usually to /dev/null). They are a simple shortcut for interactive use, but cumbersome even in Bash-only scripts. It's easier and clearer to save location into a variable or to use a subshell. I don't think that the BashPitfalls item is strongly recommending pushd; it says, "The subshell version is simpler and cleaner"... I wouldn't say that using pushd and popd in scripts is "incorrect", but I think it's a poor practice. \$\endgroup\$ – Toby Speight Nov 27 '19 at 20:20
5
\$\begingroup\$

Make the process approach atomic

The script first wipes out ${war}/${domain}, then populates it with content from wget commands, and restores content from backup.

The wget commands are a serious risk here. If any of the source sites is down or experiencing instability at the time the script runs, your site can end up in an inconsistent state.

At the minimum, it would be better to fetch all external resources before you start rewriting your content.

An even better way would be to try to make the process as atomic as possible: build up the new content in a dedicated directory isolated from production, but on the same partition, and when all is ready, make the switch in fast directory rename operations.

Separate your backups from your production environment

The $war directory looks clearly in the heart of the production environment. In that directory, you create backup directories side by side with the $domain directory. I suggest to create the backups elsewhere. Ideally on a different filesystem partition.

Closely related to this, the backup directories are somewhere in $war/..., and the original content is also somewhere in $war/.... I think this invites mistakes. If you had distinct variable name prefixes such as $backup_... and $content, that would reduce the risk of mistakes.

Syntax error

I don't understand what this is doing in a Bash script:

db-username_and_db-name="DB_CREDENTIALS"

This is an invalid statement in all Bash versions that I know.

Bash scripts should have a shebang

The posted code has no shebang. Since the question is tagged bash, maybe it's implied it's really there in your version, but just for the record, every Bash script should start with a shebang.

Watch out for things that may go wrong

Toby already mentioned this in his review, but I think it deserves to be stressed. Many things can go wrong in this script:

  • If mysqldump fails for some reason, the script will keep going without a backup created. That can lead to data loss, and it's to be avoided.
  • If any of the wget commands fail for some reason, the script will keep going and update the sitemap and the DB. That can lead to inconsistent state of your site that's difficult to debug.

My suggestion is similar to Toby's: put set -euo pipefail near the top of the script, to make it stop executing when something goes wrong unexpectedly.

Double-quote variables used in command arguments

You did that correctly for the most part, you missed just one here:

cp "${war}/${domain}"/${domain}.png               "${war}/mediawiki_specific_backups"/${domain}.png

Use more variables

The term ${war}/mediawiki_general_backups comes up repeatedly. It would be better if it was in a variable. Editors with Bash support will make it possible to rename, see, find easily all occurrences.

Make long pipelines easier to read and understand

Take for example this:

wget "${latest_mediawiki_core}" -O - | tar -xzv --strip-components 1 -C "${war}/${domain}"

I think it's better to write flags before arguments. The text closer to the left is naturally easier to read. And the flags pack a lot of important details in little space, and the URL argument is long and not very interesting to read.

Secondly, it's easier to understand code when there is a single statement per line. A good way to achieve that is by splitting the line at each command in the pipeline.

That is, I suggest writing like this:

wget -O- "${latest_mediawiki_core}" \
| tar -xzv --strip-components 1 -C "${war}/${domain}"

Simplify

Instead of this:

rm "${war}"/mediawiki_specific_backups/*
rm "${war}"/mediawiki_specific_backups/.* # If I won't run this, a specific backup of .htaccess, in that directory, won't get deleted;

Why not simply:

rm -r "${war}"/mediawiki_specific_backups
|improve this answer|||||
\$\endgroup\$
  • \$\begingroup\$ Hello, I saw you asked me questions inside the answer; I think clarification questions should be asked only in comments as it is unclear where I should answer... I invite you to comment on the question directly. \$\endgroup\$ – JohnDoea Nov 28 '19 at 20:06
  • 1
    \$\begingroup\$ @JohnDoea Sometimes I like to lead users to discover solutions by making them ask themselves questions. I removed these rhetorical questions to avoid ambiguity. \$\endgroup\$ – janos Dec 2 '19 at 16:03
  • \$\begingroup\$ Thank you --- as I understand you mean there surly isn't any literal if-else but not surly a philosophical one... \$\endgroup\$ – JohnDoea Dec 2 '19 at 16:05
  • \$\begingroup\$ @JohnDoea When you write, "the script keeps the original state", my expectation is that the script doesn't modify the environment. The script does modify the environment, if the packages are the same version as before, the result should be identical, but that's not the same thing as not modifying the environment at all. \$\endgroup\$ – janos Dec 2 '19 at 17:16
  • 1
    \$\begingroup\$ @JohnDoea I accepted your edit. \$\endgroup\$ – janos Mar 26 at 5:52
4
\$\begingroup\$

You should save all files that are downloaded by wget in a local directory, either called distfiles, or download, or even cache. This makes upgrading independent of the internet being available, and also prevents unnecessary duplicate downloads. Plus, it allows the downloaded files to be checked whether they are cryptographically signed.

|improve this answer|||||
\$\endgroup\$
  • \$\begingroup\$ Hello Roland, This makes upgrading independent of the internet being available ; I didn't understand this. I further misunderstood why the destination directory allows checking or not checking if a file is cryptographically signed. \$\endgroup\$ – JohnDoea Mar 25 at 10:08
  • 1
    \$\begingroup\$ The first step should be to download all necessary files. After that step, you can verify that the files are indeed from the correct publisher. Downloading the files should be a separate step from upgrading since the internet might fail just at the moment you want to upgrade. If you download the files first, you can just skip that step later. It also saves bandwidth. For example, in pkgsrc a software package is installed by following these steps: fetch, checksum, tools, extract, patch, configure, build, install, package. Each of these steps can be run on its own. \$\endgroup\$ – Roland Illig Mar 25 at 18:13
  • \$\begingroup\$ Did you mean to say that I both download and upgrade because of the piping and tar as with wget "${latest_mediawiki_core}" -O - | tar -xzv --strip-components 1 -C "${war}/${domain}" AND wget "${latest_contactpage_extension}" -O - | tar -xzv -C "${war}/${domain}"/extensions/? \$\endgroup\$ – JohnDoea Mar 26 at 12:21
  • \$\begingroup\$ If not, than I misunderstand how did I both downloaded and upgraded instead doing both separately (if I'm not wrong, piping to tar would only succeed if the download itself succeeded, but I might be wrong). \$\endgroup\$ – JohnDoea Mar 26 at 12:22
  • \$\begingroup\$ Yes, exactly. And sorry for giving you the too generic link to pkgsrc first. The relevant part is better described here: netbsd.org/docs/pkgsrc/build.html \$\endgroup\$ – Roland Illig Mar 27 at 6:14
3
\$\begingroup\$

Repeating a bit what Fólkvangr said in his excellent answer.

More variables

The same expressions (with potentially hardcoded strings) are used many times in many places (most of them corresponding to paths). That makes things potentially hard to understand and to update.

You should try to define more variables (actually used as constants).

Taking this chance to rename them, reorder them, redocument them, we'd get:

# Declare MediaWiki download variables (ensure latest versions are downloaded because as of 08/10/19 there aren't version-agnostic download links):
MEDIAWIKI_CORE_URL="https://releases.wikimedia.org/mediawiki/1.33/mediawiki-1.33.0.tar.gz"
CONTACTPAGE_EXTENSION_URL="https://extdist.wmflabs.org/dist/extensions/ContactPage-REL1_33-abdcab9.tar.gz"

# Credentials
DOMAIN="example.com" # Change to relevant domain
DB_USER_AND_DB_NAME="DB_CREDENTIALS" # Change to relevant DB credentials

# Constant paths
WEB_APPL_ROOT="$HOME/public_html" # Change to your Web Application Root if needed
GENERAL_BACKUP_DIR="${WEB_APPL_ROOT}/mediawiki_general_backups"
SPECIFIC_BACKUP_DIR="${WEB_APPL_ROOT}/mediawiki_specific_backups"
DOMAIN_DIR="${WEB_APPL_ROOT}/${DOMAIN}"

# Date to be used in backup filenames
DATE="$(date +%Y-%m-%d-%H-%M-%S)"

# Create backup directories:
mkdir -p "${GENERAL_BACKUP_DIR}"
mkdir -p "${SPECIFIC_BACKUP_DIR}"

# General backups:
zip -r "${GENERAL_BACKUP_DIR}/${DOMAIN}-directory-backup-${DATE}.zip" "${DOMAIN_DIR}"
mysqldump -u "${DB_USER_AND_DB_NAME}" -p "${DB_USER_AND_DB_NAME}" \
 > "${GENERAL_BACKUP_DIR}/${DB_USER_AND_DB_NAME}-${DATE}.sql"

# Specific backups:
rm "${SPECIFIC_BACKUP_DIR}"/*
rm "${SPECIFIC_BACKUP_DIR}"/.* # If I won't run this, a specific backup of .htaccess, in that directory, won't get deleted;

cp "${DOMAIN_DIR}"/.htaccess*                  "${SPECIFIC_BACKUP_DIR}/".htaccess*
cp "${DOMAIN_DIR}/LocalSettings.php"           "${SPECIFIC_BACKUP_DIR}/LocalSettings.php"
cp "${DOMAIN_DIR}/robots.txt"                  "${SPECIFIC_BACKUP_DIR}/robots.txt"
cp "${DOMAIN_DIR}/${DOMAIN}.png"               "${SPECIFIC_BACKUP_DIR}/${DOMAIN}.png"
cp "${DOMAIN_DIR}/googlec69e044fede13fdc.html" "${SPECIFIC_BACKUP_DIR}/googlec69e044fede13fdc.html"

# Downloads and configurations:
rm -rf "${DOMAIN_DIR}"
mkdir "${DOMAIN_DIR}"
wget "${MEDIAWIKI_CORE_URL}" -O - | tar -xzv --strip-components 1 -C "${DOMAIN_DIR}"
wget "${CONTACTPAGE_EXTENSION_URL}" -O - | tar -xzv -C "${DOMAIN_DIR}/extensions/"
cp -a "${SPECIFIC_BACKUP_DIR}"/* "${DOMAIN_DIR}"

# Create a new sitemap:
mkdir -p "${DOMAIN_DIR}/sitemap"
php "${DOMAIN_DIR}/maintenance/generateSitemap.php" \
--memory-limit=50M \
--fspath="/${DOMAIN_DIR}/sitemap" \
--identifier="${DOMAIN}" \
--urlpath=/sitemap/  \
--server="https://${DOMAIN}" \
--compress=yes

# Update DB (One might need to change LocalSettings.php before doing so):
php "${DOMAIN_DIR}/maintenance/update.php"

Cleaning things the easy way

Instead of

mkdir -p "${SPECIFIC_BACKUP_DIR}"

(...)

# Specific backups:
rm "${SPECIFIC_BACKUP_DIR}"/*
rm "${SPECIFIC_BACKUP_DIR}"/.* # If I won't run this, a specific backup of .htaccess, in that directory, won't get deleted;

You could do:

# Specific backups:
rm -r "${SPECIFIC_BACKUP_DIR}"
mkdir -p "${SPECIFIC_BACKUP_DIR}"

without having to worry about hidden files.

Copying the easy way

Instead of "cp abcd/foo efgh/foo", you can just "cp abcd/foo efgh"

Thus, you get:

cp "${DOMAIN_DIR}"/.htaccess*                  "${SPECIFIC_BACKUP_DIR}/"
cp "${DOMAIN_DIR}/LocalSettings.php"           "${SPECIFIC_BACKUP_DIR}/"
cp "${DOMAIN_DIR}/robots.txt"                  "${SPECIFIC_BACKUP_DIR}/"
cp "${DOMAIN_DIR}/${DOMAIN}.png"               "${SPECIFIC_BACKUP_DIR}/"
cp "${DOMAIN_DIR}/googlec69e044fede13fdc.html" "${SPECIFIC_BACKUP_DIR}/"
|improve this answer|||||
\$\endgroup\$
  • 1
    \$\begingroup\$ Why did you use upper-case for variable names? That's normally only used for environment variables used to communicate between processes, and it loses that distinction if you name your ordinary variables like that. \$\endgroup\$ – Toby Speight Nov 27 '19 at 17:03
  • \$\begingroup\$ Oh I've seen that convention used in the past and thought it was normal. Now that I see your comment it makes a lot more sense to use lower case indeed. I'll try to update my answer. In the meantime, feel free to downvote \$\endgroup\$ – SylvainD Nov 27 '19 at 21:08
2
\$\begingroup\$

Many of the commands in the script only make sense if the previous commands were successful. But there's no checking.

The simple way to get much of that checking for free, is to set the shell's exit on error flag:

set -e

I also recommend setting error on undefined to help pick up mistyped variable names:

set -eu

Make sure you understand the power and limitations of these options; in particular, it's important to know the contexts where failing commands won't cause the script to exit.

|improve this answer|||||
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.