Defensive Scripting with bash

DEFENSIVE PROGRAMMING IN BASH

Bash is a very powerful tool, but it is important to employ strategies that catch errors early, provide clear logging, and handle unexpected situations cleanly.

The scripts you write may be subjected to unexpected behaviours and events such as a required directory not present, a required shell command not present, or, commonly, permission errors.

I have been writing Bash scripts for a while, and I have gathered the best techniques from structuring scripts to seamless error handling, which we will explore: some key practices to enhance reliability of your scripts.

This guide assumes prior exposure to Bash.

Structuring a Bash script

Bash being a flexible language as it is follows no strict structure, but here is the one I recommend.

#!/bin/bash

# global variables immutable and referenced by whole script
readonly PROGNAME=$(basename "$0")

# sourced script
source my_script.sh

# program logic grouped into functions
do_something() {
    # logic
    return $?
}

main() {
    # main procedure
    return $?
}
# calling main 
main || {
    printf "program failed" >&2
    exit 1
}

export vs local vs global

• Try to keep global variables to a minimum
• SHOUTING_SNAKE_CASE VAR NAMING
• readonly declaration
• use variable names to replace cryptic $1

Globals I recommend you always use in your programs:

readonly PROG_NAME=$(basename "$0")
readonly PROG_DIR=$(readlink -m $(dirname "$0"))
# or even better 
export SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

Sourcing and splitting a script

Splitting the logic of your script into multiple scripts will show easier to manage as the script gets larger.

All other variables should be local.

#!/usr/bin/env bash
change_owner_of_file() {
    local filename=$1
    local user=$2
    local group=$3

    chown $user:$group $filename
}

Modular coding, functions and arguments

Grouping procedures into functions and specifying its return code in case of failure or success is the way to go.
Some distinctive functions I use are:

info() -- for logging successful events

• track all changes made by script, especially cron jobs

#!/bin/bash

info() {
    echo "[INFO] [$TIME] $*" >> "$info_logs"
}

# or you can use separate info files for each event, but that might take up space when doing scripting on small-space devices like Raspberry Pi boards 

error() -- for logging failed events and warnings

• error logging very important for building daemons
• easily get reason why script may have failed mid-run

#!/bin/bash

error() {
    echo "[ERROR] [$TIME] $*" >> "$err_logs"
}

main() -- single point of entry for your program

• keeps code clean
• procedures become descriptive

    main() {
    # check req files
    # check req dirs
    # install missing deps
    # confirm installation
    # run data collections
    # run inference engine and generate results
    # run decision engine and generate results
    # run tui if -q flag isn't given, which just pretty-print results and decision
    # exit 
    check_req_files || {
        printf "Required files missing" >&2
        exit "$ERR_FAILURE"
    }
    check_req_dirs || {
        printf "Required directories missing" >&2
        exit "$ERR_FAILURE"
    }

    parse_args "$@"
    # installing missing dependencies
    install_missing_deps

    # running data collections
    write_cpu_json &
    write_memory_json
    write_thermal_json
    write_disk_json 
    write_sound_json

    # run engine
    python3 "$ENGINE_DIR"/inference.py && python3 "$ENGINE_DIR"/decision.py

    display_tui || {
        exit "$ERR_FAILURE"
    }
    exit "$ERR_SUCCESS"
}

main "$@"
# code taken from sysk

Commenting

From my experience, Bash syntax isn't really presentable. It gets messy fast as the program expands. All the cryptic ?*&&^^ symbols become confusing, because a single symbol can be used in 2 or more cases representing different things.

A simple comment # this does this will go a long way.

giving strict instructions:
# DONT TOUCH THIS # USE PRINTF NOT ECHO

Wrapping lines

When using a code editor like VS Code, they give you automatic wrap-line options.
Yes, it works, but when migrating the code to another IDE it breaks.
So if code is too long, wrap with the \.
Also makes script look clean.

jq -n \
--arg name "$names" \
-f "$file"

/dev/null and /dev/zero

Think of /dev/null as a black hole. It is essentially the equivalent of a write-only file.
Everything written to it disappears. Attempts to read or output from it result in nothing.
Suppressing stdout.

cat $filename >/dev/null
# Contents of the file will not list to stdout.

It can be very useful when you need a command to run without any output.

Debugging

Bash has no built-in debugger, but there are many workarounds.

1. using shellcheck
   • ShellCheck is one of the best ways to write a safe script. Sometimes it might be annoying to use because of the many options, but a script which can pass through ShellCheck successfully has probably no syntax error. But it can't catch runtime errors or logic errors.

2. using set -x and set +x

   • this echoes each line before producing output
   • use to debug a particular piece of code where you feel the error might be from

       set -x
       echo "use case" >> $file
       set +x
   

3. using sh -n scriptname

   • catches syntax errors without executing script
   • some syntax errors may escape

Handling failures and testing using if statements

Setting error-handling options

Bash is set by default to run to completion even when it encounters an error.

#!/bin/bash
echo "program starts"
ech "program continues" # command doesn't exist
echo "program end"

Running this gives:

program starts
./test.sh: line 3: ech: command not found
program end

This and other common unexpected behaviours can be avoided using shell options.

1. set -o errexit or set -e: this option ensures that the script exits when any command returns a non-zero status unless handled with ||
2. set -o nounset or set -u: ensures the script treats unset variables as errors
3. set -o pipefail: this option causes the pipeline to fail if any command within it fails

These commands can be combined as set -euo pipefail.

#!/bin/bash
set -euo pipefail
echo "program starts"
ech "program continues" # command doesn't exist
echo "program end"

Running this we get:

program starts
./test.sh: line 4: ech: command not found

Some problems with the set -euo pipefail

While it might seem very convenient to use the guard line, there are some unexpected behaviours which might be unwanted, especially with the set -e flag, which is very problematic.

1. Conditions disable set -e
   • the test of if, while and until
   • on the left of && and ||
   • inverted with the ! flag

false && echo never # false fails but script continues
false && echo this runs

1. Command substitution and subshells don't automatically inherit the set -e flag:

set -e 
result=$(false)
echo "gets here" # still successfully runs

This can be fixed with shopt -s inherit_errexit shell option (Bash 4.4+).

1. local, export and readonly: they mask return values so they always succeed even if substituted command fails

export answer=$(find /bin/bash) # bad
# better
declare word=$()
export word

This can be fixed by declaring and assigning on separate lines.

This demonstrates set -e can give a fake sense of safety.

Explicitly catching and handling errors

As shown above, the shell options are a good way to perform error handling in a script, but they are not perfect. That said, the best way to handle errors is to use conditionals.

• In Bash, an exit status of 0 represents success; any other represents a failure, but there are special exit codes that specify a particular event.

• The exit code of the last command to run is stored in the $? special variable.
  

echo "hello world"
echo "$?" # outputs 0 

# some special return codes
readonly ERR_SUCCESS=0  # operation successful
readonly ERR_FAILURE=1  # operation failed general 
readonly ERR_PERMISSION_DENIED=126 # permission denied
readonly ERR_COMMAND_FAILED=126 # command runs but fails
readonly ERR_NOT_FOUND=127  # command or file not found
readonly ERR_BAD_USAGE=2    # bad use of script or function

• Catching these codes with || and && is the way to go.

find "$file"* || {
    echo "no file with name $file"
    exit 127
}

Handling unexpected exit events using traps

Trap exit events to perform cleanup tasks or handle unexpected
trap [command] [signals]

Let's take a script designed to run non-stop for 60 seconds, maybe performing data movement or log parsing, and we don't want any interruption, maybe from Ctrl + C.

#!/bin/bash

trap "echo Booh!" SIGINT SIGTERM
echo "pid is $$"

while true; do
    sleep 60    # script last 60 seconds
done

You can do more research on traps.

Conclusion

Following these practices will ensure to a certain level that your Bash script will be robust.

Reference

• Defensive Programming to create a robust shell script by prabesh on medium

• stop ignoring errors in bash

Note: this article summarises my discoveries with Bash. There are many pitfalls and gotchas which have not been mentioned here.

Remember there is no perfect guide to Bash. The best way to learn is to practice and encounter the errors yourself.

Thanks for reading to this level. If you have any doubts, please don't hesitate to share in the comments. Happy scripting.