Diff for "BashFAQ/050"

I'm trying to put a command in a variable, but the complex cases always fail!

Some people attempt to do things like this:

    # Non-working example
    args="-s 'The subject' $address"
    mail $args < $body

This fails because of WordSplitting. When $args is expanded, it becomes four words. 'The is the second word, and subject' is the third word.

So, how do we do this? That all depends on what "this" is!

There are at least three situations in which people try to shove commands, or command arguments, into variables and then run them. Each case needs to be handled separately.

1. I'm constructing a command based on information that is only known at run time

The root of the issue described above is that you need a way to maintain each argument as a separate word, even if that argument contains spaces. Quotes won't do it, but an array will.

Suppose your script wants to send email. You might have places where you want to include a subject, and others where you don't. The part of your script that sends the mail might check a variable named subject to determine whether you need to supply additional arguments to the mail command. A naive programmer may come up with something like this:

    # Don't do this.
    args=$recipent
    if [[ $subject ]]; then
        args+=" -s $subject"
    fi
    mail $args < $bodyfilename

As we have seen, this approach fails when the subject contains whitespace. It simply is not robust enough.

As such, if you really need to create a command dynamically, put each argument in a separate element of an array, like so:

    # Working example, bash 3.1 or higher
    args=("$recipient")
    if [[ $subject ]]; then
        args+=(-s "$subject")
    fi
    mail "${args[@]}" < "$bodyfilename"

(See FAQ #5 for more details on array syntax.)

Often, this question arises when someone is trying to use dialog to construct a menu on the fly. The dialog command can't be hard-coded, because its parameters are supplied based on data only available at run time (e.g. the number of menu entries). For an example of how to do this properly, see FAQ #40.

2. I want to generalize a task, in case the low-level tool changes later

You generally do NOT want to put command names or command options in variables. Variables should contain the data you are trying to pass to the command, like usernames, hostnames, ports, text, etc. They should NOT contain options that are specific to one certain command or tool. Those things belong in functions.

In the mail example, we've got hard-coded dependence on the syntax of the Unix mail command -- and in particular, versions of the mail command that permit the subject to be specified after the recipient, which may not always be the case. Someone maintaining the script may decide to fix the syntax so that the recipient appears last, which is the most correct form; or they may replace mail altogether due to internal company mail system changes, etc. Having several calls to mail scattered throughout the script complicates matters in this situation.

What you probably should be doing, is this:

    # POSIX

    # Send an email to someone.
    # Reads the body of the mail from standard input.
    #
    # sendto address [subject]
    #
    sendto() {
        # mail ${2:+-s "$2"} "$1"
        MailTool ${2:+--subject="$2"} --recipient="$1"
    }

    sendto "$address" "The Subject" < "$bodyfile"

Here, the parameter expansion checks if $2 (the optional subject) has expanded to anything. If it has, the expansion adds the -s "$2" to the mail command. If it hasn't, the expansion doesn't add the -s option at all.

The original implementation uses mail(1), a standard Unix command. Later, this is commented out and replaced by something called MailTool, which was made up on the spot for this example. But it should serve to illustrate the concept: the function's invocation is unchanged, even though the back-end tool changes.

3. I want a log of my script's actions

Another reason people attempt to stuff commands into variables is because they want their script to print each command before it runs it. If that's all you want, then simply use the set -x command, or invoke your script with #!/bin/bash -x or bash -x ./myscript. Note that you can turn it off and back on inside the script with set +x and set -x.

It's worth noting that you cannot put a pipeline command into an array variable and then execute it using the "${array[@]}" technique. The only way to store a pipeline in a variable would be to add (carefully!) a layer of quotes if necessary, store it in a string variable, and then use eval or sh to run the variable. This is not recommended, for security reasons. The same thing applies to commands involving redirection, if or while statements, and so on.

Some people get into trouble because they want to have their script print their commands including redirections before it runs them. set -x shows the command without redirections. People try to work around this by doing things like:

    # Non-working example
    command="mysql -u me -p somedbname < file"
    ((DEBUG)) && echo "$command"
    "$command"

(This is so common that I include it explicitly, even though it's repeating what I already wrote.)

Once again, this does not work. Not even using an array works here. The only thing that would work is rigorously escaping the command to be sure no metacharacters will cause serious security problems, and then using eval or sh to re-read the command. Please don't do that!

If your head is SO far up your ass that you still think you need to write out every command you're about to run before you run it, AND that you must include all redirections, then just do this:

    # Working example
    echo "mysql -u me -p somedbname < file"
    mysql -u me -p somedbname < file

Don't use a variable at all. Just copy and paste the command, wrap an extra layer of quotes around it (sometimes tricky), and stick an echo in front of it.

My personal recommendation would be just to use set -x and not worry about it.

4. How to add testing capability to a programs

[By Jari Aalto] If you are developing a longer program, the possibility to test (what would it do) before actual use would can help to spot problems before they happen in real. Here we define function Run() that is used to proxy all commands. If the TEST mode is on, the commands are not executed for real but only printed to the screen for review. The test mode is activated with program's command line option -t which is read read via Bash's built-in getopt.

The heart of the demonstration is function 'Demo()' where we see how the calls make use of testing. Pay attention to how the quotes are used when command contains any shell meta characters. Notice also how you need the 'Run' call also inside subshell calls. The represetation of subshell calls is limited under test mode as you can see from the last output.

You can add similar testing approach to your programs by: 1) copying the Run() 2) utilizing variable TEST 3) modifying all shell command calls to go through Run(). In practice it is very difficult to completely put shell program under pure testing mode, because programs may use very complex shell structures and depend on outputs that are generated by previous commands. Still, the possibility to improve testing capabilities from nothing gives a better chance to be able to review the program execution before anything is done for real.

The output:

$ bash test-demo.sh -t

test-demo.sh -- Demonstrate how testing feature can be implemented in program
# DEMO: a command
ls -l

# DEMO: a command with pipe
ls -l | sort

# DEMO: a command with pipe and redirection
ls -l | sort > /tmp/jaalto.1396-ls.lst

# DEMO: a command with pipe and redirection using quotes
ls -l | sort > "/tmp/jaalto.1396-ls.lst"

# DEMO: a command and subshell call.
echo ls -l

#
#   test-demo.sh -- Demonstrate how testing feature can be implemented in program
#
#       Copyright (C) 2009 Jari Aalto <jari.aalto@cante.net>
#
#   License
#
#       This program is free software; you can redistribute it and/or
#       modify it under the terms of the GNU General Public License as
#       published by the Free Software Foundation; either version 2 of
#       the License, or (at your option) any later version.
#
#       This program is distributed in the hope that it will be useful, but
#       WITHOUT ANY WARRANTY; without even the implied warranty of
#       MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
#       General Public License for more details.
#
#       You should have received a copy of the GNU General Public License
#       along with program. If not, write to the Free Software
#       Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
#       02110-1301, USA.
#
#       Visit <http://www.gnu.org/copyleft/gpl.html>
#
#   Description
#
#       To enable debugging and testing capabilities in shell
#       scripts, add function Run() and use it to proxy all commands.
#
#   Notes
#
#       The functions in the program are "defined before used". It is only
#       possible to call a function (= command) if it exists (= is defined).
#
#       There is explicit Main() where program starts. This follows
#       the convention of good programming style. By putting the code
#       inside functions, it also makes one think about modularity and
#       reusable components.

DESC="$0 -- Demonstrate how testing feature can be implemented in program"
TEMPDIR=${TEMPDIR:-/tmp}
TEMPPATH=$TEMPDIR/${LOGNAME:-foo}.$$

#   This variable is best to be undefined, not TEST="" or anything.
unset TEST

Help ()
{
    echo "\
$DESC

Available options:

-d      Debug. Before command is run, show it.
-t      Test mode. Show commands, do not really execute.

The -t option takes precedence over -d option."

    exit ${1:-0}
}

Run ()
{
    if [ "$TEST" ]; then
        echo "$*"
        return 0
    fi

    eval "$@"
}

Echo ()
{
    echo "# DEMO: $*"
}

Demo ()
{
    Echo "a command"
    Run ls -l

    Echo "a command with pipe"
    Run "ls -l | sort"

    Echo "a command with pipe and redirection"
    Run "ls -l | sort > $TEMPPATH-ls.lst"

    Echo "a command with pipe and redirection using quotes"
    Run "ls -l | sort > \"$TEMPPATH-ls.lst\""

    #   You need to put Run() call also into subshell, otherwise
    #   it would be run "for real" and defeat the test mode.

    Echo "a command and subshell call."
    Run "echo $( Run ls -l )"
}

Main ()
{
    echo "$DESC"

    OPTIND=1
    local arg

    while getopts "hdt" arg "$@"
    do
        case "$arg" in
            h)  Help
                ;;
            t)  TEST="test"
                ;;
        esac
    done

    #   Remove found options from command line arguments.
    shift $(($OPTIND - 1))

    #   Run the demonstration
    Demo
}

Main "$@"

# End of file