Diff for "BashFAQ/050"

I'm trying to put a command in a variable, but the complex cases always fail!

Variables hold data. Functions hold code. Don't put code inside variables! There are many situations in which people try to shove commands, or command arguments, into variables and then run them. Each case needs to be handled separately.

For the simple case in bash, you can use an array to store arguments to pass to a command (similar to constructing a command using args only known at runtime, below):

args=(-s "$subject" --flag "arg with spaces")
mail "${args[@]}"

1. Things that do not work

Some people attempt to do things like this:

# Example of BROKEN code, DON'T USE THIS.
args=$address1
if [[ $subject ]]; then
    args+=" -s $subject"
fi
mail $args < "$body"

Adding quotes won't help, either:

# Example of BROKEN code, DON'T USE THIS.
args="$address1 $address2"
if [[ $subject ]]; then args+=" -s '$subject'"; fi
mail $args < "$body"

This fails because of WordSplitting and because the single quotes inside the variable are literal, not syntactical. If $subject contains internal whitespace, it will be split at those points. The mail command will receive -s as one argument, then the first word of the subject (with a literal ' in front of it) as the next argument, and so on.

Read Arguments to get a better understanding of how the shell figures out what the arguments in your statement are.

Here's another thing that won't work:

# BROKEN code.  Do not use!
redirs=">/dev/null 2>&1"
if ((debug)); then redirs=; fi
some command $redirs

Here's yet another thing that won't work:

# BROKEN code.  Do not use!
runcmd() { if ((debug)); then echo "$@"; fi; "$@"; }

The runcmd function can only handle simple commands with no redirections. It can't handle redirections, pipelines, for/while loops, if statements, etc.

Now let's look at how we can perform some of these tasks.

2. I'm trying to save a command so I can run it later without having to repeat it each time

Just use a function:

pingMe() {
    ping -q -c1 "$HOSTNAME"
}

[...]
if pingMe; then ..

3. I only want to pass options if the runtime data needs them

You can use the ${var:+..} parameter expansion for this:

ping -q ${count:+-c "$count"} "$HOSTNAME"

Now the -c option (with its "$count" argument) is only added to the command when $count is not empty. Notice the quoting: No quotes around ${var:+...} but quotes on expansions INSIDE!

This would also work well for our mail example:

addresses=("$address1" "$address2")
mail ${subject:+-s "$subject"} "${addresses[@]}" < body

There is one important caveat with this construct: the unquoted pieces of the expansion (e.g. '-s ' here) are subject to word-splitting using IFS. Everything should work as you expect so long as IFS is set to its default value (or something sane). If IFS has been changed, see the gory details here.

4. I want to generalize a task, in case the low-level tool changes later

Again, variables hold data; functions hold code.

In the mail example, we've got hard-coded dependence on the syntax of the Unix mail command. The version in the previous section is an improvement over the original broken code, but what if the internal company mail system changes? Having several calls to mail scattered throughout the script complicates matters in this situation.

What you probably should be doing, paying very close attention at how to quote your expansions, is this:

# Bash 3.1

# Send an email to someone.
# Reads the body of the mail from standard input.
#
# sendto subject address [address ...]
#
sendto() {
    # Used to be standard mail, but the fucking HR department
    # said we have to use this crazy proprietary shit....
    # mailx -s "$@"

    local subject=$1
    shift
    local addr addrs=()
    for addr; do addrs+=(--recipient="$addr"); done
    MailTool --subject="$subject" "${addrs[@]}"
}

sendto "The Subject" "$address" <"$bodyfile"

The original implementation uses mailx(1), a standard Unix command. Later, this is commented out and replaced by something called MailTool, which was made up on the spot for this example. But it should serve to illustrate the concept: the function's invocation is unchanged, even though the back-end tool changes.

5. I'm constructing a command based on information that is only known at run time

The root of the issue described above is that you need a way to maintain each argument as a separate word, even if that argument contains spaces. Quotes won't do it, but an array will. (We saw a bit of this in the previous section, where we constructed the addrs array on the fly.)

If you need to create a command dynamically, put each argument in a separate element of an array. A shell with arrays (like Bash) makes this much easier.

# Bash 3.1 / ksh93
args=("my arguments" "$go" here)
if ((foo)); then args+=(--foo); fi    # and so on
somecommand "${args[@]}"

POSIX sh has no arrays, so the closest you can come is to build up a list of elements in the positional parameters. Here's a POSIX sh version of the sendto function from the previous section:

# POSIX sh
# Usage: sendto subject address [address ...]
sendto() {
    subject=$1
    shift
    first=1
    for addr; do
        if [ "$first" = 1 ]; then set --; first=0; fi
        set -- "$@" --recipient="$addr"
    done
    if [ "$first" = 1 ]; then
        echo "usage: sendto subject address [address ...]"
        return 1
    fi
    MailTool --subject="$subject" "$@"
}

Note that we overwrite the positional parameters inside a loop that is iterating over the previous set of positional parameters (because we can't make a second array, not even to hold a copy of the original parameters). This appears to work in at least 3 different /bin/sh implementations (tested in Debian's dash, HP-UX's sh and OpenBSD's sh).

Another example of this is using dialog to construct a menu on the fly. The dialog command can't be hard-coded, because its parameters are supplied based on data only available at run time (e.g. the number of menu entries). For an example of how to do this properly, see FAQ #40.

It's worth noting that you cannot put anything other than a list of arguments into an array variable when using the "${array[@]}" technique to evaluate a command. Pipelines, redirection, assignments, and any other shell keywords or syntax will not be evaluated correctly.

In bash, the only ways to generate, manipulate, or store code more complex than a simple command at runtime involve storing the code's plain text in a variable, file, stream, or function, and then using eval or sh to evaluate the stored code. Directly manipulating raw code strings is among the least robust of metaprogramming techniques and most common sources of bugs and security issues. That's because predicting all possible ways code might come together to form a valid construct and restricting it to never operate outside of what's expected requires great care and detailed knowledge of language quirks. Bash lacks all the usual kinds of abstractions that allow doing this safely. Excessive use can also obfuscate your code.

6. I want a log of my script's actions

Another reason people attempt to stuff commands into variables is because they want their script to print each command before it runs it. If that's all you want, then simply use set -x command, or invoke your script with #!/bin/bash -x or bash -x ./myscript.

if ((DEBUG)); then set -x; fi
mysql -u me -p somedbname < file
...

Note that you can turn it off and back on inside the script with set +x and set -x.

Some people get into trouble because they want to have their script print their commands including redirections. set -x shows the command without redirections. People try to work around this by doing things like:

# Non-working example
command="mysql -u me -p somedbname < file"
((DEBUG)) && echo "$command"
"$command"

(This is so common that I include it here explicitly.)

Once again, this does not work. You can't make it work. Even the array trick won't work here.

One way to log the whole command, without resorting to the use of eval or sh (don't do that!), is the DEBUG trap. A practical code example:

trap 'printf %s\\n "$BASH_COMMAND" >&2' DEBUG

Assuming you're logging to standard error.

Note that redirect representation by BASH_COMMAND may still be affected by this bug.

If you STILL think you need to write out every command you're about to run before you run it, AND that you must include all redirections, AND you can't use a DEBUG trap, then just do this:

# Working example
echo "mysql -u me -p somedbname < file"
mysql -u me -p somedbname < file

Don't use a variable at all. Just copy and paste the command, wrap an extra layer of quotes around it (can be tricky -- that's why we do not recommend trying to use eval here), and stick an echo in front of it.

However, consider that echoing your commands verbatim is really ugly. Why are you doing this? Are you debugging the script? If so, how is the output of set -x insufficient? All you have to do is find the bug and fix it. Surely you won't leave this debugging code in place once the bug has been fixed.

If you intend to create a log of your script's actions, every time it is run, for accountability or other reasons, then that log should be human-readable. In that case, don't just echo your commands (especially if you have to bend over backwards to do so)! Write out meaningful (possibly even date-stamped) lines describing what you're doing.

echo "Populating database table"
mysql -u me -p somedbname < file

7. I cannot use Bash arrays. How can I pass arguments in a string?

You can use printf and xargs to encode and decode strings

# build a string of arguments
# use printf to escape each argument
argstring=""
arg="'a a'"; argstring+=" $(printf '%q' "$arg")"
arg='"b b"'; argstring+=" $(printf '%q' "$arg")"
arg="c\\c"; argstring+=" $(printf '%q' "$arg")"
echo "$argstring"
#  \'a\ a\' \"b\ b\" c\\c

# print arguments with xargs
echo "$argstring" | xargs printf 'arg: %s\n'
# arg: 'a a'
# arg: "b b"
# arg: c\c

# loop arguments with xargs
while read -d '' arg; do
    echo "arg: $arg"
done < <(<<<"$argstring" xargs printf '%s\0')
# arg: 'a a'
# arg: "b b"
# arg: c\c

# create a bash array with xargs and readarray
readarray -d '' argarray < <(<<<"$argstring" xargs printf '%s\0')
# loop the bash array
for arg in "${argarray[@]}"; do
    echo "arg: $arg"
done
# arg: 'a a'
# arg: "b b"
# arg: c\c

Alternative: You can replace $(printf '%q' "$arg") with ${arg@Q}. This will use single-quotes to quote the $arg string. This is less pretty, because it will always add quotes, also when not needed.

Limitation: xargs cannot parse "\"" and '\''

• Note that xargs handles quoting in a different way from shells (its handling is closer to the Mashey shell's from the late 70s which is not in use anymore these days). For instance, it would choke on "foo\"bar".

  - Stéphane Chazelas https://unix.stackexchange.com/questions/676249

# problem
argstring="\"a \\\" a\""
echo "$argstring"
# "a \" a"
echo "$argstring" | xargs printf 'arg: %s\n'
# xargs: unmatched double quote

# solution
arg="\"a \\\" a\""
argstring="$(printf '%q' "$arg") ${arg@Q}"
echo "$argstring"
# \"a\ \\\"\ a\" '"a \" a"'
echo "$argstring" | xargs printf 'arg: %s\n'
# arg: "a \" a"
# arg: "a \" a"

Same problem with single quotes:

# problem
argstring="'c \\' c'"
echo "$argstring"
# 'c \' c'
echo "$argstring" | xargs printf 'arg: %s\n'
# xargs: unmatched single quote

# solution
arg="'c \\' c'"
argstring="$(printf '%q' "$arg") ${arg@Q}"
echo "$argstring"
# \'c\ \\\'\ c\' ''\''c \'\'' c'\'''
echo "$argstring" | xargs printf 'arg: %s\n'
# arg: 'c \' c'
# arg: 'c \' c'

CategoryShell