Diff for "ComplexOptionParsing"

This page is a split-off from Bash FAQ 35. It holds all of the crazy nonstandard ideas that people have come up with to write option-parsing code. These ideas tend to be over-engineered, half-baked, inefficient, unreliable, unsafe, or downright insane, or any combination of the above. Use at your own risk.

If you find yourself writing a bash script which is so large that you think you need a third-party option-processing module, then you are writing in the wrong language.

util-linux's special getopt

Traditional Unix systems have a program named getopt(1). This is a massively broken tool. You should never use it.

The purpose of getopt was to rewrite a script's command arguments into a normalized form, splitting single-letter options apart (e.g. -abc to -a -b -c), and separating an option and its argument into two arguments (-ffile to -f file), so that a Bourne shell script could write a reasonable option parsing loop. Unfortunately, getopt was written in the Days Of Olde, when shells didn't have arrays, lists were stored in strings, and it was all fine and dandy because filenames didn't have spaces either.

In the real world, filenames (and even non-filename arguments like CFLAGS="-g -O") contain spaces, and empty arguments are used, and both of these cause getopt to fail. The failure is so inextricably baked into the design of getopt that it cannot be fixed (see example below). Every Unix system which still ships getopt includes warnings in its man page telling you not to use it.

And then... there's Linux.

Linux is so special that its developers decided to rewrite getopt. They made one that handles arguments with spaces in them, and empty arguments, and they did so by breaking compatibility with the Unix getopt program. But they did not change the name. So now there are two completely incompatible programs in the world, both named getopt, and one of them is a live minefield.

The most reasonable response to this situation would be to avoid using getopt completely. Right? Well, not if you're a Linux-head. There are a few people in the world who apparently love their util-linux version of getopt so much that they are incapable of heeding the standard advice. They have to use it. They have to tell other people to use it, at every opportunity.

I got so sick of trying to remove util-linux getopt propaganda from the Bash FAQ in a respectful manner that I finally gave up and banned it from the FAQ, and wrote this rant.

So, this is your spot, util-linux zealots. Here's where you can tell the whole world how great getopt is on your one operating system, and how it can save you oodles of work writing your option parsing loops. Go for it.

To get you started, here is an example of the Unix getopt failing to do its job:

   1 #!/bin/sh
   2 
   3 # Do NOT use this.  This example is historical only.
   4 
   5 if args=`getopt abo: $*`
   6 then
   7     :
   8 else
   9     echo "usage: ..." >&2
  10     exit 1
  11 fi
  12 set -- $args
  13 
  14 a=0
  15 b=0
  16 out=default.out
  17 
  18 while [ $# -gt 0 ]
  19 do
  20     case $1
  21     in
  22         -a) a=1;;
  23         -b) b=1;;
  24         -o) out="$2"; shift;;
  25         --) shift; break;;
  26         *)  break;;
  27     esac
  28     shift
  29 done

As you can see, this code will fail completely when you feed it any arguments that contain whitespace, or which are empty strings. They get split or dropped by the several layers of false assumptions.

Now, here's the same option set, using the util-linux version of getopt:

   1 #!/bin/bash
   2 
   3 getopt -T >/dev/null 2>&1
   4 if (($? != 4)); then
   5     echo "util-linux getopt is not installed.  Aborting." >&2
   6     exit 2
   7 fi
   8 
   9 a=0
  10 b=0
  11 out=default.out
  12 
  13 eval set -- "$(getopt -o abo: -- "$@")"
  14 while (($# > 0)); do
  15     case $1 in
  16         -a) a=1;;
  17         -b) b=1;;
  18         -o) out="$2"; shift;;
  19         --) shift; break;;
  20         *) break;;
  21     esac
  22     shift
  23 done

I'll let the util-linux fans explain why this is worthy of such dedicated fervor.

getopts long option trickery

IMPORTANT NOTE BEFORE YOU READ ANY FURTHER AND TRY TO APPLY THIS TO YOUR SCRIPT

Suppose that you want to check for --hello and --world and for -a, -b and -c. You pass this optspec to getopts: :abc-:, and then you check if $OPTARG is hello or world.

This will be parsed as correct: ./myscript -a --hello -b --world -c but so will this: ./myscript -ab-hello -c- world

DO YOU REALLY WANT THAT?

Sure, you can work around this thing by doing triple backflips with $OPTIND, but please stop using this blindly and posting it on the internet where uninformed noobs will read it.

Here is an example which claims to parse long options with getopts. The basic idea is quite simple: just put "-:" into the optstring. This trick requires a shell which permits the option-argument (i.e. the filename in "-f filename") to be concatenated to the option (as in "-ffilename") as POSIX requires. The POSIX standard says you should pass arguments to options as separate options (to avoid problems with empty arguments) but requires getopts to handle the case where the argument is stuck to the option in the same argument. However, POSIX leaves the behaviour unspecified when an option is not alpha-numerical, so POSIX does not guarantee this to work. However, it should work in all of bash, dash, yash, zsh and all implementations of ksh, the only exception reported so far being posh.

   1 #!/bin/bash --
   2 # Uses bash extensions.  Not portable as written.
   3 
   4 usage() {
   5     echo "Usage:"
   6     echo " $0 [ --loglevel=<value> | --loglevel <value> | -l <value> ] [ --range <beginning> <end> ] [--] [non-option-argument]..."
   7     echo " $0 [ --help | -h ]"
   8     echo
   9     echo "Default loglevel is 0. Default range is 0 to 0"
  10 }
  11 
  12 # set defaults
  13 loglevel=0
  14 r1=0 # beginning of range
  15 r2=0 # end of range
  16 
  17 i=$(($# + 1)) # index of the first non-existing argument
  18 declare -A longoptspec
  19 # Use associative array to declare how many arguments a long option
  20 # expects. In this case we declare that loglevel expects/has one
  21 # argument and range has two. Long options that aren't listed in this
  22 # way will have zero arguments by default.
  23 longoptspec=( [loglevel]=1 [range]=2 )
  24 optspec=":l:h-:"
  25 while getopts "$optspec" opt; do
  26 while true; do
  27     case "${opt}" in
  28         -) #OPTARG is name-of-long-option or name-of-long-option=value
  29             if [[ ${OPTARG} =~ .*=.* ]] # with this --key=value format only one argument is possible
  30             then
  31                 opt=${OPTARG/=*/}
  32                 ((${#opt} <= 1)) && {
  33                     echo "Syntax error: Invalid long option '$opt'" >&2
  34                     exit 2
  35                 }
  36                 if ((longoptspec[\$opt] != 1))
  37                 then
  38                     echo "Syntax error: Option '$opt' does not support this syntax." >&2
  39                     exit 2
  40                 fi
  41                 OPTARG=${OPTARG#*=}
  42             else #with this --key value1 value2 format multiple arguments are possible
  43                 opt="$OPTARG"
  44                 ((${#opt} <= 1)) && {
  45                     echo "Syntax error: Invalid long option '$opt'" >&2
  46                     exit 2
  47                 }
  48                 OPTARG=("${@:OPTIND:longoptspec[\$opt]}")
  49                 ((OPTIND+=longoptspec[\$opt]))
  50                 echo "$OPTIND"
  51                 ((OPTIND > i)) && {
  52                     echo "Syntax error: Not all required arguments for option '$opt' are given." >&2
  53                     exit 3
  54                 }
  55             fi
  56 
  57             continue #now that opt/OPTARG are set we can process them as
  58             # if getopts would've given us long options
  59             ;;
  60         l|loglevel)
  61             loglevel=$OPTARG
  62             ;;
  63         range)
  64             r1=${OPTARG[0]}
  65             r2=${OPTARG[1]}
  66             ;;
  67         h|help)
  68             usage
  69             exit 0
  70             ;;
  71         ?)
  72             echo "Syntax error: Unknown short option '$OPTARG'" >&2
  73             exit 2
  74             ;;
  75         *)
  76             echo "Syntax error: Unknown long option '$opt'" >&2
  77             exit 2
  78             ;;
  79     esac
  80 break; done
  81 done
  82 
  83 echo "Loglevel: $loglevel"
  84 echo "Range: $r1 to $r2"
  85 echo "First non-option-argument (if exists): ${!OPTIND-}"
  86 
  87 # End of file

With this version you can have long and short options side by side and you shouldn't need to modify the code from line 25 to 59. This solution can also handle multiple arguments for long options, just use ${OPTARG} or ${OPTARG[0]} for the first argument, ${OPTARG[1]} for the second argument, ${OPTARG[2]} for the third argument and so on. It has the same disadvantage of its predecessor in not being portable and specific to bash. Additionally, it breaks arguments with whitespace (line 48) for long options, when given using the syntax that does not include equal sign (=).

Silly repeated brute-force scanning

Another approach is to check options with if statements "on demand". A function like this one may be useful:

   1 #!/bin/bash
   2 
   3 HaveOpt ()
   4 {
   5     local needle=$1
   6     shift
   7 
   8     while [[ $1 == -* ]]
   9     do
  10         # By convention, "--" means end of options.
  11         case "$1" in
  12             --)      return 1 ;;
  13             $needle) return 0 ;;
  14         esac
  15 
  16         shift
  17     done
  18 
  19     return 1
  20 }
  21 
  22 HaveOpt --quick "$@" && echo "Option quick is set"
  23 
  24 # End of file

and it will work if script is run as:

• YES: ./script --quick
• YES: ./script -other --quick

but will stop on first argument with no "-" in front (or on --):

• NO: ./script -bar foo --quick
• NO: ./script -bar -- --quick

Of course, this approach (iterating over the argument list every time you want to check for one) is far less efficient than just iterating once and setting flag variables.

It also spreads the options throughout the program. The literal option --quick may appear a hundred lines down inside the main body of the program, nowhere near any other option name. This is a nightmare for maintenance.

Complex nonstandard add-on utilities

process-getopt

bhepple suggests the use of process-getopt (GPL licensed) and offers this example code:

PROG=$(basename $0)
VERSION='1.2'
USAGE="A tiny example using process-getopt(1)"

# call process-getopt functions to define some options:
source process-getopt

SLOT=""
SLOT_func()   { [ "${1:-""}" ] && SLOT="yes"; }      # callback for SLOT option
add_opt SLOT "boolean option" s "" slot

TOKEN=""
TOKEN_func()  { [ "${1:-""}" ] && TOKEN="$2"; }      # callback for TOKEN option
add_opt TOKEN "this option takes a value" t n token number

add_std_opts     # define the standard options --help etc:

TEMP=$(call_getopt "$@") || exit 1
eval set -- "$TEMP" # just as with getopt(1)

# remove the options from the command line
process_opts "$@" || shift "$?"

echo "SLOT=$SLOT"
echo "TOKEN=$TOKEN"
echo "args=$@"

Here, all information about each option is defined in one place making for much easier authoring and maintenance. A lot of the dirty work is handled automatically and standards are obeyed as in getopt(1) - because it calls getopt for you.

• Actually, what the author forgot to say was that it's actually using getopts semantics, rather than getopt. I ran this test:

   ~/process-getopt-1.6$ set -- one 'rm -rf /' 'foo;bar' "'"
   ~/process-getopt-1.6$ call_getopt "$@"
    -- 'rm -rf /' 'foo;bar' ''\'''

• It appears to be intelligent enough to handle null options, whitespace-containing options, and single-quote-containing options in a manner that makes the eval not blow up in your face. But this is not an endorsement of the process-getopt software overall; I don't know it well enough. -GreyCat

It's written and tested on Linux where getopt(1) supports long options. For portability, it tests the local getopt(1) at runtime and if it finds an non-GNU one (ie one that does not return 4 for getopt --test) it only processes short options. It does not use the bash builtin getopts(1) command. -bhepple

Argbash

Argbash is a simple-to-use yet feature-rich code generator that can either generate the parsing code for your script, tailor-made. The project features extensive documentation. Argbash can be downloaded and installed locally, or one can try a script generator online.

The sample project from above (script accepting a -f|--file|--verbose|...) would use the following template:

   1 #!/bin/bash
   2 
   3 # ARG_OPTIONAL_SINGLE([file],[f],[input file])
   4 # ARG_VERBOSE()
   5 # ARG_POSITIONAL_DOUBLEDASH()
   6 # ARG_LEFTOVERS([other args])
   7 # ARGBASH_GO()
   8 
   9 # [ <-- needed because of Argbash
  10 
  11 if [ "$_arg_verbose" -gt 0 ]; then
  12         echo "Input file: $_arg_file"
  13         echo "Other args: ${_arg_leftovers[*]}"
  14 fi
  15 
  16 # ] <-- needed because of Argbash

Then, executing the result script as ./script.sh -f my-file --verbose -- one two three --file foo would yield

    Input file: my-file
    other args: one two three --file foo

The project also features a quickstart utility that can generate a minimal template for you like this: argbash-init --opt file --pos arg-positional | argbash - -o basic_script.sh && ./basic_script.sh -h

<The general help message of my script>
Usage: ./basic_script.sh [--file <arg>] [-h|--help] <arg-positional>
        <arg-positional>: <arg-positional's help message goes here>
        --file: <file's help message goes here> (no default)
        -h,--help: Prints help

One can then just fine-tune the template and get a script with argument parsing capabilities with little effort.

Rearranging arguments

If you want to mix flags and arguments here's a function to rearrange all options to place flags first. Use before doing your normal getopts parsing.

   1 arrange_opts() {
   2     local flags args optstr=$1
   3     shift
   4 
   5     while (($#)); do
   6         case $1 in
   7             --) args+=("$@")
   8                 break;
   9                 ;;
  10             -*) flags+=("$1")
  11                 if [[ $optstr == *"${1: -1}:"* ]]; then
  12                     flags+=("$2")
  13                     shift
  14                 fi
  15                 ;;
  16             * ) args+=("$1")
  17                 ;;
  18         esac
  19         shift
  20     done
  21     OPTARR=("${flags[@]}" "${args[@]}")
  22 }
  23 
  24 example() {
  25     local OPTIND OPTARR optstring=ab:cd
  26 
  27     printf 'before arrange:'
  28     printf '<%s>' "$@"
  29     echo
  30 
  31     arrange_opts "$optstring" "$@"
  32     set -- "${OPTARR[@]}"
  33 
  34     printf 'after  arrange:'
  35     printf '<%s>' "$@"
  36     printf \\n\\n
  37 
  38     printf flag:arg\\n
  39 
  40     OPTIND=1
  41     while getopts "$optstring" opt; do
  42         case $opt in
  43             a|c|d) printf        %s\\n "$opt"    ;;
  44             b    ) printf      b:%s\\n "$OPTARG" ;;
  45             \?   ) printf badopt:%s\\n "$OPTARG" ;;
  46             :    ) printf  noarg:%s\\n "$OPTARG" ;;
  47             *    ) printf    wtf:%s\\n "$opt"    ;;
  48         esac
  49     done
  50     shift $((OPTIND-1))
  51 
  52     printf \\nargs:
  53     printf '<%s>' "$@"
  54     echo
  55 }
  56 
  57 example -ad foo -b bar baz

CategoryShell