Arrays

As mentioned earlier, BASH provides three types of parameters: Strings, Integers and Arrays.

Strings are without a doubt the most used parameter type. But they are also the most mis-used parameter type. It is important to remember that a string holds just one element. Capturing the output of a command, for instance, and putting it in a string parameter means that parameter holds just one string of characters. Whether that string represents twenty filenames, twenty numbers or twenty names of people.

And as is always the case when you put multiple items in a single string: these multiple items are somehow delimited from each other. We, humans, can usually decipher which the different filenames are when looking at a string. We assume that, perhaps, each line in the string represents a filename. Or each word represents a filename. While this assumption is understandable, it is also inheritly flawed: Each single filename can contain every character you might want to use to separate the filenames from each other in a string. That means there's technically no telling where the first filename in the string ends, because there's no character that can say: "I denote the end of this filename" because that character itself could be part of the filename.

Often, people make this mistake:

    $ files=$(ls); cp $files /backups/

This attempt at backing up our files in the current directory is flawed. We put the output of ls in a string called files and then use the unquoted $files parameter expansion to cut that string into arguments (relying on Word Splitting). As mentioned before; argument and word splitting cuts a string into pieces wherever there is whitespace. Relying on it means we assume that none of our filenames will contain any whitespace. If they do, the filename will be cut in half or more. Conclusion: bad.

The only safe way to represent multiple string elements in bash is through the use of arrays. Arrays are types that map integers to strings. That basically means that they hold a numbered list of strings. Since each of these strings is a separate entity (element), it can safely contain any character.

For the best results and the least headaches, remember that if you have a list of things that you need to remember: Always put it in an array.

Also note that since arrays map integers to strings you can't use them to do any other mapping. For instance, you cannot use strings as keys or other arrays as values. Usually, if you think you need something like this, you can change the logic of your script such that you don't need it anymore. If you can't, you should probably consider switching to a language that's built for this sort of job (perl, python, ruby) or use multiple arrays with a shared index in bash.



Creating Arrays

There are several ways you can create or fill your array with data. There is no one single true way: the method you'll need depends on where your data comes from and what it is.

The easiest way to create a simple array with data is by using the =() syntax:

    $ names=("Bob" "Peter" "$USER" "Big Bad John")

This syntax is great for creating arrays with static data or a known set of string parameters. Though it gives us very little flexibility for adding lots of array elements.

When the elements in the arrays are filenames, then you'll probably want to use Globs in there:

    $ photos=(~/"My Photos"/*jpg)

Notice here that we quoted the My Photos part because it contains a space. If we hadn't quoted it, bash would have split it up into photos=('~/My' 'Photos/'*.jpg ) which is obviously not what we want. Also notice that we quoted only the part that contained the space. That's because we cannot quote the ~ or the *; if we do, they'll become literal and bash won't treat them as special characters anymore.

Creating arrays with a bunch of filenames becomes really easy like this. So remember to never use ls:

    $ files=$(ls)  # BAD, BAD, BAD!
    $ files=(*)    # Good!

The first would create a string with the output of ls. That string cannot possibly be used safely for reasons mentioned in the Arrays introduction. The second statement gives us an array where each filename is a separate string. Perfect.

Now, sometimes we want to build an array from a string or the output of a command. Commands (generally) just output strings: for instance, running a find command will enumerate filenames, and separate these filenames with newlines (putting each filename on a separate line). So to parse that one big string into an array we need to tell bash what character delimits the parts of the string that we want to put in separate array elements.

That's what IFS is used for:

    $ IFS=. read -a ip_elements <<< "127.0.0.1"

Here we use IFS with the value . to cut the given ip address into array elements wherever there's a ., resulting in an array with the elements 127, 0, 0 and 1.

Very often, though, it's impossible to safely tell what the delimiter is. For instance, when find outputs filenames separated by a newline, using a newline to delimit our parsing of that output string is flawed in that a filename can itself contain a newline. We cannot assume that each newline means: "A new filename follows", which means we cannot parse the output safely. And if we can't even parse it safely, there's little point in parsing it badly and then putting it in a safe array: The array would just contain badly parsed data.

The answer to this problem is NULL bytes. The main difference between strings and the output of commands is that the latter is a stream, not a string. Streams are just like strings with one big difference: They can contain NULL bytes, strings cannot. A NULL byte is a byte which is just all zeros: 00000000. The reason that they can't be used in strings is an artifact of the C programming language: NULL bytes are used in your computer memory to mark the end of a string. That way, when you output a string, you don't output everything in your memory starting at the string's location until the end of your memory but just until it reaches a NULL byte.

Streams can contain NULL bytes, and we will use them to delimit our data. Since strings can't contain NULL bytes, neither can filenames or anything else we may want to put in an array element. That makes it the perfect candidate for safe array creation. Usually, the command you want to read the output of has an option that makes it output its data separated by NULL bytes rather than newlines or something else. find has the option -print0, which we'll use in this example:

    files=()
    while read -r -d $'\0'; do
        files+=("$REPLY")
    done < <(find /foo -print0)

This is the only safe way of parsing a command's output into a string. Understandably, it looks a little confusing and convoluted at first. So let's take it apart:

We're using a while loop that runs a read command each time. The read command uses the -d $'\0' option, which means that instead of reading a line (up to a newline), we're reading up to a NULL byte (\0).

Once read has read some data and encountered its NULL byte, the while loop's body is executed where we put what we read (which is in REPLY) into our array.

To do this, we use the +=() syntax. This syntax adds an (or more) element(s) to the end of our array.

Finally, the < <(..) syntax is a combination of File Redirection (<) and Process Substitution (<(..)) which is used to redirect the output of the find command into our while loop.

The find command itself uses the -print0 option as mentioned before to tell it to separate the filenames it finds with a NULL byte.





Using Arrays

Walking over array elements is really easy. Because arrays are such a safe medium of storage, we can simply use a for loop to iterate over its elements:

    $ for file in "${myfiles[@]}"; do
    >     cp "$file" /backups/
    > done

Notice the syntax used to expand the array here. We use the quoted form: "${arrayname[@]}". That causes bash to replace it with every single element in the array, properly quoted. For instance, these are identical:

    $ names=("Bob" "Peter" "$USER" "Big Bad John")
    $ for name in "${names[@]}"; do :; done
    $ for name in "Bob" "Peter" "$USER" "Big Bad John"; do :; done

Remember to quote the ${arrayname[@]} properly. If you don't you loose all benefit of using an array at all: You're telling bash it's OK to wordsplit your array elements to pieces and break everything again.

Another use of "${arrayname[@]}" is simplifying the above loop, for instance:

    cp "${myfiles[@]}" /backups/

This runs the cp command, replaces the "${myfiles[@]}" part by each filename in the myfiles array, properly quoted, causing cp to safely copy them to your backups.

You can also expand single array elements by referencing their element number (called index). Though remember that arrays are zero-based, which means that their first element has the index zero:

    $ echo "The first name is: ${names[0]}"
    $ echo "The second name is: ${names[1]}"

There is also a second form of expanding all array elements, which is "${arrayname[*]}". This form is ONLY useful for converting arrays back into a single string. The only real purpose for this is outputting the array to humans:

    $ names=("Bob" "Peter" "$USER" "Big Bad John")
    $ echo "Today's contestants are: ${names[*]}"
    Today's contestants are: Bob Peter lhunath Big Bad John

Remember to still keep everything nicely quoted! If you don't keep ${arrayname[*]} quoted, once again bash's wordsplitting will cut it into bits.

You can combine IFS with ${arrayname[*]} to indicate the character to use to delimit your array elements as you merge them into a single string. This is handy, for example, when you want to comma delimit names:

    $ names=("Bob" "Peter" "$USER" "Big Bad John")
    $ ( IFS=,; echo "Today's contestants are: ${names[*]}" )
    Today's contestants are: Bob,Peter,lhunath,Big Bad John

Notice how in this example we put the IFS=,; echo ... statement in a Subshell by wrapping ( and ) around it. We do this because we don't want to change the default value of IFS in the main shell. As soon as the subshell exits, IFS is still its default value and no longer just a comma. This is important because IFS is used for a lot of things, and changing its value to something non-default will result in very odd behaviour if you don't expect it!