|
Size: 134373
Comment:
|
Size: 7955
Comment:
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| #acl GreyCat:read,write,revert,admin,delete Lhunath:read,write,revert pgas:read,write,revert All:read [[BashGuide/CommandsAndArguments|Commands and Arguments ->]] ---- |
|
| Line 2: | Line 5: |
| '''This guide is currently still a work in progress. It grows a little every day. You are invited to make additions or modifications so long as you can keep them accurate (and linguistically correct).''' | '''You are invited to make additions or modifications so long as you can keep them accurate. Please test any code samples you write.''' |
| Line 6: | Line 10: |
| ["BASH"] is a BourneShell compatible shell, which adds many new features to its ancestor. Most of them are available in the 'KornShell', too. (Bourne Again - you will be ;-) ) | [[BASH]] is a BourneShell compatible shell, which adds many new features to its ancestor. Most of them are available in the 'KornShell', too. |
| Line 8: | Line 12: |
| ---- . [[TableOfContents]] |
|
| Line 11: | Line 13: |
| [[Anchor(About)]] | <<Anchor(About)>> = About This Guide = |
| Line 13: | Line 16: |
| == About This Guide == This guide aims to become a point of reference for people interested in learning to work with ["BASH"]. It aspires to teach its readers good practice techniques in developing scripts for the ["BASH"] interpreter and educate them about the internal operation of ["BASH"]. |
|
| Line 16: | Line 17: |
| This guide is targeted at beginning users. It assumes no basic knowledge, but rather expects you to have enough common sense to put two and two together. If something is unclear to you, you should report this so that it may be clarified in this document for future readers. | This guide aims to become a starting point for people interested in learning to work with [[BASH]]. It aspires to teach its readers good practice techniques for developing scripts for the [[BASH]] interpreter, and to educate them about the internal operation of [[BASH]]. This guide is targeted at beginning users. It assumes no basic knowledge, but rather expects you to have enough common sense to put two and two together. If something is unclear to you, you are invited to report this (use BashGuideFeedback, or the `#bash` channel on `irc.freenode.org`) so that it may be clarified in this document for future readers. |
| Line 22: | Line 25: |
| * -- ["Lhunath"] -------- [[Anchor(Definition)]] |
* -- [[Lhunath]] (primary author) * -- GreyCat |
| Line 26: | Line 28: |
| == A Definition == ["BASH"] is an acronym for '''''B'''ourne '''A'''gain '''Sh'''ell''. It is based on the ''Bourne'' shell and is mostly compatible with its features. Shells are applications that provide users with the ability to give commands to their operating system interactively, or to allow them to execute batch processes quickly. In no way are they required for execution of processes; they are merely a layer between system function calls and the user. |
The guide is also available in [[http://stuff.lhunath.com/BashGuide.pdf|PDF format]]. Alternatively, you can just hit print after going to [[FullBashGuide]]. This guarantees you'll be printing the latest version of this document. |
| Line 32: | Line 32: |
| . '''In The Manual: [http://www.gnu.org/software/bash/manual/bashref.html#SEC1 Introduction]''' | <<Anchor(Definition)>> = A Definition = [[BASH]] is an acronym for '''''B'''ourne '''A'''gain '''Sh'''ell''. It is based on the ''Bourne'' shell and is mostly compatible with its features. Shells are command interpreters. They are applications that provide users with the ability to give commands to their operating system interactively, or to execute batch processes quickly. In no way are they required for the execution of processes; they are merely a layer between system function calls and the user. Think of a shell as a way for you to speak to your system. Your system doesn't need it for most of its work, but it is an excellent interface between you and what your system can offer. It allows you to perform basic math, run basic tests and execute applications. More importantly, it allows you to combine these operations and connect applications to each other to perform complex and automated tasks. [[BASH]] is '''not''' your operating system. It is not your window manager. It is not your terminal (but it oftens runs ''inside'' your terminal). It does not control your mouse or keyboard. It does not configure your system, activate your screensaver, or open your files when you double-click them. It is generally not involved in launching applications from your window manager or desktop environment. It's important to understand that [[BASH]] is only the interface for you to execute statements (using [[BASH]] syntax), either at the interactive [[BASH]] prompt or via [[BASH]] scripts. -------- . '''In The Manual: [[http://www.gnu.org/software/bash/manual/bashref.html#Introduction|Introduction]]''' |
| Line 34: | Line 47: |
| . ''Shell'': A (possibly interactive) layer between the user and the system. [[BR]] ''["BASH"]'': The Bourne Again Shell, a ''Bourne'' compatible shell. | . ''Shell'': A (possibly interactive) command interpreter, acting as a layer between the user and the system. <<BR>> ''[[BASH]]'': The Bourne Again Shell, a ''Bourne'' compatible shell. |
| Line 36: | Line 50: |
| [[Anchor(Using_Bash)]] | <<Anchor(Using_Bash)>> = Using Bash = |
| Line 38: | Line 53: |
| == Using Bash == Most users that think of ["BASH"] think of it as a prompt and a command line. That is ["BASH"] in ''interactive mode''. ["BASH"] can also run in ''non-interactive mode'' through scripts. We can use scripts to automate certain logic. Scripts are basically lists of commands that you can type on the command line. When such a script is executed, all these commands are executed sequentially, one after another. |
Most users that think of [[BASH]] think of it as a prompt and a command line. That is [[BASH]] in ''interactive mode''. [[BASH]] can also run in ''non-interactive mode'' through scripts. We can use scripts to automate certain logic. Scripts are basically lists of commands that you can type on the command line. When such a script is executed, all these commands are (generally) executed sequentially, one after another. |
| Line 43: | Line 58: |
| '''Important! [[BR]] You should make yourself familiar with the `man` and `apropos` commands on the shell. They will be vital to your self-tutoring.''' | '''Important! <<BR>> You should make yourself familiar with the `man` and `apropos` commands on the shell. They will be vital to your self-tutoring.''' |
| Line 49: | Line 64: |
| -------- . '''In the FAQ: [[BR]] [http://wooledge.org/mywiki/BashFAQ/061 Is there a list of which features were added to specific releases (versions) of Bash?]''' ---- . ''Interactive mode'': A mode of operation where a prompt asks you for one command at a time. ''Script'': A file that contains a sequence of commands to execute one after the other. -------- [[Anchor(Scripts)]] |
|
| Line 56: | Line 65: |
| == Scripts == A script is basically a sequence of commands that ["BASH"] processes in order. It only moves on to the next command when the current one has ended, unless the current one has been executed asynchronously (in the background). Don't worry too much about the latter case yet -- you'll learn about how that works later on. |
In this guide, the `$` at the beginning of a line represents your [[BASH]] prompt. Traditionally, a shell prompt either ends with `$`, `%` or `#`. If it ends with `$`, this indicates a shell that's compatible with the Bourne shell (such as a POSIX shell, or a Korn shell, or [[BASH]]). If it ends with `%`, this indicates a ''C shell'' (csh or tcsh); this guide does not cover C shell. If it ends with `#`, this indicates that the shell is running as the system's superuser account (`root`), and that you should be extra careful. |
| Line 59: | Line 67: |
| Virtually any example that you see in this guide can be used in a script just as well as on the command line. | Your actual [[BASH]] prompt will probably be much longer than `$`. Prompts are often highly individualized. |
| Line 61: | Line 69: |
| Making a script is easy. You just make a new file, and put this in it at the top: | The `man` command opens documentation (so-called "man pages") on a certain topic. You use it by running the command `man [topic]` at the [[BASH]] prompt, where `[topic]` is the name of the "page" you wish to read. Note that many of these "pages" are considerably longer than one printed page; nevertheless, the name persists. Each command (application) on your system is likely to have a man page. There are pages for other things too, such as system calls or specific configuration files. In this guide, we will only be covering commands. Note that if you're looking for information on [[BASH]] built-ins (commands provided by [[BASH]], not by external applications) you should look in `man bash` instead. [[BASH]]'s manual is extensive and detailed. It is an excellent reference, albeit more technical than this guide. [[BASH]] also offers a `help` command which contains brief summaries of its built-in commands (which we'll discuss in depth later on). |
| Line 64: | Line 76: |
| #!/bin/bash | $ help $ help read |
| Line 66: | Line 79: |
| This header makes sure that whenever your script is executed, ["BASH"] will be used as its interpreter. '''Please''' do not be fooled by examples on the Internet that use `/bin/sh` as interpreter. '''`sh` is not `bash`'''. Even though `sh`'s syntax and `bash`'s look very much alike and even though most `bash` scripts will run in `sh`, a lot of the examples in this guide only apply to `bash` and will just break or cause unexpected behaviour in `sh`. Also, please refrain from giving your scripts that stupid `.sh` extension. It serves no purpose, and it's completely misleading (since it's going to be a `bash` script, not an `sh` script). And by the way, it's perfectly fine if you use ''Windows'' to write your scripts, but if at all possible, '''avoid using ''Notepad'' for writing scripts'''. ''Microsoft Notepad'' can only make files with DOS-style line-endings. That means that each line you make in notepad will be ended by two characters: a ''Carriage Return'' and a ''Newline'' character. ["BASH"] reads lines as terminated by ''Newline'' characters only. As a result, the ''Carriage Return'' character will cause you incredible headache if you don't know it's there (very weird error messages). If at all possible, ''use a decent editor'' like [http://www.vim.org/ Vim], [http://www.gnu.org/software/emacs/ Emacs], kate, GEdit, GVIM or xemacs. If you don't, then you will need to remove the carriage returns from your scripts before running them. Once your script file has been made, you can run it like this: {{{ $ bash myscript }}} In this example, we execute ["BASH"] and tell it to run our script. Alternatively, you can give your script executable permissions. When you do this, you can actually execute the script instead of executing ["BASH"] with it: {{{ $ chmod +x myscript $ ./myscript }}} Some people like to keep their scripts in a personal directory. Others like to keep their scripts somewhere in the `PATH` variable. Most like to do both at once. Here's what I suggest you do: {{{ $ mkdir -p "$HOME/bin" $ echo 'PATH="$HOME/bin:$PATH"' >> "$HOME/.bashrc" }}} The first command will make a directory called `bin` in your home directory. The second command will add a line to your `.bashrc` file which adds the directory we just made to the beginning of the `PATH` variable. Every new instance of ["BASH"] will now check for executable scripts in your `bin` directory. To apply the changes we added to `.bashrc` we obviously need to actually process `.bashrc` first. You can do that by closing your existing terminal and opening a new one. ["BASH"] will then initialize itself again by reading `.bashrc` among others. Alternatively you can just execute that line of code on the command line (`PATH="$HOME/bin:$PATH"`) or manually process your `.bashrc` file in the running shell by running `source "$HOME/.bashrc"`. As a result, we can now put our script in our `bin` directory and execute it as a normal command (we no longer need to prepend our script's name with its path, which was the `./` part in the previous examples): {{{ $ mv myscript "$HOME/bin" $ myscript }}} -------- . '''Tip: [[BR]] While you're defining the interpreter in your header, you might want to take the time to explain your script's function and expected arguments a little too:''' {{{ #! /bin/bash # # scriptname argument [argument] ... # # A short explanation of your script's purpose. # # Copyright [date], [name] }}} ---- . '''Tip: [[BR]] You can use this header to specify up to one word of optional arguments that you want to pass to the interpreter. For example, the following arguments will turn on some verbose debugging:''' {{{ #! /bin/bash -xv }}} ---- . ''Header'': The header of a script determines the application that will function as its interpreter (e.g. `bash`, `sh`, `perl`, ...). -------- [[Anchor(Basics)]] = The Basics = [[Anchor(Commands_Arguments)]] == Commands And Arguments == ["BASH"] reads commands from its input (which is either a terminal or a file). In ''interactive mode'', its input is your terminal. These commands can be aliases, functions, builtin commands, or executable applications. * '''Aliases''': ["BASH"] can use aliases to make it easier to quickly execute complex commands. An alias is a ''name'' that is mapped to a certain ''string''. Whenever that ''name'' is used as a command in bash, it is replaced by the ''string''. * '''Functions''': Functions in ["BASH"] are much like aliases, but more powerful and general. A function contains shell commands, very much like a script. When a function is called, the commands in it are executed. * '''Builtin Commands''': ["BASH"] has some basic commands built into it, such as `cd` (change directory), `if` (conditional command execution), and so on. * '''Executable Applications''': ["BASH"] uses a variable that tells it where to find other applications. This variable is called `PATH`, and it is a set of directory names separated by colons -- for example, `/bin:/usr/bin`. Each directory can contain executables. When a command is specified in ["BASH"] without a pathname (e.g. `ls`), and it isn't an alias, function or builtin, ["BASH"] searches the paths in `PATH`, in order from left to right, for this command. Since these commands are not part of ["BASH"] and run as separate processes, they are also called ''external commands''. Each command can be followed by arguments. It is very important that you understand how this works exactly. If you don't grasp these concepts well, the quality of your code will degrade significantly and you will introduce very dangerous bugs. So, pay close attention in the next few chapters. {{{ $ ls a b c }}} `ls` is a command that lists files in the current directory. It's intended to be used only for producing human-readable results. Please don't try to parse, pipe, grep, capture, read, or loop over the output of `ls` in a script. It's dangerous and there's always a better way. {{{ $ mkdir d $ cd d $ ls }}} `mkdir` is a command that creates a new directory. We specified the argument `d` to that command. This way, the application `mkdir` is instructed to create a directory called `d`. After that, we use the builtin command `cd` to change the shell's current directory to `d`. `ls` shows us that the current directory (which is now `d`) is empty, since it doesn't display any filenames. |
|
| Line 144: | Line 81: |
| . '''Tip: [[BR]] You can use the `type` command to figure out the type of a command. [[BR]] For example:''' {{{ $ type rm rm is hashed (/bin/rm) $ type cd cd is a shell builtin }}} |
. '''In the FAQ: <<BR>> [[BashFAQ/061|Is there a list of which features were added to specific releases (versions) of Bash?]]''' |
| Line 152: | Line 83: |
| . '''In The Manual: [http://www.gnu.org/software/bash/manual/bashref.html#SEC16 Simple Commands]''' ---- . ''Alias'': A name that is mapped to a string. Whenever that name is used as a command, it is replaced by the string it has mapped. [[BR]] ''Function'': A name that is mapped to a script. Whenever that name is used as a command, the script is called with the arguments provided to the function's name on the command line. [[BR]] ''Builtin'': Certain features have been built into ["BASH"]. These are handled internally whenever they are executed on the command line (and often do not create a new process). [[BR]] ''Application'': A binary that can be executed by referring to it (`/bin/ls`) or if its location is in your `PATH` variable, you can execute it simply by using its name (`ls`). -------- [[Anchor(Argument_Splitting)]] == Argument Splitting == Commands in ["BASH"] can take multiple arguments. These arguments are used to tell the command exactly what it's supposed to do. In ["BASH"], you separate these arguments by spaces and tabs. {{{ $ ls $ touch a b c $ ls a b c }}} `touch` is an application that changes the 'Last Modified'-time of a certain file to the current time. If the filename that it's given does not exist yet, it simply creates that file, as a new and empty file. In this example, we passed three arguments. `touch` creates a file for each argument. `ls` shows us that three files have been created. {{{ $ rm * $ ls $ touch a b c $ ls a b c }}} `rm` is an application that removes all the files that it was given. ''*'' is a ''glob''. It basically means ''all files in the current directory''. You will read more about this later on. Now, did you notice that there are several spaces between `a` and `b`, and only one between `b` and `c`? Also, notice that the files that were created by `touch` are no different than the first time. You now know that the amount of whitespace between arguments does not matter. This is important to know. For example: {{{ $ echo This is a test. This is a test. $ echo This is a test. This is a test. }}} In this case, we provide the `echo` command with four arguments. 'This', 'is', 'a' and 'test.'. `echo` takes these arguments, and prints them out one by one with a space in between. In the second case, the exact same thing happens. The extra spaces make no difference. If we actually want the extra whitespace, we need to pass the sentence as one single argument. We can do this by using quotes: {{{ $ echo "This is a test." This is a test. }}} Quotes group everything inside them into a single argument. This argument is '`This is a test.`', properly spaced. `echo` prints this single argument out just like it always does. Be very careful to avoid the following: {{{ $ ls The secret voice in your head.mp3 secret $ rm The secret voice in your head.mp3 rm: cannot remove `The': No such file or directory rm: cannot remove `voice': No such file or directory rm: cannot remove `in': No such file or directory rm: cannot remove `your': No such file or directory rm: cannot remove `head.mp3': No such file or directory $ ls The secret voice in your head.mp3 }}} You need to make sure you quote filenames properly. If you don't you'll end up deleting the wrong things! `rm` takes filenames as arguments. If you do not quote filenames with spaces, `rm` thinks that each argument is a separate file. Since ["BASH"] splits your arguments at the spaces, `rm` will try to remove each word. Please have a good look at http://bash-hackers.org/wiki/doku.php?id=syntax:words if all this isn't very clear to you yet. -------- . '''Good Practice: [[BR]] You should ''always'' quote sentences or strings that belong together, even if it's not absolutely necessary. This will keep you alert and reduce the risk of human error in your scripts. [[BR]] For example, you should always quote arguments to the `echo` command.''' ---- . '''In the FAQ: [[BR]] [http://wooledge.org/mywiki/BashFAQ/050 I'm trying to construct a command dynamically, but I can't figure out how to deal with quoted multi-word arguments.]''' ---- . ''Arguments'': These are the optional additional words you can specify when running commands. They are appended to the command's name ('`ls -l foo`' executes `ls` with two arguments). [[BR]] ''Quotes'': The two forms of quotes (`'` and `"`) are used to protect certain special characters inside them from being interpreted as special by ["BASH"]. The difference between `'` and `"` will be discussed later. -------- [[Anchor(Special_Characters)]] == Special Characters == There are several special characters in ["BASH"] that have a non-literal meaning. When we use these characters, ["BASH"] evaluates these characters and their meaning, but usually does not pass them on to the underlying commands. Here are a few of those special characters, and what they do: * '''''[whitespace]''''': Whitespace (spaces, tabs and newlines). ["BASH"] uses whitespace to determine where words begin and end. The first word of each command is used as the command name; any additional words become arguments to that command. * '''"text"''': Double quotes. Double quotes protect the text inside from being split into multiple words or arguments. They also prevent the special meaning of single quotes inside. However, other special characters retain their special meanings. * ''''text'''': Single quotes. Single quotes protect the text inside from any kind of expansion by the shell and keeps it from being split into multiple words or arguments. They also prevent the special meaning of all special characters inside. * '''# text''': Comment character. Any word beginning with `#` begins a ''comment'' that extends to the next newline. Comments are not processed by the shell. * ''';''': Command separator. The semicolon is used to separate multiple commands from each other if the user chooses to keep them on the same line. It's ''basically'' the same thing as a newline. * '''\''': Escape character. The backslash protects the next character from being used in any special sort of way. * '''>''' or '''<''': Redirection character. These characters are used to modify (redirect) the input and/or output of a command. * '''[[ expression ]]''': Test expression. This evaluates the conditional expression. * '''{ commands; }''': Command Group. This executes the commands inside the braces as though they were only one command. It is convenient for places where ["BASH"] syntax requires only one command to be present. * '''{{{`command`}}}''', '''{{{$(command)}}}''': Command substitution (The latter form is '''highly''' preferred). Command substitution executes the command inside the substitution form first, and replaces itself by that command's output. * '''(command)''': Subshell Execution. This executes the command in a new bash shell, instead of in the current. * '''((expression))''': Arithmetic Evaluation. Inside the parentheses, operators such as +, -, * and / are seen as mathematical operators. * '''$((expression))''': Arithmetic Expansion. Comparable to the above, but this expression is replaced with the result of its arithmetic evaluation. * '''$''': Expansion character. This character is used for any form of parameter expansion. More about this later. Some examples: {{{ $ echo "I am $LOGNAME" I am lhunath $ echo 'I am $LOGNAME' I am $LOGNAME $ # boo $ echo An open\ \ \ space An open space $ echo "My computer is $(hostname)" My computer is Lyndir $ echo boo > file $ echo $(( 5 + 5 )) 10 $ (( 5 > 0 )) && echo "Five is bigger than zero." Five is bigger than zero. }}} -------- . '''In The Manual: [http://www.gnu.org/software/bash/manual/bashref.html#SEC6 Shell Syntax]''' ---- . ''Special Characters'': Characters that have a special meaning to ["BASH"]. Usually their meaning is interpreted and afterwards they are removed from the command before executing it. -------- [[Anchor(Parameters)]] == Parameters == Parameters should be seen as a sort of named space in memory where you can store your data. Generally speaking, they will store string data, but can also be used to store integers or arrays. -------- . ''Parameters'': Parameters store data that can be retrieved through a symbol or a name. |
. ''Interactive mode'': A mode of operation where a prompt asks you for one command at a time. . ''Script'': A file that contains a sequence of commands to execute one after the other. |
| Line 272: | Line 87: |
| === Special Parameters and Variables === Let's get our vocabulary straight before we get into the real deal. There are ''Parameters'' and ''Variables''. Variables are actually just a kind of parameters: parameters that are denoted by a name. Parameters that aren't variables are called ''Special Parameters''. I'm sure you'll understand things better with a few examples: |
= Contents = |
| Line 275: | Line 89: |
| {{{ $ # Some parameters that aren't variables: $ echo My shell is $0, and was started with these options: $- My shell is -bash, and was started with these options: himB $ # Some parameters that ARE variables: $ echo I am $LOGNAME, and I live at $HOME. I am lhunath, and I live at /home/lhunath. }}} '''Please note: Unlike PHP/Perl/... parameters do NOT start with a $-sign. The $-sign you see in the examples merely causes the parameter that follows it to be ''expanded''. Expansion basically means that the shell replaces it by its content. As such, `LOGNAME` is the parameter (variable), that contains your username. `$LOGNAME` will be replaced with its content; which in my case, is `lhunath`.''' |
The guide has been divided into sections, which are intended to be read roughly in the order presented. If you skip ahead to a specific section, you might find yourself missing some background information from previous sections. (Links to relevant sections are not always provided when a topic is mentioned.) |
| Line 285: | Line 91: |
| I think you've got the drift now. Here's a summary of most ''Special Parameters'': * '''0''': Contains the name of the script. (This is not always reliable.) * '''Positional Parameters''': 1, 2, ...; They contain the arguments that were passed to the current script. * '''*''': Expands to all the words of all the positional parameters. If double quoted, it expands to a single string containing all the positional parameters separated by the first character of the '''IFS''' variable (which will be discussed later). * '''@''': Expands to all the words of all the positional parameters. If double quoted, it expands to a list of all the positional parameters as individual words. * '''#''': Expands to the number of positional parameters that are currently set. * '''?''': Expands to the exit code of the most recently completed foreground command. * '''$''': Expands to the ["PID"] (process ID number) of the current shell. * '''!''': Expands to the ["PID"] of the application most recently executed in the background. * '''_''': Expands to the last argument of the last command that was executed. And here are some examples of ''Variables'' that the shell initializes for you: * '''BASH_VERSION''': Contains a string describing the version of ["BASH"]. * '''HOSTNAME''': Contains the hostname of your computer, I swear. * '''PPID''': Contains the ["PID"] of the parent process of this shell. * '''PWD''': Contains the current directory. * '''RANDOM''': Each time you expand this variable, a random number between 0 and 32767 is generated. * '''UID''': The ID number of the current user. * '''COLUMNS''': The number of characters that fit on one line in your terminal. (The width of your terminal in characters.) * '''LINES''': The number of lines that fit in your terminal. (The height of your terminal in lines.) * '''HOME''': The current user's home directory. * '''PATH''': A colon-separated list of paths that will be searched to find the executable for a command that is executed, if it is not an alias, function or builtin command (or absolutely referenced). * '''PS1''': Contains a string that describes the format of your shell prompt. * '''TMPDIR''': Contains the directory that is used to store temporary files (by the shell). Of course, you aren't restricted to only these variables. Feel free to define your own: {{{ $ country=Canada $ echo "I am $LOGNAME and I currently live in $country." I am lhunath and I currently live in Canada. }}} Notice what we did to assign the value `Canada` to the variable `country`. Remember that you are '''NOT allowed to have any spaces before or after that equals sign'''! {{{ $ language = PHP -bash: language: command not found $ language=PHP $ echo "I'm far too used to $language." I'm far too used to PHP. }}} Remember that ["BASH"] is not Perl or PHP. You need to be very well aware of how ''expansion'' works to avoid '''big''' trouble. If you don't, you'll end up creating very dangerous situations in your scripts, especially when making this mistake with `rm`: {{{ $ ls no secret secret $ file='no secret' $ rm $file rm: cannot remove `no': No such file or directory }}} Imagine we have two files, `no secret` and `secret`. The first contains nothing useful, but the second contains the secret that will save the world from impending doom. Unthoughtful as you are, you forgot to '''quote''' your parameter expansion of `file`. ["BASH"] expands the parameter and the result is `rm no secret`. ["BASH"] splits the arguments up by their whitespace as it normally does, and `rm` is passed two arguments; 'no' and 'secret'. As a result, it fails to find the file `no` and it deletes the file `secret`. ''The secret is lost!'' -------- . '''Good Practice: [[BR]] You should always keep parameter expansions well quoted. This prevents the whitespace or the possible globs inside of them from giving you gray hair or unexpectedly wiping stuff off your computer. The only good PE, is a quoted PE.''' ---- . '''In The Manual: [http://www.gnu.org/software/bash/manual/bashref.html#SEC23 Shell Parameters], [http://www.gnu.org/software/bash/manual/bashref.html#SEC60 Shell Variables]''' ---- . '''In the FAQ: [[BR]] [http://wooledge.org/mywiki/BashFAQ/013 How can I concatenate two variables? How do I append a string to a variable?] [[BR]] [http://wooledge.org/mywiki/BashFAQ/025 How can I access positional parameters after $9?]''' ---- . ''Variable'': A variable is a kind of parameter that you can create and modify directly. It is denoted by a name, which must begin with a letter, and must consist only of letters, digits, and the underscore (`_`). Variable names are case-sensitive. [[BR]] ''Expansion'': Expansion happens when a parameter is prefixed by a dollar sign. ["BASH"] takes the parameter's value and replaces the parameter's expansion by its value before executing the command. -------- [[Anchor(Variables)]] === Variable Types === Although ["BASH"] is not a typed language, it does have a few different types of variables. These types define the kind of content they have. They are stored in the variable's attributes. ''Attributes'' are settings for a variable. They define the way the variable will behave. Here are the attributes you can assign to a variable: * '''Array''': (`declare -a [variable]`): The variable is an array of strings. * '''Integer''': (`declare -i [variable]`): The variable holds an integer. Assigning values to this variable automatically triggers ''Arithmetic Evaluation''. * '''Read Only''': (`declare -r [variable]`): The variable can no longer be modified or unset. * '''Export''': (`declare -x [variable]`): The variable is marked for export which means it will be inherited by any subshell. Arrays are basically lists of strings. They are very convenient for their ability to store different elements without relying on a delimiter. That way, you don't need to worry about the fact that this delimiter could possibly end up being part of an element's content and thus split that element up: {{{ $ files='one:two:three:four' }}} Here we try to use a string to contain a list of files. To do that, we need to rely on a delimiter to keep the files apart. We choose '`:`'. As a result, we cannot add any files to the list that have a '`:`' in their filename. That's why arrays are so convenient: {{{ $ files=( 'one' 'two' 'three' 'four' '5: five' ) }}} As shown above, you can assign arrays using `(...)`. In this case, elements are separated by whitespace; but you can protect an element's whitespace with quotes. If you want to use some form of expansion to assign values to an array, rather than literal, be aware that ["BASH"] will obviously need to perform some form of word splitting to figure out which parts of your expansion should be put in which elements of the array: {{{ $ files='one:two:three:four' $ IFS=: $ files=( $files ) }}} For this word splitting, ["BASH"] looks at the first character in `IFS` again. There, it finds the delimiter to use for splitting the result of the expansion up in elements. Defining variables as integers has the advantage that you can leave out some syntax when trying to assign or modify them: {{{ $ a=5; a+=2; echo $a; unset a 52 $ a=5; let a+=2; echo $a; unset a 7 $ declare -i a=5; a+=2; echo $a; unset a 7 $ a=5+2; echo $a; unset a 5+2 $ declare -i a=5+2; echo $a; unset a 7 }}} -------- . ''String'': A string is a sequence of characters. [[BR]] ''Array'': An array is a list of strings that does not use a delimiter to separate them. [[BR]] ''Integer'': An integer is a form of data that can only contain digits. [[BR]] ''Read Only'': Parameters that are read-only cannot be modified or unset. [[BR]] ''Export'': Variables that are marked for export will be inherited by any subshell. -------- [[Anchor(Parameter_Expansion)]] === Parameter Expansion === ''Parameter Expansion'' is the term that refers to any operation that causes a parameter to be expanded (replaced by content). In its most basic appearance, the parameter expansion of a parameter is achieved by prefixing that parameter with a `$` sign. In certain situations, additional curly brackets around the parameter's name are required: {{{ $ echo "'$USER', '$USERs', '${USER}s'" 'lhunath', '', 'lhunaths' }}} This example illustrates what basic parameter expansions look like. The second PE results in an empty string. That's because the parameter `USERs` is empty. We did not intend to have the `s` be part of the parameter name. Since there's no way for ["BASH"] to determine whether you want the `s` appended to the name of the parameter or its value you need to use curly brackets to mark the beginning and end of the parameter name. That's what we do in the third PE in our example above. ''Parameter Expansion'' can also be used to modify the string that will be expanded. These operations are terribly convenient: {{{ $ for file in *.JPG *.jpeg > do mv "$file" "${file%.*}.jpg" > done }}} The code above can be used to rename all `JPEG` files with a `JPG` or a `jpeg` extension to have a normal `jpg` extension. The PE (`${file%.*}`) cuts off everything from the end until it finds a period (`.`). Then, in the same quotes, a new extension is appended to the expansion result. Here's a summary of most PEs that are available: * '''${parameter:-word}''': Use Default Values. If '`parameter`' is unset or null, the expansion of '`word`' is substituted. Otherwise, the value of '`parameter`' is substituted. * '''${parameter:=word}''': Assign Default Values. If '`parameter`' is unset or null, the expansion of '`word`' is assigned to '`parameter`'. The value of '`parameter`' is then substituted. * '''${parameter:?word}''': Display Error if Null or Unset. If '`parameter`' is null or unset, the expansion of '`word`' (or a message to that effect if '`word`' is not present) is written to the standard error and the shell, if it is not interactive, exits. Otherwise, the value of '`parameter`' is substituted. * '''${parameter:+word}''': Use Alternate Value. If '`parameter'` is null or unset, nothing is substituted, otherwise the expansion of '`word`' is substituted. * '''${parameter:offset:length}''' Substring Expansion. Expands to up to '`length`' characters of '`parameter`' starting at the character specified by '`offset`'. If '`length`' is omitted, expands to the substring of '`parameter`' starting at the character specified by '`offset`'. * '''${#parameter}''': The length in characters of the value of '`parameter`' is substituted. * '''${parameter#pattern}''': The '`pattern`' is anchored to the beginning of '`parameter`'. The result of the expansion is the expanded value of '`parameter`' with the shortest match deleted. * '''${parameter##pattern}''': The '`pattern`' is anchored to the beginning of '`parameter`'. The result of the expansion is the expanded value of '`parameter`' with the longest match deleted. * '''${parameter%pattern}''': The '`pattern`' is anchored to the end of '`parameter`'. The result of the expansion is the expanded value of '`parameter`' with the shortest match deleted. * '''${parameter%%pattern}''': The '`pattern`' is anchored to the end of '`parameter`'. The result of the expansion is the expanded value of '`parameter`' with the longest match deleted. * '''${parameter/pattern/string}''': The '`pattern`' is not anchored but evaluated from left to right in the value of '`parameter`'. The result of the expansion is the expanded value of '`parameter`' with the first match of '`pattern`' replaced by '`string`'. * '''${parameter//pattern/string}''': As above, but every match of '`pattern`' is replaced. You will learn them all through experience. They will come in handy far more often than you think they might. Here's a few examples to kickstart you: {{{ $ file="$HOME/.secrets/007"; \ > echo "File location: $file"; \ > echo "Filename: ${file##*/}"; \ > echo "Directory of file: ${file%/*}"; \ > echo "Non-secret file: ${file/secrets/not_secret}"; \ > echo; \ > echo "Other file location: ${other:-There is no other file}"; \ > echo "Using file if there is no other file: ${other:=$file}"; \ > echo "Other filename: ${other##*/}"; \ > echo "Other file location length: ${#other}" File location: /home/lhunath/.secrets/007 Filename: 007 Directory of file: /home/lhunath/.secrets Non-secret file: /home/lhunath/.not_secret/007 Other file location: There is no other file Using file if there is no other file: /home/lhunath/.secrets/007 Other filename: 007 Other file location length: 26 }}} Remember the difference between `${v#p}` and `${v##p}`. The doubling of the `#` character means metacharacters will become greedy. The same goes for `%`: {{{ $ version=1.5.9; echo "MAJOR: ${version%%.*}, MINOR: ${version#*.}." MAJOR: 1, MINOR: 5.9. $ echo "Dash: ${version/./-}, Dashes: ${version//./-}." Dash: 1-5.9, Dashes: 1-5-9. }}} '''Note: You cannot nest PEs together. If you need to execute multiple PEs on a parameter, you will need to use multiple statements:''' {{{ $ file=$HOME/image.jpg; file=${file##*/}; echo "${file%.*}" image }}} -------- . '''Good Practice: [[BR]] You may be tempted to use external applications such as `sed`, `awk`, `cut`, `perl` or others to modify your strings. Be aware that all of these require an extra process to be started, which in some cases can cause slowdowns. Parameter Expansions are the perfect alternative.''' ---- . '''In The Manual: [http://www.gnu.org/software/bash/manual/bashref.html#SEC29 Shell Parameter Expansion]''' ---- . '''In the FAQ: [[BR]] [http://wooledge.org/mywiki/BashFAQ/007 Is there a function to return the length of a string?] [[BR]] [http://wooledge.org/mywiki/BashFAQ/073 How can I use parameter expansion? How can I get substrings? How can I get a file without its extension, or get just a file's extension?] [[BR]] [http://wooledge.org/mywiki/BashFAQ/074 How do I get the effects of those nifty Bash Parameter Expansions in older shells?]''' ---- . ''Parameter Expansion'': Any expansion (see earlier definition) of a parameter. Certain operations are possible during this expansion that are performed on the value that will be expanded. -------- [[Anchor(Patterns)]] == Patterns == Patterns are strings that are used to match a whole range of strings. They have a special format depending on the pattern dialect which describes the kinds of strings that they match. ''Regular Expression'' patterns can even be used to grab certain pieces out of the strings they match. On the command line you will mostly use ''Glob Patterns''. They are a fairly straight-forward form of patterns that can easily be used to match a range of files. Since version `3.0`, ["BASH"] also supports ''Regular Expression'' patterns. These will be useful mainly in scripts to test user input or parse data. -------- . ''Pattern'': A pattern is a string with a special format designed to be a sort of key that matches several other strings of a kind. -------- [[Anchor(Globs)]] === Glob Patterns === Globs are a very important concept in ["BASH"], if only for their incredible convenience. Properly understanding globs will benefit you in many ways. Globs are basically patterns that can be used to match filenames or other strings. Globs are composed of normal characters and meta characters. Meta characters are characters that have a special meaning. These are the basic meta characters: * '''*''': Matches any string, including the null string. * '''?''': Matches any single character. * '''[...]''': Matches any one of the enclosed characters. Here's an example of how we can use glob patterns to expand to filenames: {{{ $ ls a abc b c $ echo * a abc b c $ echo a* a abc }}} ["BASH"] sees the glob, for example `a*`. It ''expands'' this glob, by looking in the current directory and matching it against all files there. Any filenames that match the glob, are enumerated and used in place of the glob. As a result, the statement `echo a*` is replaced by the statement `echo a abc`, and is then executed. ["BASH"] will always make sure that whitespace and special characters are escaped properly when expanding the glob. For example: {{{ $ touch "a b.txt" $ ls a b.txt $ rm * $ ls }}} Here, `rm *` is expanded into `rm a\ b.txt`. This makes sure that the string `a b.txt` is passed as a single argument to `rm`, since it represents a single file. It is important to understand that using globs to enumerate files is nearly '''always''' a better idea than using `ls` for that purpose. Here's an example with some more complex syntax which we will cover later on, but it will illustrate the problem very well: {{{ $ ls a b.txt $ for file in `ls`; do rm "$file"; done rm: cannot remove `a': No such file or directory rm: cannot remove `b.txt': No such file or directory $ for file in *; do rm "$file"; done $ ls }}} Here we use the `for` command to go through the output of the `ls` command. The `ls` command results in a string `a b.txt`. The `for` command splits that string into arguments over which it iterates. As a result, for iterates over `a` and `b.txt`. Naturally, this is '''not''' what we want. The glob however expands in the proper form. It results in the string `a\ b.txt`, which `for` takes as a single argument. ["BASH"] also supports a feature called `Extended Globs`. These globs are more powerful in nature. This feature is turned off by default, but can be turned on with the `shopt` command, which is used to toggle '''sh'''ell '''opt'''ions: {{{ $ shopt -s extglob }}} * '''?(list)''': Matches zero or one occurrence of the given patterns. * '''*(list)''': Matches zero or more occurrences of the given patterns. * '''+(list)''': Matches one or more occurrences of the given patterns. * '''@(list)''': Matches one of the given patterns. * '''!(list)''': Matches anything except one of the given patterns. The list inside the parentheses is a list of globs separated by the `|` character. Here's an example: {{{ $ ls names.txt tokyo.jpg california.bmp $ echo !(*jpg|*bmp) names.txt }}} Our glob now expands to anything that does not match the `*jpg` or the `*bmp` pattern. Only the text file passes for that, so it is expanded. Then, there is ''Brace Expansion''. Brace Expansion technically does not fit in the category of Globs, but it is similar. Globs only expand to actual filenames, where brace expansion will expand to any permutation of the pattern. Here's how they work: {{{ $ echo th{e,a}n then than $ echo {/home/*,/root}/.*profile /home/axxo/.bash_profile /home/lhunath/.profile /root/.bash_profile /root/.profile $ echo {1..9} 1 2 3 4 5 6 7 8 9 $ echo {0,1}{0..9} 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 }}} -------- . '''Good Practice: [[BR]] You should always use globs instead of `ls` (or similar) to enumerate files. Globs will always expand safely and minimize the risk for bugs. [[BR]] You can sometimes end up with some very weird filenames. Generally speaking, scripts aren't always tested against all the odd cases that they may end up being used with.''' ---- . '''In The Manual: [http://www.gnu.org/software/bash/manual/bashref.html#SEC35 Pattern Matching]''' ---- . '''In the FAQ: [[BR]] [http://wooledge.org/mywiki/BashFAQ/016 How can I use a logical AND in a shell pattern (glob)?]''' ---- . ''Glob'': A glob is a string composed of glob meta characters that can match certain strings or filenames. -------- [[Anchor(Regex)]] === Regular Expressions === ''Regular Expressions'' (regex) are similar to ''Glob Patterns'' but cannot be used for filename matching in ["BASH"]. Since `3.0` ["BASH"] supports the `=~` operator to the `[[` built-in. This operator matches the string that comes before it against the regex pattern that follows it. When the string matches the pattern, `[[` returns with an exit code of `0` ("true"). If the string does not match the pattern, an exit code of `1` ("false") is returned. In case the pattern's syntax is invalid, `[[` will abort the operation and return an exit code of `2`. ["BASH"] uses the ''Extended Regular Expression'' (`ERE`) dialect. I will not teach you about regex in this guide, but if you are interested in this concept, please read up on [http://www.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap09.html#tag_09_04 Extended Regular Expressions] or Google for a tutorial. ''Regular Expression'' patterns that use capturing groups will have their captured strings assigned to the `BASH_REMATCH` variable for later retrieval. Let's illustrate how regex can be used in ["BASH"]: {{{ $ if [[ $LANG =~ (..)_(..) ]] > then echo "You live in ${BASH_REMATCH[2]} and speak ${BASH_REMATCH[1]}." > else echo "Your locale was not recognised" > fi }}} Be aware that regex parsing in ["BASH"] has changed between releases `3.1` and `3.2`. Before `3.2` it was safe to wrap your regex pattern in quotes but this has changed in `3.2`. Since then, regex should always be unquoted. You should protect any special characters by escaping it using a backslash. {{{ $ [[ "My sentence" =~ My\ sentence ]] }}} Be careful to escape any characters that the shell could misinterpret, such as whitespace, dollar signs followed by text, braces, etc. -------- . '''Good Practice: [[BR]] Since the way regex is used in `3.2` is also valid in `3.1` we ''highly'' recommend you just never quote your regex. Remember to keep special characters properly escaped!''' ---- . '''In The Manual: [http://www.daemon-systems.org/man/regex.3.html Regex(3)]''' ---- . '''In the FAQ: [[BR]] [http://wooledge.org/mywiki/BashFAQ/066 I want to check if [[ $var == foo || $var == bar || $var == more ... without repeating $var n times.]''' ---- . ''Regular Expression'': A regular expression is a more complex pattern that can be used to match specific strings (but unlike globs cannot expand to filenames). -------- [[Anchor(Conditionals)]] == Tests and Conditionals == Sequential execution of applications is one thing, but to achieve a sort of logic in your scripts or your command line one-liners, you'll need variables and conditionals. Conditionals are used to make decisions which determine the execution flow of a script. -------- [[Anchor(Exit_Status)]] === Exit Status === Every application results in an exit code whenever it terminates. This exit code is used by whatever application started it to evaluate whether everything went OK. This exit code is like a return value from functions. It's an integer between 0 and 255 (inclusive). Convention dictates that we use 0 to denote success, and any other number to denote failure of some sort. The specific number is entirely application-specific, and is used to hint as to what exactly went wrong. For example, the `ping` command sends ICMP packets over the network to a certain host. That host normally responds to this packet by sending the exact same one right back. This way, we can check whether we can communicate with a remote host. `ping` has a range of exit codes which can tell us what went wrong, if anything did: '''From the `ping` manual:''' . '''If ping does not receive any reply packets at all it will exit with code 1. If a packet count and deadline are both specified, and fewer than count packets are received by the time the deadline has arrived, it will also exit with code 1. On other error it exits with code 2. Otherwise it exits with code 0. This makes it possible to use the exit code to see if a host is alive or not.''' The parameter `?` shows us the exit code of the last foreground process that terminated. Let's play around a little with `ping` to see its exit codes: {{{ $ ping God ping: unknown host God $ echo $? 2 $ ping -c 1 -W 1 1.1.1.1 PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data. --- 1.1.1.1 ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0ms $ echo $? 1 }}} -------- . '''Good Practice: [[BR]] You should make sure that your scripts always return a non-zero exit code if something unexpected happened in their execution. You can do this with the `exit` builtin:''' {{{ rm file || { echo "Could not delete file!"; exit 1; } }}} ---- . '''In The Manual: [http://www.gnu.org/software/bash/manual/bashref.html#SEC52 Exit Status]''' ---- . ''Exit Code'' / ''Exit Status'': Whenever a command ends it notifies its parent (which in our case will always be the shell that started it) of its exit status. This is represented by a number ranging from 0 to 255. This code is a hint as to the success of the command's execution. -------- [[Anchor(Control_Operators)]] === Control Operators === Now that we know what exit codes are, and that an exit code of '0' means the command's execution was successful, we'll learn to use this information. The easiest way of performing a certain action depending on the success of a previous command is through the use of 'control operators'. These operators are `&&` and `||`, which respectively represent a logical AND and a logical OR. These operators are used between two commands, and they are used to control whether the second command should be executed depending on the success of the first. Let's put that theory in practice: {{{ $ mkdir d && cd d }}} This simple example has two commands, `mkdir d` and `cd d`. You could easily just use a semi-colon there to separate both commands and execute them sequentially; but we want something more. In the above example, ["BASH"] will execute `mkdir d`, then `&&` will check the result of the `mkdir` application as it finishes. If the `mkdir` application resulted in a success (exit code 0), then `&&` will execute the next command, `cd d`. If `mkdir d` failed, and returned a non-0 exit code, `&&` will skip the next command, and we will stay in the current directory. {{{ $ echo "$PWD" && cd - }}} Another example: {{{ $ rm /etc/some_file.conf || echo "I couldn't remove the file!" rm: cannot remove `/etc/some_file.conf': No such file or directory I couldn't remove the file! }}} `||` is much like `&&`, but it does the exact opposite. It only executes the next command if the first '''failed'''. As such, the message is only echoed if the `rm` command was unsuccessful. You can make a sequence with these operators, but you have to be very careful when you do. Remember what exit code the operator is '''really''' going to be checking against! Here's an example that might cause confusion: {{{ $ false && true || echo "Riddle, riddle?" Riddle, riddle? $ true && false || echo "Riddle, riddle?" Riddle, riddle? }}} `true` is obviously always going to be successful. `false` is obviously always going to be unsuccessful. Can you guess why the `echo` statement is executed in both occasions? The key to understanding how to sequence these operators properly is by evaluating exit codes from left to right. In the first example, `false` is unsuccessful, so `&&` does not execute the next command (which is `true`), but the next `||` gets a shot too. `||` still sees that the last exit code was that from `false`, and `||` executes the next command when the previous was unsuccessful. As a result, the `echo` statement is executed. The same for the second statement again. `true` is successful, so the `&&` executes the next statement. That is `false`; the last exit code now becomes unsuccessful. After that, `||` is evaluated, which sees the unsuccessful exit code from `false` and executes the `echo` statement. It's all easy with `true`s and `false`s; but how about real commands? {{{ $ rm file && touch file || echo "File not found!" }}} All seems well with this piece of code, and when you test it, I'm sure you'll see that it actually does what it's supposed to. It tries to delete a file, and if it succeeds, it creates it again as a new and empty file; if something goes wrong we get the error message. What's the catch? Perhaps you guessed, perhaps not, but here's a hint: Imagine the file system fills up (or some other disaster occurs) in between `rm` and the `touch`; the `rm` will succeed in deleting our file, but `touch` will fail to create it anew. As a result, we get a strange error message saying that the file wasn't found while we were actually trying to '''create''' it. What's up with that? The next section will show a safer way to put both "AND" and "OR" conditions after a command. -------- . '''Good Practice: [[BR]] It's best not to get overcourageous when dealing with conditional operators. They can make your script hard to understand, especially for a person that's assigned to maintain it and didn't write it himself.''' ---- . '''In The Manual: [http://www.gnu.org/software/bash/manual/bashref.html#SEC18 List of Commands]''' ---- . ''Control Operators'': These operators are used to link commands together. They check the exit code of the command that was previously ran to determine whether or not to execute the next command in the sequence. -------- [[Anchor(Conditionals)]] === Conditional Blocks === `if` is a shell builtin that executes a command, and checks that command's exit code to see whether its execution was successful. Depending on that exit code, `if` executes a specific block of code. {{{ $ if true > then echo "It was true." > else echo "It was false!" > fi It was true. }}} Here you see the basic outline of an ''if-statement''. We start by calling `if` with the argument `true`. `true` is a built-in command that always ends successfully. `if` runs that command, and once the command is done, `if` checks the exit code. Since `true` always exits successfully, `if` continues to the `then`-block, and executes that code. Should the `true` application have failed somehow, and returned an unsuccessful exit code, the `if` statement would have skipped the `then` code, and executed the `else` code block instead. There are commands that can help us a lot in doing conditional checks. They are `[` (also named `test`) and `[[`. `[` is a normal application that reads its arguments and does some checks with them. `[[` is much like `[`, but it's special, and it offers far more versatility. Let's get practical: {{{ $ if [ a = b ] > then echo "a is the same as b." > else echo "a is not the same as b." > fi a is not the same as b. }}} `if` executes the command `[` with the arguments '`a`', '`=`', '`b`' and '`]`'. `[` uses these arguments to determine what must be checked. In this case, it checks whether the string 'a' is identical to the string 'b', and if this is the case, it will exit successfully. However, since we know this is not the case, `[` will not exit successfully (its exit code will be 1). `if` sees that `[` terminated unsuccessfully and executes the code in the `else` block. Now, to see why `[[` is so much more interesting and trustworthy than `[`, let us highlight some possible problems with `[`: {{{ $ if [ my dad = my dog ] > then echo "I have a problem." > fi -bash: [: too many arguments }}} Can you guess what caused the problem? [[BR]] `[` was executed with the arguments '`my`', '`dad`', '`=`', '`my`', '`dog`' and '`]`'. `[` doesn't understand what test it's supposed to execute, because it expects the second argument to be the operator. In our case, the operator is the third argument. Yet another reason why '''quotes''' are so terribly important. Whenever we type whitespace in bash that belongs together with the words before or after it, '''we need to quote the whole string''': {{{ $ if [ 'my dad' = 'my dog' ] > then echo "I have a problem." > fi }}} This time, `[` sees an operator (`=`) in the second argument and it can continue with its work. Now, this may be easy to see and avoid, but it gets just a little trickier when we put the strings in variables, rather than literally in the statement: {{{ $ dad='my dad'; dog='my dog' $ if [ $dad = $dog ] > then echo "I have a problem." > fi -bash: [: too many arguments }}} How did we mess up this time? [[BR]] Here's a hint: ["BASH"] takes our ''if-statement'' and expands all the parameters in it. The result is `if [ my dad = my dog ]`. Boom, game over. Here's how it's supposed to look like: {{{ $ if [ "$dad" = "$dog" ] > then echo "I have a problem." > fi }}} To help us out a little, ["BASH"] introduced a new style of conditional test. Original as the ["BASH"] authors are, they called it `[[`. `[[` is loaded with several very interesting features which are missing from `[`. One of them helps us in dealing with parameter expansions: {{{ $ if [[ $dad = $dog ]] > then echo "I have a problem." > fi $ if [[ I want $dad = I want $dog ]] > then echo "I want too much." > fi -bash: conditional binary operator expected -bash: syntax error near `want' }}} This time, `$dad` and `$dog` did not need quotes. Since `[[` isn't an application (while `[` is), but a ''shell keyword'', it has special magical powers. It parses its arguments before they are expanded by bash and does the expansion itself, taking the result as a single argument, even if that result contains whitespace. ''However'', be aware that simple strings still have to be quoted properly. `[[` can't know whether your literal whitespace in the statement is intentional or not; so it splits it up just like ["BASH"] normally would. Let's fix our last example: {{{ $ if [[ "I want $dad" = "I want $dog" ]] > then echo "I want too much." > fi }}} You could also combine several `if` statements into one using `elif` instead of `else`; where each test indicates another possibility: {{{ $ name=lhunath $ if [[ $name = "George" ]] > then echo "Bonjour, $name!" > elif [[ $name = "Hans" ]] > then echo "Goeie dag, $name!" > elif [[ $name = "Jack" ]] > then echo "Good day, $name!" > else > echo "You're neither George, Hans or Jack. Who the hell are you, $name?" > fi }}} Now that you've got a decent understanding of quoting issues that may arise, let's have a look at some of the other features that `[` and `[[` were blessed with: * Tests supported by `[` (also known as `test`): * '''-e FILE''': True if file exists. * '''-f FILE''': True if file is a regular file. * '''-d FILE''': True if file is a directory. * '''-h FILE''': True if file is a symbolic link. * '''-r FILE''': True if file is readable by you. * '''-s FILE''': True if file exists and is not empty. * '''-t FD ''': True if FD is opened on a terminal. * '''-w FILE''': True if the file is writable by you. * '''-x FILE''': True if the file is executable by you. * '''-O FILE''': True if the file is effectively owned by you. * '''-G FILE''': True if the file is effectively owned by your group. * '''FILE -nt FILE''': True if the first file is newer than the second. * '''FILE -ot FILE''': True if the first file is older than the second. * '''-z STRING''': True if the string is empty (it's length is zero). * '''-n STRING''': True if the string is not empty (it's length is not zero). * '''STRING = STRING''': True if the first string is identical to the second. * '''STRING != STRING''': True if the first string is not identical to the second. * '''STRING < STRING''': True if the first string sorts before the second. * '''STRING > STRING''': True if the first string sorts after the second. * '''EXPR -a EXPR''': True if both expressions are true (logical AND). * '''EXPR -o EXPR''': True if either expression is true (logical OR). * '''! EXPR''': Inverts the result of the expression (logical NOT). * '''INT -eq INT''': True if both integers are identical. * '''INT -ne INT''': True if the integers are not identical. * '''INT -lt INT''': True if the first integer is less than the second. * '''INT -gt INT''': True if the first integer is greater than the second. * '''INT -le INT''': True if the first integer is less than or equal to the second. * '''INT -ge INT''': True if the first integer is greater than or equal to the second. * Additional tests supported only by `[[`: * '''STRING = (or ==) PATTERN''': Not string comparison like with `[` (or `test`), but ''pattern matching'' is performed. True if the string matches the glob pattern. * '''STRING =~ REGEX''': True if the string matches the regex pattern. * '''( EXPR )''': Parantheses can be used to change the evaluation precedence. * '''EXPR && EXPR''': Much like the '-a' operator of `test`, but does not evaluate the second expression if the first already turns out to be false. * '''EXPR || EXPR''': Much like the '-o' operator of `test`, but does not evaluate the second expression if the first already turns out to be true. You want some examples? Sure: {{{ $ test -e /etc/X11/xorg.conf && echo "Your Xorg is configured!" Your Xorg is configured! $ test -n "$HOME" && echo "Your homedir is set!" Your homedir is set! $ [[ boar != bear ]] && echo "Boars aren't bears!" Boars aren't bears! $ [[ boar != b?ar ]] && echo "Boars don't look like bears!" $ [[ $DISPLAY ]] && echo "Your DISPLAY variable is not empty, you probably have Xorg running." Your DISPLAY variable is not empty, you probably have Xorg running. $ [[ ! $DISPLAY ]] && echo "Your DISPLAY variable is not not empty, you probably don't have Xorg running." }}} -------- . '''Good Practice: [[BR]] Whenever you're making a ["BASH"] script, you should always use `[[` (unless if for some reason you need very specific and rare functionality from `[`). [[BR]] Whenever you're making a Shell script, which may end up being used in an environment where ["BASH"] is not available, you should use `[`, because it is far more compatible. (In addition to being built in to ["BASH"] and some other shells, `[` should be available as an external application.)''' ---- . '''In The Manual: [http://www.gnu.org/software/bash/manual/bashref.html#SEC20 Conditional Constructs]''' ---- . '''In the FAQ: [[BR]] [http://wooledge.org/mywiki/BashFAQ/017 How can I group expressions, e.g. (A AND B) OR C?] [[BR]] [http://wooledge.org/mywiki/BashFAQ/031 What is the difference between the old and new test commands ([ and [[)?] [[BR]] [http://wooledge.org/mywiki/BashFAQ/041 How do I determine whether a variable contains a substring?] [[BR]] [http://wooledge.org/mywiki/BashFAQ/054 How can I tell whether a variable contains a valid number?]''' ---- . ''If (builtin)'': This command executes the command that is given on its arguments and depending on that command's exit code executes the code in the following `then` (or optionally `else`) block. -------- [[Anchor(Loops)]] === Conditional Loops === You've learned how to code some basic logic flow for your scripts. It's important that you understand a thing or two about keeping scripts healthy before we start introducing more complexity. ["BASH"] scripts, much like any other kind of scripts, should never be overrated. Although they have great potential once you fully understand the shell's features, they aren't always the right tool for the job. At the same time, when you make scripts, you should remember to keep them light, both in length and in complexity. Very long and/or very complex scripts are most often also very bad scripts. Those that aren't yet soon will be, because they are always very difficult to maintain and adapt/extend. A technique that we can use to try and keep code length and complexity down is the ''loop''. There are two kinds of loops. Using the correct kind of loop will help you keep your scripts readable and healthy. ["BASH"] supports `while` loops and `for` loops. The `for` loops can appear in three different forms. Here's a summary: * '''`while` ''command''''': Repeat so long as command is executed successfully (exit code: `0`). * '''`until` ''command''''': Repeat so long as command is executed unsuccessfully (exit code: `>0`). * '''`for` ''variable'' `in` ''words''''': Repeat the loop for each word, putting it into the variable. * '''`for ((` ''expression''`;` ''expression''`;` ''expression'' `))`''': Starts by evaluating the first expression, repeats the loop so long as the second expression is valid and at the end of each loop evaluates the third expression. Let's put that in practice; here are some examples to illustrate the differences but also the similarities between the loops: {{{ $ while true > do echo "Infinite loop!" > done $ (( i=10 )); while (( i > 0 )) > do echo "$i empty cans of beer." > (( i-- )) > done $ for (( i=10; i > 0; i-- )) > do echo "$i empty cans of beer." > done $ for i in {10..0} > do echo "$i empty cans of beer." > done }}} The last three loops achieve exactly the same result, using different syntax. You'll encounter this many times in your shell scripting experience. There will nearly always be multiple approaches to solving a problem. The test of your skill soon won't be about solving a problem as much as about how best to solve it. You need to learn to pick the best angle of approach for the job. Usually, the main factors to keep into account will be the simplicity and flexibility of the resulting code. My personal favorite is the last of the examples. In that example I used ''Brace Expansion'' to generate the words; but there are other ways, too. Let's take a closer look at that last example, because although it looks the easier of the two `for`s, it can often be the trickier, if you don't know exactly how it works. As I mentioned before: `for` runs through a list of words and puts each in the variable, one at a time, then loops through the body with it. The tricky part is how ["BASH"] decides what the words are. Let me explain myself by expanding the braces from that previous example: {{{ $ for i in 10 9 8 7 6 5 4 3 2 1 0 > do echo "$i empty cans of beer." > done }}} ["BASH"] takes the characters between `in` and the end of the line, and splits them up into words. You shouldn't confuse the splitting that happens here with the splitting that happens with ''Commandline Arguments'', even though they look exactly the same at first sight. Commandline arguments are split at ''spaces'', ''tabs'' and ''newlines''. The splitting in this `for` statement happens at ''spaces'' by default, but this default behaviour can be changed. The way `for` determines what delimiter to use for the splitting is by the contents of the `IFS` variable. `IFS` is an acronym for ''Internal Field Separator''; and by default it contains a ''space'', a ''tab'' and a ''newline''. Since the ''space'' is one of the characters there, `for` uses it to split up the words in our sequence, and feeds each word to the variable `i`, one at a time. '''As a result; be VERY careful not to make the following mistake:''' {{{ $ ls The best song in the world.mp3 $ for file in $(ls *.mp3) > do rm "$file" > done rm: cannot remove `The': No such file or directory rm: cannot remove `best': No such file or directory rm: cannot remove `song': No such file or directory rm: cannot remove `in': No such file or directory rm: cannot remove `the': No such file or directory rm: cannot remove `world.mp3': No such file or directory }}} You should already know to quote the `$file` in the `rm` statement; but what's going wrong here? ["BASH"] expands the command substitution (`$(ls *.mp3)`), replaces it by its output, and as a result executes `for file in The best song in the world.mp3`. ["BASH"] splits that up into words by using ''spaces'' and tries to `rm` each word. ''Boom, you are dead''. You want to quote it, you say? Let's add another song: {{{ $ ls The best song in the world.mp3 The worst song in the world.mp3 $ for file in "$(ls *.mp3)" > do rm "$file" > done rm: cannot remove `The best song in the world.mp3 The worst song in the world.mp3': No such file or directory }}} Quotes will indeed protect the whitespace in your filenames; but they will do more than that. The quotes will protect '''all the whitespace''' from the output of `ls`. There is no way ["BASH"] can know which parts of the output of `ls` represent filenames; it's not psychic. The output of `ls` is a simple string, and ["BASH"] treats it as such. The `for` puts the whole quoted output in `i` and runs the `rm` command with it. ''Damn, dead again''. So what do we do? As suggested earlier, globs are your best friend: {{{ $ for file in *.mp3 > do rm "$file" > done }}} This time, ["BASH"] '''does''' know that it's dealing with filenames, and it '''does''' know what the filenames are, and as such it can split them up nicely. The result of expanding the glob is this: `for file in "The best song in the world.mp3" "The worst song in the world.mp3"`. Problem resolved. Let's talk about changing that delimiter. Say you've got yourself a nice cooking recipe, and you want to write a script that tells you how to use it. Sure, let's get right at it: {{{ $ recipe='2 c. all purpose flour > 6 tsp. baking powder > 2 eggs > 2 c. milk > 1/3 c. oil' $ for ingredient in $recipe > do echo "Take $ingredient; mix well." > done }}} Can you guess what the result will look like? I recommend you run the code if you can't and ponder the reason first. It will help you understand things. Yes, as explained earlier, `for` splits its stuff up into words by using the delimiters from `IFS`. To read the recipe correctly, we need to split it up by newlines alone, instead of by spaces and newlines. Here's how we do that: {{{ $ recipe='2 c. all purpose flour > 6 tsp. baking powder > 2 eggs > 2 c. milk > 1/3 c. oil' $ IFS=$'\n' $ for ingredient in $recipe > do echo "Take $ingredient; mix well." > done Take 2 c. all purpose flour; mix well. Take 6 tsp. baking powder; mix well. Take 2 eggs; mix well. Take 2 c. milk; mix well. Take 1/3 c. oil; mix well. }}} Beautiful. '''Note: This delimiter is only used when the words consist of an expansion. Not when they're literal. Literal words are always split at spaces:''' {{{ $ PATH=/bin:/usr/bin $ IFS=: $ for i in $PATH > do echo "$i" > done /bin /usr/bin $ for i in $PATH:/usr/local/bin > do echo "$i" > done /bin /usr/bin /usr/local/bin $ for i in /bin:/usr/bin:/usr/local/bin > do echo "$i" > done /bin:/usr/bin:/usr/local/bin }}} Lets focus a little more on the `while` loop. It promises even more simplicity than this `for` loop, so long as you don't need any `for` specific features. The `while` loop is very interesting for its capacity of executing commands and basing the loop's progress on the result of them. Here are a few examples of how `while` loops are very often used: {{{ $ # The sweet machine; hand out sweets for a cute price. $ while read -p $'The sweet machine.\nInsert 20c and enter your name: ' name > do echo "The machine spits out three lollipops at $name." > done $ # Check your email every five minutes. $ while sleep 5m > do kmail --check > done $ # Wait for a host to come back online. $ while ! ping -c 1 -W 1 "$host" > do echo "$host is still unavailable." > done; echo -e "$host is available again!\a" }}} The `until` loop is barely ever used, if only because it is pretty much exactly the same as the `while` loop except for the fact that its command is not checked for an exit status of zero but for an exit status of non-zero. As a result, we could rewrite our last example using an until loop as such: {{{ $ # Wait for a host to come back online. $ until ping -c 1 -W 1 "$host" > do echo "$host is still unavailable." > done; echo -e "$host is available again!\a" }}} -------- . '''In The Manual: [http://www.gnu.org/software/bash/manual/bashref.html#SEC19 Looping Constructs]''' ---- . '''In the FAQ: [[BR]] [http://wooledge.org/mywiki/BashFAQ/015 How can I run a command on all files with the extention .gz?] [[BR]] [http://wooledge.org/mywiki/BashFAQ/018 How can I use numbers with leading zeros in a loop, e.g. 01, 02?] [[BR]] [http://wooledge.org/mywiki/BashFAQ/020 How can I find and deal with file names containing newlines, spaces or both?] [[BR]] [http://wooledge.org/mywiki/BashFAQ/030 How can I rename all my *.foo files to *.bar, or convert spaces to underscores, or convert upper-case file names to lower case?] [[BR]] [http://wooledge.org/mywiki/BashFAQ/034 Can I do a spinner in Bash?] [[BR]] [http://wooledge.org/mywiki/BashFAQ/046 I want to check to see whether a word is in a list (or an element is a member of a set).]''' ---- . ''Loop'': A loop is a structure that is designed to repeat the code within until a certain condition has been fulfilled. At that point, the loop stops and the code beyond it is executed. [[BR]] ''For (builtin)'': A `for`-loop is a type of loop that sets a variable to each of a list of values in turn, and repeats until the list is exhausted. [[BR]] ''While (builtin)'': A `while`-loop is a type of loop that continues to run its code so long as a certain command (run before each iteration) executes successfully. [[BR]] ''Until (builtin)'': An `until`-loop is a type of loop that continues to run its code so long as a certain command (run before each iteration) executes unsuccessfully. -------- [[Anchor(Choices)]] === Choices === Sometimes you just want to built application logic depending on the content of a variable but you already know all of the possible choices of content the variable can contain. You no longer need `if`'s two ''yes or no'' blocks, but you want a block of code for every possible choice of content a variable can have. You could obviously attempt to emulate this behaviour using several `if`-statements: {{{ $ if [[ $LANG = en* ]] > then echo "Hello!"; fi $ if [[ $LANG = fr* ]] > then echo "Salut!"; fi $ if [[ $LANG = de* ]] > then echo "Guten Tag!"; fi $ if [[ $LANG = nl* ]] > then echo "Hallo!"; fi $ if [[ $LANG = it* ]] > then echo "Ciao!"; fi }}} But let's say we like to actually make clean code, shall we? ["BASH"] provides a construct called `case` exactly for this kind of situation. A `case` construct basically enumerates several possible ''Glob Patterns'' and checks the content of your parameter against these: {{{ $ case $LANG in > en*) echo "Hello!" ;; > fr*) echo "Salut!" ;; > de*) echo "Guten Tag!" ;; > nl*) echo "Hallo!" ;; > it*) echo "Ciao!" ;; > *) echo "I do not speak your language." ;; > esac }}} Each choice in a `case` statement starts with a pattern followed by a block of code that is to be executed if the parameter's content matches that pattern, which in turn is followed by two semi-colons. `case` stops matching patterns as soon as one was successful. Therefore, we can use the `*` pattern in the end to match any case that has not been caught by the other choices. Another construct of choice is the `select` construct. This statement smells like a loop and is a convenience statement for generating a menu of choices that the user can choose from. The user is presented by choices and asked to enter a number reflecting his choice. The code in the `select` block is then executed with a variable set to the choice the user made. If the user's choice was invalid, the variable is made empty: {{{ $ echo "Which of these does not belong in the group?"; \ > select choice in Apples Pears Crisps Lemons Kiwis; do > if [[ $choice = Crisps ]] > then echo "Correct! Crisps are not fruit."; break; fi > echo "Errr... no. Try again." > done }}} The menu reappears so long as the `break` statement is not executed. In the example the `break` statement is only executed when the user makes the correct choice. We can also use the `PS3` variable to define the prompt the user replies on. Instead of showing the question before executing the `select` statement, we could choose to set the question as our prompt: {{{ $ PS3="Which of these does not belong in the group (#)? " \ > select choice in Apples Pears Crisps Lemons Kiwis; do > if [[ $choice = Crisps ]] > then echo "Correct! Crisps are not fruit."; break; fi > echo "Errr... no. Try again." > done }}} -------- . '''In The Manual: [http://www.gnu.org/software/bash/manual/bashref.html#SEC20 Conditional Constructs]''' ---- . '''In the FAQ: [[BR]] [http://wooledge.org/mywiki/BashFAQ/066 I want to check if [[ $var == foo || $var == bar || $var = more ... without repeating $var n times.]''' ---- . ''Case (builtin)'': This statement evaluates a parameter's value based on several given patterns (choices). [[BR]] ''Select (builtin)'': The `select` statement offers the user the choice between several options and executes a block of code for the user's choice. -------- [[Anchor(IO)]] == Input And Output == This basic principle of computer science applies just as well to applications started through ["BASH"]. ["BASH"] makes it fairly easy to play around with the input and output of commands, which gives us great flexibility and incredible opportunities for automation. -------- [[Anchor(FDs)]] === File Descriptors === Input and output from and to processes always occurs via so called ''File Descriptors'' (in short: FDs). FDs are kind of like pointers to sources of data. When something reads from or writes to that FD, the data is being read from or written to the FD's data source. FDs can point to regular files, but they can also point to more abstract data sources, like the input and output source of a process. By default, every new process has three FDs. They are referred to by the names ''Standard Input'', ''Standard Output'' and ''Standard Error''. In short, they are respectively called `stdin`, `stdout` and `stderr`. The ''Standard Input'' is where the characters you type on your keyboard usually come from. The ''Standard Output'' is where the program sends most of its normal information to so that the user can see it, and the ''Standard Error'' is where the program sends its error messages to. Be aware that GUI applications work in the same way; but the actual GUI doesn't work via these FDs. GUI applications can still read and write from and to the standard FDs, but they usually don't. Usually, they do all the user interaction via that GUI; making it hard to control for ["BASH"]. As a result, we'll stick to simple console applications. Those we can easily feed data on the "Standard Input" and read data from on its "Standard Output" and "Standard Error". Let's make these definitions a little more concrete. Here's a demonstration of how "Standard Input" and "Standard Output" work: {{{ $ read -p "What is your name? " name; echo "Good day, $name. Would you like some tea?" What is your name? lhunath Good day, lhunath. Would you like some tea? }}} `read` is a command that reads information from `stdin` and stores it in a variable. We specified `name` to be that variable. Once `read` has read a line of information from `stdin`, it finished and lets `echo` display a message. `echo` uses `stdout` to send its output to. `stdin` is connected to your terminal's input device; which is probably going to be your keyboard. `stdout` is connected to your terminal's output device; which I assume is a computer monitor. As a result; you can type in your name and are then greeted with a friendly message on your monitor, offering you a cup of tea. So what is `stderr`? Let's demonstrate: {{{ $ rm secrets rm: cannot remove `secrets': No such file or directory }}} Unless if you had a file called `secrets` in your current directory; that `rm` command will fail and show an error message explaining what went wrong. Error messages like these are by convention displayed on `stderr`. `stderr` is also connected to your terminal's output device, just like `stdout`. As a result, error messages display on your monitor just like the messages on `stdout`. However, this separation makes it easy to keep errors separated from the application's normal messages. Some people like to use wrappers to make all the output on `stderr` red, so that they can see the error messages more clearly. This is not generally advisable, but it is a simple example of the many options this separation provides us with. -------- . '''Good Practice: [[BR]] Remember that when you create scripts, you should send your custom error messages to the `stderr` FD. This is a convention and it is very convenient when applications follow the convention. As such, so should you! You're about to learn redirection soon, but let me show you quickly how it's done:''' {{{ echo "Uh oh. Something went really bad.." >&2 }}} ---- . ''File Descriptor'': Basically this is just a "description" of where to read input from and where to send output to. Each command has at least three basic descriptors: `FD 0` is `stdin`, `FD 1` is `stdout` and `FD 2` is `stderr`. -------- [[Anchor(Redirection)]] === Redirection === The most basic form of input/output manipulation in ["BASH"] is ''Redirection''. ''Redirection'' is used to change the data source or destination of an application's FDs. That way, you can send the application's output to a file instead of the terminal, or have the application read from a file instead of from the keyboard. Redirection, too, comes in different shapes. There's ''File Redirection'', ''File Descriptor manipulation'', ''Heredocs'' and ''Herestrings''. -------- . '''In The Manual: [http://www.gnu.org/software/bash/manual/bashref.html#SEC37 Redirections]''' ---- . ''Redirection'': This is the practice of changing a certain FD to read its input from or send its output to elsewhere. -------- [[Anchor(File_Redirection)]] ==== File Redirection ==== ''File Redirection'' is probably the most basic form of redirection. I'll start with this so you can grasp the concept of redirection well. {{{ $ echo "The story of William Tell. > > It was a cold december night. Too cold to write." > story $ cat story The story of William Tell. It was a cold december night. Too cold to write. }}} As a result; the `echo` command will not send its output to the terminal, but the `> story` operation '''changes the destination of the `stdout` FD''' so that it now points to a file called `story`. Be aware that before the `echo` command is executed, ["BASH"] first checks to see whether that file `story` actually exists. If it doesn't, it is created as an empty file, so that the FD can be pointed to it. This behaviour can be toggled with ''Shell Options'' (see later). We then use the application `cat` to print out the contents of that file. `cat` is an application that reads the contents of all the files you pass it as arguments. It then outputs each file one after another on `stdout`. In essence, it con'''cat'''enates the contents of all the files you pass it as arguments. '''Warning:''' Far too many code examples and shell tutorials on the Internet tell you to use `cat` whenever you need to read the contents of a file. '''This is highly ill-adviced!''' `cat` only serves well to contatenate contents of multiple files together, or as a quick tool on the shell prompt to see what's inside a file. You should '''NOT''' use `cat` to read from files in your scripts. There will almost always be far better ways to do this. Please keep this warning in mind. Useless usage of `cat` will merely result in an extra process to create, and often results in poorer read speed because `cat` cannot determine the context of what it's reading and the purpose for that data. When we use `cat` without passing any kind of arguments, it obviously doesn't know what files to read the content for. In this case, `cat` will just read from `stdin` instead of from a file (much like `read`). Since `stdin` is normally not a regular file, starting `cat` without any arguments will seem to do nothing: {{{ $ cat }}} It doesn't even give you back your shell prompt! What's going on? `cat` is still reading from `stdin`, which is your keyboard. Anything you type now will be sent to `cat`. As soon as you hit the ''Enter'' key, `cat` will do what it normally does; it will display what it reads on `stdout`, just the same way as when it displayed our story on `stdout`: {{{ $ cat test? test? }}} Why does it say `test?` twice now? Well, as you type, your terminal shows you all the characters that you send to `stdin` before sending them there. That results in the first `test?` that you see. As soon as you hit ''Enter'', `cat` has read a line from `stdin`, and shows it on `stdout`, which is also your terminal; hence, resulting in the second line: `test?`. You can press ''Ctrl+D'' to send `cat` the ''End of File'' character. That'll cause `cat` to think the file `stdin` has closed. It will stop reading from it and return you to your prompt. Let's use file redirection to attach a file to `stdin`, so that `stdin` is no longer reading from our keyboard, but instead, now reads from the file: {{{ $ cat < story The story of William Tell. It was a cold december night. Too cold to write. }}} The result of this is exactly the same as the result from our previous `cat story`; except this time, the way it works is a little different. In our first example, `cat` opened an FD to the file `story` and read its contents through that FD. In this recent example, `cat` simply reads from `stdin`, just like it did when it was reading from our keyboard. However, this time, the `< story` operation has '''modified''' `stdin` so that its data source is the file `story` rather than our keyboard. Let's summarize: * '''`command > file`''': Send the `stdout` of command to `file`. * '''`command < file`''': Use the contents of `file` when `command` reads from `stdin`. * '''`command 1> file`''': Send the `stdout` of command to `file`. * '''`command <0 file`''': Use the contents of `file` when `command` reads from `stdin`. Redirection operators can take a number. That number denotes the FD that it changes. If the number is not present, the `>` operator uses FD 1 by default, because that is the number for `stdout`. `<` uses FD 0 by default, because that is the number for `stdin`. The number for the `stderr` FD is 2. So, let's try sending the output of `stderr` to a file: {{{ $ for homedir in /home/* > do rm "$homedir/secret" > done 2> errors }}} In this example, we're looping over each file in `/home`. We then try to delete the file `secret` in each of them. Some `homedir`s may not have a secret. As a result, the `rm` operation will fail and send an error message on `stderr`. You may have noticed that our redirection operator isn't on `rm`, but it's on that `done` thing. Why is that? Well, this way, the redirection applies to all output to `stderr` made inside the whole loop. Let's see what the result of our loop was? {{{ $ cat errors rm: cannot remove `/home/axxo/secret': No such file or directory rm: cannot remove `/home/lhunath/secret': No such file or directory }}} Two error messages in our error log file. Two people that didn't have a `secret` file in their home directory. If you're writing a script, and you expect that running a certain command may fail on occasion, but don't want the script's user to be bothered by the possible error messages that command may produce, you can silence an FD. Silencing it is as easy as normal ''File Redirection''. We're just going to send all output to that FD into the system's black hole: {{{ $ for homedir in /home/* > do rm "$homedir/secret" > done 2> /dev/null }}} The file `/dev/null` is '''always''' empty, no matter what you write or read from it. As such, when we write our error messages to it, they just disappear. The `/dev/null` file remains as empty as ever before. That's because it's not a normal file, it's a ''virtual'' device. There is one last thing you should learn about ''File Redirection''. It's interesting that you can make error log files like this to keep your error messages; but as I mentioned before, ["BASH"] makes sure that the file exists before trying to redirect to it. ["BASH"] '''also makes sure the file is empty''' before redirecting to it. As a result, each time we run our loop to delete secret files, our log file will be truncated empty before we fill it up again with new error messages. What if we'd like to keep a record of any error messages generated by our loop? What if we don't want that file to be truncated each time we start our loop? The solution is achieved by doubling the redirection operator. `>` becomes `>>`. `>>` will not empty a file, it will just append new data to the end of it! {{{ $ for homedir in /home/* > do rm "$homedir/secret" > done 2>> errors }}} Hooray! -------- . '''Good Practice: [[BR]] It's a good idea to use redirection whenever an application needs file data and is built to read data from `stdin`. A lot of bad examples on the Internet tell you to pipe (see later) the output of `cat` into processes; but this is nothing more than a very ''bad'' idea.''' ---- . '''In The Manual: [http://www.gnu.org/software/bash/manual/bashref.html#SEC38 Redirecting Input], [http://www.gnu.org/software/bash/manual/bashref.html#SEC39 Redirecting Output], [http://www.gnu.org/software/bash/manual/bashref.html#SEC40 Appending Redirected Output], [http://www.gnu.org/software/bash/manual/bashref.html#SEC41 Redirecting Standard Output and Standard Error]''' -------- [[Anchor(FD_Manipulation)]] ==== File Descriptor Manipulation ==== Now that you know how to manipulate process input and output by sending it to and reading it from files, let's make it a little more interesting still. It's possible to change the source and desination of FDs to point to or from files, as you know. It's also possible to copy one FD to another. Let's prepare a simple testbed: {{{ $ echo "I am a proud sentence." > file }}} We've made a file called `file`, and written a proud sentence into it. It's time I introduce a new application to you. Its name is `grep`, and it's increadibly powerful. `grep` is that one thing that you need more than anything else in your household. It basically takes a ''search string'' as its first argument and one or more files as extra arguments. Just like `cat`, `grep` also uses `stdin` if you don't specify any files as extra arguments. `grep` reads the files (or `stdin` if none were provided) and searches for the ''search string'' you gave it. Most versions of `grep` even support a `-r` switch, which makes it take directories as well as files as extra arguments, and then searches all the files and directories in those directories that you gave it. Here's an example of how `grep` can work: {{{ $ ls house/ house/drawer house/closet house/dustbin house/sofa $ grep -r socks house/ house/sofa:socks }}} In this silly example we have a directory called `house` with several pieces of furniture in it as files. If we're looking for our `socks` in each of those files, we send `grep` to search the directory `house/`. `grep` will search everything in there, open each file and look through its contents. In our example, `grep` finds `socks` in the file `house/sofa`; presumably tucked away under a pillow. You want a more realistic example? Sure: {{{ $ grep "$HOSTNAME" /etc/* /etc/hosts:127.0.0.1 localhost Lyndir }}} Here we instruct `grep` to search for whatever `$HOSTNAME` expands to in whatever `/etc/*` expands to. It finds my hostname, which is `Lyndir` in the file `/etc/hosts`, and shows me the line in that file that contains the ''search string''. OK, now that you understand `grep`, let's continue with our ''File Descriptor Manipulation''. Remeber that we created a file called `file`, and wrote a proud sentence to it? Let's use `grep` to find where that proud sentence is now: {{{ $ grep proud * file:I am a proud sentence. }}} Good! `grep` found our sentence in `file`. It writes the result of its operation to `stdout` which is shown on our terminal. Now let's see if we can make `grep` send an error message, too: {{{ $ grep proud file 'not a file' file:I am a proud sentence. grep: not a file: No such file or directory }}} This time, we instruct `grep` to search for the string `proud` in the files '`file`' and '`not a file`'. `file` exists, and the sentence is in there, so `grep` happily writes the result to `stdout`. It moves on to the next file to scan, which is '`not a file`'. `grep` can't open this file to read its content, because it doesn't exist. As a result, `grep` emits an error message on `stderr` which is still connected to our terminal. Now, how would you go about silencing this `grep` statement completely? We'd like to send all the output that appears on the terminal to a file instead; let's call it `proud.log`: {{{ $ grep proud file 'not a file' > proud.log 2> proud.log }}} Does that look about right? We first use `>` to send `stdout` to `proud.log`, and then use `2>` to send `stderr` to `proud.log` as well. Almost, but not quite. If you run this command and then look in `proud.log`, you'll see there's only an error message, not the output from `stdout`. We've created a very bad condition here. After `stdout` has written its output to the log file, the error message needs to be written to the log file. The `stderr` redirection truncates the log file and writes its own information there. Things would go even more wrong if after that new information arrives on `stdout`. `stdout`'s write operation might cancel an active write operation from `stderr` to continue writing its own information. We need to prevent having two FDs working on the same destination or source. We can do this by duplicating FDs: {{{ $ grep proud file 'not a file' > proud.log 2>&1 }}} You need to remember to always read file redirections from left to right. This is the order in which ["BASH"] assigns and processes them. First, `stdout` is changed so that it points to our `proud.log`. Then, we use the `>&` syntax to duplicate FD 1 and put this duplicate in FD 2's stead. If this is hard for you to grasp, you could read this as: `stdout` becomes `proud.log` and `stderr` becomes `stdout` (which is `proud.log`). As a result, `stdout` obviously writes its information to `proud.log`, but `stderr` writes its information to whatever `stdout` was when the `>&` redirector was used; which was `proud.log` as well. In this case, the handle that writes to `proud.log` is the same for both `stdout` and `stderr`, and no collisions occur. Be careful not to confuse the order: {{{ $ grep proud file 'not a file' 2>&1 > proud.log }}} This would read as `stderr` becomes `stdout` (which is the terminal), and then `stdout` becomes `proud.log`. As a result, `stdout`'s messages will be logged, but the error messages will still go to the terminal. ''Oops''. '''Note: [[BR]] For compatibility reasons with other shells, ["BASH"] also makes yet another form of redirection available to you. The `&>` redirection operator is actually just a shorter version of what we did here; redirecting both `stdout` and `stderr` to a file:''' {{{ $ grep proud file 'not a file' &> proud.log }}} This is the same as `> proud.log 2>&1`, but not portable to BourneShell. It is not recommended practice. '''''TODO: Moving FDs and Opening FDs RW.''''' -------- . '''In The Manual: [http://www.gnu.org/software/bash/manual/bashref.html#SEC44 Dublicating File Descriptors], [http://www.gnu.org/software/bash/manual/bashref.html#SEC45 Moving File Descriptors], [http://www.gnu.org/software/bash/manual/bashref.html#SEC46 Opening File Descriptors for Reading and Writing]''' ---- . '''In the FAQ: [[BR]] [http://wooledge.org/mywiki/BashFAQ/014 How can I redirect the output of multiple commands at once?] [[BR]] [http://wooledge.org/mywiki/BashFAQ/032 How can I redirect the output of 'time' to a variable or file?] [[BR]] [http://wooledge.org/mywiki/BashFAQ/040 How do I use dialog to get input from the user?] [[BR]] [http://wooledge.org/mywiki/BashFAQ/047 How can I redirect stderr to a pipe?] [[BR]] [http://wooledge.org/mywiki/BashFAQ/055 Tell me all about 2>&1 -- what's the difference between 2>&1 >foo and >foo 2>&1, and when do I use which?]''' -------- [[Anchor(Heredocs_Herestrings)]] ==== Heredocs And Herestrings ==== Files aren't all that. They're boring, really. Strings are so much more interesting. They're not permanent like files on a hard disk, but they're easy to work with, easy to make and easy to manipulate. ''Heredocs'' and ''Herestrings'' allow you to perform ''Redirection'' as you would with files, just by using strings instead. Let's try it out! {{{ $ grep proud <<END > I am a proud sentence. > END I am a proud sentence. }}} This is a ''Heredoc'' (or ''Here Document''). ''Heredoc''s aren't really useful unless you're trying to embed long strings of several lines inside your scripts, which is bad practice. You should keep your logic (your code) and your input (your data) separated, preferably in different files. The way ''Heredoc''s work, is by adding the `<<STRING` operator at the end of a command. That'll instruct that command's `stdin` that it has to start reading from the script (or the command line, if you're not in a script). The input of the ''Heredoc'' stops as soon as you repeat whatever string you added to the end of the `<<`. In the example above, I used the string `END`; but it can really be anything (so long as you quote it if it has whitespace). Beware that all text following the ''Heredoc'' operator is sent to the command's `stdin` literally. That also means any spaces you use for indenting your script. The terminator string (in our case `END`) must be in the beginning of the line. {{{ echo "Let's test abc:" if [[ abc = a* ]]; then cat <<END abc seems to start with an a! END fi }}} Will result in: {{{ Let's test abc: abc seems to start with an a! }}} You can avoid this by temporarily removing the indentation for the lines of your ''Heredoc''s. However, that distorts your pretty and consistent indendation. There is an alternative. If you use `<<-END` instead of `<<END` as your ''Heredoc'' operator, ["BASH"] removes any `tab` characters in the beginning of each line of your ''Heredoc'' content before sending it to the command. That way you can still use tabs to indent your ''Heredoc'' content with the rest of your code. Those tabs will not be sent to the command that receives your ''Heredoc''. This also means you can use tabs to indent your terminator string. Let's check out the very similar but more interesting ''Herestring''s: {{{ $ grep proud <<<"I am a proud sentence" I am a proud sentence. }}} This time, `stdin` reads its information straight from the string you put after the `<<<` operator. This is very convenient to send data that's in variables into processes: {{{ $ grep proud <<<"$USER sits proudly on his throne in $HOSTNAME." lhunath sits proudly on his throne in Lyndir. }}} ''Herestring''s are shorter, less intrusive and overall more convenient than their bulky ''Heredoc'' counterpart. Later on, you will learn about pipes and how they can be used to send the output of a command into another command's `stdin`. Many people use pipes to send the output of a variable as `stdin` into a command. However, for this purpose, ''Herestring''s should be preferred. They do not create a subshell and are lighter both to the shell and to the style of your shell script: {{{ $ echo 'Wrap this silly sentence.' | fmt -t -w 20 Wrap this silly sentence. $ fmt -t -w 20 <<< 'Wrap this silly sentence.' Wrap this silly sentence. }}} -------- . '''Good Practice: [[BR]] Long heredocs are usually a bad idea because scripts should contain logic, not data. If you have a large document that your script needs, you should ship it in a separate file along with your script. Herestrings, however, come in handy quite often, especially for sending variable content to processes like `grep` or `sed` instead of files.''' ---- . '''In The Manual: [http://www.gnu.org/software/bash/manual/bashref.html#SEC42 Here Documents], [http://www.gnu.org/software/bash/manual/bashref.html#SEC43 Here Strings]''' -------- [[Anchor(Pipes)]] === Pipes === Now that you can effortlessly manipulate ''File Descriptors'' to direct certain types of output to certain files, it's time you learn some more ingenious tricks available through I/O redirection. You can use ''File Redirection'' to write output to files or read input from files. But what if you want to connect the output of one application directly to the input of another? That way, you could build a sort of chain to process output. If you already know about `FIFO`s, you could use something like this to that end: {{{ $ ls $ mkfifo myfifo; ls myfifo $ grep bea myfifo & [1] 32635 $ echo "rat > cow > deer > bear > snake" > myfifo bear }}} We use the `mkfifo` command to create a new file in the current directory named '`myfifo`'. This is no ordinary file, however, but a `FIFO`. `FIFO`s are files that serve data on a `First In, First Out`-basis. When you read from a `FIFO`, you will only receive data as soon as another process writes to it. As such, a `FIFO` never really contains any data. So long as no process writes data to a file that can be read, any read operation on the `FIFO` will '''block''' as it waits for data to become available. The same works for writes to the file. They will block until another process reads from the `FIFO`. In our example, the `FIFO` called `myfifo` is read from by `grep`. `grep` waits for data to become available on the `FIFO`. That's why we append the grep command with the `&` operator, which puts it in the background. That way, we can continue typing and executing commands while `grep` runs and waits for data. Our `echo` statement feeds data to the `FIFO`. As soon as this data becomes available, the running `grep` command reads it in and processes it. The result is displayed. We have successfully sent data from the `echo` command to the `grep` command. But these temporary files are a real annoyance. You may not have write permissions. You need to remember to clean up any temporary files you create. You need to make sure that data is going in and out, or the `FIFO` might just end up blocking for no reason. For these reasons, another feature is made available. This feature is called `Pipes`. It basically just connects the `stdout` of one process to the `stdin` of another; effectively ''piping'' the data from one process into another. Let's try our above example again, but using pipes: {{{ $ echo "rat > cow > deer > bear > snake" | grep bea bear }}} The pipe is created using the `|` operator inbetween two commands that are connected with the pipe. The former command's `stdout` is connected to the latter command's `stdin`. As a result, `grep` can read `echo`'s output and display the result of it's operation, which is `bear`. Pipes are widely used as a means of postprocessing application output. `FIFO`s are, in fact, also referred to as `named pipes`. They accomplish the same results as the pipe operator, but through a filename. '''Note: [[BR]] The pipe operator creates a subshell environment to run the second process in. This is important to know because any variables that you modify or initialize inside the second command will appear unmodified outside of it. Let's illustrate:''' {{{ $ message=Test $ echo "Salut, le monde!" | { read message; echo "The message is: $message"; } The message is: Salut, le monde! $ echo "The message is: $message" The message is: Test }}} Once the piped command ends, so does the subshell that was created for it. Along with that subshell, any modifications made in that subshell are lost. So be careful! -------- . '''Good Practice: [[BR]] Pipes are a very attractive means of post-processing application output. You should, however, be careful not to over-use pipes. If you end up making a pipe-chain that consists of three or more applications, it is time to ask yourself whether you're doing things a smart way. You might be able to use more application features of one of the post-processing applications you've used earlier in the pipe. Each new element in a pipe chain causes a new subshell and a new application to be loaded. It also makes it very hard to follow the logic in your script!''' ---- . '''In The Manual: [http://www.gnu.org/software/bash/manual/bashref.html#SEC17 Pipelines]''' ---- . '''In the FAQ: [[BR]] [http://wooledge.org/mywiki/BashFAQ/024 I set variables in a loop. Why do they suddenly disappear after the loop terminates? Or, why can't I pipe data to read?] [[BR]] [http://wooledge.org/mywiki/BashFAQ/027 How can two processes communicate using named pipes (fifos)?] [[BR]] [http://wooledge.org/mywiki/BashFAQ/047 How can I redirect stderr to a pipe?] [[BR]] [http://wooledge.org/mywiki/BashFAQ/001 How can I read a file line-by-line?] [[BR]] [http://wooledge.org/mywiki/BashFAQ/055 Tell me all about 2>&1 -- what's the difference between 2>&1 >foo and >foo 2>&1, and when do I use which?]''' -------- [[Anchor(Miscellaneous_Operators)]] === Miscellaneous Operators (stub) === [[Anchor(Structural_Constructs)]] == Structural Constructs (stub) == [[Anchor(Compound_Commands)]] === Compound Commands (stub) === [[Anchor(Subshells)]] === Subshells (stub) === [[Anchor(Functions)]] === Functions === Functions are very nifty in bash scripts. They are effectively no different than normal lines of code you might write, except they only get called when you decide to call it. Take this for example: {{{ $ sum() { > echo $1 + $2 = $(($1 + $2)) > } }}} This will do absolutely nothing when run. This is because it has only been stored in memory, much like a variable, but it has never been called. To run the function, you would do this: {{{ $ sum 1 4 1 + 4 = 5 }}} Amazing! We now have a basic calculator, and potentially a more economic replacement for a five year-old. A note on scope: if you choose to embed functions within script files, as many will find more convenient, then you need to understand that the parameters you pass to the script are not necessarily the parameters that are passed to the function. To wrap this function inside a script, we would write a file contain this: {{{ #!/bin/bash sum() { echo $1 + $2 = $(($1 + $2)) } sum $1 $2 }}} As you can see, we passed the script's two parameters to the function within, but we could have passed anything we wanted (though, doing so in this situation would only confuse users trying to use the script). [[Anchor(Aliases)]] === Aliases (stub) === [[Anchor(Sourcing)]] == Sourcing == When you call a script from another, the new script inherits the environment of the original script, just like running any other program in UNIX. Explaining what this means is out of the scope of this guide, but for the most part you can consider the environment to be the current working directory and the exported parameters, which you van view using the `export` command. When the script that you ran (or any other program, for that matter) finishes executing, its environment is discarded. The environment of the first script will be same as it was before the second script was called, although of course some of bash's special parameters may have changed (such as the value of `$?`, the return value of the most recent command). This means, for example, you can't simply run a script to change your current working directory for you. |
* [[/CommandsAndArguments|Commands and Arguments]] * Types of commands; argument splitting; writing scripts. * [[/SpecialCharacters|Special Characters]] * [[/Parameters|Parameters]] * Variables; special parameters; parameter types; parameter expansion. * [[/Patterns|Patterns]] * Globs; filename matching; extended globs; brace expansion; regular expressions. * [[/TestsAndConditionals|Tests and Conditionals]] * Exit status; {{{&&}}} and {{{||}}}; if, test and {{{[[}}}; while, until and for; case and select. * [[/Arrays|Arrays]] * Arrays; associative arrays. * [[/InputAndOutput|Input and Output]] * Redirection; here documents; here strings; pipes; process substitution. * [[/CompoundCommands|Compound Commands]] * Subshells; command grouping; arithmetic evaluation; functions; aliases. * [[/Sourcing|Sourcing]] * Reading commands from other files. * [[/JobControl|Job Control]] * [[/Practices|Practices]] * Choosing your shell; quoting; readability; debugging. |
| Line 1473: | Line 113: |
| . '''In the FAQ:''' [[BR]] [http://wooledge.org/mywiki/BashFAQ/060 I'm trying to write a script that will change directory (or set a variable), but after the script finishes, I'm back where I started (or my variable isn't set)!] | [[BashGuide/CommandsAndArguments|Commands and Arguments ->]] |
| Line 1475: | Line 115: |
| What you can do instead is to source the script instead of running it directly, using the `.` (dot) command: {{{ . myscript #runs the commands from the file myscript in this environment }}} This makes the commands in ''myscript'' run similarly to the way they would if they were in a function defined in the original Bash process, and able to change its environment. Note that Bash has a second name for this command, `source`, but since this works identically to the `.` command, it's probably easier to just forget about it and use the `.` command, as that will work everywhere. [[Anchor(Practices)]] = Practices = [[Anchor(Choose_Your_Shell)]] == Choose Your Shell == The first thing you should do before starting a shell script is enumerate the requirements and the goal of that script. Pick the right tool for the job. ["BASH"] may be easy to learn and write, but it isn't always fitted to do the work for you. There are a lot of tools in the basic toolset made available in most *NIX flavors that can help you. If you need only use AWK, then don't make a ["BASH"] script that invokes AWK, just make an AWK script. If you need to retrieve data from an HTML or XML file, ["BASH"] is also '''the wrong tool for the job'''. You should considder XSLT instead, or a language that has a library available for parsing XML or HTML. If you decide to use ["BASH"], then ask yourself these questions first: * Does your script need to run in environments where it's possible ["BASH"] is not available? . '''Then choose Sh instead!''' ''(Keep in mind that this guide does not apply to Sh!)'' * Are you certain about the version of ["BASH"] that will be available in all environments that your script will run in? . '''If not, you should limit yourself to features of ["BASH"] 2 ONLY.''' If the above questions do not limit your choices, '''then use ["BASH"] 3'''. Keep these points in mind: * '''Do not inherit ugly and evil shell scripting techniques from retarded Sh scripting guides and examples on the net; ["BASH"] 3 often offers far better alternatives!''' * Do not use a `#!/bin/sh` header in your script. Be honest and use `#!/bin/bash` instead. Otherwise you disable some ["BASH"] specific features that are not available in Sh. * Forget `[` for once and for all. ["BASH"] features an alternative that's '''far''' better for many of reasons. It's called `[[`. Using `[` will only cause headaches and break your script when you least expect it. * It's also time you forgot about {{{`}}}. It isn't consistent with the syntax of ''Expansion'' and is terribly hard to nest without a dose of painkillers. Use `$()` instead. * ''And for heaven's sake, '''quotes, quotes, quotes and more quotes'''!'' Protect your strings from ''Word Splitting''. ''Word splitting'' will eat your babies if you don't quote them properly. -------- [[Anchor(Quoting)]] == Quoting == ''Word Splitting'' is the demon inside ["BASH"] that is out to get unsuspecting newcomers or any other scripters off guard. If you do not understand how ''Word Splitting'' works or when it is aplied you should be very careful with your strings and your ''Parameter Expansion''s. I suggest you read up on ''Word Splitting'' if you doubt your knowledge. The best way to protect yourself from this beast is to quote all your strings. Quotes keep your strings in one piece and prevent ''Word Splitting'' from tearing them open. Allow me to illustrate: {{{ $ echo Push that word away from me. Push that word away from me. $ echo "Push that word away from me." Push that word away from me. }}} Now, don't think ''Word Splitting'' is about collapsing spaces. What really happened in this example is that the first command passed each word in our sentence as a separate argument to `echo`. ["BASH"] split our sentence up in words using the whitespace inbetween them to determine where each argument begins and ends. In the second example ["BASH"] is forced to keep the whole quoted string together. This means it's not split up into arguments and the whole string is passed to `echo` '''as the first argument'''. `echo` prints out each argument it gets with a space inbetween them. You should understand the basics of ''Word Splitting'' now. This is where it gets dangerous! ''Word Splitting'' does not just happen on literal strings, it also happens '''after ''Parameter Expansion'''''! As a result, on a dull and tired day, you might just be stupid enough to make this mistake: {{{ $ sentence="Push that word away from me." $ echo $sentence Push that word away from me. $ echo "$sentence" Push that word away from me. }}} As you can see, in the first `echo` command, we were neglectant and left out the quotes. That was a mistake. ["BASH"] expanded our sentence and then used ''Word Splitting'' to split the resulting expansion up into arguments to use for `echo`. In our second example, the quotes around the ''Parameter Expansion'' of `sentence` make sure ["BASH"] '''does not''' split it up into multiple arguments around the whitespace. It's not just spaces you need to protect. ''Word Splitting'' occurs on spaces, tabs and newlines. In certain context it aplies to the characters in the `IFS` variable. Here's another example to show you just how badly you can break things if you neglect to use quotes: {{{ $ echo "$(ls -al)" total 8 drwxr-xr-x 4 lhunath users 1 2007-06-28 13:13 "."/ drwxr-xr-x 102 lhunath users 9 2007-06-28 13:13 ".."/ -rw-r--r-- 1 lhunath users 0 2007-06-28 13:13 "a" -rw-r--r-- 1 lhunath users 0 2007-06-28 13:13 "b" -rw-r--r-- 1 lhunath users 0 2007-06-28 13:13 "c" drwxr-xr-x 2 lhunath users 1 2007-06-28 13:13 "d"/ drwxr-xr-x 2 lhunath users 1 2007-06-28 13:13 "e"/ $ echo $(ls -al) total 8 drwxr-xr-x 4 lhunath users 1 2007-06-28 13:13 "."/ drwxr-xr-x 102 lhunath users 9 2007-06-28 13:13 ".."/ -rw-r--r-- 1 lhunath users 0 2007-06-28 13:13 "a" -rw-r--r-- 1 lhunath users 0 2007-06-28 13:13 "b" -rw-r--r-- 1 lhunath users 0 2007-06-28 13:13 "c" drwxr-xr-x 2 lhunath users 1 2007-06-28 13:13 "d"/ drwxr-xr-x 2 lhunath users 1 2007-06-28 13:13 "e"/ }}} In some very rare occasions it may be desired to leave out the quotes. That's if you '''need''' ''Word Splitting'' to take place: {{{ $ friends="Marcus JJ Thomas Michelangelo" $ for friend in $friends > do echo "$friend is my friend!"; done Marcus is my friend! JJ is my friend! Thomas is my friend! Michelangelo is my friend! }}} But honestly? You should use arrays for nearly all of these cases. Arrays have the benefit that they separate strings without the need for an explicit delimitor. That means your strings in the array can contain '''any''' valid character, without the worry that it might be your string delimitor (like the space is in our example above). Using arrays in our example above gives us the ability to add middle or last names of our friends: {{{ $ friends=( "Marcus The Rich" "JJ The Short" "Timid Thomas" "Michelangelo The Mobster" ) $ for friend in "${friends[@]}" > do echo "$friend is my friend!"; done }}} Note that in our previous `for` we used an ''unquoted'' `$friends`. This allowed ["BASH"] to split our `friends` string up into words. In this last example, we quoted the `${friends[@]}` ''Parameter Expansion''. Quoting an expansion of an array through the `@` index makes ["BASH"] expand that array into a sequence of its elements, '''where each element is wrapped into quotes'''. [[Anchor(Readability)]] == Readability == Almost just as important as the result of your code is the '''readability of your code'''. Chances are that you aren't just going to write this script once and then forget about it. If so, you might as well delete it after using it. If you plan to continue using it, you should also plan to continue maintaining it. Unlike your room your code won't get dirty over time, but you will learn new techniques and new approaches constantly. You will also gain experience about how your script is used. All that new information you gather since the completion of your initial codebase should be used to maintain your code in such a way that it constantly improves. Your code should keep growing more userfriendly and more stable. . Trust me when I say, no piece of code is ever 100% ready; with exception to some very short and useless chucks of nothingness. To make it easier for yourself to keep your code healthy and improve it regularly you should keep an eye on the readability of what you write. When you return to a long loop after a year has passed since your last visit to it and you wish to improve it, add a feature, or debug something about it, believe me when I say you'd rather see this: {{{ friends=( "Marcus The Rich" "JJ The Short" "Timid Thomas" "Michelangelo The Mobster" ) # Say something significant about my friends. for name in "${friends[@]}"; do # My first friend (in the list). if [[ $name = ${friends[0]} ]]; then echo $name was my first friend. # My friends those names start with M. elif [[ $name = M* ]]; then echo "$name starts with an M" # My short friends. elif [[ " $name " = *" Short "* ]]; then echo "$name is a shorty." # Friends I kind of didn't bother to remember. else echo "I kind of forgot what $name is like." fi done }}} Than being confronted with something like this: {{{ x=( Marcus\ The\ Rich JJ\ The\ Short Timid\ Thomas Michelangelo\ The\ Mobster) for name in "${x[@]}" do if [ "$name" = "$x" ]; then echo $name was my first friend. elif echo $name | \ grep -qw Short then echo $name is a shorty. elif [ "x${name:0:1}" = "xM" ] then echo $name starts with an M; else echo I kind of forgot what $name \ is like.; fi; done }}} And yes, I know this is an exagerated example, but I've seen some authentic code that actually has a lot in common with that last example. For your own health, keep these few points in mind: * ''Healthy whitespace gives you breathing space''. Indent your code properly and consistently. Use blank lines to separate paragraphs or logic blocks. * ''Avoid backslash-escaping''. It's counter-intuitive and boggles the mind when overused. Even in small examples it takes your mind more effort to understand `a\ b\ c` than it takes to understand `'a b c'`. * ''Comment your way of thinking before you forget''. You might find that even code that looks totally common sense right now could become subject of ''"What the hell was I thinking when I wrote this?"'' or ''"What is '''this''' supposed to do?"''. * ''Consistency prevents mind boggles''. Be consistent in your naming style. Be consistent in your use of capitals. Be consistent in your use of shell features. No matter what your girlfriend says, it's good to be predictable and transparant. [[Anchor(Bash_Tests)]] == Bash Tests == The `test` application, also known as `[`, is an application that usually resides somewhere in `/usr/bin` or `/bin` and is used a lot by shell programmers to perform certain tests on variables. In a number of shells, including bash, [ is implemented as a shell builtin. It can produce surprising results, especially for people starting shell scripting that consider [ ] as part of the shell syntax. If you use `sh`, you have little choice but to use test as it is the only way to do most of your testing. If however you are using ["BASH"] to do your scripting (and I presume you are since you're reading this guide), then you can also use the newer [[ ]] keywords that, though [[ still behaves in many ways like a command, present several advantages over the traditional test command. Let me illustrate how `[[` can be used to replace `test`, and how it can help you to avoid some of the common mistakes made by using `test`: {{{ $ var='' $ [ $var = '' ] && echo True -bash: [: =: unary operator expected $ [ "$var" = '' ] && echo True True $ [[ $var = '' ]] && echo True True }}} `[ $var = '' ]` expands into `[ = '' ]`. The first thing `test` sees, is the equals sign. It takes this to be the left hand string of the test. The next thing `test` sees is the `''`. It expects an operator to perform some kind of test on `=` and explodes. Yes, `test` did not see our empty `$var` because ["BASH"] expanded it into nothingness before `test` could even see it. Morale of the story? '''Use more quotes!''' Using quotes, `[ "$var" = '' ]` expands into `[ "" = '' ]` and `test` has no problem. Now, `[[` can see the whole command before it's being expanded. It sees `$var`, and not the expansion of `$var`. As a result, there is no need for the quotes at all! '''`[[` is safer'''. {{{ $ var= $ [ "$var" < a ] && echo True -bash: a: No such file or directory $ [ "$var" \< a ] && echo True True $ [[ $var < a ]] && echo True True }}} In this example we attempted a string comparison between an empty variable and '`a`'. We're surprised to see the first attempt does not yield `True` even though it should. Instead, we get some weird error that implies ["BASH"] is trying to open a file called '`a`'. We've been bitten by ''File Redirection''. Since `test` is just an application, the `<` character in our command is interpreted (as it should) as a ''File Redirection'' operator instead of the string comparison operator of `test`. (Furthermore, the proper operator would have been -lt instead of < anyway.) ["BASH"] is instructed to open a file '`a`' and connect it to `stdin` for reading. To prevent this, we need to escape `<` so that `test` receives the operator rather than ["BASH"]. This makes our second attempt work. Using `[[` we can avoid the mess altogether. `[[` sees the `<` operator before ["BASH"] gets to use it for ''Redirection''; problem fixed. Once again, '''`[[` is safer'''. Even more dangerous is using the `>` operator instead of our previous example with the `<` operator. Since `>` triggers output ''Redirection'' it will create a file called '`a`'. As a result, '''there will be no error message warning us that we've committed a sin'''! Instead, our script will just break. Up to us to ''guess'' where the problem is: {{{ $ var=a $ [ "$var" > b ] && echo True || echo False True $ [[ "$var" > b ]] && echo True || echo False False }}} Two different results, great. Trust me when I say you can always trust `[[` more than `[`. `[ "$var" > b ]` is expanded into `[ "a" ]` and the output of that is being redirected into a new file called '`b`'. Since `[ "a" ]` is the same as `[ -n "a" ]` and that basically tests whether the `"a"` string is non-empty, the result is a success and the `echo True` is executed. Using `[[` we get our expected scenario where `"a"` is tested against `"b"` and since we all know `"a"` sorts before `"b"` this triggers the `echo False` statement. And this is how you can break your script without realizing it. You will however have a suspiciously empty file called '`b`' in your current directory. So believe me when I say, '''`[[` is safer than `[`'''. Because everybody inevitably makes programming errors. People usually don't intend to introduce bugs in their code. It just happens. So don't pretend you can use `[` and ''"You'll be careful not to make these mistakes"'', because I can assure you that you '''will'''. Besides `[[` provides the following features over `[`: * '''`[[` can do glob pattern matching:''' . `[[ abc = a* ]]` * '''`[[` can do regex pattern matching (Bash 3.1+):''' . `[[ abb =~ ab+ ]]` The only advantage of `test` is its portability. [[Anchor(Dont_Ever_Do_These)]] == Don't Ever Do These == The Bash shell allows you to do quite a lot of things, offering you considderable flexibility. Unfortunately, it does very little to discourage misuse and other ill-adviced behavior. It hopes people will find out for themselves that certain things should be avoided at all costs. Unfortunately many people don't care enough to want to find out for themselves. They write without thinking things through and many aweful and dangerous scripts end up in production environments or in Linux Distro scripts. The result of these, but even your very own scripts written in a time of neglect, can very often be '''DISASTROUS'''. That said, for the good of your scripts and for the rest of mankind; '''Never Ever Do Anything Along These Lines''': * '''`ls -l | awk '{ print $8 }'`''' . '''DON'T EVER parse the output of `ls`!''' The `ls` command's output cannot be trusted for a number of reasons.[[BR]] For one, `ls` will mangle the filenames of files if they contain characters unsupported by your locale. As a result, parsing filenames out of `ls` is NEVER guaranteed to actually give you a file that you will be able to find. `ls` might have replaced certain characters in the filename by question marks.[[BR]] Secondly, `ls` separates lines of data by newlines. This way, every bit of information on a file is on a new line. '''UNFORTUNATELY filenames themselves can ALSO contain newlines'''. This means that if you have a filename that contains a newline in the current directory, it will completely break your parsing and as a result, break your script![[BR]] Last but not least, the output format of `ls` is NOT guaranteed across platforms.[[BR]] If you need to work with file times; then I highly recommend you implement something using the features provided by `test`. See the `Bash Tests` chapter before this one. The truth is that there is '''no reliable way''' of working with file metadata in Bash. If you have considdered all the alternatives (backuping? Use `rsync`) and still '''really''' need this; ''I recommend you switch to a different language, like python or perl''. * '''`if echo "$file" | fgrep .txt; then`'''[[BR]] '''`ls *.txt | grep story`''' . '''DON'T EVER test or filter filenames `grep`!''' Unless your grep pattern is really smart it will probably not be trustworthy.[[BR]] In my example above, the test would match both `story.txt` AND `story.txt.exe`. If you make grep patterns that are smart enough, they'll probably be so ugly, massive and unreadable that you shouldn't use it anyway.[[BR]] The alternative is called '''GLOBS'''. Bash has a feature called ''Pathname Expansion''. This will help you enumerate all files that match a certain pattern! Alternatively you can use globs to test whether a certain filename matches a glob pattern. * '''`cat file | grep pattern`''' . Don't use `cat` as a default method for feeding file content to a process. `cat` is a utility used to concatenate the contents of several files together.[[BR]] To feed the contents of a file to a process you will probably be able to pass the filename as an argument to the process (`grep 'pattern' /my/file`, `sed 'expression' /my/file, ...).[[BR]] If the manual of the process does not specify any way to do this, you should use redirection (`read column1 column2 < /my/file`, `tr ' ' '\n' < /my/file`, ...). * '''`for number in $(seq 1 10); do`''' . For the love of god and all that is holy, do not use `seq` to count.[[BR]] Bash is able enough to do the counting for you. You do not need to spawn an external application to do some counting and then pass that application's output to Bash for word splitting. Learn the syntax of `for` already![[BR]] You should use this in Bash 3.x: `for number in {1..10}`, or this in Bash 2.x: `for ((i=1; i<10; i++))`. * '''`i= `{{{`}}}`expr $i + 1`{{{`}}}''' . `expr` is a relic of ancient Rome. Do not wield it.[[BR]] It was used very often in scripts written for shells with very limited capabilities. You're basically spawning a new process, calling another C program to do some math for you and return the result as a string to bash. Bash can do all this itself and so much faster, more reliable (no numbers->string->number conversions) and in all, better.[[BR]] You should use this in Bash: `let i++;`. * '''`i= `{{{`}}}`expr $i + 1`{{{`}}}''' . `expr` is a relic of ancient Rome. Do not wield it.[[BR]] It was used very often in scripts written for shells with very limited capabilities. You're basically spawning a new process, calling another C program to do some math for you and return the result as a string to bash. Bash can do all this itself and so much faster, more reliable (no numbers->string->number conversions) and in all, better.[[BR]] You should use this in Bash: `let i++;`. [[Anchor(Debugging)]] == Debugging == Very often you will find yourself clueless as to why your script isn't acting the way you want it to. Resolving this problem is always just a matter of common sense and debugging techniques. [[BR]] ||<tablewidth="100%"> ||<style="text-align: center;">'''Diagnose The Problem''' || || Unless you know what exactly the '''problem''' is, you most likely won't come up with a ''solution'' anytime soon. So make sure you understand what exactly goes wrong. Evaluate the symptoms and/or error messages. If you must, try and formulate the problem as a sentence. This will also be vital if you're going to ask for help with other people about your problem. You don't want them to rewrite your whole script. You also don't want them to have to go through your whole script or run it so that they understand what's going on. No; '''you''' need to make the problem perfectly clear to '''yourself''' and to anybody trying to help you. ''This requirement stands until the day the human race inventes means of telepathy''. [[BR]] ||<tablewidth="100%"> ||<style="text-align: center;">'''Minimize The Codebase''' || || If staring at your code doesn't give you a devine intervention, the next thing you should do is try to minimize your codebase while maintaining the same problem. Don't worry about preserving the functionality of your script. The only thing you want to preserve is the logic of the code block that seems buggy. Often, the best way to do this is by copying your script to a new file and start deleting everything that seems irrelevant from it. Alternatively, you can make a new script that does something similar in the same code fashion. As soon as you delete something that makes the problem go away, you'll have isolated the problem. If not, at least you're not staring at a massive script anymore, but hopefully at a stub of no more than 5-7 lines. For example; if you have a script that lets you browse images in your image folder by date, and for some reason you can't manage to iterate over your images in the folder properly, it suffices to reduce the script to this part in it: {{{ for image in $(ls -R "$imgFolder"); do echo "$image" done }}} Your actual script will be far more complex, and the inside of the `for` loop will also be far longer. But the essence of the problem is this code. Once you've reduced your problem to this it may be easier to see the problem you're facing. Your `echo` spits out parts of image names; it looks like all whitespace is replaced by newlines. That must be because `echo` is ran for each chunk terminated by whitespace, not for every image name (as a result, it seems the output has split open image names with whitespace in them). With this reduced code; it's easier to see that the cause is actually your `for` statement that splits up the output of `ls` into words. That's because `ls` is '''UNPARSABLE''' in a bugless manner (do not ever use `ls` in scripts, unless if you want to show its output to a user). The fix would be: {{{ find "$imgFolder" -print0 | while read -d $'\0' image; do echo "$image" done }}} Now that you've fixed the problem in this tiny example, it's easy to merge it back into the original script. [[BR]] ||<tablewidth="100%"> ||<style="text-align: center;">'''Activate ["BASH"]'s Debug Mode''' || || If you still don't see the error of your ways, ["BASH"]'s debugging mode might help you see the problem through the code. When ["BASH"] runs with the `x` option turned on, it prints out every command it executes before executing it. This is, '''after any expansions have been applied'''. As a result, you can see exactly what's happening as each line in your code is executed. Pay very close attention to the quoting used. ["BASH"] uses quotes to show you exactly which strings are passed as a single argument. There are three ways of turning on this mode. * Run the script with `bash -x`: {{{ $ bash -x ./mybrokenscript }}} * Modify your script's header: {{{ #! /bin/bash -x [.. script ..] }}} * Or add `set -x` somewhere in your code to turn on this mode for only a specific block of your code: {{{ #! /bin/bash [..irrelevant script code..] set -x [..relevant script code..] set +x [..irrelevant script code..] }}} [[BR]] ||<tablewidth="100%"> ||<style="text-align: center;">'''Reread The Manual''' || || If your script still doesn't seem to agree with you, maybe your perception of the way things work is wrong. Try going back to the manual (or this guide) to re-evaluate whether commands do exactly what you think they do, or the syntax is what you think it is. Very often people misunderstand what `for` does, how ''Word Splitting'' works, or how often they should use quotes. Keep the tips and good practice guidelines in this guide in mind as well. They often help you avoid bugs and problems with scripts. I mentioned this in the ''Scripts'' section of this guide too, but it's worth repeating it here. First of all, make sure your script's '''header''' is actually '''`#! /bin/bash`'''. If it is '''missing''' or if it's something like '''`#! /bin/sh`''' then you deserve the problems you're having. That means you're probably not even using ["BASH"] to run your code. Obviously that'll cause issues. Also, make sure you have no ''Carriage Return'' characters at the ends of your lines. This is the cause of scripts written in ''Microsoft Windows(tm)''. You can get rid of these fairly easily like this: {{{ $ tr -d '\r' < myscript > tmp && mv tmp myscript }}} [[BR]] ||<tablewidth="100%"> ||<style="text-align: center;">'''Read the FAQ / Pitfalls''' || || The ["BashFAQ"] and BashPitfalls pages explain common misconceptions and issues encountered by other ["BASH"] scripters. It's very likely that your problem will be described there in some shape or form. To be able to find your problem in there, you'll obviously need to have ''Diagnosed'' it properly. You'll need to know what you're looking for. [[BR]] ||<tablewidth="100%"> ||<style="text-align: center;">'''Ask Us On IRC''' || || There are people in the `#bash` channel almost 24/7. This channel resides on the `freenode` IRC network. To reach us, you need an IRC client. Connect it to `irc.freenode.net`, and `/join #bash`. Make '''sure''' that you know what your real problem is and have stepped through it on paper, so you can explain it well. We don't like having to guess at things. Start by explaining what you're trying to do with your script. Either way, please have a look at this page before entering `#bash`: XyProblem. |
CategoryShell |
Introduction
You are invited to make additions or modifications so long as you can keep them accurate. Please test any code samples you write.
All the information here is presented without any warranty or guarantee of accuracy. Use it at your own risk. When in doubt, please consult the man pages or the GNU info pages as the authoritative references.
BASH is a BourneShell compatible shell, which adds many new features to its ancestor. Most of them are available in the 'KornShell', too.
About This Guide
This guide aims to become a starting point for people interested in learning to work with BASH. It aspires to teach its readers good practice techniques for developing scripts for the BASH interpreter, and to educate them about the internal operation of BASH.
This guide is targeted at beginning users. It assumes no basic knowledge, but rather expects you to have enough common sense to put two and two together. If something is unclear to you, you are invited to report this (use BashGuideFeedback, or the #bash channel on irc.freenode.org) so that it may be clarified in this document for future readers.
You are invited to contribute to the development of this document by extending it or correcting invalid or incomplete information.
The maintainer(s) of this document:
The guide is also available in PDF format. Alternatively, you can just hit print after going to FullBashGuide. This guarantees you'll be printing the latest version of this document.
A Definition
BASH is an acronym for Bourne Again Shell. It is based on the Bourne shell and is mostly compatible with its features.
Shells are command interpreters. They are applications that provide users with the ability to give commands to their operating system interactively, or to execute batch processes quickly. In no way are they required for the execution of processes; they are merely a layer between system function calls and the user.
Think of a shell as a way for you to speak to your system. Your system doesn't need it for most of its work, but it is an excellent interface between you and what your system can offer. It allows you to perform basic math, run basic tests and execute applications. More importantly, it allows you to combine these operations and connect applications to each other to perform complex and automated tasks.
BASH is not your operating system. It is not your window manager. It is not your terminal (but it oftens runs inside your terminal). It does not control your mouse or keyboard. It does not configure your system, activate your screensaver, or open your files when you double-click them. It is generally not involved in launching applications from your window manager or desktop environment. It's important to understand that BASH is only the interface for you to execute statements (using BASH syntax), either at the interactive BASH prompt or via BASH scripts.
In The Manual: Introduction
Shell: A (possibly interactive) command interpreter, acting as a layer between the user and the system.
BASH: The Bourne Again Shell, a Bourne compatible shell.
Using Bash
Most users that think of BASH think of it as a prompt and a command line. That is BASH in interactive mode. BASH can also run in non-interactive mode through scripts. We can use scripts to automate certain logic. Scripts are basically lists of commands that you can type on the command line. When such a script is executed, all these commands are (generally) executed sequentially, one after another.
We'll start with the basics in an interactive shell. Once you're familiar with those, you can put them together in scripts.
Important!
You should make yourself familiar with the man and apropos commands on the shell. They will be vital to your self-tutoring.
$ man man
$ man aproposIn this guide, the $ at the beginning of a line represents your BASH prompt. Traditionally, a shell prompt either ends with $, % or #. If it ends with $, this indicates a shell that's compatible with the Bourne shell (such as a POSIX shell, or a Korn shell, or BASH). If it ends with %, this indicates a C shell (csh or tcsh); this guide does not cover C shell. If it ends with #, this indicates that the shell is running as the system's superuser account (root), and that you should be extra careful.
Your actual BASH prompt will probably be much longer than $. Prompts are often highly individualized.
The man command opens documentation (so-called "man pages") on a certain topic. You use it by running the command man [topic] at the BASH prompt, where [topic] is the name of the "page" you wish to read. Note that many of these "pages" are considerably longer than one printed page; nevertheless, the name persists. Each command (application) on your system is likely to have a man page. There are pages for other things too, such as system calls or specific configuration files. In this guide, we will only be covering commands.
Note that if you're looking for information on BASH built-ins (commands provided by BASH, not by external applications) you should look in man bash instead. BASH's manual is extensive and detailed. It is an excellent reference, albeit more technical than this guide.
BASH also offers a help command which contains brief summaries of its built-in commands (which we'll discuss in depth later on).
$ help
$ help read
Interactive mode: A mode of operation where a prompt asks you for one command at a time.
Script: A file that contains a sequence of commands to execute one after the other.
Contents
The guide has been divided into sections, which are intended to be read roughly in the order presented. If you skip ahead to a specific section, you might find yourself missing some background information from previous sections. (Links to relevant sections are not always provided when a topic is mentioned.)
- Types of commands; argument splitting; writing scripts.
- Variables; special parameters; parameter types; parameter expansion.
- Globs; filename matching; extended globs; brace expansion; regular expressions.
Exit status; && and ||; if, test and [[; while, until and for; case and select.
- Arrays; associative arrays.
- Redirection; here documents; here strings; pipes; process substitution.
- Subshells; command grouping; arithmetic evaluation; functions; aliases.
- Reading commands from other files.
- Choosing your shell; quoting; readability; debugging.