Differences between revisions 76 and 102 (spanning 26 versions)
Revision 76 as of 2013-07-21 13:57:23
Size: 21355
Editor: 59
Comment: Updated nmap switches
Revision 102 as of 2015-07-02 15:21:08
Size: 20902
Editor: c-24-4-69-166
Comment:
Deletions are marked like this. Additions are marked like this.
Line 10: Line 10:
[[BASH]] reads commands from its input (which is usually either a terminal or a file). Each line of input that it reads is treated as a ''command'' -- an instruction to be carried out. (There are a few advanced cases, such as commands that span multiple lines, but we won't worry about that just yet.)

Bash divides each line into ''words'' at each whitespace character (spaces and tabs). The first word it finds is the name of the command to be executed. All the remaining words become ''arguments'' to that command (options, filenames, etc.).

Assume you're in an empty directory. (If you want to try this code out, you can create and go into an empty directory called `test` by running: `mkdir test; cd test`.)

{{{
$ ls   # List files in the current directory (no output: no files).
$ touch a b c   # Create files 'a', 'b' and 'c'.
$ ls   # List all files again; this time the output shows 'a', 'b' and 'c'.
[[BASH]] reads commands from its input (which is usually either a terminal or a file). Each line of input that it reads is treated as a ''command'' an instruction to be carried out. (There are a few advanced cases, such as commands that span multiple lines, that will be gotten to later.)

Bash divides each line into ''words'' that are demarcated by a whitespace character (spaces and tabs). The first word of the line is the name of the command to be executed. All the remaining words become ''arguments'' to that command (options, filenames, etc.).

Assume we're in an empty directory... (to try these commands, create an empty directory called `test` and enter that directory by running: `mkdir test; cd test`):

{{{
$ ls # List files in the current directory (no output, no files).
$ touch a b c # Create files 'a', 'b', and 'c'.
$ ls # List files again, and this time outputs: 'a', 'b', and 'c'.
Line 22: Line 22:
The `ls` command prints out the names of the files in the current directory. The first time we run `ls`, we get no output, because there are no files yet.

The `#` character, when it is at the beginning of a word, indicates a ''comment''. Comments are ignored by the shell; they are meant only for humans to read. Everything from the `#` to the end of the current line is considered a comment, and ignored. If you're running these examples in your own shell, you don't have to type the comments; but even if you do, things will still work.

`touch` is an application that changes the ''Last Modified'' time of a file to the current time. If the filename that it's given does not exist yet, it simply creates that file, as a new and empty file. In this example, we passed three arguments. `touch` creates a file for each argument. `ls` shows us that three files have been created.

{{{
$ rm *   # Remove all files in the current directory.
$ ls   # List files in the current directory (no output: no files).
$ touch a b c   # Create files 'a', 'b' and 'c'.
$ ls   # List all files again; this time the output shows 'a', 'b' and 'c'.
The command `ls` prints out the names of the files in the current directory. The first time we run the list command we get no output, because there are no files yet.

The `#` character at the start of a word indicates a ''comment''. Any words following the comment are ignored by the shell, meant only for reading. If we run these examples in our own shell, we don't have to type the comments; but even if we do, the command will still work.

`touch` is an application that changes the ''Last Modified'' time of a file. If the filename that it is given does not exist yet, it creates a file of that name as a new and empty file. In this example, we passed three arguments. `touch` creates a file for each argument. `ls` shows us that three files have been created.

{{{
$ rm * # Remove all files in the current directory.
$ ls # List files in the current directory (no output, no files).
$ touch a b c # Create files 'a', 'b' and 'c'.
$ ls # List files again; this time the outputs: 'a', 'b' and 'c'.
Line 35: Line 35:
`rm` is an application that removes all the files that it was given. `*` is a [[glob]]. It basically means ''all files in the current directory''. You will read more about globs later.

Now, did you notice that there are several spaces between `a` and `b`, and only one between `b` and `c`? Also, notice that the files that were created by `touch` are no different than the first time. You now know that the amount of whitespace between arguments does not matter. This is important to know. For example:
`rm` is an application that removes all the files that it was given. `*` is a [[glob]]. It basically means ''all'' and in this case means all files in the current directory. We will talk more about globs later.

Now, did we notice that there are several spaces between `a` and `b`, and only one between `b` and `c`? Also, notice that the files that were created by `touch` are no different than the first time. The amount of whitespace between arguments does not matter! This is important to know. For example:
Line 45: Line 45:
`echo` is a command that writes its arguments to ''standard output'' (which in our case is the terminal). In this example, we provide the `echo` command with four arguments: 'This', 'is', 'a' and 'test.'. `echo` takes these arguments, and prints them out one by one with a space in between. In the second case, the exact same thing happens. The extra spaces make no difference. If we actually want the extra whitespace, we need to pass the sentence as one single argument. We can do this by using [[Quotes|quotes]]: `echo` is a command that prints its arguments to ''standard output'' (which in our case is the terminal). In this example, we provide the `echo` command with four arguments: 'This', 'is', 'a', and 'test.'. `echo` takes these arguments, and prints them out one by one with a space in between. In the second case, the exact same thing happens. The extra spaces make no difference. If we ''want'' the extra whitespace, we need to pass the sentence as one single argument. We can do this by using [[Quotes|quotes]]:
Line 51: Line 51:
Quotes group everything inside them into a single argument. This argument is '`This is a test.`', properly spaced. Note that the quotes are not part of the argument; BASH removes them before handing the argument to `echo`. `echo` prints this single argument out just like it always does. Quotes group everything inside them into a single argument. The argument is: '`This is a test.`'... specifically spaced. Note that the quotes are not part of the argument — Bash removes them before handing the argument to `echo`. `echo` prints this single argument out just like it always does.
Line 67: Line 67:
You need to make sure you quote filenames properly. If you don't you'll end up deleting the wrong things! `rm` takes filenames as arguments. If your filenames have spaces and you do not quote them, BASH thinks each word is a separate argument. BASH hands each argument to `rm` separately, like individually wrapped slices of processed cheese. `rm` treats each argument as a separate file.

The above example tried to delete a file for each word in the filename of the song, instead of keeping the filename intact. That caused our file `secret` to be deleted, and our song to remain behind!
We need to make sure we quote filenames properly. If we don't, we'll end up deleting the wrong things! `rm` takes filenames as arguments. If our filenames have spaces and we do not quote them, Bash thinks each word is a separate argument. Bash hands each argument to `rm` separately. Like individually wrapped slices of processed cheese, `rm` treats each argument as a separate file.

In the above example, `rm` tried to delete a file for each word in the filename of the song, rather than keeping the filename intact. That caused our file `secret` to be deleted, and our song to remain behind!
Line 76: Line 76:
Arguments are separated from the command name and from each other by white space. This is important to remember. For example, the following is '''wrong''': Arguments are separated from the command name and from each other by a whitespace. This is important to remember. For example, the following is '''wrong''':
Line 81: Line 81:
You want the `[` command name to be separated from the arguments `-f`, `file` and `]`. If you do not separate `[` and `-f` from each other with whitespace, BASH will think you are trying to execute the command name `[-f` and look in `PATH` for a program named `[-f`. Additionally, the arguments `file` and `]` need to be separated by spaces. The `[` command expects its last argument to be `]`. The correct command separates all arguments with spaces: We want the command `[` (a synonym for test) to be separated from the arguments: `-f`, `file`, and `]`. If we do not separate `[` and `-f` from each other with a whitespace, Bash will think we are trying to execute a command named `[-f`. The arguments `file` and `]` will so need to be separated by spaces. Additionally, the command `[` requires a closing argument of `]`. The correct command separates all arguments with whitespaces:
Line 86: Line 86:
(We'll cover the `[` command in more detail later. We see a lot of people who are confused by it, and think that they can omit the white space between it and its arguments, so we need to present this particular example very early.)

And of course, if your filename contains whitespace or other [[BashGuide/SpecialCharacters|special characters]], it should be quoted:
(We'll cover the `[` command in more detail later. We see a lot of people who are confused by this behavior; they think that they can omit the whitespace between it and its arguments, so we need to present this particular example early.)

And, if our ''filename'' contains whitespace or other [[BashGuide/SpecialCharacters|special characters]], it should also be quoted:
Line 93: Line 93:
'''NOTE:''' Please have a good look at [[Arguments]], [[Quotes]], WordSplitting and http://wiki.bash-hackers.org/syntax/words if all this isn't very clear to you yet. It is important that you have a good grasp of how the shell interprets the statements you give it before you continue this guide.

--------
 . '''Good Practice: <<BR>> You should ''always'' quote sentences or strings that belong together, even if it's not absolutely necessary. This will keep you alert and reduce the risk of human error in your scripts. <<BR>> For example, you should always quote arguments to the `echo` command.'''

----
 . '''In the FAQ: <<BR>> [[BashFAQ/050|I'm trying to put a command in a variable, but the complex cases always fail!]] <<BR>> [[BashFAQ/035|How can I handle command-line arguments (options) to my script easily?]]'''

----
 . ''Arguments'': These are the optional additional words you can specify when running commands. They are given after the command's name ('`ls -l foo`' executes `ls` with two arguments).
 . ''Quotes'': The two forms of quotes (`'` and `"`) are used to protect certain special characters inside them from being interpreted as special by Bash. The difference between `'` and `"` will be discussed later.

--------
Have a good look at [[Arguments]], [[Quotes]], WordSplitting and http://wiki.bash-hackers.org/syntax/words if all this isn't very clear yet. It is important to have a good grasp of how the shell interprets the whitespace and special characters before continuing with this guide.

'''Review:'''

 * ''Arguments'': These are additional words specified after the command ('`ls -l foo`' executes `ls` with two arguments).
 * ''Quotes'': The two forms of quotes, single and double (`'` and `"`), are used to group words and can protect special characters. The difference between `'` and `"` will be discussed later.

'''Additionally:'''

 * Tip — ''Always'' quote sentences or strings that belong together, even if it's not absolutely necessary. This developed practice will reduce the risk of human error in the scripts. (e.g. quoting a sentence of an `echo` command).
 * FAQ — I'm trying to put a command in a variable, but the complex cases always fail! [[BashFAQ/050]].
 * FAQ — How can I handle command line arguments (options) to my script easily? [[BashFAQ/035]].
Line 110: Line 109:
In BASH programming, almost ''everything'' is a string. When you type a command, the command's name is a string, and each argument is a string. Variable names are strings, and the contents of variables are strings as well. A filename is a string, and most files ''contain'' strings. They're everywhere! In Bash programming, almost ''everything'' is a string. When we type a command, the command's name is a string and each argument is a string; variable names are strings, and the contents of variables are strings as well; a filename is a string, and most files ''contain'' strings. They're everywhere!
Line 116: Line 115:
Let's try another example. With your favorite editor, write a shopping list and save it with the file name "list", and use `cat` to show it: Let's try another example.  With the editor, write a shopping list and save it with the filename "list", and use `cat` to print it:
Line 124: Line 123:
We typed a command: `cat list`. The shell reads this command as a string, and then divides it into the substrings `cat` and `list`. As far as the shell is concerned, `list` has no meaning. It's just a string with four characters in it. `cat` receives the argument `list`, which is a string that it interprets as a filename. The string `list` has become meaningful because of ''how it was used''.

The file happens to contain some text, which we see on our terminal. The entire file content, taken as a whole, is a string, but that string is not meaningful. However, if we divide the file into ''lines'' (and therefore treat each line as a separate string), then we see each individual ''line'' has meaning.

We can divide the final line into words, but these words are not meaningful by themselves. We can't buy `(skim` at the store, and we might get the wrong kind of `milk`. Dividing the lines into words is not a useful thing to do in this example. But the shell doesn't know any of this -- only ''you'' do!

So, when you are dealing with commands, and data, and variables -- all of which are just strings to the shell -- you have all the responsibility. You need to be sure everything that needs to be separated can be separated properly, and everything that needs to stay together stays together properly.

We'll touch on these concepts repeatedly as we continue.

== Types of Commands ==
We typed a command: `cat list`. The shell reads this command as a string, and then divides it into the substrings `cat` and `list`. As far as the shell is concerned, `list` has no meaning, it's just a string with four characters in it. `cat` receives the argument `list`, which is a string of a filename. The string `list` has become meaningful because of ''how it was used''.

The file happens to contain some text, which we see on our terminal. The entire file content — taken as a whole — is a string, but that string is not meaningful. However, if we divide the file into ''lines'' (and therefore treat each line as a separate string), then we see each individual ''line'' has meaning.

We can divide the final line into words, but these words are not meaningful by themselves. We can't buy "`(skim`" at the store, and we might get the wrong kind of "`milk`". Dividing the lines into words is not a useful thing to do in this example. But the shell doesn't know any of this only ''we'' do!

So, when we are dealing with commands, data, and variables all of which are just strings to the shell — we have ''all'' the responsibility. We need to be sure everything that needs to be separated is separated properly, and everything that needs to stay together stays together properly.  We'll touch on these concepts repeatedly as we continue.

== Types of commands ==
Line 137: Line 134:
 * '''Aliases''': Aliases are a way of shortening commands. They are only used in '''interactive''' shells, not in '''scripts'''. (This is one of the ''very'' few differences between a script and an interactive shell.) An alias is a ''name'' that is mapped to a certain ''string''. Whenever that ''name'' is used as a command name, it is replaced by the ''string'' before executing the command. So, instead of executing:
 {{{
 $ nmap -Pn -A --osscan-limit 192.168.0.1
}}}
 You could use an alias like this:
 {{{
 $ alias nmapp='nmap -Pn -A --osscan-limit'
 $ nmapp 192.168.0.1
}}}
 Aliases are limited in power; the replacement only happens in the first word. If you want more flexibility, use a function. Aliases are only useful as simple textual shortcuts.

 * '''Functions''': Functions in Bash are somewhat like aliases, but more powerful. Unlike aliases, they can be used in '''scripts'''. A function contains shell commands, very much like a small script; they can even take arguments and create local variables. When a function is called, the commands in it are executed. Functions will be covered in depth [[BashGuide/CompoundCommands#Functions|later in the guide]].

 * '''Builtins''': Bash has some basic commands built into it, such as `cd` (change directory), `echo` (write output), and so on. You can think of them as functions that are provided already.

 * '''Keywords''': Keywords are quite like builtins, but the main difference is that special parsing rules apply to them. For example, `[` is a bash builtin, while `[[` is a bash keyword. They are both used for testing stuff, but since `[[` is a keyword rather than a builtin, it benefits from a few special parsing rules which make it a lot easier to use:
 {{{
 $ [ a < b ]
 -bash: b: No such file or directory
 $ [[ a < b ]]
}}}
 The first example returns an error because bash tries to redirect the file `b` to the command `[ a ]` (See [[BashGuide/InputAndOutput#File_Redirection|File Redirection]]). The second example actually does what you expect it to. The character `<` no longer has its special meaning of ''File Redirection'' operator when it's used in a `[[` command.

 * '''Executables''': The last kind of command that can be executed by bash is an ''executable'', also called an ''external command'' or ''application''. Executables are invoked by using a pathname. If the executable is in the current directory, use `./myprogram`. If it's in the `/usr/local/bin` directory, use `/usr/local/bin/myprogram`.
 To make life a little easier for you, though, Bash uses a variable that tells it where to find applications in case you just know the name of the application but not its full pathname. This variable is called `PATH`, and it is a set of directory names separated by colons -- for example, `/bin:/usr/bin`. When a command is specified in Bash without a pathname (e.g. `myprogram`, or `ls`), and it isn't an alias, function, builtin or keyword, Bash searches through the directories in `PATH`, in order from left to right, to see whether they contain an executable of the name you typed.

--------
 . '''Tip: <<BR>> You can use the `type` command to figure out the type of a command.<<BR>> For example:'''
=== Aliases ===
An alias is a way of shortening a command. (They are only used in '''interactive''' shells and ''not'' in scripts — this is one of the ''very'' few differences between a script and an interactive shell.) An alias is a ''word'' that is mapped to a certain ''string''. Whenever that ''word'' is used as a command name, it is replaced by the ''string'' before executing the command. So, instead of executing:

{{{
$ nmap -Pn -A --osscan-limit 192.168.0.1
}}}
We could use an alias like this:

{{{
$ alias nmapp="nmap -Pn -A --osscan-limit"
$ nmapp 192.168.0.1
}}}
Aliases are limited in power; the replacement only happens in the first word. To have more flexibility, use a function. Aliases are only useful as simple textual shortcuts.

=== Functions ===
Functions in Bash are somewhat like aliases, but more powerful. Unlike aliases, they can be used in '''scripts'''. A function contains shell commands, and acts very much like a small script; they can even take arguments and create local variables. When a function is called, the commands in it are executed. Functions will be covered in depth [[BashGuide/CompoundCommands#Functions|later in the guide]].

=== Builtins ===
Bash has some basic commands built into it, such as: `cd` (change directory), `echo` (write output), and so on. They can be thought of as functions that are already provided.

=== Keywords ===
Keywords are like builtins, with the main difference being that special parsing rules apply to them. For example, `[` is a Bash builtin, while `[[` is a Bash keyword. They are both used for testing stuff. `[[` is a keyword rather than a builtin and is therefore able to offer an extended test:

{{{
$ [ a < b ]
-bash: b: No such file or directory
$ [[ a < b ]]
}}}
The first example returns an error because Bash tries to redirect the file `b` to the command `[ a ]` (see [[BashGuide/InputAndOutput#File_Redirection|File Redirection]] for additional explanation). The second example actually does what we expect it to, alphabetical comparision. The character `<` no longer has its typical meaning of a ''File Redirection'' operator when it's used with the extended test `[[` keyword.

=== Executables ===
The last kind of command that can be executed by Bash is an ''executable''. (Executables may also be called ''external commands'' or ''applications''.) Executables are commonly invoked by typing only their name. This can be done because a pre-defined variable makes known to Bash a list of common, executable, file paths. This variable is called `PATH`. It is a set of directory names separated by colons (e.g.`/usr/bin:/bin`). When a command is specified (e.g. `myprogram`, or `ls`) without a file path (and it isn't an alias, function, builtin or keyword), Bash searches through the directories in `PATH`. The search is done in order, from left to right, to see whether they contain an executable of the command typed.

If the executable is ''outside'' a known path... the executables file path will need to be defined. For an executable in the current directory, use `./myprogram`; if it's in the directory `/opt/somedirectory`, use `/opt/somedirectory/myprogram`.

'''Review:'''

 * ''Alias'' — a word that is mapped to a string. Whenever that word is used as a command, it is replaced by the string it has mapped.
 * ''Function'' — a name that is mapped to a set of commands. Whenever the function is used as a command, it is called with the arguments proceding it. Functions are the basic method of making new commands.
 * ''Builtin'' — certain commands have been built into Bash. These are handled directly by the Bash executable and do not create a new process.
 * ''Executable'' — a program that can be executed by referring to its file path (e.g. `/bin/ls`), or simply by its name if its location is in the `PATH` variable.

'''Additionally:'''

 * Manual — [[http://www.gnu.org/software/bash/manual/bashref.html#Simple-Commands|Simple Commands]]
 * FAQ — What is the difference between `test`, `[` and `[[`? [[BashFAQ/031]].
 * FAQ — How can I make an alias that takes an argument? [[BashFAQ/080]].
 * Tip — The `type` command can be used to get a description of the command type:
Line 172: Line 189:
----
 . '''In The Manual: [[http://www.gnu.org/software/bash/manual/bashref.html#Simple-Commands|Simple Commands]]'''

----
 . '''In the FAQ: <<BR>> [[BashFAQ/031|What is the difference between test, [ and [[ ?]] <<BR>> [[BashFAQ/080|How can I make an alias that takes an argument?]]'''

----
 . ''Alias'': A name that is mapped to a string. Whenever that name is used as a command, it is replaced by the string it has mapped.
 . ''Function'': A name that is mapped to a set of commands. Whenever that name is used as a command, the function is called with the arguments provided after the function's name on the command line. Functions are the basic method of making new commands.
 . ''Builtin'': Certain commands have been built into Bash. These are handled internally whenever they are executed on the command line (and do not create a new process).
 . ''Application'': A program that can be executed by referring to its pathname (`/bin/ls`), or simply by its name if its location is in your `PATH` variable. Running an application creates a new process.

--------
Line 187: Line 190:
A script is basically a sequence of commands in a file. Bash reads the file and processes the commands in order. It only moves on to the next command when the current one has ended, unless the current one has been executed asynchronously (in the background). Don't worry too much about the latter case yet -- you'll learn about how that works later on.

Virtually any example that you see in this guide can be used in a script just as well as on the command line.

Making a script is easy. Just make a new file, and put this in it at the top:
A script is basically a sequence of commands in a file. Bash reads the file and processes the commands '''in order'''. It moves on to the next command only when the current one has '''ended'''. (The exception being if a command has been specified to run asynchronously, in the background. Don't worry too much about the this case yet — we'll learn about how that works later on.)

Virtually any example that exists on the command line in this guide can also be used in a script.

Making a script is easy. Begin by making a new file, and put this on the first line:
Line 196: Line 199:
This header (also called the ''hashbang'' or ''shebang'') makes sure that whenever your script is executed, Bash will be used as its interpreter. The way it works, is that when the kernel executes a non-binary, it looks at the first line of the file. If the line begins with `#!`, the kernel uses the line to determine the interpreter that the code should be passed to. (There are other valid ways to do this, as well -- see below.) The `#!` must be at the very start of the file, with no spaces or blank lines before them. Your script's commands should all appear on separate lines below this.

'''Please''' do not be fooled by examples on the Internet that use `/bin/sh` as interpreter. '''`sh` is not `bash`'''. Even though `sh`'s syntax and `bash`'s look very much alike and even though most `bash` scripts will run in `sh`, a lot of the examples in this guide only apply to `bash` and will just break or cause unexpected behaviour in `sh`.

Also, please refrain from giving your scripts that stupid `.sh` extension. It serves no purpose, and it's completely misleading (since it's going to be a `bash` script, not an `sh` script).

And by the way, it's perfectly fine if you use ''Windows'' to write your scripts, but if at all possible, '''avoid using ''Notepad'' for writing scripts'''. ''Microsoft Notepad'' can only make files with DOS-style line-endings. That means that each line you make in notepad will be ended by two characters: a ''Carriage Return'' and a ''Newline'' character. Bash reads lines as terminated by ''Newline'' characters only. As a result, the ''Carriage Return'' character will cause you incredible headache if you don't know it's there (very weird error messages). If at all possible, ''use a decent editor'' like [[http://www.vim.org/|Vim]], [[http://www.gnu.org/software/emacs/|Emacs]], kate, GEdit, GVIM or xemacs. If you don't, then you will need to remove the carriage returns from your scripts before running them.

Once your script file has been made, you can call it like this:
The header is called an ''interpreter directive'' (it is also called a ''hashbang'' or ''shebang''). It makes sure that whenever our script is executed, Bash will be used as its interpreter. The way it works is this: when the kernel executes a non-binary file, it reads the first line of the file; if the line begins with `#!`, the kernel uses the line to determine the interpreter to relay the file to. (There are other valid ways to do this as well, see below.) The `#!` must be at the very start of the file, with no spaces or blank lines before it. Our script's commands will appear on separate lines below this.

'''Please''' do not be fooled by scripts or examples on the Internet that use `/bin/sh` as the interpreter. '''`sh` is not `bash`'''! Bash itself is a "sh-compatible" shell (meaning that it can run most 'sh' scripts and carries much of the same syntax) however, the opposite is not true; some features of Bash will break or cause unexpected behavior in `sh`.

Also, please refrain from giving scripts a `.sh` extension. It serves no purpose, and it's completely misleading (since it's going to be a `bash` script, not an `sh` script).

{{{#!wiki comment
Some reasons for using ".sh":

* It is more specific than not using any extension.
* Most people probably think of it as short for "shell" rather than "Bourne or POSIX equivalent shell."
* It's a convention. Following conventions is useful to ''at least'' everybody except the original developer(s).
* Obscure file systems (not counting FAT) where there can't be more than three letters in the file extension.
* People don't use ".bash3" (or, dog forbid, ".bash4.2.45(2)-release-x86_64-unknown-linux-gnu") even if it describes their interpreter more than the standard extensions.
}}}
''It is perfectly fine to use '''Windows''' to write scripts. Avoid, however, using '''Notepad'''. "Microsoft Notepad" can only make files with DOS-style line-endings. DOS-style line-endings end with two characters: a '''Carriage Return''' and a '''Newline''' character. Bash understands line-endings with only '''Newline''' characters. As a result, the '''Carriage Return''' character will cause an unexpected surprise if one doesn't know it's there (very weird error messages). If at all possible, use a more capable editor like [[http://www.vim.org/|Vim]], [[http://www.gnu.org/software/emacs/|Emacs]], kate, GEdit... If one doesn't, the carriage returns will need to be removed from the scripts before running them.''

Once the script file has been created, it can be executed by doing:
Line 209: Line 221:
In this example, we execute `bash` and tell it to read our script. When we do this, the `#!` line is just a comment. Bash does not do anything at all with it.

Alternatively, you can give your script executable permissions. When you do this, you can actually execute the script as an application instead of calling Bash manually:
In this example, we execute `bash` and tell it to read the script. When we do it like this, the `#!` line is just a comment, Bash does not do anything at all with it.

Alternatively, we can give our scripts executable permissions. With this method, instead of calling Bash manually, we can execute the script as an application:
Line 217: Line 229:
When executed in this way, the `#!` line tells the operating system (OS) what interpreter to use. The OS runs `/usr/bin/env`, which in turn runs `bash`, which reads our script.  BASH itself ignores the `#` header line, just like last time.

Some people like to keep their scripts in a personal directory. Others like to keep their scripts somewhere in the `PATH` variable. Most like to do both at once. Here's what I suggest you do:
When executed in this way, the `#!` line tells the operating system (OS) what interpreter to use. The OS runs `/usr/bin/env`, which in turn runs `bash`, which reads our script.

To decide where to put the script, a couple alternatives exists: generally people like to keep their scripts in a personally-defined directory; a few, like to keep their scripts in an existent, executable `PATH`. To use a personal directory:
Line 224: Line 236:
$ exec bash
}}}
The first command will make a directory called `bin` in your home directory. It is traditional for directories that contain commands to be named `bin`, even when those commands are scripts and not compiled ("''binary''") programs. The second command will add a line to your `.bashrc` file which adds the directory we just made to the beginning of the `PATH` variable. Every new instance of Bash will now check for executable scripts in your `bin` directory. Finally, the third line replaces our current instance of Bash with a new one, which reads the `.bashrc` file.

Changes to DotFiles (such as `.bashrc`) never have an immediate effect. You have to take some step to re-read the files. In the example above, we used `exec bash` to replace the running shell. If you wanted, you could close your existing terminal and open a new one. Bash would then initialize itself again by reading `.bashrc` (and possibly other files). Or, you could just execute that line of code on the command line (`PATH="$HOME/bin:$PATH"`) or manually process your `.bashrc` file in the running shell by running `source "$HOME/.bashrc"` .

In any case, we can now put our script in our `bin` directory and execute it as a normal command (we no longer need to prepend our script's name with its path, which was the `./` part in the previous examples):
$ source "$HOME/.bashrc"
}}}
The first command will make a directory called `bin` in the home, local, directory. It is traditional for directories that contain commands to be named `bin`, even when those commands are scripts and not compiled ''binary'' programs. The second command will add the directory's file path to the previous PATH variable and add that to the Bash configuration file (`.bashrc`). Every new instance of Bash will now check for executable scripts in defined directory. The third line reloads the Bash configuration file.

(Some people prefer to use a different directory to hold their personal scripts, such as `$HOME/.config/bin` or `$HOME/.local/bin`. You can use whatever you prefer, as long as you are consistent.)

Changes to Bash configuration file will not have an immediate effect, we have to make the step to re-read the file. In the example above, we used `source "$HOME/.bashrc"`, we could have also used `exec bash`, or we could close the existing terminal and open a new one. Bash would then initialize itself again by reading `.bashrc` (and possibly other files).

In any case, we can now put our script in our `bin` directory and execute it as a normal command — we no longer need to prepend our script's name with the file path (which was the "`./`" part in the previous example):
Line 236: Line 250:
--------
 . '''Tip: <<BR>> While you're defining the interpreter in your header, you might want to take the time to explain your script's function and expected arguments a little too:'''
 {{{#!text
#! /usr/bin/env bash
'''Additionally:'''

 * Tip — If the system has multiple versions of bash installed, specifying the interpreter by absolute path may be desirable in order to ensure the right bash version is used. For example: `#!/usr/bin/bash`. Type "`type -a bash`" to print the paths to all Bash executables in `PATH`.
 * Tip — The interpreter can optionally be followed by one word of interpreter's options. For example, the following options will turn on verbose debugging: "`#!/usr/bin/bash -xv`". To learn more, see [[BashGuide/Practices#Debugging|Debugging]].
 * Tip — After defining the interpreter in the header, it is recommended to summarize the scripts purpose. If desired, copyright information can be listed...:
 {{{#!highlight bash
#!/usr/bin/env bash
# scriptname - a short explanation of the scripts purpose.
Line 241: Line 259:
# scriptname argument [argument] ... # Copyright (C) <date> <name>...
Line 243: Line 261:
# A short explanation of your script's purpose.
#
# Copyright [date], [name]
}}}

----
 . '''Tip: <<BR>> You can use this header to specify up to one word of optional arguments that you want to pass to the interpreter. For example, the following arguments will turn on some verbose debugging:'''
 {{{#!text
#! /bin/bash -xv
}}}
 . '''But that requires knowing where Bash is installed, and it's not always in `/bin`. Unfortunately, you can't use this:'''
 {{{#!text
#! /usr/bin/env bash -xv
}}}
 . '''because that would require two words of arguments in the header line. Unix doesn't allow that. You can do this instead:'''
 {{{#!text
#! /usr/bin/env bash
set -xv
}}}
 . '''For more hints, see [[BashGuide/Practices#Debugging|Debugging]].'''

----
 . ''Header'': The header of a script determines the application that will function as its interpreter (e.g. `bash`, `sh`, `perl`, ...). Colloquially, this is also called a ''shebang'' -- a slang term derived by combining ''hash'' (`#`) and ''bang'' (`!`). You will probably see the word ''shebang'' used more often than ''header'', particularly since ''header'' has several other meanings in different contexts, while ''shebang'' only means one thing.
# scriptname [option] [argument] ...
}}}

<- Contents | Special Characters ->


Commands and Arguments

BASH reads commands from its input (which is usually either a terminal or a file). Each line of input that it reads is treated as a command — an instruction to be carried out. (There are a few advanced cases, such as commands that span multiple lines, that will be gotten to later.)

Bash divides each line into words that are demarcated by a whitespace character (spaces and tabs). The first word of the line is the name of the command to be executed. All the remaining words become arguments to that command (options, filenames, etc.).

Assume we're in an empty directory... (to try these commands, create an empty directory called test and enter that directory by running: mkdir test; cd test):

$ ls              # List files in the current directory (no output, no files).
$ touch a b c     # Create files 'a', 'b', and 'c'.
$ ls              # List files again, and this time outputs: 'a', 'b', and 'c'.
a  b  c

The command ls prints out the names of the files in the current directory. The first time we run the list command we get no output, because there are no files yet.

The # character at the start of a word indicates a comment. Any words following the comment are ignored by the shell, meant only for reading. If we run these examples in our own shell, we don't have to type the comments; but even if we do, the command will still work.

touch is an application that changes the Last Modified time of a file. If the filename that it is given does not exist yet, it creates a file of that name as a new and empty file. In this example, we passed three arguments. touch creates a file for each argument. ls shows us that three files have been created.

$ rm *            # Remove all files in the current directory.
$ ls              # List files in the current directory (no output, no files).
$ touch a   b c   # Create files 'a', 'b' and 'c'.
$ ls              # List files again; this time the outputs: 'a', 'b' and 'c'.
a  b  c

rm is an application that removes all the files that it was given. * is a glob. It basically means all and in this case means all files in the current directory. We will talk more about globs later.

Now, did we notice that there are several spaces between a and b, and only one between b and c? Also, notice that the files that were created by touch are no different than the first time. The amount of whitespace between arguments does not matter! This is important to know. For example:

$ echo This is a test.
This is a test.
$ echo This    is    a    test.
This is a test.

echo is a command that prints its arguments to standard output (which in our case is the terminal). In this example, we provide the echo command with four arguments: 'This', 'is', 'a', and 'test.'. echo takes these arguments, and prints them out one by one with a space in between. In the second case, the exact same thing happens. The extra spaces make no difference. If we want the extra whitespace, we need to pass the sentence as one single argument. We can do this by using quotes:

$ echo "This    is    a    test."
This    is    a    test.

Quotes group everything inside them into a single argument. The argument is: 'This    is    a    test.'... specifically spaced. Note that the quotes are not part of the argument — Bash removes them before handing the argument to echo. echo prints this single argument out just like it always does.

Be very careful to avoid the following:

$ ls                                          # There are two files in the current directory.
The secret voice in your head.mp3  secret
$ rm The secret voice in your head.mp3        # Executes rm with 6 arguments; not 1!
rm: cannot remove `The': No such file or directory
rm: cannot remove `voice': No such file or directory
rm: cannot remove `in': No such file or directory
rm: cannot remove `your': No such file or directory
rm: cannot remove `head.mp3': No such file or directory
$ ls                                          # List files in the current directory: It is still there.
The secret voice in your head.mp3             # But your file 'secret' is now gone!

We need to make sure we quote filenames properly. If we don't, we'll end up deleting the wrong things! rm takes filenames as arguments. If our filenames have spaces and we do not quote them, Bash thinks each word is a separate argument. Bash hands each argument to rm separately. Like individually wrapped slices of processed cheese, rm treats each argument as a separate file.

In the above example, rm tried to delete a file for each word in the filename of the song, rather than keeping the filename intact. That caused our file secret to be deleted, and our song to remain behind!

This is what we should have done:

$ rm "The secret voice in your head.mp3"

Arguments are separated from the command name and from each other by a whitespace. This is important to remember. For example, the following is wrong:

$ [-f file]

We want the command [ (a synonym for test) to be separated from the arguments: -f, file, and ]. If we do not separate [ and -f from each other with a whitespace, Bash will think we are trying to execute a command named [-f. The arguments file and ] will so need to be separated by spaces. Additionally, the command [ requires a closing argument of ]. The correct command separates all arguments with whitespaces:

$ [ -f file ]

(We'll cover the [ command in more detail later. We see a lot of people who are confused by this behavior; they think that they can omit the whitespace between it and its arguments, so we need to present this particular example early.)

And, if our filename contains whitespace or other special characters, it should also be quoted:

$ [ -f "my file" ]

Have a good look at Arguments, Quotes, WordSplitting and http://wiki.bash-hackers.org/syntax/words if all this isn't very clear yet. It is important to have a good grasp of how the shell interprets the whitespace and special characters before continuing with this guide.

Review:

  • Arguments: These are additional words specified after the command ('ls -l foo' executes ls with two arguments).

  • Quotes: The two forms of quotes, single and double (' and "), are used to group words and can protect special characters. The difference between ' and " will be discussed later.

Additionally:

  • Tip — Always quote sentences or strings that belong together, even if it's not absolutely necessary. This developed practice will reduce the risk of human error in the scripts. (e.g. quoting a sentence of an echo command).

  • FAQ — I'm trying to put a command in a variable, but the complex cases always fail! BashFAQ/050.

  • FAQ — How can I handle command line arguments (options) to my script easily? BashFAQ/035.

Strings

The term string refers to a sequence of characters which is treated as a single unit. The term is used loosely throughout this guide, as well as in almost every other programming language.

In Bash programming, almost everything is a string. When we type a command, the command's name is a string and each argument is a string; variable names are strings, and the contents of variables are strings as well; a filename is a string, and most files contain strings. They're everywhere!

An entire command can also be considered a string. This is not normally a useful point of view, but it illustrates the fact that parts of strings can sometimes be considered strings in their own right. A string which is part of a larger string is called a substring.

Strings do not have any intrinsic meaning. Their meaning is defined by how and where they are used.

Let's try another example. With the editor, write a shopping list and save it with the filename "list", and use cat to print it:

$ cat list
shampoo
tissues
milk (skim, not whole)

We typed a command: cat list. The shell reads this command as a string, and then divides it into the substrings cat and list. As far as the shell is concerned, list has no meaning, it's just a string with four characters in it. cat receives the argument list, which is a string of a filename. The string list has become meaningful because of how it was used.

The file happens to contain some text, which we see on our terminal. The entire file content — taken as a whole — is a string, but that string is not meaningful. However, if we divide the file into lines (and therefore treat each line as a separate string), then we see each individual line has meaning.

We can divide the final line into words, but these words are not meaningful by themselves. We can't buy "(skim" at the store, and we might get the wrong kind of "milk". Dividing the lines into words is not a useful thing to do in this example. But the shell doesn't know any of this — only we do!

So, when we are dealing with commands, data, and variables — all of which are just strings to the shell — we have all the responsibility. We need to be sure everything that needs to be separated is separated properly, and everything that needs to stay together stays together properly. We'll touch on these concepts repeatedly as we continue.

Types of commands

Bash understands several different types of commands: aliases, functions, builtins, keywords, and executables.

Aliases

An alias is a way of shortening a command. (They are only used in interactive shells and not in scripts — this is one of the very few differences between a script and an interactive shell.) An alias is a word that is mapped to a certain string. Whenever that word is used as a command name, it is replaced by the string before executing the command. So, instead of executing:

$ nmap -Pn -A --osscan-limit 192.168.0.1

We could use an alias like this:

$ alias nmapp="nmap -Pn -A --osscan-limit"
$ nmapp 192.168.0.1

Aliases are limited in power; the replacement only happens in the first word. To have more flexibility, use a function. Aliases are only useful as simple textual shortcuts.

Functions

Functions in Bash are somewhat like aliases, but more powerful. Unlike aliases, they can be used in scripts. A function contains shell commands, and acts very much like a small script; they can even take arguments and create local variables. When a function is called, the commands in it are executed. Functions will be covered in depth later in the guide.

Builtins

Bash has some basic commands built into it, such as: cd (change directory), echo (write output), and so on. They can be thought of as functions that are already provided.

Keywords

Keywords are like builtins, with the main difference being that special parsing rules apply to them. For example, [ is a Bash builtin, while [[ is a Bash keyword. They are both used for testing stuff. [[ is a keyword rather than a builtin and is therefore able to offer an extended test:

$ [ a < b ]
-bash: b: No such file or directory
$ [[ a < b ]]

The first example returns an error because Bash tries to redirect the file b to the command [ a ] (see File Redirection for additional explanation). The second example actually does what we expect it to, alphabetical comparision. The character < no longer has its typical meaning of a File Redirection operator when it's used with the extended test [[ keyword.

Executables

The last kind of command that can be executed by Bash is an executable. (Executables may also be called external commands or applications.) Executables are commonly invoked by typing only their name. This can be done because a pre-defined variable makes known to Bash a list of common, executable, file paths. This variable is called PATH. It is a set of directory names separated by colons (e.g./usr/bin:/bin). When a command is specified (e.g. myprogram, or ls) without a file path (and it isn't an alias, function, builtin or keyword), Bash searches through the directories in PATH. The search is done in order, from left to right, to see whether they contain an executable of the command typed.

If the executable is outside a known path... the executables file path will need to be defined. For an executable in the current directory, use ./myprogram; if it's in the directory /opt/somedirectory, use /opt/somedirectory/myprogram.

Review:

  • Alias — a word that is mapped to a string. Whenever that word is used as a command, it is replaced by the string it has mapped.

  • Function — a name that is mapped to a set of commands. Whenever the function is used as a command, it is called with the arguments proceding it. Functions are the basic method of making new commands.

  • Builtin — certain commands have been built into Bash. These are handled directly by the Bash executable and do not create a new process.

  • Executable — a program that can be executed by referring to its file path (e.g. /bin/ls), or simply by its name if its location is in the PATH variable.

Additionally:

  • Manual — Simple Commands

  • FAQ — What is the difference between test, [ and [[? BashFAQ/031.

  • FAQ — How can I make an alias that takes an argument? BashFAQ/080.

  • Tip — The type command can be used to get a description of the command type:

     $ type rm
     rm is hashed (/bin/rm)
     $ type cd
     cd is a shell builtin

Scripts

A script is basically a sequence of commands in a file. Bash reads the file and processes the commands in order. It moves on to the next command only when the current one has ended. (The exception being if a command has been specified to run asynchronously, in the background. Don't worry too much about the this case yet — we'll learn about how that works later on.)

Virtually any example that exists on the command line in this guide can also be used in a script.

Making a script is easy. Begin by making a new file, and put this on the first line:

#!/usr/bin/env bash

The header is called an interpreter directive (it is also called a hashbang or shebang). It makes sure that whenever our script is executed, Bash will be used as its interpreter. The way it works is this: when the kernel executes a non-binary file, it reads the first line of the file; if the line begins with #!, the kernel uses the line to determine the interpreter to relay the file to. (There are other valid ways to do this as well, see below.) The #! must be at the very start of the file, with no spaces or blank lines before it. Our script's commands will appear on separate lines below this.

Please do not be fooled by scripts or examples on the Internet that use /bin/sh as the interpreter. sh is not bash! Bash itself is a "sh-compatible" shell (meaning that it can run most 'sh' scripts and carries much of the same syntax) however, the opposite is not true; some features of Bash will break or cause unexpected behavior in sh.

Also, please refrain from giving scripts a .sh extension. It serves no purpose, and it's completely misleading (since it's going to be a bash script, not an sh script).

It is perfectly fine to use Windows to write scripts. Avoid, however, using Notepad. "Microsoft Notepad" can only make files with DOS-style line-endings. DOS-style line-endings end with two characters: a Carriage Return and a Newline character. Bash understands line-endings with only Newline characters. As a result, the Carriage Return character will cause an unexpected surprise if one doesn't know it's there (very weird error messages). If at all possible, use a more capable editor like Vim, Emacs, kate, GEdit... If one doesn't, the carriage returns will need to be removed from the scripts before running them.

Once the script file has been created, it can be executed by doing:

$ bash myscript

In this example, we execute bash and tell it to read the script. When we do it like this, the #! line is just a comment, Bash does not do anything at all with it.

Alternatively, we can give our scripts executable permissions. With this method, instead of calling Bash manually, we can execute the script as an application:

$ chmod +x myscript
$ ./myscript

When executed in this way, the #! line tells the operating system (OS) what interpreter to use. The OS runs /usr/bin/env, which in turn runs bash, which reads our script.

To decide where to put the script, a couple alternatives exists: generally people like to keep their scripts in a personally-defined directory; a few, like to keep their scripts in an existent, executable PATH. To use a personal directory:

$ mkdir -p "$HOME/bin"
$ echo 'PATH="$HOME/bin:$PATH"' >> "$HOME/.bashrc"
$ source "$HOME/.bashrc"

The first command will make a directory called bin in the home, local, directory. It is traditional for directories that contain commands to be named bin, even when those commands are scripts and not compiled binary programs. The second command will add the directory's file path to the previous PATH variable and add that to the Bash configuration file (.bashrc). Every new instance of Bash will now check for executable scripts in defined directory. The third line reloads the Bash configuration file.

(Some people prefer to use a different directory to hold their personal scripts, such as $HOME/.config/bin or $HOME/.local/bin. You can use whatever you prefer, as long as you are consistent.)

Changes to Bash configuration file will not have an immediate effect, we have to make the step to re-read the file. In the example above, we used source "$HOME/.bashrc", we could have also used exec bash, or we could close the existing terminal and open a new one. Bash would then initialize itself again by reading .bashrc (and possibly other files).

In any case, we can now put our script in our bin directory and execute it as a normal command — we no longer need to prepend our script's name with the file path (which was the "./" part in the previous example):

$ mv myscript "$HOME/bin"
$ myscript

Additionally:

  • Tip — If the system has multiple versions of bash installed, specifying the interpreter by absolute path may be desirable in order to ensure the right bash version is used. For example: #!/usr/bin/bash. Type "type -a bash" to print the paths to all Bash executables in PATH.

  • Tip — The interpreter can optionally be followed by one word of interpreter's options. For example, the following options will turn on verbose debugging: "#!/usr/bin/bash -xv". To learn more, see Debugging.

  • Tip — After defining the interpreter in the header, it is recommended to summarize the scripts purpose. If desired, copyright information can be listed...:
       1 #!/usr/bin/env bash
       2 # scriptname - a short explanation of the scripts purpose.
       3 #
       4 # Copyright (C) <date> <name>...
       5 #
       6 # scriptname [option] [argument] ...
    


<- Contents | Special Characters ->

BashGuide/CommandsAndArguments (last edited 2023-06-20 18:36:11 by larryv)