Differences between revisions 40 and 41
Revision 40 as of 2015-10-30 13:01:33
Size: 18965
Editor: GreyCat
Comment: ${path##*/} is not about the number of bytes of code. It is about efficiency. Don't fork a whole process to manipulate a string. Also, don't change the style of one page of a multi-page guide.
Revision 41 as of 2016-02-11 03:44:23
Size: 18892
Editor: llua
Comment: made line spanning expressions copy/pasteable
Deletions are marked like this. Additions are marked like this.
Line 3: Line 3:
----
<<TableOfContents>>
<<Anchor(StartOfContent)>>

----
<<TableOfContents>> <<Anchor(StartOfContent)>>
Line 8: Line 8:
Line 20: Line 19:
Line 24: Line 22:
Line 38: Line 37:
Line 45: Line 43:
Line 56: Line 53:
Line 62: Line 58:
Line 65: Line 60:
Line 68: Line 62:
Line 71: Line 66:
Line 82: Line 76:
Line 86: Line 79:

|| '''Parameter Name''' || '''Usage''' || '''Description''' ||
||
'''0''' || `"$0"` || Contains the name, or the path, of the script. This is not always reliable. ||
|| '''1''' '''2''' etc. || `"$1"` etc. || ''Positional Parameters'' contain the arguments that were passed to the current script or function. ||
|| '''*''' || `"$*"` || Expands to all the words of all the positional parameters. Double quoted, it expands to a single string containing them all, separated by the first character of the '''IFS''' variable (discussed later). ||
|| '''@''' || `"$@"` || Expands to all the words of all the positional parameters. Double quoted, it expands to a list of them all as individual words. ||
|| '''#''' || `$#` || Expands to the number of positional parameters that are currently set. ||
|| '''?''' || `$?` || Expands to the exit code of the most recently completed foreground command. ||
|| '''$''' || `$$` || Expands to the [[ProcessManagement|PID]] (process ID number) of the current shell. ||
|| '''!''' || `$!` || Expands to the PID of the command most recently executed in the background. ||
|| '''_''' || `"$_"` || Expands to the last argument of the last command that was executed. ||
||'''Parameter Name''' ||'''Usage''' ||'''Description''' ||
||'''0''' ||`"$0"` ||Contains the name, or the path, of the script. This is not always reliable. ||
||'''1''' '''2''' etc. ||`"$1"` etc. ||''Positional Parameters'' contain the arguments that were passed to the current script or function. ||
||'''*''' ||`"$*"` ||Expands to all the words of all the positional parameters. Double quoted, it expands to a single string containing them all, separated by the first character of the '''IFS''' variable (discussed later). ||
||'''@''' ||`"$@"` ||Expands to all the words of all the positional parameters. Double quoted, it expands to a list of them all as individual words. ||
||'''#''' ||`$#` ||Expands to the number of positional parameters that are currently set. ||
||'''?''' ||`$?` ||Expands to the exit code of the most recently completed foreground command. ||
||'''$''' ||`$$` ||Expands to the [[ProcessManagement|PID]] (process ID number) of the current shell. ||
||'''!''' ||`$!` ||Expands to the PID of the command most recently executed in the background. ||
||'''_''' ||`"$_"` ||Expands to the last argument of the last command that was executed. ||


Line 142: Line 137:
Line 144: Line 140:
Line 146: Line 143:
Line 149: Line 147:
--------


--------
Line 153: Line 150:
Line 154: Line 152:
Line 179: Line 176:
Line 190: Line 186:
Line 192: Line 189:
--------


--------
Line 196: Line 192:
Line 197: Line 194:
Line 210: Line 206:
> do mv -- "$file" "${file%.*}.jpg"
> done
do mv -- "$file" "${file%.*}.jpg"
done
Line 216: Line 212:

||
'''Syntax''' || '''Description''' ||
|| `${parameter:-word}` || '''Use Default Value'''. If '`parameter`' is unset or null, '`word`' (which may be an expansion) is substituted. Otherwise, the value of '`parameter`' is substituted. ||
|| `${parameter:=word}` || '''Assign Default Value'''. If '`parameter`' is unset or null, '`word`' (which may be an expansion) is assigned to '`parameter`'. The value of '`parameter`' is then substituted. ||
|| `${parameter:+word}` || '''Use Alternate Value'''. If '`parameter'` is null or unset, nothing is substituted, otherwise '`word`' (which may be an expansion) is substituted. ||
|| `${parameter:offset:length}` || '''Substring Expansion'''. Expands to up to '`length`' characters of '`parameter`' starting at the character specified by '`offset`' (0-indexed). If '`:length`' is omitted, go all the way to the end. If '`offset`' is negative (use parentheses!), count backward from the end of '`parameter`' instead of forward from the beginning. If '`parameter`' is @ or an indexed array name subscripted by @ or *, the result is '`length`' positional parameters or members of the array, respectively, starting from '`offset`'. ||
|| `${#parameter}` || The length in characters of the value of '`parameter`' is substituted. ||
|| `${parameter#pattern}` || The '`pattern`' is matched against the '''beginning''' of '`parameter`'. The result is the expanded value of '`parameter`' with the shortest match deleted. ||
|| `${parameter##pattern}` || As above, but the ''longest'' match is deleted. ||
|| `${parameter%pattern}` || The '`pattern`' is matched against the '''end''' of '`parameter`'. The result is the expanded value of '`parameter`' with the shortest match deleted. ||
|| `${parameter%%pattern}` || As above, but the ''longest'' match is deleted. ||
|| `${parameter/pat/string}` || Results in the expanded value of '`parameter`' with the first (unanchored) match of '`pat`' replaced by '`string`'. ||
|| `${parameter//pat/string}` || As above, but every match of '`pat`' is replaced. ||
||'''Syntax''' ||'''Description''' ||
||`${parameter:-word}` ||'''Use Default Value'''. If '`parameter`' is unset or null, '`word`' (which may be an expansion) is substituted. Otherwise, the value of '`parameter`' is substituted. ||
||`${parameter:=word}` ||'''Assign Default Value'''. If '`parameter`' is unset or null, '`word`' (which may be an expansion) is assigned to '`parameter`'. The value of '`parameter`' is then substituted. ||
||`${parameter:+word}` ||'''Use Alternate Value'''. If '`parameter'` is null or unset, nothing is substituted, otherwise '`word`' (which may be an expansion) is substituted. ||
||`${parameter:offset:length}` ||'''Substring Expansion'''. Expands to up to '`length`' characters of '`parameter`' starting at the character specified by '`offset`' (0-indexed). If '`:length`' is omitted, go all the way to the end. If '`offset`' is negative (use parentheses!), count backward from the end of '`parameter`' instead of forward from the beginning. If '`parameter`' is @ or an indexed array name subscripted by @ or *, the result is '`length`' positional parameters or members of the array, respectively, starting from '`offset`'. ||
||`${#parameter}` ||The length in characters of the value of '`parameter`' is substituted. ||
||`${parameter#pattern}` ||The '`pattern`' is matched against the '''beginning''' of '`parameter`'. The result is the expanded value of '`parameter`' with the shortest match deleted. ||
||`${parameter##pattern}` ||As above, but the ''longest'' match is deleted. ||
||`${parameter%pattern}` ||The '`pattern`' is matched against the '''end''' of '`parameter`'. The result is the expanded value of '`parameter`' with the shortest match deleted. ||
||`${parameter%%pattern}` ||As above, but the ''longest'' match is deleted. ||
||`${parameter/pat/string}` ||Results in the expanded value of '`parameter`' with the first (unanchored) match of '`pat`' replaced by '`string`'. ||
||`${parameter//pat/string}` ||As above, but every match of '`pat`' is replaced. ||


Line 234: Line 232:
> echo "File location: $file"; \
> echo "Filename: ${file##*/}"; \
> echo "Directory of file: ${file%/*}"; \
> echo "Non-secret file: ${file/secrets/not_secret}"; \
> echo; \
>
echo "Other file location: ${other:-There is no other file}"; \
> echo "Using file if there is no other file: ${other:=$file}"; \
> echo "Other filename: ${other##*/}"; \
> echo "Other file location length: ${#other}"
echo "File location: $file"; \
echo "Filename: ${file##*/}"; \
echo "Directory of file: ${file%/*}"; \
echo "Non-secret file: ${file/secrets/not_secret}"; \
echo; \
echo "Other file location: ${other:-There is no other file}"; \
echo "Using file if there is no other file: ${other:=$file}"; \
echo "Other filename: ${other##*/}"; \
echo "Other file location length: ${#other}"
Line 253: Line 251:
Line 262: Line 259:
Line 271: Line 267:
Line 273: Line 270:
Line 279: Line 277:
Line 281: Line 280:
Line 282: Line 282:

<- Special Characters | Patterns ->


Parameters

Parameters are a sort of named space in memory you can use to retrieve or store information. Generally speaking, they will store string data, but can also be used to store integers, indexed and associative arrays.

Parameters come in two flavors: variables and special parameters. Special parameters are read-only, pre-set by BASH, and used to communicate some type of internal status. Variables are parameters that you can create and update yourself. Variable names are bound by the following rule:

  • Name: A word consisting only of letters, digits and underscores, and beginning with a letter or an underscore. Also referred to as an identifier.

To store data in a variable, we use the following assignment syntax:

$ varname=vardata

This command assigns the data vardata to the variable by name of varname.

Please note that you cannot use spaces around the = sign in an assignment. If you write this:

# This is wrong!
$ varname = vardata

BASH will not know that you are attempting to assign something. The parser will see varname with no = and treat it as a command name, and then pass = and vardata to it as arguments.

To access the data stored in a variable, we use parameter expansion. Parameter expansion is the substitution of a parameter by its value, which is to say, the syntax tells bash that you want to use the contents of the variable. After that, Bash may still perform additional manipulations on the result. This is a very important concept to grasp correctly, because it is very much unlike the way variables are handled in other programming languages!

To illustrate what parameter expansion is, let's use this example:

$ foo=bar
$ echo "Foo is $foo"

When Bash is about to execute your code, it first changes the command by taking your parameter expansion (the $foo), and replacing it by the contents of foo, which is bar. The command becomes:

$ echo "Foo is bar"
Foo is bar

Now, Bash is ready to execute the command. Executing it shows us the simple sentence on screen.

It is important to understand that parameter expansion causes the $parameter to be replaced by its contents. Note the following case, which relies on an understanding of the previous chapter on argument splitting:

$ song="My song.mp3"
$ rm $song
rm: My: No such file or directory
rm: song.mp3: No such file or directory

Why did this not work? Because Bash replaced your $song by its contents, being My song.mp3; then it performed word splitting; and only THEN executed the command. It was as if you had typed this:

$ rm My song.mp3

And according to the rules of word splitting, Bash thought you meant for My and song.mp3 to mean two different files, because there is white space between them and it wasn't quoted. How do we fix this? We remember to put double quotes around every parameter expansion!


  • Parameters: Parameters store data that can be retrieved through a symbol or a name.


Special Parameters and Variables

Let's get our vocabulary straight before we get into the real deal. There are Parameters and Variables. Variables are actually just one kind of parameter: parameters that are denoted by a name. Parameters that aren't variables are called Special Parameters. I'm sure you'll understand things better with a few examples:

$ # Some parameters that aren't variables:
$ echo "My shell is $0, and has these options set: $-"
My shell is -bash, and has these options set: himB
$ # Some parameters that ARE variables:
$ echo "I am $LOGNAME, and I live at $HOME."
I am lhunath, and I live at /home/lhunath.

Please note: Unlike PHP/Perl/... parameters do NOT start with a $-sign. The $-sign you see in the examples merely causes the parameter that follows it to be expanded. Expansion basically means that the shell replaces the parameter by its content. As such, LOGNAME is the parameter (variable) that contains your username. $LOGNAME is an expression that will be replaced with the content of that variable, which in my case is lhunath.

I think you've got the drift now. Here's a summary of most of the Special Parameters:

Parameter Name

Usage

Description

0

"$0"

Contains the name, or the path, of the script. This is not always reliable.

1 2 etc.

"$1" etc.

Positional Parameters contain the arguments that were passed to the current script or function.

*

"$*"

Expands to all the words of all the positional parameters. Double quoted, it expands to a single string containing them all, separated by the first character of the IFS variable (discussed later).

@

"$@"

Expands to all the words of all the positional parameters. Double quoted, it expands to a list of them all as individual words.

#

$#

Expands to the number of positional parameters that are currently set.

?

$?

Expands to the exit code of the most recently completed foreground command.

$

$$

Expands to the PID (process ID number) of the current shell.

!

$!

Expands to the PID of the command most recently executed in the background.

_

"$_"

Expands to the last argument of the last command that was executed.

And here are a few examples of Variables that the shell provides for you:

  • BASH_VERSION: Contains a string describing the version of Bash.

  • HOSTNAME: Contains the hostname of your computer, I swear. Either short or long form, depending on how your computer is set up.

  • PPID: Contains the PID of the parent process of this shell.

  • PWD: Contains the current working directory.

  • RANDOM: Each time you expand this variable, a (pseudo)random number between 0 and 32767 is generated.

  • UID: The ID number of the current user. Not reliable for security/authentication purposes, alas.

  • COLUMNS: The number of characters that fit on one line in your terminal. (The width of your terminal in characters.)

  • LINES: The number of lines that fit in your terminal. (The height of your terminal in characters.)

  • HOME: The current user's home directory.

  • PATH: A colon-separated list of paths that will be searched to find a command, if it is not an alias, function, builtin command, or shell keyword, and no pathname is specified.

  • PS1: Contains a string that describes the format of your shell prompt.

  • TMPDIR: Contains the directory that is used to store temporary files (by the shell).

(There are many more -- see the manual for a comprehensive list.) Of course, you aren't restricted to only these variables. Feel free to define your own:

$ country=Canada
$ echo "I am $LOGNAME and I currently live in $country."
I am lhunath and I currently live in Canada.

Notice what we did to assign the value Canada to the variable country. Remember that you are NOT allowed to have any spaces before or after that equals sign!

$ language = PHP
-bash: language: command not found
$ language=PHP
$ echo "I'm far too used to $language."
I'm far too used to PHP.

Remember that Bash is not Perl or PHP. You need to be very well aware of how expansion works to avoid big trouble. If you don't, you'll end up creating very dangerous situations in your scripts, especially when making this mistake with rm:

$ ls
no secret  secret
$ file='no secret'
$ rm $file
rm: cannot remove `no': No such file or directory

Imagine we have two files, no secret and secret. The first contains nothing useful, but the second contains the secret that will save the world from impending doom. Unthoughtful as you are, you forgot to quote your parameter expansion of file. Bash expands the parameter and the result is rm no secret. Bash splits the arguments up by their whitespace as it normally does, and rm is passed two arguments: 'no' and 'secret'. As a result, it fails to find the file no and it deletes the file secret. The secret is lost!


  • Good Practice:
    You should always keep parameter expansions properly quoted. This prevents the whitespace or the possible globs inside of them from giving you gray hair or unexpectedly wiping stuff off your computer. The only good PE, is a quoted PE.




  • Variable: A variable is a kind of parameter that you can create and modify directly. It is denoted by a name, which must begin with a letter or underscore (_), and must consist only of letters, digits, and the underscore. Variable names are case-sensitive.

  • Expansion: Expansion happens when a parameter is prefixed by a dollar sign. Bash takes the parameter's value and replaces the parameter's expansion by its value before executing the command. This is also called substitution.


Variable Types

Although Bash is not a typed language, it does have a few different types of variables. These types define the kind of content they are allowed to have. Type information is stored internally by Bash.

  • Array: declare -a variable: The variable is an array of strings.

  • Associative array: declare -A variable: The variable is an associative array of strings (bash 4.0 or higher).

  • Integer: declare -i variable: The variable holds an integer. Assigning values to this variable automatically triggers Arithmetic Evaluation.

  • Read Only: declare -r variable: The variable can no longer be modified or unset.

  • Export: declare -x variable: The variable is marked for export which means it will be inherited by any child process.

Arrays are basically indexed lists of strings. They are very convenient for their ability to store multiple strings together without relying on a delimiter to split them apart (which is tedious when done correctly and error-prone when not).

Defining variables as integers has the advantage that you can leave out some syntax when trying to assign or modify them:

$ a=5; a+=2; echo "$a"; unset a
52
$ a=5; let a+=2; echo "$a"; unset a
7
$ declare -i a=5; a+=2; echo "$a"; unset a
7
$ a=5+2; echo "$a"; unset a
5+2
$ declare -i a=5+2; echo "$a"; unset a
7

However, in practice the use of declare -i is exceedingly rare. In large part, this is because it creates behavior that can be surprising to anyone trying to maintain the script, who misses the declare statement. Most experienced shell scripters prefer to use explicit arithmetic commands (with ((...)) or let) when they want to perform arithmetic.

It is also rare to see an explicit declaration of an array using declare -a. It is sufficient to write array=(...) and Bash will know that the variable is now an array. The exception to this is the associative array, which must be declared explicitly: declare -A myarray.


  • String: A string is a sequence of characters.

  • Array: An array is a list of strings indexed by numbers.

  • Integer: An integer is a whole number (positive, negative or zero).

  • Read Only: Parameters that are read-only cannot be modified or unset.

  • Export: Variables that are marked for export will be inherited by any child process. Variables inherited in this way are called Environment Variables.



Parameter Expansion

Parameter Expansion is the term that refers to any operation that causes a parameter to be expanded (replaced by content). In its most basic appearance, the expansion of a parameter is achieved by prefixing that parameter with a $ sign. In certain situations, additional curly braces around the parameter's name are required:

$ echo "'$USER', '$USERs', '${USER}s'"
'lhunath', '', 'lhunaths'

This example illustrates what basic parameter expansions (PE) look like. The second PE results in an empty string. That's because the parameter USERs is empty. We did not intend to have the s be part of the parameter name. Since there's no way Bash could know you want a literal s appended to the parameter's value, you need to use curly braces to mark the beginning and end of the parameter name. That's what we do in the third PE in our example above.

Parameter Expansion also gives us tricks to modify the string that will be expanded. These operations can be terribly convenient:

$ for file in *.JPG *.jpeg
do mv -- "$file" "${file%.*}.jpg"
done

The code above can be used to rename all JPEG files with a .JPG or a .jpeg extension to have a normal .jpg extension. The expression ${file%.*} cuts off everything from the end starting with the last period (.). Then, in the same quotes, a new extension is appended to the expansion result.

Here's a summary of most of the PE tricks that are available:

Syntax

Description

${parameter:-word}

Use Default Value. If 'parameter' is unset or null, 'word' (which may be an expansion) is substituted. Otherwise, the value of 'parameter' is substituted.

${parameter:=word}

Assign Default Value. If 'parameter' is unset or null, 'word' (which may be an expansion) is assigned to 'parameter'. The value of 'parameter' is then substituted.

${parameter:+word}

Use Alternate Value. If 'parameter' is null or unset, nothing is substituted, otherwise 'word' (which may be an expansion) is substituted.

${parameter:offset:length}

Substring Expansion. Expands to up to 'length' characters of 'parameter' starting at the character specified by 'offset' (0-indexed). If ':length' is omitted, go all the way to the end. If 'offset' is negative (use parentheses!), count backward from the end of 'parameter' instead of forward from the beginning. If 'parameter' is @ or an indexed array name subscripted by @ or *, the result is 'length' positional parameters or members of the array, respectively, starting from 'offset'.

${#parameter}

The length in characters of the value of 'parameter' is substituted.

${parameter#pattern}

The 'pattern' is matched against the beginning of 'parameter'. The result is the expanded value of 'parameter' with the shortest match deleted.

${parameter##pattern}

As above, but the longest match is deleted.

${parameter%pattern}

The 'pattern' is matched against the end of 'parameter'. The result is the expanded value of 'parameter' with the shortest match deleted.

${parameter%%pattern}

As above, but the longest match is deleted.

${parameter/pat/string}

Results in the expanded value of 'parameter' with the first (unanchored) match of 'pat' replaced by 'string'.

${parameter//pat/string}

As above, but every match of 'pat' is replaced.

You will learn them through experience. They come in handy far more often than you think they might. Here are a few examples to kickstart you:

$ file="$HOME/.secrets/007"; \
echo "File location: $file"; \
echo "Filename: ${file##*/}"; \
echo "Directory of file: ${file%/*}"; \
echo "Non-secret file: ${file/secrets/not_secret}"; \
echo; \
echo "Other file location: ${other:-There is no other file}"; \
echo "Using file if there is no other file: ${other:=$file}"; \
echo "Other filename: ${other##*/}"; \
echo "Other file location length: ${#other}"
File location: /home/lhunath/.secrets/007
Filename: 007
Directory of file: /home/lhunath/.secrets
Non-secret file: /home/lhunath/.not_secret/007

Other file location: There is no other file
Using file if there is no other file: /home/lhunath/.secrets/007
Other filename: 007
Other file location length: 26

Remember the difference between ${v#p} and ${v##p}. The doubling of the # character means patterns will become greedy. The same goes for %:

$ version=1.5.9; echo "MAJOR: ${version%%.*}, MINOR: ${version#*.}."
MAJOR: 1, MINOR: 5.9.
$ echo "Dash: ${version/./-}, Dashes: ${version//./-}."
Dash: 1-5.9, Dashes: 1-5-9.

Note: You cannot use multiple PEs together. If you need to execute multiple PEs on a parameter, you will need to use multiple statements:

$ file=$HOME/image.jpg; file=${file##*/}; echo "${file%.*}"
image


  • Good Practice:
    You may be tempted to use external applications such as sed, awk, cut, perl or others to modify your strings. Be aware that all of these require an extra process to be started, which in some cases can cause slowdowns. Parameter Expansions are the perfect alternative.




  • Parameter Expansion: Any expansion (see earlier definition) of a parameter. Certain operations are possible during this expansion that are performed on the value that will be expanded.


<- Special Characters | Patterns ->

BashGuide/Parameters (last edited 2017-04-07 00:32:44 by pool-71-188-87-188)