Differences between revisions 27 and 59 (spanning 32 versions)
Revision 27 as of 2013-08-02 00:51:01
Size: 10644
Editor: 114
Comment:
Revision 59 as of 2022-10-13 13:52:20
Size: 17694
Editor: emanuele6
Comment: mention that `.*' matching `.' and `..' is not a problem in bash 5.2+; minor formatting fixes
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
"Glob" is the common name for a set of Bash features that match or expand specific types of patterns. Some synonyms for globbing (depending on the context in which it appears) are [[http://tiswww.case.edu/php/chet/bash/bashref.html#SEC36|pattern matching]], pattern expansion, filename expansion, and so on. A glob may look like {{{*.txt}}} and, when used to match filenames, is sometimes called a "wildcard". <<TableOfContents>>

= Globs =
"Glob" is the common name for a set of Bash features that match or expand specific types of patterns. Some synonyms for globbing (depending on the context in which it appears) are [[http://tiswww.case.edu/php/chet/bash/bashref.html#Pattern-Matching|pattern matching]], pattern expansion, filename expansion, and so on. A glob may look like {{{*.txt}}} and, when used to match filenames, is sometimes called a "wildcard".
Line 4: Line 7:
Line 12: Line 14:


Line 14: Line 19:
{{{ {{{#!highlight bash
Line 19: Line 24:
Line 22: Line 26:
{{{ {{{#!highlight bash
Line 33: Line 37:
Line 36: Line 39:
A glob that ends in `/` will only match directories (the `/` will be included in the results):

{{{#!highlight bash
touch file1 file2
mkdir dir1 dir2

printf '<%s>' *; printf '\n'
# <dir1><dir2><file1><file2>
printf '<%s>' */; printf '\n'
# <dir1/><dir2/>
}}}
Line 38: Line 53:
{{{ {{{#!highlight bash
Line 45: Line 60:
Line 50: Line 64:
{{{ {{{#!highlight bash
Line 53: Line 67:
Line 56: Line 69:
{{{ {{{#!highlight bash
Line 63: Line 76:

(Reference: [[BashGuide/Arrays|Arrays]] [[Quotes]] [[http://bash-hackers.org/wiki/doku.php/commands/builtin/printf|printf]].)
(Reference: [[BashGuide/Arrays|Arrays]] [[Quotes]] [[http://wiki.bash-hackers.org/commands/builtin/printf|printf]].)
Line 67: Line 79:
Line 69: Line 80:

||`[abcd]`        ||Matches `a` or `b` or `c` or `d`||
||`[a-d]`         ||The same as above, if your [[locale]] is C or POSIX.  Otherwise, implementation-defined.||
||`[!aeiouAEIOU]` ||Matches any character ''except'' `a`, `e`, `i`, `o`, `u` and their uppercase counterparts||
||`[[:alnum:]]`   ||Matches any alphanumeric character in the current locale (letter or number)||
||`[[:space:]]`   ||Matches any whitespace character||
||`[![:space:]]`  ||Matches any character that is ''not'' whitespace||
||`[[:digit:]_.]` ||Matches any digit, or `_` or `.`||

''Implementation-defined'' means it may work as you expect on one machine, but give completely different results on another machine. Do not use the `m-n` syntax unless you have explicitly set your locale to C first, or you may get unexpected results. The [[http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap09.html#tag_09_03_05|POSIX character class expressions]] should be preferred whenever possible.
||`[abcd]` ||Matches `a` or `b` or `c` or `d` ||
||`[a-d]` ||The same as above, if ''globasciiranges'' is set or your [[locale]] is C or POSIX. Otherwise, implementation-defined. ||
||`[!aeiouAEIOU]` ||Matches any character ''except'' `a`, `e`, `i`, `o`, `u` and their uppercase counterparts ||
||`[[:alnum:]]` ||Matches any alphanumeric character in the current locale (letter or number) ||
||`[[:space:]]` ||Matches any whitespace character ||
||`[![:space:]]` ||Matches any character that is ''not'' whitespace ||
||`[[:digit:]_.]` ||Matches any digit, or `_` or `.` ||




In most shell implementations, one may also use `^` as the range negation character, e.g. `[^[:space:]]`. However, POSIX specifies `!` for this role, and therefore `!` is the standard choice.

=== globasciiranges (since bash 4.3-alpha) ===
Interprets [a-d] as [abcd]. To match a literal -, include it as first or last character.

=== For older versions ===
Note that
''Implementation-defined'' means it may work as you expect on one machine, but give completely different results on another machine. Do not use the `m-n` syntax unless you have explicitly set your locale to C first, or you may get unexpected results. The [[http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap09.html#tag_09_03_05|POSIX character class expressions]] should be preferred whenever possible.
Line 81: Line 100:
Line 83: Line 101:
Line 86: Line 103:
{{{ {{{#!highlight bash
Line 89: Line 106:
Line 102: Line 118:
{{{ {{{#!highlight bash
Line 108: Line 124:

{{{
{{{#!highlight bash
Line 113: Line 128:
Line 116: Line 130:
{{{ {{{#!highlight bash
Line 120: Line 134:
Line 123: Line 136:
{{{
[[ $fruit = @(ba*(na)|a+(p)le) ]] && echo 'Nice fruit'
}}}

Because the `extglob` option changes the way certain characters are parsed, it is necessary to have a newline (not just a semicolon) between the `shopt` command and any subsequent commands that use extended globs. Likewise, you cannot put `shopt -s extglob` inside a statement block that uses extended globs, because the block as a whole must be parsed when it's defined; the `shopt` command won't take effect until the block is ''evaluated'', at which point it's too late. In fact as bash parses the entire statement block before evaluating any of it, you need to set extglob outside of the outermost block.

Therefore, if you use this option in a script, it's best to put it right under the shebang line, or as close as you can get it while still making your boss happy.

{{{#!highlight text numbers=disable
{{{#!highlight bash
[[ $fruit = @(ba*(na)|a+(p)le) ]] && echo "Nice fruit"
}}}
'''`extglob` changes the way certain characters are parsed. It is necessary to have a newline (not just a semicolon) between `shopt -s extglob` and any subsequent commands to use it.''' You cannot enable extended globs inside a [[BashGuide/CompoundCommands#Command.2BAFw_Grouping|group command]] that uses them, because the entire block is parsed before the `shopt` is ''evaluated''. Note that the typical [[BashGuide/CompoundCommands#Functions|function]] body ''is'' a ''group command''. An unpleasant workaround could be to use a ''subshell command list'' as the function body.

Therefore, if you use this option in a script, it is best put right under the shebang line.

{{{#!highlight bash
Line 133: Line 145:
# Copyright (c) 2012 Foo Corporation
Line 136: Line 147:

If your code isn't a ''script'', but is instead being sourced, and must set `extglob` itself:
{{{
SourcedFile.sh
    if ! shopt extglob; then
      ClearExtGlob_SourcedFile_sh=1
      shopt -s extglob
    fi
   # The basic Concept behind the following options is to delay parsing of the
   # extglob until evaluation.

   declare -a s='( !(x) )'
   echo "${s[@]}"

   echo "${InvalidVar:-!(x)}"

   eval 'echo !(x)' # using eval if no other option.

   if [ "${ClearExtGlob_SourcedFile_sh}" == "1" ]; then
     unset ClearExtGlob_SourcedFile_sh
     shopt -u extglob
   fi

test.sh
   shopt -u extglob
   if true; then
     source ./SourcedFile.sh
   fi
}}}
If your code must be sourced and needs `extglob`, ensure it preserves the original setting from your shell:

{{{#!highlight bash
# remember whether extglob was originally set, so we know whether to unset it
shopt -q extglob; extglob_set=$?
# set extglob if it wasn't originally set.
((extglob_set)) && shopt -s extglob
# Note, 0 (true) from shopt -q is "false" in a math context.

# The basic concept behind the following is to delay parsing of the globs until evaluation.
# This matters at group commands, such as functions in { } blocks

declare -a s='( !(x) )'
echo "${s[@]}"

echo "${InvalidVar:-!(x)}"

eval 'echo !(x)' # using eval if no other option.

# unset extglob if it wasn't originally set
((extglob_set)) && shopt -u extglob
}}}
This should also apply for other shell options.
Line 167: Line 172:

If a glob fails to match any filenames, the shell normally leaves it alone. This means the raw glob will be passed on to the command, as in:

{{{
$ ls *.ttx
ls: cannot access *.ttx: No such file or directory
}}}

This allows the command to see the glob you used, and to use it in an error message. If the Bash option '''nullglob''' is set, however, a glob which matches no files will be removed entirely. This is [[BashFAQ/004|useful in scripts]], but somewhat confusing at the command line, since it "breaks" the expectations of many of the standard tools (see failglob below for a better alternative):

{{{
# Good in scripts!
shopt -s nullglob
oggs=(*.ogg)
for ogg in "${oggs[@]}"; do ...

# Bad at the command line!
shopt -s nullglob
ls *.ttx
`nullglob` expands non-matching globs to zero arguments, rather than to themselves.

{{{#!highlight bash
$ ls *.c
ls: cannot access *.c: No such file or directory

# with nullglob set
shopt -s nullglob
ls *.c
Line 188: Line 183:

=== failglob ===

If a pattern fails to match, bash reports an expansion error. This can be useful at the commandline:

{{{
# Good at the command line!
$ touch *.foo # creates file '*.foo' if glob fails to match
$ shopt -s failglob
$ touch *.foo # touch doesn't get executed
-bash: no match: *.foo
}}}
Typically, `nullglob` is used to [[BashFAQ/004|count the number of files]] matching a pattern:

{{{#!highlight bash
shopt -s nullglob
files=(*)
echo "There are ${#files[@]} files in this directory."
}}}
Without `nullglob`, the glob would expand to a literal `*` in an empty directory, resulting in an erroneous count of 1.

It can also be used to iterate the matches of a glob with a `for` loop.

{{{#!highlight bash
shopt -s nullglob
for f in ./*.mp4; do
    command1 "$f"
    command2 "$f"
done
}}}
Without `nullglob`, if there were no matches for {{{./*.mp4}}}, the loop would be still run once with {{{f='./*.mp4'}}}.

==== Warning ====
Enabling `nullglob` on a wide scope can trigger bugs caused by bad programming practices. It "breaks" the expectations of many utilities.

Removing array elements:

{{{#!highlight bash
shopt -s nullglob
unset array[1]
#unsets nothing
unset -v "array[1]"
#correct
}}}
Array member assignments in compound form using subscripts:

{{{#!highlight bash
shopt -s nullglob
array=([1]=*)
#results in an empty array
}}}
This was reported as a [[http://lists.gnu.org/archive/html/bug-bash/2012-08/msg00032.html|bug]] in 2012, yet is unchanged to this day.

  EDIT: It has been changed in bash4.3; now, it results in an array that only contains {{{array[1]='*'}}}.

Apart from few builtins that use modified parsing under special conditions (e.g. declare) '''always use [[Quotes]]''' when arguments to simple commands could be interpreted as globs.

Enabling `failglob`, `nullglob`, or both during development and testing can help catch mistakes early. To prevent ''pathname expansion'' occuring in unintended places, you can set [[#failglob|failglob]]. However, you must then guarantee all intended globs match at least one file. Also note that the result of a glob expansion does not always differ from the glob itself. `failglob` won't distinguish `echo ?` from `echo '?'` in a directory containing only a file named `?`. `nullglob` will.

==== Portability ====
"null globbing" is not specified by POSIX. In portable scripts, you must explicitly check that a glob match was successful by checking that the files actually exist.

{{{#!highlight bash
# POSIX

for x in *; do
    [ -e "$x" ] || break
    ...
done

f() {
    [ -e "$1" ] || return 1

    for x do
        ...
    done
}

f * || echo "No files found"
}}}
Some modern POSIX-compatible shells allow null globbing as an extension.

{{{#!highlight bash
# Bash
shopt -s nullglob
}}}
In ksh93, there is no toggle-able option. Rather, that the "nullglob" behavior is to be enabled is specified inline using the "N" option to the `∼()` sub-pattern syntax.

{{{#!highlight bash
# ksh93

for x in ~(N)*; do
    ...
done
}}}
In zsh, a toggle-able option (NULL_GLOB) or a glob qualifier(N) can be used.

{{{#!highlight bash
# zsh
for x in *(N); do ...; done # or setopt NULL_GLOB
}}}
mksh doesn't yet support nullglob (maintainer says he'll think about it).
Line 202: Line 275:
Line 205: Line 277:
{{{ {{{#!highlight bash
Line 210: Line 282:

It should be noted that when `dotglob` is enabled, `*` will match files like `.bashrc` but ''not'' the `.` or `..` directories. This is orthogonal to the problem of matching "just the dot files" -- a glob of `.*` ''will'' match `.` and `..`, typically causing problems. See next section.
It should be noted that when `dotglob` is enabled, `*` will match files like `.bashrc` but ''not'' the `.` or `..` directories. This is orthogonal to the problem of matching "just the dot files" (prior to bash 5.2; see '''globskipdots''' below) -- a glob of `.*` ''will'' match `.` and `..`, typically causing problems.

=== globskipdots (since bash 5.2) ===
When '''globskipdots''' is on, pathname expansion will never expand to `.` or `..`.

{{{#!highlight bash
LC_COLLATE=C
shopt -s extglob nullglob

touch .a ..b ...c ....d ......

# with globskipdots on
declare -p files=( .* ) # filenames that start with "."
# declare -a files=([0]="......" [1]="....d" [2]="...c" [3]="..b" [4]=".a")
declare -p files=( ..* ) # filenames that start with ".."
# declare -a files=([0]="......" [1]="....d" [2]="...c" [3]="..b")
declare -p files=( +(..)?([!.]*) ) # filenames that start with an even number of "."
# declare -a files=([0]="......" [1]="....d" [2]="..b")

# with globskipdots off
shopt -u globskipdots
declare -p files=( .* ) # filenames that start with "."
# declare -a files=([0]="." [1]=".." [2]="......" [3]="....d" [4]="...c" [5]="..b" [6]=".a")
declare -p files=( ..* ) # filenames that start with ".."
# declare -a files=([0]=".." [1]="......" [2]="....d" [3]="...c" [4]="..b")
declare -p files=( +(..)?([!.]*) ) # filenames that start with an even number of "."
# declare -a files=([0]=".." [1]="......" [2]="....d" [3]="..b")
}}}

This option is enabled by default; before bash 5.2, pathname expansion behaved as if '''globskipdots''' was disabled.

=== globstar (since bash 4.0-alpha) ===
globstar recursively repeats a pattern containing `**`.

{{{#!highlight bash
$ shopt -s globstar; tree
.
├── directory2
│ ├── directory3
│ ├── file1.c
│ └── file2
├── file1
└── file2.c

# Suppose that for the following examples.
}}}
Matching files:

{{{#!highlight bash
$ files=(**)
# equivalent to: files=(* */* */*/*)
# finds all files recursively

$ files=(**/*.c)
# equivalent to: files=(*.c */*.c */*/*.c)
# finds all *.c files recursively
# corresponds to: find -name "*.c"
# Caveat: **.c will not work, it is equivalent to *.c
}}}
Just like '*', '**' followed by a '/' will only match directories:

{{{#!highlight bash
$ files=(**/)
# finds all subdirectories

$ files=(. **/)
# finds all subdirectories, including the current directory
# corresponds to: find -type d
}}}
=== failglob ===
If a pattern fails to match, bash reports an expansion error. This can be useful at the commandline:

{{{#!highlight bash
# Good at the command line!
$ > *.foo # creates file '*.foo' if glob fails to match
$ shopt -s failglob
$ > *.foo # doesn't get executed
-bash: no match: *.foo
}}}
Line 214: Line 362:
Line 217: Line 364:
{{{ {{{#!highlight bash
Line 224: Line 371:
Unset GLOBIGNORE

{{{#!highlight bash
$ GLOBIGNORE=
$ echo .*
. .. .bash_history .bash_logout .bashrc .inputrc .vimrc
}}}
=== nocasematch ===
Globs inside [[ and case commands are matched case-insensitive:

{{{#!highlight bash
foo() {
   local f r=0 nc=0
   shopt -q nocasematch && nc=1 || shopt -s nocasematch
   for f; do
      [[ $f = *.@(txt|jpg) ]] || continue
      cmd -on "$f" || r=1
   done
   ((nc)) || shopt -u nocasematch
   return $r
}
}}}
This is conventionally done this way:

{{{#!highlight bash
case $f in
    *.[Tt][Xx][Tt]|*.[Jj][Pp][Gg]) : ;;
    *) continue
esac
}}}
and in earlier versions of bash we'd use a similar glob:

{{{#!highlight bash
[[ $f = *.@([Tt][Xx][Tt]|[Jj][Pp][Gg]) ]] || continue
}}}
or with no extglob:

{{{#!highlight bash
[[ $f = *.[Tt][Xx][Tt] ]] || [[ $f = *.[Jj][Pp][Gg] ]] || continue
}}}
Here, one might keep the tests separate for maintenance; they can be easily reused and dropped,

 . without having to concern oneself with where they fit in relation to an internal ||.

Note also:

{{{#!highlight bash
[[ $f = *.@([Tt][Xx][Tt]|[Jj][Pp]?([Ee])[Gg]) ]]
}}}
Variants left as an exercise.

=== nocaseglob (since bash 2.02-alpha1) ===
This option makes ''pathname expansion'' case-insensitive. In contrast, [[#nocasematch|nocasematch]] operates on matches in ''[[BashGuide/TestsAndConditionals#Conditional_Blocks|[[]]'' and ''[[BashGuide/TestsAndConditionals#Choices|case]]'' commands.

Globs

"Glob" is the common name for a set of Bash features that match or expand specific types of patterns. Some synonyms for globbing (depending on the context in which it appears) are pattern matching, pattern expansion, filename expansion, and so on. A glob may look like *.txt and, when used to match filenames, is sometimes called a "wildcard".

Traditional shell globs use a very simple syntax, which is less expressive than a RegularExpression. Most characters in a glob are treated literally, but a * matches 0 or more characters, a ? matches precisely one character, and [...] matches any single character in a specified set (see Ranges below). All globs are implicitly anchored at both start and end. For example:

*

Matches any string, of any length

foo*

Matches any string beginning with foo

*x*

Matches any string containing an x (beginning, middle or end)

*.tar.gz

Matches any string ending with .tar.gz

*.[ch]

Matches any string ending with .c or .h

foo?

Matches foot or foo$ but not fools

Bash expands globs which appear unquoted in commands, by matching filenames relative to the current directory. The expansion of the glob results in 1 or more words (0 or more, if certain options are set), and those words (filenames) are used in the command. For example:

   1 tar xvf *.tar
   2 # Expands to: tar xvf file1.tar file2.tar file42.tar ...
   3 # (which is generally not what one wants)

Even if a file contains internal whitespace, the expansion of a glob that matches that file will still preserve each filename as a single word. For example,

   1 # This is safe even if a filename contains whitespace:
   2 for f in *.tar; do
   3     tar tvf "$f"
   4 done
   5 
   6 # But this one is not:
   7 for f in $(ls | grep '\.tar$'); do
   8     tar tvf "$f"
   9 done

In the second example above, the output of ls is filtered, and then the result of the whole pipeline is divided into words, to serve as iterative values for the loop. This word-splitting will occur at internal whitespace within each filename, which makes it useless in the general case. The first example has no such problem, because the filenames produced by the glob do not undergo any further word-splitting. For more such examples, see BashPitfalls.

A glob that ends in / will only match directories (the / will be included in the results):

   1 touch file1 file2
   2 mkdir dir1 dir2
   3 
   4 printf '<%s>' *; printf '\n'
   5 # <dir1><dir2><file1><file2>
   6 printf '<%s>' */; printf '\n'
   7 # <dir1/><dir2/>

Globs are also used to match patterns in a few places in Bash. The most traditional is in the case command:

   1 case "$input" in
   2     [Yy]|'') confirm=1;;
   3     [Nn]*) confirm=0;;
   4     *) echo "I don't understand.  Please try again.";;
   5 esac

Patterns (which are separated by | characters) are matched against the first word after the case itself. The first pattern which matches, "wins", causing the corresponding commands to be executed.

Bash also allows globs to appear on the right-hand side of a comparison inside a [[ command:

   1 if [[ $output = *[Ee]rror* ]]; then ...

Finally, globs are used during parameter expansion to indicate patterns which may be stripped out, or replaced, during a substitution. Simple examples (there are many more on the previously referenced page):

   1 filename=${path##*/}    # strip leading pattern that matches */ (be greedy)
   2 dirname=${path%/*}      # strip trailing pattern matching /* (non-greedy)
   3 
   4 printf '%s\n' "${arr[@]}"          # dump an array, one element per line
   5 printf '%s\n' "${arr[@]/error*/}"  # dump array, removing error* if matched

(Reference: Arrays Quotes printf.)

Ranges

Globs can specify a range or class of characters, using square brackets. This gives you the ability to match against a set of characters. For example:

[abcd]

Matches a or b or c or d

[a-d]

The same as above, if globasciiranges is set or your locale is C or POSIX. Otherwise, implementation-defined.

[!aeiouAEIOU]

Matches any character except a, e, i, o, u and their uppercase counterparts

[[:alnum:]]

Matches any alphanumeric character in the current locale (letter or number)

[[:space:]]

Matches any whitespace character

[![:space:]]

Matches any character that is not whitespace

[[:digit:]_.]

Matches any digit, or _ or .

In most shell implementations, one may also use ^ as the range negation character, e.g. [^[:space:]]. However, POSIX specifies ! for this role, and therefore ! is the standard choice.

globasciiranges (since bash 4.3-alpha)

Interprets [a-d] as [abcd]. To match a literal -, include it as first or last character.

For older versions

Note that Implementation-defined means it may work as you expect on one machine, but give completely different results on another machine. Do not use the m-n syntax unless you have explicitly set your locale to C first, or you may get unexpected results. The POSIX character class expressions should be preferred whenever possible.

Options which change globbing behavior

extglob

In addition to the traditional globs (supported by all Bourne-family shells) that we've seen so far, Bash (and Korn Shell) offers extended globs, which have the expressive power of regular expressions. Korn shell enables these by default; in Bash, you must run the command

   1 shopt -s extglob

in your shell (or at the start of your script -- see note on parsing below) to use them. The pattern matching reference describes the syntax, which is reproduced here:

?(pattern-list)
Matches zero or one occurrence of the given patterns.
*(pattern-list)
Matches zero or more occurrences of the given patterns.
+(pattern-list)
Matches one or more occurrences of the given patterns.
@(pattern-list)
Matches one of the given patterns.
!(pattern-list)
Matches anything except one of the given patterns.

Patterns in a list are separated by | characters.

Extended globs allow you to solve a number of problems which otherwise require a rather surprising amount of ugly hacking; for example,

   1 # To remove all the files except ones matching *.jpg:
   2 rm !(*.jpg)
   3 # All except *.jpg and *.gif and *.png:
   4 rm !(*.jpg|*.gif|*.png)

   1 # To copy all the MP3 songs except one to your device
   2 cp !(04*).mp3 /mnt

To use an extglob in a parameter expansion (this can also be done in one BASH statement with read):

   1 # To trim leading and trailing whitespace from a variable
   2 x=${x##+([[:space:]])}; x=${x%%+([[:space:]])}

Extended glob patterns can be nested, too.

   1 [[ $fruit = @(ba*(na)|a+(p)le) ]] && echo "Nice fruit"

extglob changes the way certain characters are parsed. It is necessary to have a newline (not just a semicolon) between shopt -s extglob and any subsequent commands to use it. You cannot enable extended globs inside a group command that uses them, because the entire block is parsed before the shopt is evaluated. Note that the typical function body is a group command. An unpleasant workaround could be to use a subshell command list as the function body.

Therefore, if you use this option in a script, it is best put right under the shebang line.

   1 #!/usr/bin/env bash
   2 shopt -s extglob   # and others, such as nullglob dotglob

If your code must be sourced and needs extglob, ensure it preserves the original setting from your shell:

   1 # remember whether extglob was originally set, so we know whether to unset it
   2 shopt -q extglob; extglob_set=$?
   3 # set extglob if it wasn't originally set.
   4 ((extglob_set)) && shopt -s extglob
   5 # Note, 0 (true) from shopt -q is "false" in a math context.
   6 
   7 # The basic concept behind the following is to delay parsing of the globs until evaluation.
   8 # This matters at group commands, such as functions in { } blocks
   9 
  10 declare -a s='( !(x) )'
  11 echo "${s[@]}"
  12 
  13 echo "${InvalidVar:-!(x)}"
  14 
  15 eval 'echo !(x)'  # using eval if no other option.
  16 
  17 # unset extglob if it wasn't originally set
  18 ((extglob_set)) && shopt -u extglob

This should also apply for other shell options.

nullglob

nullglob expands non-matching globs to zero arguments, rather than to themselves.

   1 $ ls *.c
   2 ls: cannot access *.c: No such file or directory
   3 
   4 # with nullglob set
   5 shopt -s nullglob
   6 ls *.c
   7 # Runs "ls" with no arguments, and lists EVERYTHING

Typically, nullglob is used to count the number of files matching a pattern:

   1 shopt -s nullglob
   2 files=(*)
   3 echo "There are ${#files[@]} files in this directory."

Without nullglob, the glob would expand to a literal * in an empty directory, resulting in an erroneous count of 1.

It can also be used to iterate the matches of a glob with a for loop.

   1 shopt -s nullglob
   2 for f in ./*.mp4; do
   3     command1 "$f"
   4     command2 "$f"
   5 done

Without nullglob, if there were no matches for ./*.mp4, the loop would be still run once with f='./*.mp4'.

Warning

Enabling nullglob on a wide scope can trigger bugs caused by bad programming practices. It "breaks" the expectations of many utilities.

Removing array elements:

   1 shopt -s nullglob
   2 unset array[1]
   3 #unsets nothing
   4 unset -v "array[1]"
   5 #correct

Array member assignments in compound form using subscripts:

   1 shopt -s nullglob
   2 array=([1]=*)
   3 #results in an empty array

This was reported as a bug in 2012, yet is unchanged to this day.

  • EDIT: It has been changed in bash4.3; now, it results in an array that only contains array[1]='*'.

Apart from few builtins that use modified parsing under special conditions (e.g. declare) always use Quotes when arguments to simple commands could be interpreted as globs.

Enabling failglob, nullglob, or both during development and testing can help catch mistakes early. To prevent pathname expansion occuring in unintended places, you can set failglob. However, you must then guarantee all intended globs match at least one file. Also note that the result of a glob expansion does not always differ from the glob itself. failglob won't distinguish echo ? from echo '?' in a directory containing only a file named ?. nullglob will.

Portability

"null globbing" is not specified by POSIX. In portable scripts, you must explicitly check that a glob match was successful by checking that the files actually exist.

   1 # POSIX
   2 
   3 for x in *; do
   4     [ -e "$x" ] || break
   5     ...
   6 done
   7 
   8 f() {
   9     [ -e "$1" ] || return 1
  10 
  11     for x do
  12         ...
  13     done
  14 }
  15 
  16 f * || echo "No files found"

Some modern POSIX-compatible shells allow null globbing as an extension.

   1 # Bash
   2 shopt -s nullglob

In ksh93, there is no toggle-able option. Rather, that the "nullglob" behavior is to be enabled is specified inline using the "N" option to the ∼() sub-pattern syntax.

   1 # ksh93
   2 
   3 for x in ~(N)*; do
   4     ...
   5 done

In zsh, a toggle-able option (NULL_GLOB) or a glob qualifier(N) can be used.

   1 # zsh
   2 for x in *(N); do ...; done # or setopt NULL_GLOB

mksh doesn't yet support nullglob (maintainer says he'll think about it).

dotglob

By convention, a filename beginning with a dot is "hidden", and not shown by ls. Globbing uses the same convention -- filenames beginning with a dot are not matched by a glob, unless the glob also begins with a dot. Bash has a dotglob option that lets globs match "dot files":

   1 shopt -s dotglob nullglob
   2 files=(*)
   3 echo "There are ${#files[@]} files here, including dot files and subdirs"

It should be noted that when dotglob is enabled, * will match files like .bashrc but not the . or .. directories. This is orthogonal to the problem of matching "just the dot files" (prior to bash 5.2; see globskipdots below) -- a glob of .* will match . and .., typically causing problems.

globskipdots (since bash 5.2)

When globskipdots is on, pathname expansion will never expand to . or ...

   1 LC_COLLATE=C
   2 shopt -s extglob nullglob
   3 
   4 touch .a ..b ...c ....d ......
   5 
   6 # with globskipdots on
   7 declare -p files=( .* )            # filenames that start with "."
   8 # declare -a files=([0]="......" [1]="....d" [2]="...c" [3]="..b" [4]=".a")
   9 declare -p files=( ..* )           # filenames that start with ".."
  10 # declare -a files=([0]="......" [1]="....d" [2]="...c" [3]="..b")
  11 declare -p files=( +(..)?([!.]*) ) # filenames that start with an even number of "."
  12 # declare -a files=([0]="......" [1]="....d" [2]="..b")
  13 
  14 # with globskipdots off
  15 shopt -u globskipdots
  16 declare -p files=( .* )            # filenames that start with "."
  17 # declare -a files=([0]="." [1]=".." [2]="......" [3]="....d" [4]="...c" [5]="..b" [6]=".a")
  18 declare -p files=( ..* )           # filenames that start with ".."
  19 # declare -a files=([0]=".." [1]="......" [2]="....d" [3]="...c" [4]="..b")
  20 declare -p files=( +(..)?([!.]*) ) # filenames that start with an even number of "."
  21 # declare -a files=([0]=".." [1]="......" [2]="....d" [3]="..b")

This option is enabled by default; before bash 5.2, pathname expansion behaved as if globskipdots was disabled.

globstar (since bash 4.0-alpha)

globstar recursively repeats a pattern containing **.

   1 $ shopt -s globstar; tree
   2 .
   3 ├── directory2
   4 │   ├── directory3
   5 │   ├── file1.c
   6 │   └── file2
   7 ├── file1
   8 └── file2.c
   9 
  10 # Suppose that for the following examples.

Matching files:

   1 $ files=(**)
   2 # equivalent to: files=(* */* */*/*)
   3 # finds all files recursively
   4 
   5 $ files=(**/*.c)
   6 # equivalent to: files=(*.c */*.c */*/*.c)
   7 # finds all *.c files recursively
   8 # corresponds to: find -name "*.c"
   9 # Caveat: **.c will not work, it is equivalent to *.c

Just like '*', '**' followed by a '/' will only match directories:

   1 $ files=(**/)
   2 # finds all subdirectories
   3 
   4 $ files=(. **/)
   5 # finds all subdirectories, including the current directory
   6 # corresponds to: find -type d

failglob

If a pattern fails to match, bash reports an expansion error. This can be useful at the commandline:

   1 # Good at the command line!
   2 $ > *.foo # creates file '*.foo' if glob fails to match
   3 $ shopt -s failglob
   4 $ > *.foo # doesn't get executed
   5 -bash: no match: *.foo

GLOBIGNORE

The Bash variable (not shopt) GLOBIGNORE allows you to specify patterns a glob should not match. This lets you work around the infamous "I want to match all of my dot files, but not . or .." problem:

   1 $ echo .*
   2 . .. .bash_history .bash_logout .bashrc .inputrc .vimrc
   3 $ GLOBIGNORE=.:..
   4 $ echo .*
   5 .bash_history .bash_logout .bashrc .inputrc .vimrc

Unset GLOBIGNORE

   1 $ GLOBIGNORE=
   2 $ echo .*
   3 . .. .bash_history .bash_logout .bashrc .inputrc .vimrc

nocasematch

Globs inside [[ and case commands are matched case-insensitive:

   1 foo() {
   2    local f r=0 nc=0
   3    shopt -q nocasematch && nc=1 || shopt -s nocasematch
   4    for f; do
   5       [[ $f = *.@(txt|jpg) ]] || continue
   6       cmd -on "$f" || r=1
   7    done
   8    ((nc)) || shopt -u nocasematch
   9    return $r
  10 }

This is conventionally done this way:

   1 case $f in
   2     *.[Tt][Xx][Tt]|*.[Jj][Pp][Gg]) : ;;
   3     *) continue
   4 esac

and in earlier versions of bash we'd use a similar glob:

   1 [[ $f = *.@([Tt][Xx][Tt]|[Jj][Pp][Gg]) ]] || continue

or with no extglob:

   1 [[ $f = *.[Tt][Xx][Tt] ]] || [[ $f = *.[Jj][Pp][Gg] ]] || continue

Here, one might keep the tests separate for maintenance; they can be easily reused and dropped,

  • without having to concern oneself with where they fit in relation to an internal ||.

Note also:

   1 [[ $f = *.@([Tt][Xx][Tt]|[Jj][Pp]?([Ee])[Gg]) ]]

Variants left as an exercise.

nocaseglob (since bash 2.02-alpha1)

This option makes pathname expansion case-insensitive. In contrast, nocasematch operates on matches in [[ and case commands.


CategoryShell

glob (last edited 2022-10-13 13:52:20 by emanuele6)