Differences between revisions 10 and 11
Revision 10 as of 2011-03-24 16:56:26
Size: 7538
Editor: 174
Comment:
Revision 11 as of 2012-01-05 11:01:24
Size: 8371
Editor: Lhunath
Comment: re-organise a little
Deletions are marked like this. Additions are marked like this.
Line 29: Line 29:
Also notice how `ls` sometimes garbles your filename data (in our case, it turned the newline character in between the words "`a`" and "`newline`" into a question mark). Generally speaking, though, it doesn't do this when its output isn't a terminal (which is why it didn't do it for the first example which piped `ls` into `cat`). All in all, you really can't and shouldn't trust the output of `ls` to be a true representation of the filenames that you want to work with. So don't. Also notice how `ls` sometimes garbles your filename data (in our case, it turned the newline character in between the words "`a`" and "`newline`" into a question mark. Some systems put a `\n` instead.). On some systems it doesn't do this when its output isn't a terminal, on others it always mangles the filename. All in all, you really can't and shouldn't trust the output of `ls` to be a true representation of the filenames that you want to work with. So don't.
Line 33: Line 33:
As mentioned previously, sometimes people are trying to use `ls` to get some specific piece of metadata, either for a single file, or for multiple files. With a single file, it's actually not too bad: == Enumerating files or doing stuff with files ==
Line 35: Line 35:
{{{
read _ _ owner _ < <(ls -l "$file")
}}}

(See [[BashFAQ/024|Bash FAQ 24]] for the rationale behind the ProcessSubstitution there.)

This is a little bit messier than non-portable means of acquiring the same data (like the `stat(1)` command), but even here, we still run into problems. The way `ls` reports timestamps, for example, is an unholy disaster.

{{{
# Debian unstable:
$ ls -l
-rw-r--r-- 1 wooledg wooledg 240 2007-12-07 11:44 file1
-rw-r--r-- 1 wooledg wooledg 1354 2009-03-13 12:10 file2

# OpenBSD 4.4:
$ ls -l
-rwxr-xr-x 1 greg greg 1080 Nov 10 2006 file1
-rw-r--r-- 1 greg greg 1020 Mar 15 13:57 file2
}}}

On OpenBSD, as on most versions of Unix, `ls` shows the timestamps in three fields -- month, day, and year-or-time, with the last field being the time (hours:minutes) if the file is less than 6 months old, or the year if the file is more than 6 months old. On Debian unstable, with a fairly recent version of GNU coreutils, `ls` shows the timestamps in two fields, with the first being Y-M-D and the second being H:M, no matter how old the file is. So, it should be pretty obvious we ''never'' want to have to parse the output of `ls` if we want a timestamp from a file. But for the fields before that, it's ''usually'' pretty reliable.

(Note: some versions of `ls` don't print the group ownership of a file by default, and require a `-g` flag to do so. Others print the group by default, and `-g` suppresses it. You've been warned.)

However, if we wanted to get metadata from ''more than one file'' in the same `ls` command, we run into the same problem we had before -- files can have newlines in their names, which screws up our output. Imagine how code like this would break if we have a file with a newline in its name:

{{{
# Don't do this
{ read 'perms[1]' 'links[1]' 'owner[1]' 'group[1]' _
  read 'perms[2]' 'links[2]' 'owner[2]' 'group[2]' _
} < <(ls -l "$file1" "$file2")
}}}

Similar code that uses two separate `ls` calls would probably be OK, since the second `read` command would be guaranteed to start reading at the beginning of an `ls` command's output, instead of possibly in the middle of a filename.

See [[BashFAQ/087|Bash FAQ 87]] for some ways of getting file metadata without parsing `ls` output at all.

Now, the bigger problem is when people try to use `ls` to get a list of filenames (either all files, or files that match a [[glob]], or files sorted in some way). This is where things fail disastrously.
When people try to use `ls` to get a list of filenames (either all files, or files that match a [[glob]], or files sorted in some way) things fail disastrously.
Line 121: Line 84:

== Getting metadata on a file ==

If you're after the file's size, use `wc` instead:

{{{
read size _ < <(wc -c "$file")
}}}

(See [[BashFAQ/024|Bash FAQ 24]] for the rationale behind the ProcessSubstitution there.)

Other metadata is often hard to get at in a portable way. `stat` is not available on every platform, and when it is, it often takes a completely different syntax of arguments. There is no way to use `stat` in a way that it won't break for the next POSIX system you run the script on.

You can try to parse certain metadata out of `ls -l`'s output. Two big warnings, though: run `ls` with only '''one file at a time''' (remember, you can't reliably know where to start parsing metadata of the second file since there's no good delimiter - and no, a newline is not a good delimiter) and '''don't parse the timestamp or beyond''' (the time stamp is usually formatted in a very platform and locale-dependent manner and can thus not be parsed reliably).

{{{
read mode links owner _ < <(ls -l "$file")
}}}

Note that the mode string is also often platform-specific. Eg. OS X adds an @ for files with xattrs and a + for files with extended security information.

In case you don't believe us, here's why not to try to parse the timestamp:

{{{
# Debian unstable:
$ ls -l
-rw-r--r-- 1 wooledg wooledg 240 2007-12-07 11:44 file1
-rw-r--r-- 1 wooledg wooledg 1354 2009-03-13 12:10 file2

# OpenBSD 4.4:
$ ls -l
-rwxr-xr-x 1 greg greg 1080 Nov 10 2006 file1
-rw-r--r-- 1 greg greg 1020 Mar 15 13:57 file2
}}}

On OpenBSD, as on most versions of Unix, `ls` shows the timestamps in three fields -- month, day, and year-or-time, with the last field being the time (hours:minutes) if the file is less than 6 months old, or the year if the file is more than 6 months old. On Debian unstable, with a fairly recent version of GNU coreutils, `ls` shows the timestamps in two fields, with the first being Y-M-D and the second being H:M, no matter how old the file is. So, it should be pretty obvious we ''never'' want to have to parse the output of `ls` if we want a timestamp from a file. But for the fields before that, it's ''usually'' pretty reliable.

(Note: some versions of `ls` don't print the group ownership of a file by default, and require a `-g` flag to do so. Others print the group by default, and `-g` suppresses it. You've been warned.)

If we wanted to get metadata from ''more than one file'' in the same `ls` command, we run into the same problem we had before -- files can have newlines in their names, which screws up our output. Imagine how code like this would break if we have a file with a newline in its name:

{{{
# Don't do this
{ read 'perms[1]' 'links[1]' 'owner[1]' 'group[1]' _
  read 'perms[2]' 'links[2]' 'owner[2]' 'group[2]' _
} < <(ls -l "$file1" "$file2")
}}}

Similar code that uses two separate `ls` calls would probably be OK, since the second `read` command would be guaranteed to start reading at the beginning of an `ls` command's output, instead of possibly in the middle of a filename.

If all of this sounds like a big bag of hurt to you, you're right. It probably isn't worth trying to dodge all this lack of standardization. See [[BashFAQ/087|Bash FAQ 87]] for some ways of getting file metadata without parsing `ls` output at all.

Why you shouldn't parse the output of ls(1)

The ls(1) command is pretty good at showing you the attributes of a single file (at least in some cases), but when you ask it for a list of files, there's a huge problem: Unix allows almost any character in a filename, including whitespace, newlines, commas, pipe symbols, and pretty much anything else you'd ever try to use as a delimiter except NUL. In its default mode, if standard output isn't a terminal, ls separates filenames with newlines. This is fine until you have a file with a newline in its name. And since I don't know of any implementation of ls that allows you to terminate filenames with NUL characters instead of newlines, this leaves us unable to get a list of filenames safely with ls.

$ touch 'a space' $'a\nnewline'
$ echo "don't taze me, bro" > a
$ ls | cat
a
a
newline
a space

This output appears to indicate that we have two files called a, one called newline and one called a space.

Using ls -l we can see that this isn't true at all:

$ ls -l
total 8
-rw-r-----  1 lhunath  lhunath  19 Mar 27 10:47 a
-rw-r-----  1 lhunath  lhunath   0 Mar 27 10:47 a?newline
-rw-r-----  1 lhunath  lhunath   0 Mar 27 10:47 a space

The problem is that from the output of ls, neither you or the computer can tell what parts of it constitute a filename. Is it each word? No. Is it each line? No. There is no correct answer to this question other than: you can't tell.

Also notice how ls sometimes garbles your filename data (in our case, it turned the newline character in between the words "a" and "newline" into a question mark. Some systems put a \n instead.). On some systems it doesn't do this when its output isn't a terminal, on others it always mangles the filename. All in all, you really can't and shouldn't trust the output of ls to be a true representation of the filenames that you want to work with. So don't.

Now that we've seen the problem, let's explore various ways of coping with it. As usual, we have to start by figuring out what we actually want to do.

Enumerating files or doing stuff with files

When people try to use ls to get a list of filenames (either all files, or files that match a glob, or files sorted in some way) things fail disastrously.

If you just want to iterate over all the files in the current directory, use a for loop and a glob:

for f in *; do
 ...
done

See BashPitfalls for more details. Never do this:

# BAD!  Don't do this!
for f in $(ls); do
 ...
done

Again, BashPitfalls will tell you why, if you don't already know.

Things get more difficult if you wanted some specific sorting that only ls can do, such as ordering by mtime. If you want the oldest or newest file in a directory, don't use ls -t | head -1 -- read Bash FAQ 99 instead. If you truly need a list of all the files in a directory in order by mtime so that you can process them in sequence, switch to perl, and have your perl program do its own directory opening and sorting. Then do the processing in the perl program, or -- worst case scenario -- have the perl program spit out the filenames with NUL delimiters.

Even better, put the modification time in the filename, in YYYYMMDD format, so that glob order is also mtime order. Then you don't need ls or perl or anything. (The vast majority of cases where people want the oldest or newest file in a directory can be solved just by doing this.)

You could patch ls to support a --null option and submit the patch to your OS vendor. That should have been done about 15 years ago.

Of course, the reason that wasn't done is because very few people really need the sorting of ls in their scripts. Mostly, when people want a list of filenames, they use find(1) instead, because they don't care about the order. And BSD/GNU find has had the ability to terminate filenames with NULs for a very long time.

So, instead of this:

# Bad!  Don't!
ls | while read filename; do
  ...
done

Try this:

# Be aware that this does not do the same as above. This goes recursive and lists only on normal files (i.e. no dirs or symlinks). It may work for some situation but is not at all a replacement for the above.
find . -type f -print0 | while IFS= read -r -d '' filename; do
  ...
done

Even better, most people don't really want a list of filenames. They want to do things to files instead. The list is just an intermediate step to accomplishing some real goal, such as change www.mydomain.com to mydomain.com in every *.html file. find can pass filenames directly to another command. There is usually no need to write the filenames out in a straight line and then rely on some other program to read the stream and separate the names back out.

Getting metadata on a file

If you're after the file's size, use wc instead:

read size _ < <(wc -c "$file")

(See Bash FAQ 24 for the rationale behind the ProcessSubstitution there.)

Other metadata is often hard to get at in a portable way. stat is not available on every platform, and when it is, it often takes a completely different syntax of arguments. There is no way to use stat in a way that it won't break for the next POSIX system you run the script on.

You can try to parse certain metadata out of ls -l's output. Two big warnings, though: run ls with only one file at a time (remember, you can't reliably know where to start parsing metadata of the second file since there's no good delimiter - and no, a newline is not a good delimiter) and don't parse the timestamp or beyond (the time stamp is usually formatted in a very platform and locale-dependent manner and can thus not be parsed reliably).

read mode links owner _ < <(ls -l "$file")

Note that the mode string is also often platform-specific. Eg. OS X adds an @ for files with xattrs and a + for files with extended security information.

In case you don't believe us, here's why not to try to parse the timestamp:

# Debian unstable:
$ ls -l
-rw-r--r-- 1 wooledg wooledg       240 2007-12-07 11:44 file1
-rw-r--r-- 1 wooledg wooledg      1354 2009-03-13 12:10 file2

# OpenBSD 4.4:
$ ls -l
-rwxr-xr-x  1 greg  greg  1080 Nov 10  2006 file1
-rw-r--r--  1 greg  greg  1020 Mar 15 13:57 file2

On OpenBSD, as on most versions of Unix, ls shows the timestamps in three fields -- month, day, and year-or-time, with the last field being the time (hours:minutes) if the file is less than 6 months old, or the year if the file is more than 6 months old. On Debian unstable, with a fairly recent version of GNU coreutils, ls shows the timestamps in two fields, with the first being Y-M-D and the second being H:M, no matter how old the file is. So, it should be pretty obvious we never want to have to parse the output of ls if we want a timestamp from a file. But for the fields before that, it's usually pretty reliable.

(Note: some versions of ls don't print the group ownership of a file by default, and require a -g flag to do so. Others print the group by default, and -g suppresses it. You've been warned.)

If we wanted to get metadata from more than one file in the same ls command, we run into the same problem we had before -- files can have newlines in their names, which screws up our output. Imagine how code like this would break if we have a file with a newline in its name:

# Don't do this
{ read 'perms[1]' 'links[1]' 'owner[1]' 'group[1]' _
  read 'perms[2]' 'links[2]' 'owner[2]' 'group[2]' _
} < <(ls -l "$file1" "$file2")

Similar code that uses two separate ls calls would probably be OK, since the second read command would be guaranteed to start reading at the beginning of an ls command's output, instead of possibly in the middle of a filename.

If all of this sounds like a big bag of hurt to you, you're right. It probably isn't worth trying to dodge all this lack of standardization. See Bash FAQ 87 for some ways of getting file metadata without parsing ls output at all.


CategoryShell

ParsingLs (last edited 2023-08-12 13:05:09 by StephaneChazelas)