|
Size: 2315
Comment:
|
Size: 2931
Comment: Reorganize.
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 9: | Line 9: |
| If you're running a script on a Unix system, the line endings need to be Unix ones (LFs only), or you will have problems. You can check the kind of line endings in use by running: | If you're running a script on a Unix system, the line endings need to be Unix ones (LFs only), or you will have problems. === Testing for line terminator type === A simple check is to simply look at the output of `cat -e`: |
| Line 14: | Line 18: |
| If you see something like this: | If you see something like this, then you're dealing with CRLF style newlines: |
| Line 21: | Line 26: |
| then you need to remove the CRs. There are a plethora of ways to do this. | Another method is to use the `file` utility to guess at the file type: |
| Line 23: | Line 28: |
| All these are from the [[http://www.student.northpark.edu/pemente/sed/sed1line.txt|sed one-liners page]]: | |
| Line 25: | Line 29: |
| sed 's/.$//' dosfile # assumes that all lines end with CR/LF sed 's/^M$//' dosfile # in bash/tcsh, press Ctrl-V then Ctrl-M sed 's/\x0D$//' dosfile # GNUism - does not work with Unix sed! |
file yourscript |
| Line 30: | Line 32: |
| If you want to remove all CRs regardless of whether they are at the end of a line, you can use {{{tr}}}: | The output tells you whether the ASCII text has some CR, if that's the case. Note: this is only true on GNU/Linux. On other operating systems, the result of `file` is unpredictable, except that it should contain the word "text" somewhere in the output if the result "kind of looks like a text file of some sort, maybe". |
| Line 33: | Line 35: |
| tr -d '\r' < dosfile | imadev:~$ printf 'DOS\r\nline endings\r\n' > foo imadev:~$ file foo foo: commands text arc3:~$ file foo foo: ASCII text, with CRLF line terminators |
| Line 36: | Line 42: |
| If you want to use the second {{{sed}}} example above, but without embedding a literal CR into your script: | In a script, it's more difficult to say what the most reliable method should be. Anything you do is going to be a heuristic. In theory a non-corrupt file created by a non-broken UNIX utility should only contain LFs, and by a DOS utility, there should be no bare LFs not preceded by a CR. You can check this with a PCRE. |
| Line 39: | Line 45: |
| sed $'s/\r$//' dosfile # BASH only | # Bash / Ksh # [[ $(</dev/fd/0) == ~(P)(?<!$'\r')$'\n' ]] # ksh93 PCRE pattern=*[^$'\r']$'\n'* if [[ $(</dev/fd/0) == $pattern ]] <<<$'foo\r\nbar\r\nbaz\r'; then print 'File contains only CRLFs' else print 'File contains at least one newline not preceded by a CR' fi |
| Line 42: | Line 56: |
| All of the previous examples write the modified file to standard output. Redirect the output to a new file, and then {{{mv}}} it over top of the original. | === Converting files === |
| Line 44: | Line 58: |
| There are many more ways: * Some systems have a {{{dos2unix}}} command which can do this. Or {{{recode}}}, or {{{fromdos}}}. * You can also use col <input.txt > output.txt * In {{{vim}}}, you can use {{{:set fileformat=unix}}} to do it and save it with a ":w". * You can use Perl: {{{ perl -pi -e 's/\r\n/\n/' filename }}} This has the advantage of overwriting the original file, so you don't have to mess with temporary files. |
`ex` is a good standard way to convert CRLF to LF, and probably one of the few reasonable methods for doing it in-place from a script: |
| Line 54: | Line 60: |
| *** | {{{ ex -sc $'%s/\r$//e|x' file }}} |
| Line 56: | Line 64: |
| Another way to check it: file yourscript The output tells you whether the ASCII text has some CR, if that's the case. |
Of course, Any of the more powerful dynamic languages to do this with relative ease. |
| Line 60: | Line 66: |
| And another way to fix it: nano yourscript |
{{{ perl -pi -e 's/\r\n/\n/' filename }}} Some systems have special conversion tools available to do this automatically. `dos2unix`, `recode, and `fromdos` are some examples. It be done manually with an editor like nano: {{{ nano -w yourscript }}} |
| Line 64: | Line 79: |
| *** | Or in Vim, use `:set fileformat=unix` and save with `:w`. Ensure the value of `fenc` is correct (probably utf-8). To simply strip all CRs from some input stream, you can use `tr -d '\r' <infile >outfile`. Of course, you must ensure these are not the same file. |
How do I convert a file from DOS format to UNIX format (remove CRs from CR-LF line terminators)?
Carriage return characters (CRs) are used in line ending markers on some systems. There are three different kinds of line endings in common use:
- Unix systems use Line Feeds (LFs) only.
- MS-DOS and Windows systems use CR-LF pairs.
- Old Macintosh systems use CRs only.
If you're running a script on a Unix system, the line endings need to be Unix ones (LFs only), or you will have problems.
Testing for line terminator type
A simple check is to simply look at the output of cat -e:
cat -e yourscript
If you see something like this, then you're dealing with CRLF style newlines:
command^M$ ^M$ another command^M$
Another method is to use the file utility to guess at the file type:
file yourscript
The output tells you whether the ASCII text has some CR, if that's the case. Note: this is only true on GNU/Linux. On other operating systems, the result of file is unpredictable, except that it should contain the word "text" somewhere in the output if the result "kind of looks like a text file of some sort, maybe".
imadev:~$ printf 'DOS\r\nline endings\r\n' > foo imadev:~$ file foo foo: commands text arc3:~$ file foo foo: ASCII text, with CRLF line terminators
In a script, it's more difficult to say what the most reliable method should be. Anything you do is going to be a heuristic. In theory a non-corrupt file created by a non-broken UNIX utility should only contain LFs, and by a DOS utility, there should be no bare LFs not preceded by a CR. You can check this with a PCRE.
# Bash / Ksh
# [[ $(</dev/fd/0) == ~(P)(?<!$'\r')$'\n' ]] # ksh93 PCRE
pattern=*[^$'\r']$'\n'*
if [[ $(</dev/fd/0) == $pattern ]] <<<$'foo\r\nbar\r\nbaz\r'; then
print 'File contains only CRLFs'
else
print 'File contains at least one newline not preceded by a CR'
fi
Converting files
ex is a good standard way to convert CRLF to LF, and probably one of the few reasonable methods for doing it in-place from a script:
ex -sc $'%s/\r$//e|x' file
Of course, Any of the more powerful dynamic languages to do this with relative ease.
perl -pi -e 's/\r\n/\n/' filename
Some systems have special conversion tools available to do this automatically. dos2unix, recode, and fromdos` are some examples.
It be done manually with an editor like nano:
nano -w yourscript
Type Ctrl-O and before confirming, type Alt-D (DOS) or Alt-M (Mac) to change the format.
Or in Vim, use :set fileformat=unix and save with :w. Ensure the value of fenc is correct (probably utf-8).
To simply strip all CRs from some input stream, you can use tr -d '\r' <infile >outfile. Of course, you must ensure these are not the same file.