|
Size: 26712
Comment: oops. dublicate paragraph.
|
Size: 617
Comment:
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 4: | Line 4: |
| ## Until TableOfContents.py gets fixed, every page will have to be manually included. ## Currently, using the regex makes the ToC have links pointing to the wrong anchors. ## [[Include(^BashGuide/Practices/.*, , editlink)]] |
|
| Line 5: | Line 8: |
[[Anchor(Choose_Your_Shell)]] == Choose Your Shell == The first thing you should do before starting a shell script is enumerate the requirements and the goal of that script. Pick the right tool for the job. ["BASH"] may be easy to learn and write, but it isn't always fitted to do the work for you. There are a lot of tools in the basic toolset made available in most *NIX flavors that can help you. If you need only use AWK, then don't make a ["BASH"] script that invokes AWK, just make an AWK script. If you need to retrieve data from an HTML or XML file, ["BASH"] is also '''the wrong tool for the job'''. You should considder XSLT instead, or a language that has a library available for parsing XML or HTML. If you decide to use ["BASH"], then ask yourself these questions first: * Does your script need to run in environments where it's possible ["BASH"] is not available? . '''Then choose Sh instead!''' ''(Keep in mind that this guide does not apply to Sh!)'' * Are you certain about the version of ["BASH"] that will be available in all environments that your script will run in? . '''If not, you should limit yourself to features of ["BASH"] 2 ONLY.''' If the above questions do not limit your choices, '''then use ["BASH"] 3'''. Keep these points in mind: * '''Do not inherit ugly and evil shell scripting techniques from retarded Sh scripting guides and examples on the net; ["BASH"] 3 often offers far better alternatives!''' * Do not use a `#!/bin/sh` header in your script. Be honest and use `#!/bin/bash` instead. Otherwise you disable some ["BASH"] specific features that are not available in Sh. * Forget `[` for once and for all. ["BASH"] features an alternative that's '''far''' better for many of reasons. It's called `[[`. Using `[` will only cause headaches and break your script when you least expect it. * It's also time you forgot about {{{`}}}. It isn't consistent with the syntax of ''Expansion'' and is terribly hard to nest without a dose of painkillers. Use `$()` instead. * ''And for heaven's sake, '''quotes, quotes, quotes and more quotes'''!'' Protect your strings from ''Word Splitting''. ''Word splitting'' will eat your babies if you don't quote them properly. -------- [[Anchor(Quoting)]] == Quoting == ''Word Splitting'' is the demon inside ["BASH"] that is out to get unsuspecting newcomers or any other scripters off guard. If you do not understand how ''Word Splitting'' works or when it is aplied you should be very careful with your strings and your ''Parameter Expansion''s. I suggest you read up on ''Word Splitting'' if you doubt your knowledge. The best way to protect yourself from this beast is to quote all your strings. Quotes keep your strings in one piece and prevent ''Word Splitting'' from tearing them open. Allow me to illustrate: {{{ $ echo Push that word away from me. Push that word away from me. $ echo "Push that word away from me." Push that word away from me. }}} Now, don't think ''Word Splitting'' is about collapsing spaces. What really happened in this example is that the first command passed each word in our sentence as a separate argument to `echo`. ["BASH"] split our sentence up in words using the whitespace inbetween them to determine where each argument begins and ends. In the second example ["BASH"] is forced to keep the whole quoted string together. This means it's not split up into arguments and the whole string is passed to `echo` '''as the first argument'''. `echo` prints out each argument it gets with a space inbetween them. You should understand the basics of ''Word Splitting'' now. This is where it gets dangerous! ''Word Splitting'' does not just happen on literal strings, it also happens '''after ''Parameter Expansion'''''! As a result, on a dull and tired day, you might just be stupid enough to make this mistake: {{{ $ sentence="Push that word away from me." $ echo $sentence Push that word away from me. $ echo "$sentence" Push that word away from me. }}} As you can see, in the first `echo` command, we were neglectant and left out the quotes. That was a mistake. ["BASH"] expanded our sentence and then used ''Word Splitting'' to split the resulting expansion up into arguments to use for `echo`. In our second example, the quotes around the ''Parameter Expansion'' of `sentence` make sure ["BASH"] '''does not''' split it up into multiple arguments around the whitespace. It's not just spaces you need to protect. ''Word Splitting'' occurs on spaces, tabs and newlines. In certain context it aplies to the characters in the `IFS` variable. Here's another example to show you just how badly you can break things if you neglect to use quotes: {{{ $ echo "$(ls -al)" total 8 drwxr-xr-x 4 lhunath users 1 2007-06-28 13:13 "."/ drwxr-xr-x 102 lhunath users 9 2007-06-28 13:13 ".."/ -rw-r--r-- 1 lhunath users 0 2007-06-28 13:13 "a" -rw-r--r-- 1 lhunath users 0 2007-06-28 13:13 "b" -rw-r--r-- 1 lhunath users 0 2007-06-28 13:13 "c" drwxr-xr-x 2 lhunath users 1 2007-06-28 13:13 "d"/ drwxr-xr-x 2 lhunath users 1 2007-06-28 13:13 "e"/ $ echo $(ls -al) total 8 drwxr-xr-x 4 lhunath users 1 2007-06-28 13:13 "."/ drwxr-xr-x 102 lhunath users 9 2007-06-28 13:13 ".."/ -rw-r--r-- 1 lhunath users 0 2007-06-28 13:13 "a" -rw-r--r-- 1 lhunath users 0 2007-06-28 13:13 "b" -rw-r--r-- 1 lhunath users 0 2007-06-28 13:13 "c" drwxr-xr-x 2 lhunath users 1 2007-06-28 13:13 "d"/ drwxr-xr-x 2 lhunath users 1 2007-06-28 13:13 "e"/ }}} In some very rare occasions it may be desired to leave out the quotes. That's if you '''need''' ''Word Splitting'' to take place: {{{ $ friends="Marcus JJ Thomas Michelangelo" $ for friend in $friends > do echo "$friend is my friend!"; done Marcus is my friend! JJ is my friend! Thomas is my friend! Michelangelo is my friend! }}} But honestly? You should use arrays for nearly all of these cases. Arrays have the benefit that they separate strings without the need for an explicit delimitor. That means your strings in the array can contain '''any''' valid character, without the worry that it might be your string delimitor (like the space is in our example above). Using arrays in our example above gives us the ability to add middle or last names of our friends: {{{ $ friends=( "Marcus The Rich" "JJ The Short" "Timid Thomas" "Michelangelo The Mobster" ) $ for friend in "${friends[@]}" > do echo "$friend is my friend!"; done }}} Note that in our previous `for` we used an ''unquoted'' `$friends`. This allowed ["BASH"] to split our `friends` string up into words. In this last example, we quoted the `${friends[@]}` ''Parameter Expansion''. Quoting an expansion of an array through the `@` index makes ["BASH"] expand that array into a sequence of its elements, '''where each element is wrapped into quotes'''. [[Anchor(Readability)]] == Readability == Almost just as important as the result of your code is the '''readability of your code'''. Chances are that you aren't just going to write this script once and then forget about it. If so, you might as well delete it after using it. If you plan to continue using it, you should also plan to continue maintaining it. Unlike your room your code won't get dirty over time, but you will learn new techniques and new approaches constantly. You will also gain experience about how your script is used. All that new information you gather since the completion of your initial codebase should be used to maintain your code in such a way that it constantly improves. Your code should keep growing more userfriendly and more stable. . Trust me when I say, no piece of code is ever 100% ready; with exception to some very short and useless chucks of nothingness. To make it easier for yourself to keep your code healthy and improve it regularly you should keep an eye on the readability of what you write. When you return to a long loop after a year has passed since your last visit to it and you wish to improve it, add a feature, or debug something about it, believe me when I say you'd rather see this: {{{ friends=( "Marcus The Rich" "JJ The Short" "Timid Thomas" "Michelangelo The Mobster" ) # Say something significant about my friends. for name in "${friends[@]}"; do # My first friend (in the list). if [[ $name = ${friends[0]} ]]; then echo $name was my first friend. # My friends those names start with M. elif [[ $name = M* ]]; then echo "$name starts with an M" # My short friends. elif [[ " $name " = *" Short "* ]]; then echo "$name is a shorty." # Friends I kind of didn't bother to remember. else echo "I kind of forgot what $name is like." fi done }}} Than being confronted with something like this: {{{ x=( Marcus\ The\ Rich JJ\ The\ Short Timid\ Thomas Michelangelo\ The\ Mobster) for name in "${x[@]}" do if [ "$name" = "$x" ]; then echo $name was my first friend. elif echo $name | \ grep -qw Short then echo $name is a shorty. elif [ "x${name:0:1}" = "xM" ] then echo $name starts with an M; else echo I kind of forgot what $name \ is like.; fi; done }}} And yes, I know this is an exagerated example, but I've seen some authentic code that actually has a lot in common with that last example. For your own health, keep these few points in mind: * ''Healthy whitespace gives you breathing space''. Indent your code properly and consistently. Use blank lines to separate paragraphs or logic blocks. * ''Avoid backslash-escaping''. It's counter-intuitive and boggles the mind when overused. Even in small examples it takes your mind more effort to understand `a\ b\ c` than it takes to understand `'a b c'`. * ''Comment your way of thinking before you forget''. You might find that even code that looks totally common sense right now could become subject of ''"What the hell was I thinking when I wrote this?"'' or ''"What is '''this''' supposed to do?"''. * ''Consistency prevents mind boggles''. Be consistent in your naming style. Be consistent in your use of capitals. Be consistent in your use of shell features. No matter what your girlfriend says, it's good to be predictable and transparant. [[Anchor(Bash_Tests)]] == Bash Tests == The `test` application, also known as `[`, is an application that usually resides somewhere in `/usr/bin` or `/bin` and is used a lot by shell programmers to perform certain tests on variables. In a number of shells, including bash, [ is implemented as a shell builtin. It can produce surprising results, especially for people starting shell scripting that consider [ ] as part of the shell syntax. If you use `sh`, you have little choice but to use test as it is the only way to do most of your testing. If however you are using ["BASH"] to do your scripting (and I presume you are since you're reading this guide), then you can also use the newer [[ ]] keywords that, though [[ still behaves in many ways like a command, present several advantages over the traditional test command. Let me illustrate how `[[` can be used to replace `test`, and how it can help you to avoid some of the common mistakes made by using `test`: {{{ $ var='' $ [ $var = '' ] && echo True -bash: [: =: unary operator expected $ [ "$var" = '' ] && echo True True $ [[ $var = '' ]] && echo True True }}} `[ $var = '' ]` expands into `[ = '' ]`. The first thing `test` sees, is the equals sign. It takes this to be the left hand string of the test. The next thing `test` sees is the `''`. It expects an operator to perform some kind of test on `=` and explodes. Yes, `test` did not see our empty `$var` because ["BASH"] expanded it into nothingness before `test` could even see it. Morale of the story? '''Use more quotes!''' Using quotes, `[ "$var" = '' ]` expands into `[ "" = '' ]` and `test` has no problem. Now, `[[` can see the whole command before it's being expanded. It sees `$var`, and not the expansion of `$var`. As a result, there is no need for the quotes at all! '''`[[` is safer'''. {{{ $ var= $ [ "$var" < a ] && echo True -bash: a: No such file or directory $ [ "$var" \< a ] && echo True True $ [[ $var < a ]] && echo True True }}} In this example we attempted a string comparison between an empty variable and '`a`'. We're surprised to see the first attempt does not yield `True` even though it should. Instead, we get some weird error that implies ["BASH"] is trying to open a file called '`a`'. We've been bitten by ''File Redirection''. Since `test` is just an application, the `<` character in our command is interpreted (as it should) as a ''File Redirection'' operator instead of the string comparison operator of `test`. (Furthermore, the proper operator would have been -lt instead of < anyway.) ["BASH"] is instructed to open a file '`a`' and connect it to `stdin` for reading. To prevent this, we need to escape `<` so that `test` receives the operator rather than ["BASH"]. This makes our second attempt work. Using `[[` we can avoid the mess altogether. `[[` sees the `<` operator before ["BASH"] gets to use it for ''Redirection''; problem fixed. Once again, '''`[[` is safer'''. Even more dangerous is using the `>` operator instead of our previous example with the `<` operator. Since `>` triggers output ''Redirection'' it will create a file called '`a`'. As a result, '''there will be no error message warning us that we've committed a sin'''! Instead, our script will just break. Up to us to ''guess'' where the problem is: {{{ $ var=a $ [ "$var" > b ] && echo True || echo False True $ [[ "$var" > b ]] && echo True || echo False False }}} Two different results, great. Trust me when I say you can always trust `[[` more than `[`. `[ "$var" > b ]` is expanded into `[ "a" ]` and the output of that is being redirected into a new file called '`b`'. Since `[ "a" ]` is the same as `[ -n "a" ]` and that basically tests whether the `"a"` string is non-empty, the result is a success and the `echo True` is executed. Using `[[` we get our expected scenario where `"a"` is tested against `"b"` and since we all know `"a"` sorts before `"b"` this triggers the `echo False` statement. And this is how you can break your script without realizing it. You will however have a suspiciously empty file called '`b`' in your current directory. So believe me when I say, '''`[[` is safer than `[`'''. Because everybody inevitably makes programming errors. People usually don't intend to introduce bugs in their code. It just happens. So don't pretend you can use `[` and ''"You'll be careful not to make these mistakes"'', because I can assure you that you '''will'''. Besides `[[` provides the following features over `[`: * '''`[[` can do glob pattern matching:''' . `[[ abc = a* ]]` * '''`[[` can do regex pattern matching (Bash 3.1+):''' . `[[ abb =~ ab+ ]]` The only advantage of `test` is its portability. [[Anchor(Dont_Ever_Do_These)]] == Don't Ever Do These == The Bash shell allows you to do quite a lot of things, offering you considderable flexibility. Unfortunately, it does very little to discourage misuse and other ill-adviced behavior. It hopes people will find out for themselves that certain things should be avoided at all costs. Unfortunately many people don't care enough to want to find out for themselves. They write without thinking things through and many aweful and dangerous scripts end up in production environments or in Linux Distro scripts. The result of these, but even your very own scripts written in a time of neglect, can very often be '''DISASTROUS'''. That said, for the good of your scripts and for the rest of mankind; '''Never Ever Do Anything Along These Lines''': * '''`ls -l | awk '{ print $8 }'`''' . '''DON'T EVER parse the output of `ls`!''' The `ls` command's output cannot be trusted for a number of reasons.[[BR]] For one, `ls` will mangle the filenames of files if they contain characters unsupported by your locale. As a result, parsing filenames out of `ls` is NEVER guaranteed to actually give you a file that you will be able to find. `ls` might have replaced certain characters in the filename by question marks.[[BR]] Secondly, `ls` separates lines of data by newlines. This way, every bit of information on a file is on a new line. '''UNFORTUNATELY filenames themselves can ALSO contain newlines'''. This means that if you have a filename that contains a newline in the current directory, it will completely break your parsing and as a result, break your script![[BR]] Last but not least, the output format of `ls` is NOT guaranteed across platforms.[[BR]] If you need to work with file times; then I highly recommend you implement something using the features provided by `test`. See the `Bash Tests` chapter before this one. The truth is that there is '''no reliable way''' of working with file metadata in Bash. If you have considdered all the alternatives (backuping? Use `rsync`) and still '''really''' need this; ''I recommend you switch to a different language, like python or perl''. * '''`if echo "$file" | fgrep .txt; then`'''[[BR]] '''`ls *.txt | grep story`''' . '''DON'T EVER test or filter filenames `grep`!''' Unless your grep pattern is really smart it will probably not be trustworthy.[[BR]] In my example above, the test would match both `story.txt` AND `story.txt.exe`. If you make grep patterns that are smart enough, they'll probably be so ugly, massive and unreadable that you shouldn't use it anyway.[[BR]] The alternative is called '''GLOBS'''. Bash has a feature called ''Pathname Expansion''. This will help you enumerate all files that match a certain pattern! Alternatively you can use globs to test whether a certain filename matches a glob pattern. * '''`cat file | grep pattern`''' . Don't use `cat` as a default method for feeding file content to a process. `cat` is a utility used to concatenate the contents of several files together.[[BR]] To feed the contents of a file to a process you will probably be able to pass the filename as an argument to the process (`grep 'pattern' /my/file`, `sed 'expression' /my/file, ...).[[BR]] If the manual of the process does not specify any way to do this, you should use redirection (`read column1 column2 < /my/file`, `tr ' ' '\n' < /my/file`, ...). * '''`for number in $(seq 1 10); do`''' . For the love of god and all that is holy, do not use `seq` to count.[[BR]] Bash is able enough to do the counting for you. You do not need to spawn an external application to do some counting and then pass that application's output to Bash for word splitting. Learn the syntax of `for` already![[BR]] You should use this in Bash 3.x: `for number in {1..10}`, or this in Bash 2.x: `for ((i=1; i<10; i++))`. * '''`i= `{{{`}}}`expr $i + 1`{{{`}}}''' . `expr` is a relic of ancient Rome. Do not wield it.[[BR]] It was used very often in scripts written for shells with very limited capabilities. You're basically spawning a new process, calling another C program to do some math for you and return the result as a string to bash. Bash can do all this itself and so much faster, more reliable (no numbers->string->number conversions) and in all, better.[[BR]] You should use this in Bash: `let i++;`. [[Anchor(Debugging)]] == Debugging == Very often you will find yourself clueless as to why your script isn't acting the way you want it to. Resolving this problem is always just a matter of common sense and debugging techniques. [[BR]] ||<tablewidth="100%"> ||<style="text-align: center;">'''Diagnose The Problem''' || || Unless you know what exactly the '''problem''' is, you most likely won't come up with a ''solution'' anytime soon. So make sure you understand what exactly goes wrong. Evaluate the symptoms and/or error messages. If you must, try and formulate the problem as a sentence. This will also be vital if you're going to ask for help with other people about your problem. You don't want them to rewrite your whole script. You also don't want them to have to go through your whole script or run it so that they understand what's going on. No; '''you''' need to make the problem perfectly clear to '''yourself''' and to anybody trying to help you. ''This requirement stands until the day the human race inventes means of telepathy''. [[BR]] ||<tablewidth="100%"> ||<style="text-align: center;">'''Minimize The Codebase''' || || If staring at your code doesn't give you a devine intervention, the next thing you should do is try to minimize your codebase while maintaining the same problem. Don't worry about preserving the functionality of your script. The only thing you want to preserve is the logic of the code block that seems buggy. Often, the best way to do this is by copying your script to a new file and start deleting everything that seems irrelevant from it. Alternatively, you can make a new script that does something similar in the same code fashion. As soon as you delete something that makes the problem go away, you'll have isolated the problem. If not, at least you're not staring at a massive script anymore, but hopefully at a stub of no more than 5-7 lines. For example; if you have a script that lets you browse images in your image folder by date, and for some reason you can't manage to iterate over your images in the folder properly, it suffices to reduce the script to this part in it: {{{ for image in $(ls -R "$imgFolder"); do echo "$image" done }}} Your actual script will be far more complex, and the inside of the `for` loop will also be far longer. But the essence of the problem is this code. Once you've reduced your problem to this it may be easier to see the problem you're facing. Your `echo` spits out parts of image names; it looks like all whitespace is replaced by newlines. That must be because `echo` is ran for each chunk terminated by whitespace, not for every image name (as a result, it seems the output has split open image names with whitespace in them). With this reduced code; it's easier to see that the cause is actually your `for` statement that splits up the output of `ls` into words. That's because `ls` is '''UNPARSABLE''' in a bugless manner (do not ever use `ls` in scripts, unless if you want to show its output to a user). The fix would be: {{{ find "$imgFolder" -print0 | while read -d $'\0' image; do echo "$image" done }}} Now that you've fixed the problem in this tiny example, it's easy to merge it back into the original script. [[BR]] ||<tablewidth="100%"> ||<style="text-align: center;">'''Activate ["BASH"]'s Debug Mode''' || || If you still don't see the error of your ways, ["BASH"]'s debugging mode might help you see the problem through the code. When ["BASH"] runs with the `x` option turned on, it prints out every command it executes before executing it. This is, '''after any expansions have been applied'''. As a result, you can see exactly what's happening as each line in your code is executed. Pay very close attention to the quoting used. ["BASH"] uses quotes to show you exactly which strings are passed as a single argument. There are three ways of turning on this mode. * Run the script with `bash -x`: {{{ $ bash -x ./mybrokenscript }}} * Modify your script's header: {{{ #! /bin/bash -x [.. script ..] }}} * Or add `set -x` somewhere in your code to turn on this mode for only a specific block of your code: {{{ #! /bin/bash [..irrelevant script code..] set -x [..relevant script code..] set +x [..irrelevant script code..] }}} [[BR]] ||<tablewidth="100%"> ||<style="text-align: center;">'''Reread The Manual''' || || If your script still doesn't seem to agree with you, maybe your perception of the way things work is wrong. Try going back to the manual (or this guide) to re-evaluate whether commands do exactly what you think they do, or the syntax is what you think it is. Very often people misunderstand what `for` does, how ''Word Splitting'' works, or how often they should use quotes. Keep the tips and good practice guidelines in this guide in mind as well. They often help you avoid bugs and problems with scripts. I mentioned this in the ''Scripts'' section of this guide too, but it's worth repeating it here. First of all, make sure your script's '''header''' is actually '''`#! /bin/bash`'''. If it is '''missing''' or if it's something like '''`#! /bin/sh`''' then you deserve the problems you're having. That means you're probably not even using ["BASH"] to run your code. Obviously that'll cause issues. Also, make sure you have no ''Carriage Return'' characters at the ends of your lines. This is the cause of scripts written in ''Microsoft Windows(tm)''. You can get rid of these fairly easily like this: {{{ $ tr -d '\r' < myscript > tmp && mv tmp myscript }}} [[BR]] ||<tablewidth="100%"> ||<style="text-align: center;">'''Read the FAQ / Pitfalls''' || || The ["BashFAQ"] and BashPitfalls pages explain common misconceptions and issues encountered by other ["BASH"] scripters. It's very likely that your problem will be described there in some shape or form. To be able to find your problem in there, you'll obviously need to have ''Diagnosed'' it properly. You'll need to know what you're looking for. [[BR]] ||<tablewidth="100%"> ||<style="text-align: center;">'''Ask Us On IRC''' || || There are people in the `#bash` channel almost 24/7. This channel resides on the `freenode` IRC network. To reach us, you need an IRC client. Connect it to `irc.freenode.net`, and `/join #bash`. Make '''sure''' that you know what your real problem is and have stepped through it on paper, so you can explain it well. We don't like having to guess at things. Start by explaining what you're trying to do with your script. Either way, please have a look at this page before entering `#bash`: XyProblem. |
[[Include(BashGuide/Practices/ChooseYourShell, , editlink)]] [[Include(BashGuide/Practices/Quoting, , editlink)]] [[Include(BashGuide/Practices/Readability, , editlink)]] [[Include(BashGuide/Practices/BashTests, , editlink)]] [[Include(BashGuide/Practices/DontEverDoThese, , editlink)]] [[Include(BashGuide/Practices/Debugging, , editlink)]] |
Practices
Include(BashGuide/Practices/ChooseYourShell, , editlink) Include(BashGuide/Practices/Quoting, , editlink) Include(BashGuide/Practices/Readability, , editlink) Include(BashGuide/Practices/BashTests, , editlink) Include(BashGuide/Practices/DontEverDoThese, , editlink) Include(BashGuide/Practices/Debugging, , editlink)