|
Size: 13151
Comment:
|
Size: 237
Comment:
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| <<Anchor(faq6)>> == How can I use variable variables (indirect variables, pointers, references) or associative arrays? == This is a complex page, because it's a complex topic. It's been divided into roughly three parts: associative arrays, evaluating indirect variables, and assigning indirect variables. There are discussions of programming issues and concepts scattered throughout. === Obligatory Note === Putting variable names or any other [[BashFAQ/050|bash syntax inside parameters]] is generally a bad idea. It violates the separation between code and data, and as such puts you on a slippery slope toward bugs, security issues, etc. ''Even'' when you know you "got it right", because you "know and ''understand'' exactly what you're doing", bugs happen to all of us and it pays to respect separation practices to minimize the extent of damage they can cause. Aside from that, it also makes your code non-obvious and non-transparent. Normally, in bash scripting, you won't need indirect references at all. Generally, people look at this for a solution when they don't understand or know about [[BashGuide/Arrays|Bash Arrays]] or haven't fully considered other Bash features such as functions. === Associative Arrays === There are certain tasks for which associative arrays are a powerful and completely appropriate tool. There are others for which they are overkill, or simply unsuitable. To map from one string to another, you need arrays indexed by a string instead of a number. These exists in AWK as "associative arrays", in Perl as "hashes", and in Tcl simply as "arrays". They also exist in [[KornShell|ksh93]], where you'd use them like this: {{{ # ksh93 typeset -A homedir # Declare ksh93 associative array homedir[jim]=/home/jim homedir[silvia]=/home/silvia homedir[alex]=/home/alex for user in "${!homedir[@]}" # Enumerate all indices (user names) do echo "Home directory of user $user is ${homedir[$user]}" done }}} BASH supports them from version 4 and up: {{{ # Bash 4 and up declare -A homedir homedir[jim]=/home/jim # or homedir=( [jim]=/home/jim [silvia]=/home/silvia [alex]=/home/alex ) ... }}} Prior to Bash 4 or if you can't use ksh93, your options are limited. Either move to another interpreter (awk, perl, python, ruby, tcl, ...) or re-evaluate your problem to ''simplify it''. Suppose we have several subservient hosts with slightly different configuration, and that we want to ssh to each one and run slightly different commands. One way we could set it up would be to hard-code a bunch of ssh commands in per-hostname functions in a single script and just run them in series or in parallel. (Don't reject this out of hand! Simple is good.) Another way would be to store each group of commands as an element of an associative array keyed by the hostname: {{{ source "$conf" for host in "${!commands[@]}"; do ssh "$host" "${commands[$host]}" done # Where "$conf" is a file like this: declare -A commands commands=( [host1]="mvn clean install && cd webapp && mvn jetty:run" [host2]="..." ) }}} This is the kind of approach we'd expect in a high-level language, where we can store hierarchical information in advanced data structures. The difficulty here is that we really want each element of the associative array to be a ''list'' or ''another array'' of command strings. But the shell simply doesn't permit that kind of data structure. So, often it pays to step back and ''think in terms of shells'' rather than other programming languages. Aren't we just running a script on a remote host? Then why don't we just store the configuration sets ''as scripts''? Then it's simple: {{{ # A series of conf files named for the hosts we need to run our commands on: for conf in /etc/myapp/*; do host=${conf##*/} ssh "$host" bash < "$conf" done # /etc/myapp/hostname is just a script: mvn clean install && cd webapp && mvn jetty:run }}} Now we've removed the need for associative arrays, and also the need to maintain a bunch of extremely horrible quoting issues. It is also easy to parallelize using GNU Parallel: {{{ parallel ssh {/} bash "<" {} ::: /etc/myapp/* }}} ==== Associative array hacks in older shells ==== Before you think of using `eval` to mimic associative arrays in an older shell (probably by creating a set of variable names like `homedir_alex`), try to think of a simpler or completely different approach that you could use instead. If this hack still seems to be the best thing to do, consider the following disadvantages: 1. It's really hard to read, to keep track of, and to maintain. 2. The variable names must match the RegularExpression {{{^[a-zA-Z_][a-zA-Z_0-9]*}}} -- i.e., a variable name cannot contain arbitrary characters but only letters, digits, and underscores. We cannot have a variable's name contain Unix usernames, for instance -- consider a user named {{{hong-hu}}}. A dash '-' cannot be part of a variable name, so the entire attempt to make a variable named `homedir_hong-hu` is doomed from the start. 3. Quoting is hard to get right. If a content string (not a variable name) can contain whitespace characters and quotes, it's hard to quote it right to preserve it through both shell parsings. And that's just for ''constants'', known at the time you write the program. (Bash's `printf %q` helps, but nothing analogous is available in POSIX shells.) 4. If the program handles unsanitized user input, it can be [[BashFAQ/048|VERY dangerous]]! Read [[BashGuide/Arrays]] or [[BashFAQ/005]] for a more in-depth description and examples of how to use arrays in Bash. If you ''need'' an associative array but your shell doesn't support them, please consider using AWK instead. === Evaluating indirect/reference variables === [[BASH]] allows you to expand a parameter ''indirectly'' -- that is, one variable may contain the name of another variable: {{{ # Bash realvariable=contents ref=realvariable echo "${!ref}" # prints the contents of the real variable }}} KornShell (ksh93) has a completely different, more powerful syntax -- the `nameref` command (also known as `typeset -n`): {{{ # ksh93 realvariable=contents nameref ref=realvariable echo "$ref" # prints the contents of the real variable }}} Unfortunately, for shells other than Bash and ksh93, there is no syntax for ''evaluating'' a referenced variable. You would have to use [[BashFAQ/048|eval]], which means you would have to undergo extreme measures to sanitize your data to avoid catastrophe. It's difficult to imagine a practical use for this that wouldn't be just as easily performed by using an associative array. But people ask it all the time (it is genuinely a ''frequently'' asked question). ksh93's `nameref` allows us to work with references to [[BashFAQ/005|arrays]], as well as regular scalar variables. For example, {{{ # ksh93 myfunc() { nameref ref=$1 echo "array $1 has ${#ref[*]} elements" } realarray=(...) myfunc realarray }}} We are not aware of any trick that can duplicate that functionality in POSIX or Bourne shells (short of using [[BashFAQ/048|eval]], which is extremely difficult to do securely). Bash can ''almost'' do it -- some indirect array tricks work, and others do not, and we do not know whether the syntax involved will remain stable in future releases. So, consider this a ''use at your own risk'' hack. {{{ # Bash -- trick #1. Seems to work in bash 2 and up. realarray=(...) ref=realarray; index=2 tmp="$ref[$index]" echo "${!tmp}" # gives array element [2] # Bash -- trick #2. Seems to work in bash 3 and up. # Does NOT work in bash 2.05b. tmp="$ref[@]" printf "<%s> " "${!tmp}"; echo # Iterate whole array. }}} We do not know of any way to retrieve the array indices, or even the number of elements, through this kind of indirection. === Assigning indirect/reference variables === Sometimes you'd like to "point" from one variable to another, for purposes of writing information to a dynamically configurable place. Typically this happens when you're trying to write a "reusable" function, and you want it to [[BashFAQ/084|put its output]] in a variable of the caller's choice instead of the function's choice. ([[BashWeaknesses|Reusability of shell functions]] is dubious at best, so this is something that should not happen ''often''.) Assigning a value "through" a reference (or pointer, or indirect variable, or whatever you want to call it -- I'm going to use "ref" from now on) is more widely possible, but the means of doing so are extremely shell-specific. In ksh93, we can just use `nameref` again: {{{ # ksh93 nameref ref=realvariable ref="contents" # realvariable now contains the string "contents" }}} In Bash, we can use {{{read}}} and Bash's [[HereDocument|here string]] syntax: {{{ # Bash ref=realvariable IFS= read -r $ref <<< "contents" # realvariable now contains the string "contents" }}} However, this only works if there are no newlines in the content. If you need to assign multiline values, keep reading. A similar trick works for Bash array variables too: {{{ # Bash aref=realarray read -r -a $aref <<< "words go into array elements" echo "${realarray[1]}" # prints "go" }}} (Again, newlines in the input will break this trick. [[IFS]] is used to delimit words, so you may or may not need to set that.) Another trick is to use Bash's {{{printf -v}}} (only available in [[BashFAQ/061|recent versions]]): {{{ # Bash 3.1 or higher ref=realvariable printf -v $ref %s "contents" }}} The {{{printf -v}}} trick is handy if your contents aren't a constant string, but rather, something dynamically generated. You can use all of {{{printf}}}'s formatting capabilities. This trick also permits any string content, including embedded newlines (but not NUL bytes - no force in the universe can put NUL bytes into shell strings usefully). This is the best trick to use if you're in bash 3.1 or higher. Yet another trick is Korn shell's {{{typeset}}} or Bash's {{{declare}}}. These are roughly equivalent to each other. Both of them cause a variable to become ''locally scoped'' to a function, if used inside a function; but if used outside a function, they can operate on global variables. {{{ # Korn shell (all versions): typeset $ref="contents" # Bash: declare $ref="contents" }}} The advantage of using `typeset` or `declare` over `eval` is that the right hand side of the assignment is ''not'' parsed by the shell. If you used `eval` here, you would have to sanitize/escape the entire right hand side first. This trick also preserves the contents exactly, including newlines, so this is the best trick to use if you're in bash older than 3.1 (or ksh88) and don't need to worry about accidentally changing your variable's scope (i.e., you're not using it inside a function). ''However'', with bash, you must still be careful about what is on the ''left''-hand side of the assignment. Inside square brackets, expansions are still performed; thus, with a tainted ref, `declare` can be just as dangerous as `eval`: {{{ # Bash: ref='x[$(touch evilfile; echo 0)]' ls -l evilfile # No such file or directory declare "$ref=value" ls -l evilfile # It exists now! }}} This problem also exists with `typeset` in mksh and pdksh, but apparently not ksh93. If you aren't using Bash or Korn shell, you can do assignments to referenced variables using HereDocument syntax: {{{ # Bourne ref=realvariable read $ref <<EOF contents EOF }}} (Alas, `read` means we're back to only getting at most one line of content. This is the most portable trick, but it's limited to single-line content.) Remember that, when using a here document, if the sentinel word ({{{EOF}}} in our example) is unquoted, then parameter expansions will be performed inside the body. If the sentinel is quoted, then parameter expansions are not performed. Use whichever is more convenient for your task. Finally, some people just ''cannot'' resist throwing `eval` into the picture: {{{ # Bourne ref=myVar eval "$ref=\$value" }}} This expands to the statement that is executed: {{{ myVar=$value }}} The right-hand side is not parsed by the shell, so there is no danger of unwanted side effects. The drawback, here, is that every single shell metacharacter on the right hand side of the `=` must be escaped carefully. In the example shown here, there was only one. In a more complex situation, there could be dozens. The good news is that if you can sanitize the right hand side correctly, this trick is fully portable, has no variable scope issues, and allows all content including newlines. The bad news is that if you fail to sanitize the right hand side correctly, you have a massive security hole. Use `eval` at your own risk. ---- CategoryShell |
Im addicted to my hobby Archery. Seems boring? Not!<<BR>><<BR>> I to learn German in my free time.<<BR>><<BR>> <<BR>><<BR>> Feel free to surf to my blog post: [[https://www.facebook.com/clashofclanshackf/|hack tool clash of clans]] |
Im addicted to my hobby Archery. Seems boring? Not!
I to learn German in my free time.
Feel free to surf to my blog post: hack tool clash of clans