Differences between revisions 6 and 24 (spanning 18 versions)
Revision 6 as of 2008-07-24 01:23:01
Size: 5828
Editor: BeJonas
Comment: re mkdir on nfs
Revision 24 as of 2016-07-29 15:44:32
Size: 5884
Editor: GreyCat
Comment: link to FileDescriptor
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
[[Anchor(faq45)]]
== How can I ensure that only one instance of a script is running at a time (mutual exclusion)? ==

We need some means of '''mutual exclusion'''. One easy way is to use a "lock": any number of processes can try to acquire the lock simultaneously, but only one of them will succeed.
<<Anchor(faq45)>>
== How can I ensure that only one instance of a script is running at a time (mutual exclusion, locking)? ==
We need some means of ''mutual exclusion''. One way is to use a "lock": any number of processes can try to acquire the lock simultaneously, but only one of them will succeed.
Line 18: Line 17:
 fi}}}  fi
 
}}}
Line 20: Line 20:
This example '''does not work''', because there is a time window between checking and creating the file. Assume two processes are running the code at the same time. Both check if the lockfile exists, and both get the result that it does not exist. Now both processes assume they have acquired the lock -- a disaster waiting to happen. We need an atomic check-and-create operation, and fortunately there is one: {{{mkdir}}}, the command to create a directory: This example '''does not work''', because there is a RaceCondition: a time window between checking and creating the file, during which other programs may act. Assume two processes are running this code at the same time. Both check if the lockfile exists, and both get the result that it does not exist. Now both processes assume they have acquired the lock -- a disaster waiting to happen. We need an atomic check-and-create operation, and fortunately there is one: `mkdir`, the command to create a directory:
Line 33: Line 33:
 fi}}}  fi
 
}}}
Line 37: Line 38:
Instead of using {{{mkdir}}} we could also have used the program to create a symbolic link, {{{ln -s}}}. Instead of using {{{mkdir}}} we could also have used the program to create a symbolic link, {{{ln -s}}}.  A third possibility is to have the program delete a preexisting lock file with {{{rm}}}. The lock is released by recreating the file on exit.
Line 52: Line 53:
     trap "exit 2" 1 2 3 15 # terminate script when receiving signal
Line 61: Line 61:
 fi}}}  fi
 
}}}
Line 63: Line 64:
This example is much better. There is still the problem that a ''stale'' lock could remain when the script is terminated with a signal not caught (or signal 9, SIGKILL), or could be created by a user (either accidentally or maliciously), but it's a good step towards reliable mutual exclusion. An example that remedies the "stale lock" problem (contributed by Charles Duffy) follows: This example is much better. There is still the problem that a ''stale'' lock could remain when the script is terminated with a signal not caught (or signal 9, SIGKILL), or could be created by a user (either accidentally or maliciously), but it's a good step towards reliable mutual exclusion.  Charles Duffy has [[/contrib|contributed an example]] that may remedy the "stale lock" problem.
Line 65: Line 66:
 ''Are we sure this code's correct? There seems to be a discrepancy between the names LOCK_DEFAULT_NAME and DEFAULT_NAME; and it checks for processes in what looks to be a race condition; and it uses the Linux-specific /proc file system and the GNU-specific egrep -o to do so.... I don't trust it. It looks overly complex and fragile. And quite non-portable. -- GreyCat'' If you're using a GNU/Linux distribution, you can also get the benefit of using flock(1). flock(1) ties a [[FileDescriptor]] to a lock file. There are multiple ways to use it; one possibility to solve the multiple instance problem is:
Line 68: Line 69:
 LOCK_DEFAULT_NAME=$0
 LOCK_HOSTNAME="$(hostname -f)"

 ## function to take the lock if free; will fail otherwise
 function grab-lock {
   local PROGRAMNAME="${1:-$DEFAULT_NAME}"
   local PID=${2:-$$}
   (
     umask 000;
     mkdir -p "/tmp/${PROGRAMNAME}-lock"
     mkdir "/tmp/${PROGRAMNAME}-lock/held" || return 1
     mkdir "/tmp/${PROGRAMNAME}-lock/held/${LOCK_HOSTNAME}--pid-${PID}" && return 0 || return 1
   ) 2>/dev/null
   return $?
 }

 ## function to nicely let go of the lock
 function release-lock {
   local PROGRAMNAME="${1:-$DEFAULT_NAME}"
   local PID=${2:-$$}
   (
     rmdir "/tmp/${PROGRAMNAME}-lock/held/${LOCK_HOSTNAME}--pid-${PID}" || true
     rmdir "/tmp/${PROGRAMNAME}-lock/held" && return 0 || return 1
   ) 2>/dev/null
   return $?
 }

 ## function to force anyone else off of the lock
 function break-lock {
   local PROGRAMNAME="${1:-$DEFAULT_NAME}"
   (
     [ -d "/tmp/${PROGRAMNAME}-lock/held" ] || return 0
     for DIR in "/tmp/${PROGRAMNAME}-lock/held/${LOCK_HOSTNAME}--pid-"* ; do
       OTHERPID="$(echo $DIR | egrep -o '[0-9]+$')"
       [ -d /proc/${OTHERPID} ] || rmdir $DIR
     done
     rmdir /tmp/${PROGRAMNAME}-lock/held && return 0 || return 1
   ) 2>/dev/null
   return $?
 }

 ## function to take the lock nicely, freeing it first if needed
 function get-lock {
   break-lock "$@" && grab-lock "$@"
 }
  exec 9>/path/to/lock/file
  if ! flock -n 9 ; then
     echo "another instance is running";
     exit 1
  fi
  # this now runs under the lock until 9 is closed (it will be closed automatically when the script ends)
Line 115: Line 77:
flock can also be used to protect only a part of your script, see the man page for more information.

Line 116: Line 81:

==== Alternative Solution ====
Line 121: Line 88:
 Are you sure mkdir has problems with being atomic on NFS? I thought that affected only open, but I'm not really sure. -- BeJonas [[DateTime(2008-07-24T01:22:59Z)]]  Are you sure mkdir has problems with being atomic on NFS? I thought that affected only open, but I'm not really sure. -- BeJonas <<DateTime(2008-07-24T01:22:59Z)>>

==== Removal of locking mechanism ====
Shouldn't the example code blocks above include a `rm "$lockfile"` or `rmdir "lockdir"` directly after the `#...continue script` line? - AnthonyGeoghegan

 . The lock can't be safely removed while the script is still doing its work -- that would allow another instance to run. The longer example includes a `trap` that removes the lock when the script exits.

==== flock file descriptor uniqueness ====

The example uses file descriptor 9 with flock, i.e.

exec 9>/path/to/lock/file
  if ! flock -n 9...

Note, file descriptors are unique per-process. FD 0,1, and 2 are used for stdin,stdout, and stderr so picking a generally high value is sufficient.
(source: http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/com.ibm.aix.genprogc/doc/genprogc/fdescript.htm )

However, what if this file descriptor is already in use by a completely different process? Are we then locking on the file descriptor and not the lock file? How can we ensure we use something that is not already being used?
Line 124: Line 108:




----
CategoryShell

How can I ensure that only one instance of a script is running at a time (mutual exclusion, locking)?

We need some means of mutual exclusion. One way is to use a "lock": any number of processes can try to acquire the lock simultaneously, but only one of them will succeed.

How can we implement this using shell scripts? Some people suggest creating a lock file, and checking for its presence:

  •  # locking example -- WRONG
     lockfile=/tmp/myscript.lock
     if [ -f "$lockfile" ]
     then                      # lock is already held
         echo >&2 "cannot acquire lock, giving up: $lockfile"
         exit 0
     else                      # nobody owns the lock
         > "$lockfile"         # create the file
         #...continue script
     fi

This example does not work, because there is a RaceCondition: a time window between checking and creating the file, during which other programs may act. Assume two processes are running this code at the same time. Both check if the lockfile exists, and both get the result that it does not exist. Now both processes assume they have acquired the lock -- a disaster waiting to happen. We need an atomic check-and-create operation, and fortunately there is one: mkdir, the command to create a directory:

  •  # locking example -- CORRECT
     # Bourne
     lockdir=/tmp/myscript.lock
     if mkdir "$lockdir"
     then    # directory did not exist, but was created successfully
         echo >&2 "successfully acquired lock: $lockdir"
         # continue script
     else
         echo >&2 "cannot acquire lock, giving up on $lockdir"
         exit 0
     fi

Here, even when two processes call mkdir at the same time, only one process can succeed at most. This atomicity of check-and-create is ensured at the operating system kernel level.

Instead of using mkdir we could also have used the program to create a symbolic link, ln -s. A third possibility is to have the program delete a preexisting lock file with rm. The lock is released by recreating the file on exit.

Note that we cannot use mkdir -p to automatically create missing path components: mkdir -p does not return an error if the directory exists already, but that's the feature we rely upon to ensure mutual exclusion.

Now let's spice up this example by automatically removing the lock when the script finishes:

  •  # POSIX (maybe Bourne?)
     lockdir=/tmp/myscript.lock
     if mkdir "$lockdir"
     then
         echo >&2 "successfully acquired lock"
    
         # Remove lockdir when the script finishes, or when it receives a signal
         trap 'rm -rf "$lockdir"' 0    # remove directory when script finishes
    
         # Optionally create temporary files in this directory, because
         # they will be removed automatically:
         tmpfile=$lockdir/filelist
    
     else
         echo >&2 "cannot acquire lock, giving up on $lockdir"
         exit 0
     fi

This example is much better. There is still the problem that a stale lock could remain when the script is terminated with a signal not caught (or signal 9, SIGKILL), or could be created by a user (either accidentally or maliciously), but it's a good step towards reliable mutual exclusion. Charles Duffy has contributed an example that may remedy the "stale lock" problem.

If you're using a GNU/Linux distribution, you can also get the benefit of using flock(1). flock(1) ties a FileDescriptor to a lock file. There are multiple ways to use it; one possibility to solve the multiple instance problem is:

  •   exec 9>/path/to/lock/file
      if ! flock -n 9  ; then
         echo "another instance is running";
         exit 1
      fi
      # this now runs under the lock until 9 is closed (it will be closed automatically when the script ends)

flock can also be used to protect only a part of your script, see the man page for more information.

Discussion

Alternative Solution

I believe using if (set -C; >$lockfile); then ... is equally safe if not safer. The Bash source uses open(filename, flags|O_EXCL, mode); which should be atomic on almost all platforms (with the exception of some versions of NFS where mkdir may not be atomic either). I haven't traced the path of the flags variable, which must contain O_CREAT, nor have I looked at any other shells. I wouldn't suggest using this until someone else can backup my claims. --Andy753421

  • Using set -C does not work with ksh88. Ksh88 does not use O_EXCL, when you set noclobber (-C). --jrw32982

    Are you sure mkdir has problems with being atomic on NFS? I thought that affected only open, but I'm not really sure. -- BeJonas 2008-07-24 01:22:59

Removal of locking mechanism

Shouldn't the example code blocks above include a rm "$lockfile" or rmdir "lockdir" directly after the #...continue script line? - AnthonyGeoghegan

  • The lock can't be safely removed while the script is still doing its work -- that would allow another instance to run. The longer example includes a trap that removes the lock when the script exits.

flock file descriptor uniqueness

The example uses file descriptor 9 with flock, i.e.

exec 9>/path/to/lock/file

  • if ! flock -n 9...

Note, file descriptors are unique per-process. FD 0,1, and 2 are used for stdin,stdout, and stderr so picking a generally high value is sufficient. (source: http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/com.ibm.aix.genprogc/doc/genprogc/fdescript.htm )

However, what if this file descriptor is already in use by a completely different process? Are we then locking on the file descriptor and not the lock file? How can we ensure we use something that is not already being used?

For more discussion on these issues, see ProcessManagement.


CategoryShell

BashFAQ/045 (last edited 2024-01-28 01:21:49 by larryv)