Size: 4189
Comment:
|
Size: 9441
Comment: chattr +a
|
Deletions are marked like this. | Additions are marked like this. |
Line 1: | Line 1: |
[[Anchor(faq88)]] | <<Anchor(faq88)>> |
Line 3: | Line 3: |
This method is designed to allow you to store a complete log of all commands executed by a friendly user; it is not meant for secure auditing of commands - see [:BashFAQ#faq77:securing bash against history removal]. |
This method is designed to allow you to store a complete log of all commands executed by a friendly user; it is not meant for secure auditing of commands - see [[BashFAQ/077|securing bash against history removal]]. |
Line 12: | Line 11: |
To prevent history lines being lost if Bash terminates abnormally, we need to ensure that they are written after each command. We can use the shell builtin {{{history -a}}} to cause an immediate write of all new history lines, and we can automate this execution by adding it to the PROMPT_COMMAND environment variable. This variable contains a command to be executed before any new prompt is shown, and is therefore run after every interactive command is executed. | To prevent history lines being lost if Bash terminates abnormally, we need to ensure that they are written after each command. We can use the shell builtin {{{history -a}}} to cause an immediate write of all new history lines, and we can automate this execution by adding it to the `PROMPT_COMMAND` variable. This variable contains a command to be executed before any new prompt is shown, and is therefore run after every interactive command is executed. |
Line 15: | Line 14: |
* A new login will be able to immediately scroll back through the history of existing logins. So if you wish to run the same command in two sessions, run the command and then initiate the second login and you will be able to retrieve the command immediately. | * A new login will be able to immediately scroll back through the history of existing logins. So if you wish to run the same command in two sessions, run the command and then initiate the second login and you will be able to retrieve the command immediately. |
Line 18: | Line 17: |
To set all this, use the following in your own ~/.bashrc, or systemwide /etc/bash_profile or /etc/profile to affect all users: | To set all this, use the following in your own [[DotFiles|~/.bashrc]] file: {{{ HISTFILESIZE=400000000 HISTSIZE=10000 PROMPT_COMMAND="history -a" shopt -s histappend }}} In the above we have also increased the maximum number of lines of history that will be stored in memory, and removed any limit for the history file itself. The default for these is 500 lines, which will cause you to start to lose lines fairly quickly if you are an active user. By setting HISTFILESIZE to a large value we ensure a file big enough so that it is infinite in practice - and by setting $HISTSIZE, we limit the number of these lines to be retained in memory. Unfortunately, bash will read in the full history file before truncating its memory copy to the length of $HISTSIZE - therefore if your history file grows very large, bash's startup time can grow annoyingly high. Even worse, loading a large history file then truncating it via $HISTSIZE results in bloated resource usage; bash ends up using much more memory than if the history file contained only $HISTSIZE lines. Therefore if you expect your history file to grow very large, for example above 20,000 lines, you should archive it periodically. See ''Archiving History Files'' below. PROMPT_COMMAND may already be used in your setup, for example containing control codes to update an XTerm's display bar with your current prompt. If yours is already in use, you can append to it with: {{{PROMPT_COMMAND="${PROMPT_COMMAND:-:} ; history -a"}}} You may also want to set the variables HISTIGNORE and HISTCONTROL to control what is saved, for example to remove duplicate lines - though doing so prevents you from seeing how many times a given command was run by a user, and precisely when (if HISTTIMEFORMAT is also set). Note that because PROMPT_COMMAND executes just before a new prompt is printed, you may still lose the last command line if your shell terminates during the execution of this line. As an example, consider: {{{this_cmd_is_never_written_to_history ; kill -9 $$}}} === Using extended attributes === Even after you've convinced bash to record history without truncating the history file, it's still very easy to lose. If you ever start a shell in interactive mode without the shell sourcing your `.bashrc` for any reason (e.g. via the `--rcfile` option) bash will default to indiscriminately truncating the history, which could mean losing everything unless you archive and backup the file as detailed in the next sections. Under Linux and some other OSes, this can be prevented by setting the append-only extended attribute on the history file. Subsequently, `open(2)` calls without the `O_APPEND` flag will fail and the file cannot be deleted, moved, truncated, or otherwise modified other than to append data to the end (even by the root user) until the append-only bit is unset. Usually only a root user can set or unset this attribute, and the exact method varies by OS. Support likely varies by file system as well. {{{ # Linux example - setting the append-only flag with chattr(1) ormaaj-laptop # chattr +a .bash_history ormaaj-laptop # lsattr -a .bash_history -----a---------- .bash_history ormaaj-laptop # rm .bash_history rm: cannot remove '.bash_history': Operation not permitted ormaaj-laptop # >.bash_history bash: .bash_history: Operation not permitted }}} === Prevent mangled history with atomic writes and lock files === TODO... === Compressing History Files === The result of the above is a history file with a great many duplicate commands. Appending history causes your history file to grow by all the shell's loaded history each time. More importantly, the main thing we care about with regards to history is being able find previously executed commands. The following script will remove all commands from the history file that are already in there, while keeping the order of the commands intact in such a way that commands you most recently executed will remain at the bottom of the file (ie. keep the last occurrence of a command, not the first). {{{ awk 'NR==FNR && !/^#/{lines[$0]=FNR;next} lines[$0]==FNR' "$HISTFILE" "$HISTFILE" > "$HISTFILE.compressed" && mv "$HISTFILE.compressed" "$HISTFILE" }}} After a few months, this compressed my history file from 761474 lines to 2349. Note that this does not preserve the timestamps if you have HISTTIMEFORMAT set. === Archiving History Files === Once you have enabled these methods, you should find that your bash history becomes much more valuable, allowing you to recall any command you have executed at any time. As such, you should ensure your history file(s) are included in your regular backups. You may also want to enable regular archiving of your history file, to prevent the full history from being loaded into memory by each new bash shell. With a history file size of 10,000 entries, bash uses approximately 5.5MB of memory on Solaris 10, with no appreciable start-up delay (''with $HOME on a local disk, I assume?'' -- GreyCat). With a history size of 100,000 entries this has grown to 10MB with a noticeable 3-5 second delay on startup. Periodic archiving is advisable to remove the oldest log lines and to avoid wasting resources, particular if RAM is at a premium. (My largest ~/.bash_history is at 7500 entries after 1.5 months.) This is best done via a tool that can archive just part of the file. A simple script to do this would be: |
Line 20: | Line 77: |
HISTFILESIZE=500000 HISTSIZE=500000 PROMPT_COMMAND="history -a" export HISTFILESIZE HISTSIZE PROMPT_COMMAND |
#!/bin/bash umask 077 max_lines=10000 |
Line 25: | Line 81: |
shopt -s histappend}}} | linecount=$(wc -l < ~/.bash_history) |
Line 27: | Line 83: |
In the above we have also increased the maximum number of lines of history that will be stored in memory, and in the history file. The default is 500 lines, which will cause you to start to lose lines fairly quickly if you are an active user. The value of 500000 is arbitrarily large enough to never be exceeded in most circumstances; you may wish to use a lower figure if you are not interested in older commands. Alternatively, you could use an external log rotator tool to rotate your .bash_history on a time or length basis. | if (($linecount > $max_lines)); then prune_lines=$(($linecount - $max_lines)) head -$prune_lines ~/.bash_history >> ~/.bash_history.archive \ && sed -e "1,${prune_lines}d" ~/.bash_history > ~/.bash_history.tmp$$ \ && mv ~/.bash_history.tmp$$ ~/.bash_history fi }}} |
Line 29: | Line 91: |
PROMPT_COMMAND may already be used in your system, for example containing control codes to update an XTerm's display bar with your current prompt. If yours is already in use, you can append to it with: {{{PROMPT_COMMAND="$PROMPT_COMMAND ; history -a"}}} | This script removes enough lines from the top of the history file to truncate its size to X lines, appending the rest to ~/.bash_history.archive. This mimics the pruning functionality of HISTFILESIZE, but archives the remainder rather than deleting it - ensuring you can always query your past history by grepping ~/.bash_history*. |
Line 31: | Line 93: |
You may also want to set the variables HISTIGNORE and HISTCONTROL to control what is saved, for example to remove duplicate lines - though doing so prevents you from seeing how many times a given command was run by a user, and precisely when (if HISTTIMEFORMAT is also set.) | Such a script could be run nightly or weekly from your personal crontab to enable periodic archiving. Note that the script does not handle multiple users and will archive the history of only the current user - extending it to run for all system users (as root) is left as an exercise for the reader. |
Line 33: | Line 95: |
Finally, note that because PROMPT_COMMAND executes just before a new prompt is printed, you may still lose the last command line if your shell terminates during the execution of this line. As an example, consider: {{{this_cmd_is_never_written_to_history ; kill -9 $$}}} | === Archiving by month === |
Line 35: | Line 97: |
Once you have enabled these methods, you should find that your history resource becomes much more valuable, allowing you to recall any command you have executed at any time. As such, you should ensure your history file(s) are included in your regular backups. | {{{ # https://github.com/kaihendry/dotfiles mkdir -p ~/bash_history shopt -s histappend HISTCONTROL=ignoredups PROMPT_COMMAND="history -a; history -n; $PROMPT_COMMAND" # If your bash is older than 4.3, set these to a large number instead # else your history files will be empty HISTFILESIZE=-1 HISTSIZE=-1 HISTFILE=~/bash_history/$(date +%Y-%m) h() { grep "$@" ~/bash_history/* } }}} https://youtu.be/DJ_HdmfA72E . ''What happens when the date changes to a new month while your shell is still running? Then HISTFILE is pointing to the wrong place.'' |
How can I avoid losing any history lines?
This method is designed to allow you to store a complete log of all commands executed by a friendly user; it is not meant for secure auditing of commands - see securing bash against history removal.
By default, Bash updates its history only on exit, and it overwrites the existing history with the new version. This prevents you from keeping a complete history log, for two reasons:
- If a user is logged in multiple times, the overwrite will ensure that only the last shell to exit will save its history.
- If your shell terminates abnormally - for example because of network problems, firewall changes or because it was killed - no history will be written.
To solve the first problem, we set the shell option histappend which causes all new history lines to be appended, and ensures that multiple logins do not overwrite each other's history.
To prevent history lines being lost if Bash terminates abnormally, we need to ensure that they are written after each command. We can use the shell builtin history -a to cause an immediate write of all new history lines, and we can automate this execution by adding it to the PROMPT_COMMAND variable. This variable contains a command to be executed before any new prompt is shown, and is therefore run after every interactive command is executed.
Note that there are two side effects of running 'history -a' after every command:
- A new login will be able to immediately scroll back through the history of existing logins. So if you wish to run the same command in two sessions, run the command and then initiate the second login and you will be able to retrieve the command immediately.
- More negatively, the history commands of simultaneous interactive shells (for a given user) will be intertwined. Therefore the history is not a guaranteed sequential list of commands as they were executed in a single shell. You may find this confusing if you review the history file as a whole, looking for sections encapsulating particular tasks rather than searching for individual commands. It's probably only an issue if you have multiple people using a single account simultaneously, which is not ideal in any case.
To set all this, use the following in your own ~/.bashrc file:
HISTFILESIZE=400000000 HISTSIZE=10000 PROMPT_COMMAND="history -a" shopt -s histappend
In the above we have also increased the maximum number of lines of history that will be stored in memory, and removed any limit for the history file itself. The default for these is 500 lines, which will cause you to start to lose lines fairly quickly if you are an active user. By setting HISTFILESIZE to a large value we ensure a file big enough so that it is infinite in practice - and by setting $HISTSIZE, we limit the number of these lines to be retained in memory. Unfortunately, bash will read in the full history file before truncating its memory copy to the length of $HISTSIZE - therefore if your history file grows very large, bash's startup time can grow annoyingly high. Even worse, loading a large history file then truncating it via $HISTSIZE results in bloated resource usage; bash ends up using much more memory than if the history file contained only $HISTSIZE lines. Therefore if you expect your history file to grow very large, for example above 20,000 lines, you should archive it periodically. See Archiving History Files below.
PROMPT_COMMAND may already be used in your setup, for example containing control codes to update an XTerm's display bar with your current prompt. If yours is already in use, you can append to it with: PROMPT_COMMAND="${PROMPT_COMMAND:-:} ; history -a"
You may also want to set the variables HISTIGNORE and HISTCONTROL to control what is saved, for example to remove duplicate lines - though doing so prevents you from seeing how many times a given command was run by a user, and precisely when (if HISTTIMEFORMAT is also set).
Note that because PROMPT_COMMAND executes just before a new prompt is printed, you may still lose the last command line if your shell terminates during the execution of this line. As an example, consider: this_cmd_is_never_written_to_history ; kill -9 $$
Using extended attributes
Even after you've convinced bash to record history without truncating the history file, it's still very easy to lose. If you ever start a shell in interactive mode without the shell sourcing your .bashrc for any reason (e.g. via the --rcfile option) bash will default to indiscriminately truncating the history, which could mean losing everything unless you archive and backup the file as detailed in the next sections.
Under Linux and some other OSes, this can be prevented by setting the append-only extended attribute on the history file. Subsequently, open(2) calls without the O_APPEND flag will fail and the file cannot be deleted, moved, truncated, or otherwise modified other than to append data to the end (even by the root user) until the append-only bit is unset. Usually only a root user can set or unset this attribute, and the exact method varies by OS. Support likely varies by file system as well.
# Linux example - setting the append-only flag with chattr(1) ormaaj-laptop # chattr +a .bash_history ormaaj-laptop # lsattr -a .bash_history -----a---------- .bash_history ormaaj-laptop # rm .bash_history rm: cannot remove '.bash_history': Operation not permitted ormaaj-laptop # >.bash_history bash: .bash_history: Operation not permitted
Prevent mangled history with atomic writes and lock files
TODO...
Compressing History Files
The result of the above is a history file with a great many duplicate commands. Appending history causes your history file to grow by all the shell's loaded history each time.
More importantly, the main thing we care about with regards to history is being able find previously executed commands. The following script will remove all commands from the history file that are already in there, while keeping the order of the commands intact in such a way that commands you most recently executed will remain at the bottom of the file (ie. keep the last occurrence of a command, not the first).
awk 'NR==FNR && !/^#/{lines[$0]=FNR;next} lines[$0]==FNR' "$HISTFILE" "$HISTFILE" > "$HISTFILE.compressed" && mv "$HISTFILE.compressed" "$HISTFILE"
After a few months, this compressed my history file from 761474 lines to 2349. Note that this does not preserve the timestamps if you have HISTTIMEFORMAT set.
Archiving History Files
Once you have enabled these methods, you should find that your bash history becomes much more valuable, allowing you to recall any command you have executed at any time. As such, you should ensure your history file(s) are included in your regular backups.
You may also want to enable regular archiving of your history file, to prevent the full history from being loaded into memory by each new bash shell. With a history file size of 10,000 entries, bash uses approximately 5.5MB of memory on Solaris 10, with no appreciable start-up delay (with $HOME on a local disk, I assume? -- GreyCat). With a history size of 100,000 entries this has grown to 10MB with a noticeable 3-5 second delay on startup. Periodic archiving is advisable to remove the oldest log lines and to avoid wasting resources, particular if RAM is at a premium. (My largest ~/.bash_history is at 7500 entries after 1.5 months.)
This is best done via a tool that can archive just part of the file. A simple script to do this would be:
#!/bin/bash umask 077 max_lines=10000 linecount=$(wc -l < ~/.bash_history) if (($linecount > $max_lines)); then prune_lines=$(($linecount - $max_lines)) head -$prune_lines ~/.bash_history >> ~/.bash_history.archive \ && sed -e "1,${prune_lines}d" ~/.bash_history > ~/.bash_history.tmp$$ \ && mv ~/.bash_history.tmp$$ ~/.bash_history fi
This script removes enough lines from the top of the history file to truncate its size to X lines, appending the rest to ~/.bash_history.archive. This mimics the pruning functionality of HISTFILESIZE, but archives the remainder rather than deleting it - ensuring you can always query your past history by grepping ~/.bash_history*.
Such a script could be run nightly or weekly from your personal crontab to enable periodic archiving. Note that the script does not handle multiple users and will archive the history of only the current user - extending it to run for all system users (as root) is left as an exercise for the reader.
Archiving by month
# https://github.com/kaihendry/dotfiles mkdir -p ~/bash_history shopt -s histappend HISTCONTROL=ignoredups PROMPT_COMMAND="history -a; history -n; $PROMPT_COMMAND" # If your bash is older than 4.3, set these to a large number instead # else your history files will be empty HISTFILESIZE=-1 HISTSIZE=-1 HISTFILE=~/bash_history/$(date +%Y-%m) h() { grep "$@" ~/bash_history/* }
What happens when the date changes to a new month while your shell is still running? Then HISTFILE is pointing to the wrong place.