I saw this command somewhere: :(){ :|:& } (fork bomb). How does it work?
First of all -- and this is important -- please do not run this command. I've actually omitted the trigger from the question above, and left only the part that sets up the function.
Here is that part, but written out in normal shell coding style, rather than rammed all together:
{{{:() {
: | : &
} }}}
What this does is create a function named : which calls itself recursively. Twice. In the background. Since the function keeps calling itself over and over (forking new processes), forever, this quickly consumes a lot of system resources. That's why it's called a "fork bomb".
If you still don't see how it works, here is an equivalent, which creates a function named bomb instead of :
{{{bomb() {
bomb | bomb &
} }}}
Inside the function, the first instance of the function writes its output to the input of the second instance of the function. As a result, neither instance can terminate until the output of the first instance is closed. Since each instance only generates new instances this never happens. As a result each 'fork' of the function stays in memory.
Theoretically, anybody that has shell access to your computer; be it chrooted or otherwise limited; can use such a technique to consume all the resources they have access to. If the user's resources are unlimited then in a matter of seconds all the resources of your system will be used and it will deadlock itself. Any attempt made by the kernel to free more resources will just result in more instances of the function.
As a result, the only way to protect yourself from such abuse is by limiting the maximum allowed amount of resources (for non-root users). This can be achieved through the usage of resource quotas; for example by modifying the rlimit defaults.