Diff for "BashFAQ/110"

How do I copy a file to a remote system, and specify a remote name which may contain spaces?

All of the common tools for copying files to a remote system (ssh, scp, rsync) send the filename as part of a shell command, which the remote system interprets. This makes the issue extremely complex, because the remote shell will often mangle the filename. There are at least three ways to deal with the problem: NFS, careful encoding of the filename, or submission of the filename as part of the data stream.

First let's look at what does not work:

# Will not work
scp "my file" remote:"your file"

scp is basically a thin wrapper on top of ssh, which works by instructing the remote system's shell to open a file for writing. Since the filename is passed to the remote shell in the most naive way imaginable, the remote shell sees the space as an argument separator, and ends up creating a file named your.

Similar problems plague most of the "obvious" (but wrong) attempts to address the problem with other tools:

# Will not work
ssh remote "cat > 'your file'" < "my file"

# Will not work
rsync "my file" remote:"your file"

So, what works?

NFS

If you mount the remote host's file system onto your local host with NFS (or any other competent network file system sharing technology, including sshfs, or possibly even smbfs) then you can just perform a direct copy:

cp "my file" /remote/"your file"

Carefully encoding the remote name

Now, obviously if you know the remote name at the time you're writing the command, you can encode it in a way that you know the remote shell will be able to decipher. Usually this means adding one extra layer of quotes. For example, this works:

scp "my file" remote:"'your file'"

But in the general case, we won't know the exact remote filename at the time we're writing a script. It will be given to our script as an argument, or an environment variable, etc. In that case, we have to be clever enough to encode any possible filename.

The problem is further complicated by the fact that we don't necessarily know which shell the remote user is using. Just because you're using bash on your client workstation, that doesn't mean the remote system's sshd is going to spawn bash to parse your command. (And remember, scp sends a shell command over ssh, which some unknown remote shell is going to have to parse.) So, any solution we use must be as shell-agnostic as possible. That rules out bash's printf %q for example.

Given these constraints, the only remaining approach is to wrap single quotes around the entire filename. This means we also have to modify any existing single quotes that are already in the filename. So, our encoding goes like this:

q=\'
dest="'${dest//$q/$q\\$q$q}'"

This gives us a modified dest which has literal single quotes at the start and end, and which has replaced all internal ' characters with '\''. When this is passed to a remote shell for parsing, the result is our original filename.

So, a full copy function would look something like this:

# copyto <sourcefile> <remotehost> <remotefile>
copyto() {
    q=\'
    dest="'${3//$q/$q\\$q$q}'"
    scp "$1" "$2":"$dest"
}

Sending the filename in the data stream

This approach is a bit less portable, because it requires that bash be installed on the remote host (though not necessarily as the remote user's login shell). It is a more generalized solution, because in theory any kind of data can be passed in the stream, as long as you can write a parser for it (but remember, you have to send the parser to the remote system for execution, so it needs to be simple).

In this example, we are going to send a data stream which has two things in it: a filename, and the file's contents. They will be separated by a NUL byte. We use bash to parse this stream on the remote system, because it is one of the very few shells that can parse NUL-delimited data streams.

# copyto <sourcefile> <remotehost> <remotefile>
copyto() {
    { printf '%s\0' "$3"; cat < "$1"; } |
    ssh "$2" bash -c \''read -rd ""; cat > "$REPLY"'\'
}

Our parser is the bash command read -rd ""; cat > "$REPLY". This reads the filename (terminated by NUL) into the shell variable REPLY, then calls cat to read the remainder of the stream. There are two quoting layers around the parser, because we need to quote it for our local shell and for the remote shell. So, we avoid all use of single quotes in the parser, use single quotes for the local layer, and escaped single quotes for the remote layer.

This version does not use scp, so it doesn't copy the file's permissions. If you want to do that, you could pass the permissions as another object in the data stream, parse it out, and call chmod. (There is no portable way to retrieve a local file's permissions, so that's actually the hardest part.)