% Unix/Linux Shell Quoting for remote shells % Ian! D. Allen -- -- [www.idallen.com] % Winter 2017 - January to April 2017 - Updated 2017-01-20 00:48 EST - [Course Home Page] - [Course Outline] - [All Weeks] - [Plain Text] Why Use Remote Command Lines ============================ System administrators often need to run command lines on remote machines without doing a full remote login. We just want the output of the one command line; that is all. We don't want to log in, run the command, and log out again. Or, we might want to run a command on a remote machine and redirect the output into a local file, which can't be done using a full login. For example, one might want to verify an account disk quota on a remote machine using the `quota` command remotely, without logging in. To run a command remotely, simply add the command line after the `ssh hostname` on the command line. You may need to accept the machine's host key, and you will need to enter your remote system password: $ ssh localhost hostname abcd0001@localhost's password: idallen-ubuntu $ $ ssh cst8207.idallen.ca quota abcd0001@cst8207.idallen.ca's password: Disk quotas for user abcd0001 (uid 1000): Filesystem blocks quota limit grace files quota limit grace /dev/sda1 52 204800 512000 15 5000 8000 /dev/sdc1 568 204800 512000 237 5000 8000 $ $ ssh cst8207.idallen.ca quota >quota.txt # save output in a file abcd0001@cst8207.idallen.ca's password: $ wc quota.txt 4 30 255 quota.txt The single command line is executed by a shell on the remote machine and the output is displayed. The remote output can be redirected into a file on the local machine. No full login to the remote machine is done. Another example: Perhaps you might need to create a directory on several (dozen? hundred?) machines. A quick way to do this is with a shell `for` loop that runs an `ssh` remote command line to each machine without needing to log in, type each command manually, and then log out: $ for h in host1 host2 host3 host4 host5 host6 host7 ; do > ssh $h.example.com mkdir newdir > done The above loop works best when you have enabled password-free public key logins to the remote machines, to avoid typing many passwords, but even without that it's much faster than having to log in to each machine, execute the command, then log out! Two Shells Read Every Command Line ================================== Two shells are involved in reading and parsing every command line sent to a remote system using `ssh`: 1. One shell is running on the local machine, reading your keyboard to collect the command line for `ssh`. 2. Another shell is running on the remote machine to execute the command line collected and sent there by `ssh`. The local shell processes the command line you type in, doing such things as local GLOB and variable expansion on the local machine, and then stripping off one layer of quotes and passing the result to to the `ssh` command, which will send the arguments to the remote machine. The remote machine passes the stripped command line arguments to a remote shell, and that remote shell does *more* GLOB and variable expansion and quote stripping on those arguments on the *remote* machine. The command line is processed by *two* shells. For example, we can run multiple commands on the remote machine by passing semicolons in the command line sent to the remote shell. We have to hide these semicolons from the local shell by quoting them. Without the quoting, the semicolons would be seen and acted on by the local shell, creating two commands in the local shell, like this: $ cd /tmp $ ssh cst8207.idallen.ca hostname ; pwd # unquoted semicolon idallen-ubuntu # hostname executes on remote system /tmp # pwd executes on local system Above, the `ssh` sends only the `hostname` command to the remote system; the local shell splits the command line we typed on the semicolon and the `pwd` runs on the local system. Using quotes to hide the semicolon, all the special characters can be sent to the remote shell, creating two commands on the *remote* machine: $ ssh cst8207.idallen.ca "hostname ; pwd" # quoted semicolon idallen-ubuntu # hostname executes on remote system /home/abcd0001 # pwd also executes on remote system This is what is happening in the `ssh` command above: 1. Quotes hide the semicolon metacharacter from the local shell so that it gets passed to the `ssh` command as part of an ordinary argument. (The quotes are removed from the argument by the local shell before handing the argument to the `ssh` command. Quotes delimit the argument, but are never made part of it when the argument is passed to the command, as described in [Quoting].) 2. The `ssh` command then passes its argument string to the shell on the remote machine. (Remember: The string does not have quotes around it when `ssh` receives it and passes it to the remote machine.) 3. The shell on the remote machine receives the argument string. It splits the string into tokens on unquoted blanks and acts on any unquoted shell metacharacters in the string, such as the semicolon used to put multiple commands on one line. 4. The semicolon shell metacharacter is seen by the remote shell. It allows multiple commands to be written on one line. The remote shell executes two commands (separated by the semicolon) and sends the two outputs back to our screen. The act of using SSH to send a command to a remote machine uses two shells. Each shell recognizes and strips a layer of quoting from the line being sent. You can see this by putting an `echo` in front of the command line, to see how the local shell strips one layer of quotes. The stripped line is what is actually being passed to the `ssh` command and then sent to the remote system's shell: $ echo ssh cst8207.idallen.ca "hostname ; pwd" # insert echo in front ssh cst8207.idallen.ca hostname ; pwd Above, the echo on the screen shows us that the `ssh` command is sending the argument string `hostname ; pwd` to the remote machine. (The string received by `ssh` and passed to the remote machine has no quotes around it, because the local shell stripped the quotes.) When that (unquoted) string argument `hostname ; pwd` is received by the shell on the remote machine, the remote shell will split the argument into three tokens (because shells split on unquoted blanks). The remote shell will see the unquoted semicolon metacharacter, and execute two separate commands on the remote machine. The output of the two commands will return and display on your local screen, where you could redirect it into a local file. Two shells are involved in every `ssh` remote command. One layer of quotes used on an `ssh` command line hides the metacharacters from expansion by the local shell, but not from expansion by the remote shell. Quoting Remote Command Lines Twice ================================== Two shells are involved in every `ssh` remote command. One layer of quotes used on an `ssh` command line hides the metacharacters from expansion by the local shell, but not from expansion by the remote shell. Where things get tricky is when the command line you want to run on the remote machine contains shell metacharacters that need quoting on *both* the local machine *and* on the remote machine. One level of quoting doesn't work: $ echo "Here today ; gone tomorrow." Here today ; gone tomorrow. $ ssh localhost echo "Here today ; gone tomorrow." Here today bash: gone: command not found To see the problem more clearly, again insert an `echo` into the start of the `ssh` command line to show what arguments are being passed to the `ssh` command for execution on the remote system: $ echo ssh localhost echo "Here today ; gone tomorrow." ssh localhost echo Here today ; gone tomorrow. Above, the echo on the screen shows us that the `ssh` command is sending an unquoted semicolon to the shell on the remote system. The remote shell splits the command line in two on the unquoted semicolon and tries to execute a nonexistent command named `gone`. To quote something on the *remote* machine in the *remote* shell requires careful thought. One level of quotes isn't enough. You need one level of quotes to hide things from the shell you're typing into on the local machine, and inside those quotes you need another level of quotes that get sent to the remote shell to do the quoting and hiding remotely as well. Let's modify our example and put in single quotes, inside the double quotes, to hide the semicolon from the shell on the remote system: $ echo ssh localhost echo "Here today ';' gone tomorrow." ssh localhost echo Here today ';' gone tomorrow. Above, the echo on the screen shows us that the `ssh` command will send a single-quoted semicolon to the remote system, which will keep it from being interpreted as a shell metacharacter. We can remove the leading `echo` and confirm that it works as expected: $ ssh localhost echo "Here today ';' gone tomorrow." Here today ; gone tomorrow. Before you think we've solved the problem, let's explore a bit further. Spaces are also shell meta-characters, and the above remote `echo` is not receiving exactly the same arguments as its local version. The local `echo` is receiving only *one* argument; the remote `echo` is receiving *five*: $ echo "Here today ; gone tomorrow." # one argument to echo $ ssh localhost echo "Here today ';' gone tomorrow." # five arguments to echo The remote shell executes an `echo` command with *five* arguments, not one, because the `ssh` command line is sent to the remote machine unquoted and the remote shell splits the line into tokens separated by spaces. This is a problem if the remote command is not expecting multiple arguments. To make the problem more obvious, let's try touching a file containing a space both locally and remotely: $ touch "my file" # creates one file $ ssh localhost touch "my file" # creates two files! Let's remind ourselves what `ssh` is passing to the remote system by inserting an `echo` in front of the `ssh` command line: $ echo ssh localhost touch "my file" # insert echo in front ssh localhost touch my file # two arguments to touch Above, the echo on the screen shows us that the `ssh` command is sending a command line that will cause two separate arguments to the `touch` command, not one. There are no quotes being sent to the remote system to hide the space in the file name. To fix the problem of the space in the name on the remote system, we need to hide the blanks from both the *local* shell and the *remote* shell by putting quotes inside of quotes. On easy way to do this is by putting single quotes inside double quotes: $ echo ssh localhost touch "'my file'" # single inside double ssh localhost touch 'my file' # one argument to touch Above, the echo on the screen shows us that the `ssh` command is now sending a command line containing one single-quoted argument to the `touch` command. We can remove the leading `echo` and the command will work as expected: $ ssh localhost touch "'my file'" # now creates one file We need "quotes inside of quotes" to hide all the metacharacters on both the local system and the remote system. We can also put double quotes inside single quotes to hide them. For example, suppose we want to run the command `mkdir "Space Dir Name"` on the remote machine. The command line below doesn't work; it creates three directory names not one directory name with two spaces in it. $ ssh localhost mkdir "Space Dir Name" # creates three directories Inserting a leading `echo` command at the start helps us see this: $ echo ssh localhost mkdir "Space Dir Name" # insert echo in front ssh localhost mkdir Space Dir Name # three arguments to mkdir Above, the echo on the screen shows us that the `ssh` command is sending a command line that will cause three arguments to the `mkdir` command, not one argument inside quotes. We have to hide the double quotes from the local shell so that they get sent as double quotes to the remote shell. One way to do this is to put the double quotes inside single quotes: $ ssh localhost mkdir '"Space Dir Name"' # creates one directory Inserting a leading `echo` command at the start helps us see this: $ echo ssh localhost mkdir '"Space Dir Name"' # insert echo in front ssh localhost mkdir "Space Dir Name" # one argument to mkdir Here is another example showing how one level of quotes isn't enough to preserve metacharacters on the remote system: $ ssh locahost echo "Hello World" # only one level of quotes Hello World # no quotes; blanks disappear $ echo ssh locahost echo "Hello World" # insert echo in front ssh locahost echo Hello World # no quotes are sent $ echo ssh locahost echo "'Hello World'" # put quotes inside quotes ssh locahost echo 'Hello World' # one level of quotes remain $ ssh localhost echo "'Hello World'" # remove the leading echo Hello World # quoted blanks are preserved Quotes that we want sent to the remote shell themselves need quoting, to protect those quotes from the local shell. Here is another example. We need to use a real asterisk (`*`) on the remote system, so it needs to be quoted **on the remote system**, as well as on the local system, to hide it from the shell on both systems. The first step is to know how to hide the asterisk from the local shell. Either single or double quotes will work: $ echo '*' # works fine locally * Now we need to add the extra quoting needed to pass both the single quotes and the asterisk to the remote shell. To make this work we use `echo` to develop a correctly quoted `ssh` command line. When the `echo` command shows us a correctly quoted line, we know it will work remotely. We see this doesn't work: $ echo ssh localhost echo '*' # WRONG ssh localhost echo * # WRONG - note lack of quotes The above line echoed on the screen shows that no quotes are being passed to the remote shell; that won't work on the remote machine because the remote shell will expand the unquoted asterisk. Let's hide the single quotes and the asterisk inside double quotes, and then it works: $ echo ssh localhost echo "'*'" # hide the single quotes ssh localhost echo '*' # now it has quotes $ ssh localhost echo "'*'" # remove the leading echo * # it works Quoting Quotes in remote command lines -------------------------------------- Suppose we want to `echo` a double quote on the remote system. The first step is to know how to hide the double quote from the local shell. Single quotes will work: $ echo '"' # double inside single " The command line below does not work; the first level of single quotes is stripped by the local shell and the double quote is sent alone to the remote system where it causes a syntax error: $ ssh localhost echo '"' # doesn't work with ssh bash: -c: line 0: unexpected EOF while looking for matching '"' bash: -c: line 1: syntax error: unexpected end of file Inserting a leading `echo` command at the start helps us see this: $ echo ssh localhost echo '"' # insert echo in front ssh localhost echo " # only one quote is being sent We need to quote the single quotes so that they get passed to the remote system to protect the double quote from the remote shell. The way we did this above was to put the whole string inside double quotes, but we can't do that if there is a double quote inside the string we want to surround with double quotes. The quotes won't match up correctly and we will get an *input continuation* prompt from the local shell: $ ssh localhost echo "'"'" # unmatched quote > # unmatched double quote! One solution is to hide the internal double quote from the local shell: $ ssh localhost echo "'\"'" # hide embeded double quote " # works! Another solution is to leave single quotes around the double quote to hide it from the local shell and send two more single quotes to the remote shell using backslashes to hide them locally: $ ssh localhost echo \''"'\' # add two single quotes " # works! Inserting a leading `echo` command at the start helps us see this: $ echo ssh localhost echo \''"'\' # insert echo in front ssh localhost echo '"' # correctly quoted Above, the echo on the screen shows us that the `ssh` command is sending a single-quoted double quote to the shell on the remote system. The single quotes will hide the double quote from the remote shell, as desired. Quoting Variables in remote command lines ----------------------------------------- Things get more complicated if we are trying to protect something that looks like a variable expansion in a remote command line. The following example shows how the shell variable `$$` is protected by single quotes when used locally, but it expands to be the process ID when the command line is sent to the remote system: $ echo 'It costs $$.' # single quotes stop expansion It costs $$. $ ssh localhost echo 'It costs $$.' # doesn't work remotely It costs 18029. # $$ variable expands $ echo ssh localhost echo 'It costs $$.' # insert echo in front ssh localhost echo It costs $$. # unprotected $$ variable Above, the echo on the screen shows us that the `ssh` command is sending an unquoted `$$` variable to the remote shell, which will expand it. To prevent this expansion by the remote shell, we also need to send single quotes to the remote shell, to protect the `$$` variable. You might think that you could simply double-quote the single-quoted string to protect the single quotes, as we did above, so that the single quotes would prevent the variable expansion on the remote system, but it's not that easy: $ ssh localhost echo "'It costs $$.'" # surround with double quotes It costs 25761. # doesn't work! $ echo ssh localhost echo "'It costs $$.'" # insert echo in front ssh localhost echo 'It costs 25761.' # $$ variable expanded locally Above, the echo on the screen shows us that the `ssh` command is now getting the single quotes correctly, but it is also getting an already-expanded `$$` variable, because the local shell expanded `$$` in the now-double-quoted string! We need to tell the local shell not to expand the `$$` inside the double-quoted string, and one way to do that is to use backslashes to hide the dollar signs inside the double-quoted string: $ ssh localhost echo "'It costs $$.'" # protect $$ from expansion It costs $$. # works! $ echo ssh localhost echo "'It costs $$.'" # insert echo in front ssh localhost 'It costs $$.' # correctly quoted Another way to send single quotes to the remote shell is to individually quote (using backslashes) the single quotes on either side of the single-quoted string being sent, just as we did in the case of the single-quoted double quote earlier: $ echo ssh localhost echo \''It costs $$.'\' # add two more single quotes ssh localhost echo 'It costs $$.' # this looks good $ ssh localhost echo \''It costs $$.'\' # remove the leading echo It costs $$. # it works! $ ssh localhost echo "'"'It costs $$.'"'" # another way to use single quotes It costs $$. # it works! The last example above shows that you can also double-quote the single quotes to hide them from the local shell, instead of using backslashes to hide them. Quoting both variables and quotes in remote command lines --------------------------------------------------------- Things get much more complicated if we are trying to protect both variable expansions and different types of quotes all in a single remote command line. Getting the mixed quoting right is tricky even when you only have to quote for one local shell. Suppose we want to echo the string `It's "not" easy $$.` locally. This string contains both single and double quotes, and also dollar signs. All of these have to be hidden from the local shell: $ echo "It's \"not\" easy $$." # use backslashes It's "not" easy $$. $ echo 'It'"'"'s "not" easy $$.' # use alternating quotes It's "not" easy $$. Making this same `echo` work for a *remote* shell is not easy, since we need to quote the quotes so that the *remote* shell gets all the correct quoting shown above. A strategy to get different types of quotes sent to the remote shell is to single-quote the double quotes and double-quote the single quotes. We always start using a leading `echo` to see what would be sent to the remote shell first: $ echo ssh localhost echo "'It'"'"'"'"'"'"'s "'"not" easy $$.'"'" ssh localhost echo 'It'"'"'s "not" easy $$.' # looks good $ ssh localhost echo "'It'"'"'"'"'"'"'s "'"not" easy $$.'"'" # remove the leading echo It's "not" easy $$. # it works! You can also use backslashes to escape individual characters so that they are each sent to the remote shell: $ echo ssh localhost echo \'It\'\"\'\"\'s\ \"not\"\ easy\ $$.\' ssh localhost echo 'It'"'"'s "not" easy $$.' # looks good $ ssh localhost echo \'It\'\"\'\"\'s\ \"not\"\ easy\ $$.\' # remove the leading echo It's "not" easy $$. # it works! When the leading `echo` command shows us a correctly quoted line on the screen, we know it will work as a remote command without the leading `echo`. Always check your complex quoting with `echo`, first! Using a second shell instead of remote login ============================================ Sending a command line to a remote login shell works similarly to sending a command line from the current shell into another local `bash` shell through a pipeline. A pipeline is useful and faster for working the bugs out of a remote command line before you use it remotely through `ssh` since no passwords or remote connections are needed to pipe into `bash`: $ ssh localhost "date ; whoami ; echo 'Hello World'" # slow using ssh abcd0001@localhost's password: Tue Apr 15 23:35:29 EDT 2014 abcd0001 Hello World $ echo "date ; whoami ; echo 'Hello World'" | bash -u # fast using bash Tue Apr 15 23:36:29 EDT 2014 abcd0001 Hello World Above, we can see that in both the `ssh` case and the pipeline case two shells are involved and two levels of quotes are stripped. What works in the pipeline case will also work in a remote `ssh` command line. Use `bash` to help you debug your quoting quickly. Here is a previous example showing how one level of quotes isn't enough to preserve metacharacters on the remote system: $ echo echo "Hello World" | bash -u # only one level of quotes Hello World # no quotes; blanks disappear $ echo echo "'Hello World'" | bash -u # two layers of quotes Hello World # quoted blanks are preserved Another example: $ ssh localhost echo Eat\\\ \\\ \\\;\\\ \\\ Pray\\\ \\\ \\\;\\\ \\\ Linux Eat ; Pray ; Linux $ echo echo Eat\\\ \\\ \\\;\\\ \\\ Pray\\\ \\\ \\\;\\\ \\\ Linux | bash -u Eat ; Pray ; Linux While you are testing and developing a properly-quoted remote command line, you can get the same quoting output results more quickly using `echo` and a pipeline into a second copy of `bash` instead of using the much slower `ssh`. Once you get the `bash` quoting working, switch to using `ssh`. Using `eval` instead of remote login ==================================== The `bash` shell (and most other shells) contains a keyword that lets you strip two layers of quotes from a single command line. Use the shell keyword `eval` in place of `ssh localhost` to get the same effect. The example below shows using both remote shell, a second `bash` shell, and `eval` to test a quoted string: $ ssh localhost echo Eat\\\ \\\ \\\;\\\ \\\ Pray\\\ \\\ \\\;\\\ \\\ Linux Eat ; Pray ; Linux $ echo echo Eat\\\ \\\ \\\;\\\ \\\ Pray\\\ \\\ \\\;\\\ \\\ Linux | bash -u Eat ; Pray ; Linux $ eval echo Eat\\\ \\\ \\\;\\\ \\\ Pray\\\ \\\ \\\;\\\ \\\ Linux Eat ; Pray ; Linux Using `eval` removes two levels of quotes just the way that `ssh localhost` or a pipeline into `bash` does. You can use `eval` for testing because it's faster and doesn't need us to type in any `ssh` remote password. Using `argv.sh` to count arguments ================================== You can use the `argv.sh` script from the [Class Notes][All Weeks] to verify that a string is one single argument to the remote `echo` command: $ eval ./argv.sh Eat\\\ \\\ \\\;\\\ \\\ Pray\\\ \\\ \\\;\\\ \\\ Linux Argument 0 is [./argv.sh] Argument 1 is [Eat ; Pray ; Linux] -- | Ian! D. Allen, BA, MMath - idallen@idallen.ca - Ottawa, Ontario, Canada | Home Page: http://idallen.com/ Contact Improv: http://contactimprov.ca/ | College professor (Free/Libre GNU+Linux) at: http://teaching.idallen.com/ | Defend digital freedom: http://eff.org/ and have fun: http://fools.ca/ [Plain Text] - plain text version of this page in [Pandoc Markdown] format [www.idallen.com]: http://www.idallen.com/ [Course Home Page]: .. [Course Outline]: course_outline.pdf [All Weeks]: indexcgi.cgi [Plain Text]: 445_quotes_for_remote.txt [Quoting]: 440_quotes.html [Pandoc Markdown]: http://johnmacfarlane.net/pandoc/