Shell Command Line Quoting Mechanisms

Ian! D. Allen – www.idallen.com

Winter 2016 - January to April 2016 - Updated 2021-09-03 12:03 EDT

1 Introduction to QuotingIndexup to index

Single quotes, double quotes, and backslashes tell the shell which parts of command lines to treat as ordinary (not special) characters.

The quoting delimits (identifies) a string of characters; the quoting mechanism is removed and is not part of the string passed to the command. Only the characters being quoted are passed to the command.

Another Quote Tutorial: http://www.grymoire.com/Unix/Quote.html

1.1 The Bobby Tables ProblemIndexup to index

The Bobby Tables Problem – http://xkcd.com/327/

Below and in the next link are explanations of what happened in the above comic.

Databases are accessed using a query language, typically Structured Query Language. A database query in SQL might look like this:

SELECT * FROM Students WHERE ( NAME = '$NAME' ) ;

The variable $NAME is the name of the student, and the query would be modified by inserting that name into the query in place of the $NAME variable. For a normal student name used in place of $NAME, e.g. Leslie Smith, the query simply looks up the student by name in the database:

SELECT * FROM Students WHERE ( NAME = 'Leslie Smith' ) ;

For little Bobby Tables, his name Robert'); DROP TABLE Students; also gets inserted into the database query in place of $NAME in the same way, but the presence of quotes and other special characters in the name completely changes the meaning of the query:

SELECT * FROM Students WHERE ( NAME = 'Robert'); DROP TABLE Students;' ) ;

The SQL query now tries to find a student named 'Robert' and then (after the semicolon ;) deletes the entire Students table from the database.

“Sanitizing inputs” refers to the process of removing special characters from the input so that they don’t defeat the quoting used in the query and change the meaning of the query.

2 Quoting – hiding characters from the shellIndexup to index

Shells read input and find and run commands, passing command line arguments to the commands. Shells treat a number of characters specially on the command lines; these are called shell “meta-characters”.

The most common shell meta-character is the blank or space character that shells use to separate arguments. The shell separates arguments with one blank or a dozen blanks. Other shell meta-characters include: $, *, ;, >, ?, &, |, etc.

“Quoting” is the generic name given to the action of protecting shell meta-characters from being treated specially by the shell, usually by surrounding the string containing the special character(s) with quotes, or using backslashes in front of the individual characters:

$ echo Semicolon is special; it divides command lines.
Semicolon is special
it: command not found

$ echo Semicolon is special\; quoting hides it from the shell.
Semicolon is special; quoting hides it from the shell.

$ echo 'Semicolon is special; quoting hides it from the shell.'
Semicolon is special; quoting hides it from the shell.

Quoting hides the special character(s) from processing by the shell. The characters are passed, unchanged, to the command.

2.1 Why do we need a shell quoting mechanism?Indexup to index

Sometimes you don’t want the shell to act on its meta-characters; you want these characters passed to a command as part of the command arguments:

$ touch a b c d
$ echo *** BOOM ***                     # shell expands GLOB characters
a b c d BOOM a b c d
$ echo "*** BOOM ***"                   # quoting protects characters
*** BOOM ***

$ ls My Documents                       # shell splits command line on blanks
ls: cannot access My: No such file or directory
ls: cannot access Documents: No such file or directory
$ ls 'My Documents'                     # hide the space from the shell
resume.odt  assignment.txt

The quoting mechanism protects the special shell meta-characters from expansion by the shell. The protected characters are passed as part of command arguments.

3 Which programs have quoting mechanisms?Indexup to index

The shell treats quotes and backslashes specially when they are used on a command line or in a shell script. Shells do this because shells are designed to find and run commands, and supply arguments to the commands.

Sometimes you need to supply arguments to commands that contain special meta-characters that the shell would otherwise expand or process. The quoting mechanism stops the shell from processing the special characters.

Double quotes, single quotes, and backslashes can quote other shell meta-characters only when typed as command-line input to the Unix shells.

Quotes and backslashes are not special characters when fed as input to most other non-shell programs such as cat, head, sort, wc, etc.:

$ echo "It's a nice day"           # shell will process a quoted string
It's a nice day.                   # note how quotes have been removed

$ cat
"It's a nice day"                  # typed input to cat command
"It's a nice day"                  # unchanged program output on stdout
^D

$ head -1
"It's a nice day"                  # typed input to head command
"It's a nice day"                  # unchanged program output on stdout

Most non-shell programs do not treat quotes or backslashes specially when they read them as input. They are just normal characters.

3.1 Quoting used by other programs such as grep and findIndexup to index

Backslash-style quoting may also be used by other programs to hide special characters in their command arguments, which results in us having to quote the backslashes to hide them from the shell so that the backslashes can then quote characters used in the command arguments.

We have to hide the backslash and other shell meta-characters from the shell using shell single quoting so that the programs get and use these characters and not the shell.

For example, the grep program uses backslash quoting to turn off the meaning of Regular Expression characters such as periods:

$ grep '\.' filename              # look for a real period character

The single quotes hide the backslash from the shell, and the backslash works inside grep to hide the period Regular Expression character from grep.

The find program also uses backslashes to prevent GLOB expansion inside its --name argument, and again we have to hide the backslashes (and the GLOB characters) from the shell inside single quotes so that find gets them:

$ find . -name '\*boom.txt' -o -name '\?query.txt'

The backslashes tell find to look for a real asterisk and a real question mark, not the GLOB pattern characters asterisk and question mark.

This document focuses on shell quoting mechanisms: Hiding shell meta-characters from the shell so that commands can receive them as command line arguments.

4 Three quoting mechanisms: double, single, backslashIndexup to index

Quoting is done using either double quotes, single quotes, or backslash characters. Not all quoting mechanisms protect all shell meta-characters.

While technically we only need to apply the quoting to the individual shell meta-characters we want to protect from the shell, not to the whole command line argument, it usually looks better to surround the whole argument with matching quote characters. For example, we can use double or single quotes to “quote” the asterisk and blanks in argument to the echo command:

$ echo "*   "star              # quote only the four meta-characters
*   star

$ echo "*   star"              # quoting the whole argument looks better
*   star

$ echo '*   star'              # single quotes work, too
*   star

Backslashes may also be used to “quote” or turn off the special meaning of each immediately following character, one at a time. Each backslash only protects the one single character immediately to its right. For example:

$ echo \*\ \ \ star
*   star

On a shell command line, a “quoted” semicolon (;) is not a meta-character and it does not separate commands. A “quoted” asterisk (*) is not a GLOB meta-character and it does not match file names:

$ echo one ; echo two                    # shell splits on semicolon
one
two

$ echo one ";" echo two                  # quoting protects semicolon
one ; echo two

$ echo "one ; echo two"                  # style: quote the whole argument
one ; echo two

$ touch a b c d                          # create four file names
$ echo *                                 # shell expands GLOB character
a b c d

$ echo "*"                               # quoting protects GLOB character
*

$ echo \*                                # quoting protects GLOB character
*

Quoting is used to prevent the shell from acting on and expanding its meta-characters. The quoting causes the shell to ignore the special meaning of the character, so that the character gets passed unchanged to a command as part of a command argument.

4.1 Quoting blanks and spacesIndexup to index

The most common shell meta-character is the blank or space character that shells use to separate arguments. The shell does not normally pass any blanks or spaces to the command; the shell uses the blanks and spaces to separate and identify individual command line arguments, and the shell passes only those arguments to the command. One blank separates arguments the same way as ten or a hundred.

In both examples below, the echo command receives from the shell exactly three single-character arguments and no spaces:

$ echo a b c                             # one blank between arguments
a b c

$ echo a       b               c         # many blanks between; same thing
a b c

The same arguments would be given to any command, not just echo. A touch command would create three single-character files; a mkdir would create three single-character directories. No blanks are passed to the command; blanks serve only to separate the arguments.

A “quoted” blank is no longer a shell meta-character; it does not separate arguments. To pass the folder name My Documents to a command, the space in the name must be hidden from the shell by some form of quoting:

$ ls My" "Documents                      # quoted blank is part of name
$ ls My' 'Documents                      # quoted blank is part of name
$ ls My\ Documents                       # quoted blank is part of name

The quotes and backslashes are used by the shells to locate a string of characters to protect. The quotes and backslashes themselves are not part of the string; they are removed as the string of characters is collected. For example:

$ echo "hi  ho"
hi  ho

$ echo hi\ \ ho
hi  ho

The double quotes and backslashes above both delimit a single six-character string; the quoting characters are not part of the six characters. The shell collects one six-character command line argument, and passes that argument to the echo command. If the touch command had been used, it would create a six-character file name with two embedded spaces in the name. The quoting is removed from the argument.

The quotes and backslashes used in the quoting mechanism are not part of the string passed to the command – they only delimit the string and protect the meta-characters. The quoting mechanism is removed from the string as it is collected and before it is passed to the command.

Without the quoting mechanism, blanks act as meta-characters to separate command line arguments and in the example below the shell creates two two-character arguments for the echo command, instead of one six-character argument:

$ echo hi  ho
hi ho

Blanks are meta-characters to the shell unless they are hidden from the shell by quoting. The shell splits the command line into tokens on any unquoted blanks. You may type one blank to separate the arguments or a hundred blanks; it makes no difference to the shell or to the arguments:

$ echo a b c
a b c

$ echo a                        b                         c
a b c

The three one-character string arguments passed to the echo command in the above two cases are identical; the echo command sees none of the blanks because the shell uses unquoted blanks to separate arguments.

Blanks that are quoted are not seen as special meta-characters by the shell; they do not separate arguments; they become ordinary characters made part of the argument string passed to the command being invoked.

You can create a file name that is a single blank using any of these quoting mechanisms to turn off its meta-character meaning:

$ touch \                           # there is a space after the backslash
$ touch " "                         # there is a space inside the quotes
$ touch ' '                         # there is a space inside the quotes

5 Quoting is removed from the argument being quotedIndexup to index

Quotes and backslashes tell the shell which parts of the input to treat as ordinary (not special) characters. The quoting delimits (identifies) a string of characters; the quoting mechanism is removed and is not part of the string passed to the command. Only the characters being quoted are passed to the command, not any of the quotes or backslashes used to do the quoting:

$ echo "***"                       # argument is three characters, not five
***

$ echo '***'                       # argument is three characters, not five
***

$ echo \*\*\*                      # argument is three characters, not six
***

The shell removes the quoting mechanism before passing the delimited text as an argument to a command. In the all the examples above, the echo command receives one three-character command line argument containing three asterisks ***. The quotes and backslash characters used to protect the meta-characters are removed before the argument is passed to the command. The command sees only the one three-character argument, no matter what kind of quoting was used. All the quoting syntax is removed.

This next echo command line has six string arguments. None of the quoting mechanism is part of the arguments; it is all stripped out. The only blanks passed to the echo command are the ones hidden inside the quotes:

$ echo "a  b"     c     'd  e'    f     "g  h"     "i     j"
a  b c d  e f g  h i     j
^^^^ ^ ^^^^ ^ ^^^^ ^^^^^^^
   1 2    3 4    5       6

Only the characters being quoted are passed to the command, not any of the quotes or backslashes used to do the quoting. All the unquoted blanks are stripped away; they only serve to separate the arguments.

6 Shell Quote Processing is Left to RightIndexup to index

Quotes are processed and matched on shell command lines in strict left-to-right order. Let’s examine how this works using this example:

$ echo 'one "two" three'four"five"six"seven 'eight' nine"ten
one "two" threefourfivesixseven 'eight' nineten

The above echo command line argument is actually one single argument string containing six pieces as seen by the shell:

  1. The first quote character found by the shell, starting from the left, is a single quote. The shell collects all the characters up to the next single quote as part of the first argument to echo. All these single-quoted characters are protected from further analysis by the shell. No characters are expanded or special inside single quotes.

    The words one, two, and three and the contained blanks and double-quotes are all inside this first single-quoted string. The double quotes, inside single quotes, have no special meaning to the shell. They are simply part of the string. Same with the blanks – they are quoted and thus do not perform their meta-character function of separating shell arguments. The single quotes themselves, used to delimit the single-quoted string, are not part of the string and thus are not part of the first argument being collected.

  2. Right after the closing single-quote, the word four is outside of any quotes. The word immediately follows the closing single quote; there is no space there to indicate the start of a second argument to echo and so the four is still part of the same first argument.

  3. Right after four, with no blank to start a new argument, we start a double-quoted string that extends until the next double-quote character. The characters inside this double-quoted string are still part of the first argument; the shell has not yet seen any unquoted blank that indicates the end of the first argument. The word five is inside this double-quoted string. The double quotes themselves, used to delimit the string, are not part of the string.

  4. Right after the closing double-quote, the word six is outside of any quotes. The word immediately follows the closing double quote; there is no space there to indicate the start of a second argument to echo and so the six is still part of the same first argument.

  5. Right after six we start another double-quoted string that extends until the next double-quote character. Most, but not all, double-quoted characters are protected from further analysis by the shell. (Double quotes are weaker than single quotes; some meta-characters inside double quotes do still expand.)

    The words seven, eight, and nine, the single-quotes, and the contained blanks are all inside this double-quoted string. The single quotes, inside double quotes, have no special meaning to the shell. They are simply part of the string. Same with the quoted blanks – they are quoted and thus do not perform their meta-character function of separating shell arguments. The double quotes themselves, used to delimit the string, are not part of the string.

  6. Right after the closing double-quote, the word ten is outside of any quotes, but since it is not separated from the other characters by any shell-visible blanks, it is still part of the same first argument.

The one command line argument to echo is built up of six pieces of text that includes three quoted strings. Each quoted string connected immediately to another non-blank character, and all the blanks that are inside quoted strings are protected from the shell. Thus, the echo command is handed only one command line argument. (Command line arguments are separated by meta-character blanks. There were no unquoted blanks used.)

The fact that this one argument was made up of six pieces on the command line, including both quoted and unquoted parts, is invisible to the echo command. The shell did the work; the echo command gets its one argument.

The quotes seen by the shell are not part of the argument. The quotes delimit the string; they are not part of the string.

Even though the shell used multiple quoting mechanisms to assemble the argument above, no unquoted blanks were found to separate arguments, so the string above is passed as one argument to the echo command.

7 When to use single quotes, double quotes, backslashesIndexup to index

The shells can use single- and double-quote characters to delimit and protect strings of characters used as arguments to commands. Backslashes also protect special characters from the shell; but, sometimes they are less convenient to use. For example, one could protect all the special shell meta-characters in a line of text, each with its own backslash:

$ echo \*\*\*\ Here\ is\ a\ backslash\ \"\\\".\ \*\*\*
*** Here is a backslash "\". ***

Note that quotes and backslashes are themselves meta-characters to the shell, so if you want quotes and backslashes to appear in command line arguments you have to quote them with backslashes to protect them!

Alternately, one could surround the shell meta-characters, and the whole argument string, with matching single quotes to protect the whole string from the shell:

$ echo '*** Here is a backslash "\". ***'
*** Here is a backslash "\". ***

This works because the string of text being quoted doesn’t itself contain any single quotes, so surrounding it with single quotes is possible.

In most cases the use of quotes is easier than using a lot of backslashes.

7.1 Single vs. Double QuotesIndexup to index

Single quotes are “stronger” than double quotes. Nothing is special inside single quotes; the shell treats all the characters, no matter what they are, as part of the string being collected and it does not expand any of them inside single quotes.

Inside double quotes, the shell still sees and expands some special meta-characters. The most important character expanded by the shell inside double quotes is the dollar sign that signals the start of a shell variable:

$ echo '$SHELL'
$SHELL                           # single quotes prevent expansion

$ echo "$SHELL"
/bin/bash                        # double quotes permit expansion

Backslashes are also still special meta-characters inside double-quoted strings, so you can use backslashes inside double quotes to protect other double quotes (and other backslashes, and dollars, etc.):

$ echo "This is a \"double\" quoted \$SHELL string and backslash \\."
This is a "double" quoted $SHELL string and backslash \.

For maximum protection and maximum quoting, use single quoted strings. The only character you can’t safely hide inside a single-quoted string is another single quote, since it ends the quoting.

7.2 How to quote text containing other quotes?Indexup to index

Which quotes can we use to surround a string that contains either single and double quotes? We need to put the single quotes inside of double quotes to protect them, and put the double quotes inside of single quotes to protect them:

$ echo "single ' quote"
single ' quote

$ echo 'double " quote'
double " quote

Inside a double-quoted string, a single quote is not a meta-character had has no meaning. Inside a single-quoted string, a double quote is not a meta-character and has no meaning.

7.2.1 Quotes inside quotes are not meta-charactersIndexup to index

Single quotes are not meta-characters inside of double-quoted strings; they are just ordinary characters. Double quotes are not meta-characters inside of single-quoted strings; they are just ordinary characters.

Adding single quotes inside double quotes doesn’t turn off variable expansion, because the shell doesn’t see the single quotes as special:

$ echo "This is my $SHELL variable."
This is my /bin/bash variable.

$ echo "This is my '$SHELL' variable."
This is my '/bin/bash' variable.

Adding double quotes inside single quotes doesn’t turn on variable expansion, because the shell doesn’t see the double quotes as special:

$ echo 'This is my $SHELL variable.'
This is my $SHELL variable.

$ echo 'This is my "$SHELL" variable.'
This is my "$SHELL" variable.

How can we quote some text that contains both kinds of quotes?

7.2.2 Quoting text containing both single and double quotesIndexup to index

To quote a command line argument that contains both types of quotes, we can alternate quoting mechanisms in the same line. We still put the single quotes inside of double quotes to protect them, and put the double quotes inside of single quotes to protect them:

$ echo "single ' and "'double " quote'
single ' and double " quote

Note how the double quoted section ends at the second double quote, after which we immediately start a single-quoted section that extends to the end of the line. The single quote is treated as an ordinary character inside double quotes and the double quote is treated as an ordinary character inside single quotes.

The shell sees no spaces in the argument above: all the spaces are hidden inside quotes. Without spaces to separate arguments, there is only one single argument string to echo The only space the shell sees is the one after the command name. If touch had been used, only one file would have been created.

7.2.3 Quoting text containing variablesIndexup to index

Sometimes you want to fully quote a piece of text that already contains double quotes, but still allow a variable inside the text to expand. You can’t simply single-quote the entire text, since that would prevent the variable from expanding:

$ echo 'The value of SHELL is "$SHELL".'
The value of SHELL is "$SHELL".

To allow the variable to be seen and expanded by the shell, you have to end the single-quoted string, double-quote the variable, then resume the single-quoted string:

$ echo 'The value of SHELL is "'"$SHELL"'".'
The value of SHELL is "/bin/bash".

You might also consider double-quoting the entire string and using backslashes (which do work inside double-quotes) to further quote any double-quotes inside the string:

$ echo "The value of SHELL is \"$SHELL\"."
The value of SHELL is "/bin/bash".

7.2.4 Always double-quote variable expansionsIndexup to index

Variables must always be expanded inside double quotes, so that metacharacters contained in the variables don’t themselves expand:

$ x='* BOOM *'
$ touch a b c d
$ echo $x                            # WRONG: missing double quotes!
a b c d BOOM a b c d
$ echo "$x"                          # correct use inside double quotes
* BOOM *

$ dir='My Documents'
$ ls $dir                            # WRONG: missing double quotes!
ls: cannot access 'My': No such file or directory
ls: cannot access 'Documents': No such file or directory
$ ls "$dir"                          # correct use inside double quotes
resume.odt  assignment.txt

You must always double-quote variable expansions, to prevent the shell from further blank-splitting and GLOB expansion of the substituted value.

8 Quote all grep and find expressionsIndexup to index

The shell normally doesn’t tell you that a GLOB expansion failed (unless you set the failglob option), and so you might not notice that an argument to grep contains an unquoted and unintended GLOB expression until the contents of the file system change one day:

$ mkdir empty ; cd empty
$ echo go*                         # shell fails to expand GLOB characters
go*

$ grep go* /etc/fstab              # shell fails to expand GLOB characters
[... several lines print here matching "go*" search pattern ...]

$ touch golf good goofs            # create three file names

$ grep go* /etc/fstab              # now the shell expands the go* GLOB pattern
$                                  # nothing found !!
$ echo grep go* /etc/fstab         # use echo to see the GLOB problem
grep golf good goofs /etc/fstab    # looks for pattern "golf" in three files !!

$ grep "go*" /etc/fstab            # quoting protects GLOB characters
[... several lines print here matching "go*" search pattern ...]

The same problem can happen with a find GLOB expression that the shell expands using GLOB instead of it being passed to and expanded by find:

$ mkdir empty ; cd empty
$ echo *go*                        # shell fails to expand GLOB characters
*go*

$ find /usr/bin -name *go*         # shell fails to expand GLOB characters
/usr/bin/pbmtogo
[... find finds many pathnames matching -name *go* pattern ...]
/usr/bin/gouldtoppm

$ touch xxxxgoxxxx                 # create a name containing "go"

$ echo *go*                        # now shell expands GLOB characters
xxxxgoxxxx
$ find /usr/bin -name *go*         # now shell expands GLOB characters
$                                  # NO OUTPUT FROM FIND !! 
$ echo find /usr/bin -name *go*    # use echo to see the GLOB problem
find /usr/bin -name xxxxgoxxxxx    # xxxxgoxxxxx does not exist in /usr/bin

$ touch algorithm                  # create another name with "go" in it

$ echo *go*                        # now shell expands GLOB characters
algorithm xxxxgoxxxxx
$ find /usr/bin -name *go*         # now shell expands GLOB characters
find: paths must precede expression: algorithm   # ERROR FROM FIND !!
$ echo find /usr/bin -name *go*    # use echo to see the GLOB problem
find /usr/bin -name algorithm xxxxgoxxxxx   # bogus argument to find

$ find /usr/bin -name '*go*'       # quoting protects GLOB characters
/usr/bin/pbmtogo
[... find finds many pathnames matching protected -name '*go*' pattern ...]
/usr/bin/gouldtoppm
$ echo find /usr/bin -name '*go*'  # use echo to see what happens
find /usr/bin -name *go*           # confirms GLOB characters not expanded

Don’t let the shell expand arguments to grep or find – quote the arguments to protect them from the shell and pass them unchanged.

9 Double Quote all Variable ExpansionsIndexup to index

Double quotes are essential when expanding variables, because without them the shell will blank-split and GLOB-expand the result of a variable substitution. The unquoted and unwanted GLOB expansion is harder to notice when it comes from inside the expansion of a shell variable:

$ mkdir empty ; cd empty
$ x='*** BOOM ***'                 # variable with GLOB characters
$ echo $x                          # shell fails to expand GLOB characters
*** BOOM ***
$ touch a b c d                    # create four file names
$ echo $x                          # now shell expands GLOB characters
a b c d BOOM a b c d
$ echo "$x"                        # quoting protects GLOB characters
*** BOOM ***

$ mkdir empty ; cd empty
$ x='We *go* far!'                 # variable with GLOB characters
$ echo $x                          # shell fails to expand GLOB characters
We *go* far!
$ touch Algonquin algorithm good   # create three file names
$ echo $x                          # now shell expands GLOB characters
We Algonquin algorithm good far!
$ echo "$x"                        # quoting protects GLOB characters
We *go* far!

$ mkdir empty ; cd empty
$ x='My Documents'                 # variable with blanks in it
$ mkdir $x ; ls -l                 # shell splits on blanks: two arguments
drwxr-xr-x  2 idallen idallen   40 Dec 12 11:55 Documents/
drwxr-xr-x  2 idallen idallen   40 Dec 12 11:55 My/
$ rmdir *
$ mkdir "$x" ; ls -l               # blanks are hidden from shell
drwxr-xr-x  2 idallen idallen   40 Dec 12 11:57 My Documents/

Unwanted shell splitting on blanks after variable expansion also causes problems, especially when the blanks create extra (or missing) arguments that cause syntax errors in test conditional logic in IF and WHILE statements:

$ arg="a"                          # simple argument works fine
$ [ $arg = a ] && echo hi          # unquoted $arg used - WRONG!
hi

$ arg="a b"                        # now $arg contains blanks
$ [ $arg = a ] && echo hi          # unquoted $arg used - WRONG!
sh: [: a: unexpected operator      # shell syntax error !

$ arg=" "                          # now $arg is a blank string
$ [ $arg = a ] && echo hi          # unquoted $arg used - WRONG!
sh: [: =: unexpected operator      # shell syntax error !

$ [ "$arg" = a ] && echo hi        # quoted $arg works correctly; no errors

Always double-quote your variable expansions, to prevent GLOB expansion and blank-splitting!

You must always double-quote variable expansions, to prevent the shell from further blank-splitting and GLOB expansion of the substituted value.

10 Quotes inside variables are not specialIndexup to index

Due to the order of processing of the command line, all quoted strings are located and identified by the shell before any variables are expanded. This means quotes embedded inside variables are not seen as special characters by the shell:

$ x='aaaaa " bbbbb'
$ echo $x
aaaaa " bbbbb

In the above echo command line, the shell first looks for and identifies quoted strings before it expands the variable $x. The double quote is hidden inside the variable and is not seen. Next, the shell expands the $x variable and splits it on blanks into three arguments. The double quote hidden inside $x appears “too late” – it is not treated as a special character by the shell. The echo commands sees three arguments.

Only “exposed” quotes are treated as special on shell command lines, not quotes embedded inside variables. Quotes inside variables are not special.

Exposed quotes are special; embedded quotes inside variables are not:

$ mkdir empty ; cd empty; touch a b c d ; ls
a  b  c  d
$ echo " * "
 *                              # quotes protect the glob character
$ q='"'
$ echo $q * $q                  # double quotes are inside the variables
" a b c d "                     # quotes inside variables are not special

Above, the quotes embedded inside the variable $q are not treated as special characters by the shell and they do not stop the GLOB expansions.

Note: Blanks and glob (wildcard) characters embedded inside unquoted variables are seen as special to the shell and may cause multiple arguments to be created and other problems:

$ x="a b c"
$ touch $x ; ls          # creates three files; embedded blanks are special
a  b  c
$ touch "$x" ; ls        # creates another single file named: a b c
a  a b c  b  c  d
$ y='*'
$ echo "$y"              # embedded glob char is protected by quoting
*
$ ls $y                  # unprotected glob char matches four file names!
a  a b c  b  c

You must double-quote all uses of variables to prevent the embedded blanks and glob (wildcard) characters from being seen as special by the shell after the variable expands.

11 Studies in Quoting Special CharactersIndexup to index

Understand how the shell handles quotes and blanks:

$ echo  hi            there
hi there

$ echo "hi            there"
hi            there

$ echo 'hi            there'
hi            there

Explain the above three outputs. How does the shell find arguments? How many arguments are passed to the echo command in each case?

The touch command creates empty files by name. Try this:

$ touch "a b"
$ ls
a b
$ rm a b

Explain the error message that is output by the above rm command. How many arguments are passed to the touch and rm commands? How can you remove a file name containing a special character?

Here are some more things to try, and to understand.

$ echo "'hello'"
'hello'
$ echo '"hello"'
"hello"

$ mkdir empty ; cd empty ; touch a b c d ; ls
a b c d
$ echo '     *     '
     *     
$ echo '"    *    "'
"     *     "
$ echo '"'   *   '"'
" a b c d "
$ echo '"'"  *  "'"'
"  *  "
$ echo '  *  '  *  "  *  "
  *   a b c d   * 
$ echo \'  *  \'
' a b c d '

You must be able to predict the output of each of the above command lines without having to type them in to try them.

12 Using argv.sh to count command line argumentsIndexup to index

How many arguments are there to the following echo command?

$ echo abc  \   def  \\   ghi   \ \    jkl

How many characters are in each of the arguments? Fetch and run the argv.sh program (available in the Class Notes) to help you:

$ ./argv.sh abc  \   def  \\   ghi   \ \    jkl
Argument 0 is [./argv.sh]
Argument 1 is [abc]
Argument 2 is [ ]
Argument 3 is [def]
Argument 4 is [\]
Argument 5 is [ghi]
Argument 6 is [  ]
Argument 7 is [jkl]

Since the shell is the program that parses arguments, the number of arguments passed to the argv.sh program will be exactly the same as the number of arguments passed to the echo program (or to any program).

When in doubt about how the shell will parse a command line, use echo or argv.sh to confirm the arguments that the shell is providing.

13 ADVANCED SHELL: Quoting for remote shells (ssh)Indexup to index

This is optional advanced material for people who execute shell commands on other machines: Unix/Linux Shell Quoting for remote shells

14 ADVANCED SHELL: eval: Using quotes inside shell variablesIndexup to index

This is optional advanced material for people who write scripts.

How to make quotes inside variables work using the shell eval mechanism.

This section is taken from a reply email I sent to the OCLUG mailing list:

From: "Ian! D. Allen" <idallen@idallen.ca>
To: General Membership Discussion List <oclug@lists.oclug.on.ca>
Subject: Re: [oclug] Must be Friday.....
> This doesn't work:
> $ SUDO="/usr/bin/sudo -u root -p \"Enter password for user '%U': \""
> $ $SUDO ls

Most shell meta-characters such as quotes, backslashes, line separators, pipes, redirection, etc. don’t have special meaning coming from inside shell variables. They also mean nothing if they appear in the command line due to GLOB pattern matches. The shell only treats them specially the first time it sees them on the command line:

$ mkdir empty ; cd empty ; touch a b c      # three visible files
$ echo " * "
 *                # the shell sees the quotes and protects the GLOB pattern
$ q='"'
$ echo $q * $q
" a b c "         # the hidden quotes have no special meaning

$ echo hi ; date
hi
Sun Feb 25 18:55:34 EST 2007
$ s=';'
$ echo hi $s date
hi ; date         # the hidden semicolon has no special meaning

$ echo hi >out    # put "hi\n" into file "out"
$ r=">out"
$ echo hi $r
hi >out           # the hidden >out has no special meaning

The user continues:

> # eval $SUDO ls
> Um.. okay, that works.  <scratching few remaining hairs on head>
> What is special about this case that requires the use of 'eval'?

The eval tells the shell to process the command line twice. This gives hidden meta-characters a second chance. The first time through, the shell expands the variable. The second time through, the shell now sees the quoted strings and treats them correctly (since they aren’t hidden inside the variable any more).

The user notes that this works without problems:

> # LS="ls -l"
> # $LS /usr

The above only works because no shell meta-character processing is needed. A simple variation fails:

# LSW="ls -l | wc"
# $LSW
ls: |: No such file or directory
ls: wc: No such file or directory

Don’t put commands inside variables – use shell aliases instead.

The user says:

> eval is the shell command that executes the contents of a shell variable.

The eval causes the shell to process what was in the variable as if you had typed it directly into the shell. It may “execute” and it may not, depending what else is on the command line in front of it:

$ v="date"
$ date=foo
$ eval $v
Sun Feb 25 19:06:12 EST 2007
$ eval echo $v
date
$ eval echo \$$v
foo

The last example above is how we do crude associative arrays in older versions of the Bourne shell. Assignment also has to use eval, since the shell won’t assign to a name hidden inside a variable:

$ echo $v $date
date foo
$ $v=hoho                              # wrong way
bash: date=hoho: command not found
$ eval $v=hoho                         # right way
$ echo $v $date
date hoho
Author: 
| Ian! D. Allen, BA, MMath  -  idallen@idallen.ca  -  Ottawa, Ontario, Canada
| Home Page: http://idallen.com/   Contact Improv: http://contactimprov.ca/
| College professor (Free/Libre GNU+Linux) at: http://teaching.idallen.com/
| Defend digital freedom:  http://eff.org/  and have fun:  http://fools.ca/

Plain Text - plain text version of this page in Pandoc Markdown format

Campaign for non-browser-specific HTML   Valid XHTML 1.0 Transitional   Valid CSS!   Creative Commons by nc sa 3.0   Hacker Ideals Emblem   Author Ian! D. Allen