How a Unix/Linux Shell Script is Executed

Ian! D. Allen - idallen@idallen.ca - www.idallen.com

Winter 2013 - January to April 2013 - Updated 2012-12-20 04:19 EST

1 IntroductionIndexup to index

When you type a command name or program name into a shell, e.g. “date” or “./myscript.sh” the shell will try to find an executable file to execute.

Executable files are ones that have Unix execute permissions set. If the name you type has no slashes in it, the shell searches the directories of your $PATH to find the first executable file with that name. If the name you type has slashes, the shell tries to access that pathname directly.

When the shell finds an executable file, the shell forks itself into a second process and then tells the Unix/Linux kernel try to replace that second process with the executable code in the file, causing the file to execute.

If the executable file contains a machine code binary program (e.g. ls, sort, cat, grep, etc.), the kernel will load the program into memory and execute it. When the binary program finishes, that means the second process (forked by the shell) is finished and the first shell process issues another prompt for another command line, and the process repeats when you type the next command name or pathname to the shell.

The kernel can only load into memory and execute machine code (binary) programs. The kernel looks at the first two bytes of the file to decide if the file is in the proper machine code format.

The kernel cannot execute directly a text file, even if the text file is an executable shell script. Something else happens for executable text files (shell scripts).

If an executable file found by the shell is a text file such as a shell script (it is not a machine code binary program), one of two things will happen when the kernel tries to execute this file:

1.1 Script starts with #! lineIndexup to index

If the text script file starts with the two characters #! followed by the absolute pathname of a file containing a machine code binary program (which might be a shell, or some other binary program), the kernel execution switches to the loading of that binary program.

Instead of trying to load the executable script file itself into memory, as it would do for a binary program, the kernel actually loads the binary program named after the #! into memory, not the text of the shell script itself. (Recall that only machine code binary programs can be loaded into memory and executed.)

Once the kernel starts up that specified binary program, the kernel hands the name of the text shell script to the binary program as its first pathname argument.

Therefore, the kernel never actually executes a shell script - the kernel executes some specified binary program (whatever program is named in the first line of the script), and it is up to that program to process the script name given as its first pathname argument. See below for more details.

The #! must be the very first two characters of the script file for this to work. No blank lines or spaces are allowed in front of the #!.

1.2 Script does not start with #! lineIndexup to index

If the text script file does not start with the #! followed by the name of a binary program to use (no #! line at the start of the file), the kernel fails in its attempt to execute the executable text file.

Most shells (including all the Bourne and C shells) notice the failure. On a failure to execute an executable text file, the shell forks itself into a second process, and uses that second shell process to read the script file, line by line, and execute the commands contained in it. The shell that is used to read the commands in the script file is chosen based on some historic heuristics based on the first line of the file, and there is no guarantee that the shell chosen will correctly match the syntax of the script file. This is a Bad Thing.

Shell scripts should always start with “#!” followed by the absolute pathname of the correct program that the kernel should execute. Do not let the shell use heuristics to choose which program to run (Method #2, above). Always specify the “shebang” #! line as the first line of your scripts (Method #1, above).

2 How the kernel processes the first lineIndexup to index

If you correctly specify an executable binary program on the #! line at the start of your executable text script, the kernel will load that program into memory and pass it the name of your script as its first pathname argument when you “execute” the shell script. (In fact, as mentioned earlier, you can’t actually “execute” a text shell script; you can only execute the machine code binary program named in the first line of the script. That binary program will be given the name of your text script file as an argument.)

Here is an example of how to (1) create a short shell script that has a correct first line, (2) make it executable, and then (3) use your shell to “execute” it:

$ cat >myscript.sh              # use "cat" with output redirection
#!/bin/sh -u                    # 1. type in three lines of script
echo "This is a shell script."  # 2. type in three lines of script
date                            # 3. type in three lines of script 
^D                              # signal EOF to "cat"
$ chmod +x myscript.sh          # make it executable
$ ./myscript.sh                 # "execute" the script
This is a shell script.
Sun Feb 16 10:50:27 EST 2003
$

When you enter “./myscript.sh” on your shell command line, your shell does not search $PATH to locate the executable file; because, “./myscript.sh” has a slash in it. Shells ignore $PATH if the name you type has a slash in it. Your shell tries to execute this pathname (a pathname in the current directory) directly.

Because this executable text file starts with a correct “#!/bin/sh -u” line, the Unix kernel loads the /bin/sh shell program into memory and hands the shell the name of your script as its first pathname argument. You typed in “./myscript.sh” to your shell; but, what actually gets executed by the Unix kernel is this:

/bin/sh -u ./myscript.sh

The string “/bin/sh -u” names a machine code binary program to load; it comes directly from the first line of the shell script. The string “./myscript.sh” comes from what you typed on the command line to “execute” this shell script. Thus, the /bin/sh shell is passed an option argument of “-u” and the name of a script file to read.

The above command line, executed by the Unix kernel, causes the /bin/sh shell to read the script file “./myscript.sh” and execute the commands it contains. Note that all lines starting with “#” (including the first line) are comments to the /bin/sh program and are ignored as it reads the text file. The #! first line of the script is just a comment to the shell - it is ignored by the shell.

The “#!” line only has meaning to the Unix kernel when the kernel tries to execute the file; the line is a comment line when any shells read it.

3 Typing errors on the first lineIndexup to index

The kernel is very literal about what you type - whatever you type on the #! line at the start of the script becomes the first part of the command that gets executed. If your script starts with an incorrect line such as “#!/bin/sh u” (note the missing dash), then you will likely see the following unhelpful error message when you try to execute your script:

$ ./myscript.sh
u: u: No such file or directory.

What is happening? Well, the kernel always gets the first part of the command to execute directly from the first line of your script file; so, the kernel builds and executes the following command line (note how the kernel copied your missing dash on “u” from your script):

/bin/sh u ./myscript.sh

The above command line starts the /bin/sh shell and passes it two pathname arguments. The first one, “u”, doesn’t exist, so you get an error message. You would get exactly the same error message if you typed the above command line yourself (instead of letting the kernel build it for you out of your script file):

$ /bin/sh u ./myscript.sh
u: u: No such file or directory.

You must remember how the kernel processes the #! line to understand the error message.

3.1 Confusing kernel error messageIndexup to index

The absolute pathname after the #! characters on the first line of the script must name an executable machine code binary program for the kernel to load. You cannot name another shell script on the first line.

If you spell the name of the executable program incorrectly, you may get a very unhelpful error message when you try to execute your shell script!

$ cat >wrong.sh                 # use "cat" with output redirection
#!/bin/nosuchfile               # line 1. (no such program)
echo "This is a shell script."  # line 2.
date                            # line 3.
^D                              # signal EOF to "cat"
$ chmod +x wrong.sh             # make it executable
$ ./wrong.sh                    # "execute" the script

At this point you will see one of two error messages, depending on the version of Unix you are running. The errors are:

./wrong.sh: No such file        # confusing kernel error message
./wrong.sh: Command not found.  # another confusing message

Although you can plainly see that the shell script “./wrong.sh” exists and is executable, the error message leads you to believe that the file isn’t there. In fact, what the kernel is trying to tell you is that “/bin/nosuchfile” on the first line of the script is invalid - there is no such machine code binary program to load (“No such file or directory”). (Some shells say “Command not found” for this error.)

Unfortunately, the error messages produced for this type of spelling error don’t contain the name of the program that is causing the actual error. You have to have the experience to know that, if the file clearly exists and is executable, the problem must be with the spelling of the program named on the first line of the script. (A common error is to name the script “#!bin/sh” instead of “#!/bin/sh”.)

The name on the first line of the script must be an absolute pathname to a machine code binary executable program.

4 Passing Arguments via the KernelIndexup to index

Most Unix kernels will let you squeeze a very small amount of option text onto the #! line after the name of the binary program, e.g.

#!/bin/sh -u
#!/bin/grep pattern
#!/bin/sort -r

Since it is the Unix kernel that is parsing this line and trying to find the binary program to execute, not the shell, none of the quoting or variable expansion features of a shell work on the #! line.

Some kernels limit the entire line to 32 characters. Some kernels only accept exactly one option argument. “#!/bin/sh -u” appears to work everywhere.

(The “-u” option to a Bourne shell tells the shell to warn you about undefined variables. Always use it!)

5 Any Binary Program Will DoIndexup to index

The machine code binary program that you name on the first line of your shell script can be any binary program, not just a shell:

#!/bin/cat
#!/bin/sort
#!/bin/wc
#!/bin/head
#!/bin/tail
#!/bin/rm     <-- be careful with this one!

When you “execute” your script by typing “./myscript.sh”, the kernel will load into memory the binary program specified by the first line of the script, and then give only the name of the script to the binary program as its first argument. (The kernel never executes a text file directly.)

You should make sure that the program the kernel executes on the #! line does something reasonable with its first pathname argument. If the program is a shell program such as “#!/bin/sh”, shells take a pathname argument as a file of commands to read and execute. If the program is “#!/bin/rm”, the “rm” program will unlink the name that is its first argument (which will probably delete your script!).

6 ExercisesIndexup to index

Author: 
| Ian! D. Allen  -  idallen@idallen.ca  -  Ottawa, Ontario, Canada
| Home Page: http://idallen.com/   Contact Improv: http://contactimprov.ca/
| College professor (Free/Libre GNU+Linux) at: http://teaching.idallen.com/
| Defend digital freedom:  http://eff.org/  and have fun:  http://fools.ca/

Plain Text - plain text version of this page in Pandoc Markdown format

Campaign for non-browser-specific HTML   Valid XHTML 1.0 Transitional   Valid CSS!   Creative Commons by nc sa 3.0   Hacker Ideals Emblem   Author Ian! D. Allen