% Shell Scripts -- lists of commands, executable scripts, script header, command arguments and positional parameters % Ian! D. Allen -- -- [www.idallen.com] % Winter 2017 - January to April 2017 - Updated 2018-11-01 17:32 EDT - [Course Home Page] - [Course Outline] - [All Weeks] - [Plain Text] Shells can read from keyboards or from text files ================================================= A shell (e.g. `bash`) can be used in one of two ways: - As an interactive *command interpreter*, used by a human and reading from your keyboard - As a *programming language*, to read **shell scripts** that are stored in text files This page discusses **shell scripts**. Help on Lynda.com ----------------- You may find some parts of these advanced [Lynda.com] videos useful as you start writing your own shell scripts. These links require you to have created a free account on lynda.com via the [Algonquin Lynda.com Link]: - - Create Shell Scripts to automate sysadmin tasks =============================================== > A **shell script** is a list of commands, stored in a text file, that can > be executed by a shell. The shell reads the commands from the file instead > of reading commands from your keyboard. If we have a set of commands that we want to run on a regular basis, we could create a text file containing those commands. The text file allows us to use a single file to hold many commands. This text file is called a **Shell Script**. For example: $ cat myscript.sh # show the content of the file echo "This is a shell script." date echo "Goodbye $USER" $ sh myscript.sh # pass the file to a shell "sh" This is a shell script Wed Nov 2 04:47:44 EDT 2016 Goodbye idallen Like most commands that process files, if you give the shell a file to read, it will read from that file and not from your keyboard. Unlike most other commands, shells will treat each line in the file as a command to be run. There are two ways to run the list of commands in the script file: Run Method 1: Using the shell to read the script file ----------------------------------------------------- The most obvious way to run a list of commands stored in a shell script is to type the name of the shell program that we want to use to read the script file (e.g. `sh` or `bash`) and give it the script file name as a file name argument: $ bash -u myscript.sh $ sh -u myscript.sh > The `-u` argument to the shell tells the shell to report undefined > variables used in the script and abort if any are used. Using `-u` is > always good practice, in case there might be a typing mistake in the > script. With a file name argument, this script-reading copy of the shell program does not read your keyboard but instead reads the script file and executes each line in the script file as a command line. When it reaches the end of the script file, the script shell exits and you get your original interactive shell prompt back. A separate shell process is reading the script file, not your interactive shell that is reading your keyboard. Changes made to the environment of the shell running the script (e.g. changing directories, setting variables) will not affect your interactive shell. Your interactive shell is protected from environment changes that the shell script might make as it runs. Run Method 2: Making the script file executable ----------------------------------------------- The other, more elegant way to run the list of commands in a script file is to make the script text file **executable** (only needs to be done once) and then simply type its name as a command name to the shell: $ chmod ugo+x myscript.sh # only needs to be done once $ ./myscript.sh The name of the script can be typed into your interactive shell and a second copy of the shell will **execute** the script by reading the script file and finding and running the commands in the script file. When that script shell reaches the end of the script file, that script shell exits and you get your original shell prompt back. Again, a separate shell process is reading the script file, not your interactive shell that is reading your keyboard. Use your own `bin` directory for your scripts ============================================= If you put your scripts in your own directory, e.g. `~/bin` and append that directory name at the end of your shell `$PATH`, then your script names behave just like other command names: $ mkdir ~/bin # only need to do this once $ PATH=$PATH:~/bin # could do this in your .bashrc $ mv myscript.sh ~/bin/myscript # choose a unique name $ myscript # execute your script! As with every command name typed into the shell, the shell will search for the command name `myscript` in all the directories in your search `$PATH`, finally finding it in the file `~/bin/myscript` and executing it. Most people set their `$PATH` variable to include their personal `bin/` directory at log-in time via their `.bashrc`. In short: 1. Create an executable shell script with a unique name. 2. Move the script file to a directory that is in your search `$PATH` 3. Type the name and the script runs! > As a system administrator, you can make your job easier by writing your own > custom shell scripts containing lists of commands to help automate tasks. > You can often bring these scripts with you when you change jobs. Sysadmin > may have dozens of personal shell scripts in their own personal `bin` > directory. Many command names are actually shell scripts --------------------------------------------- Many command names in the system are not binary executable programs; they are actually shell scripts. You can find out how many scripts are in the standard system command directories `/bin` and `/usr/bin`: $ file /bin/* /usr/bin/* | fgrep 'script' | wc 446 3232 40941 $ file /bin/gunzip /bin/gunzip: POSIX shell script, ASCII text executable Above we see that the `gunzip` command is actually a shell script, not a binary executable program. Because it is a text file, you can read it and discover that all it does is call `gzip -d` with whatever arguments you give it. Standard Script Header -- three lines ===================================== Shell scripts need a consistent environment when they run. For this, course we will specify three things at the start of every shell script: 1. The shell program and options that runs the script: `/bin/sh -u` 2. The search PATH that will be used to find commands in the script: `/bin:/usr/bin` 3. The `umask` that will be used in the script: `022` To do this, use this exact set of **Standard Script Header** lines as the first three lines of shell scripts you write in this course: #!/bin/sh -u PATH=/bin:/usr/bin ; export PATH # (2) PATH line umask 022 # (3) umask line Each of the three lines above is described in detail below. - Use this **Standard Script Header** at the top of all your scripts. - You could put this header, plus some comments containing your name and contact information, in a file that you keep in a convenient place, and copy that file to be the beginnings of any new script you create. (Don't re-type all the lines every time!) - The hashtag ("number sign") character `#` indicates the start of a shell **comment**. The hashtag and everything to the right, to the end of the line, are ignored by the shell. - No added shell comments are allowed on the first line of the script, because even though this whole line is a shell comment and is ignored by the shell, this line is also read and executed by the Linux kernel, and the kernel does not understand shell comments. Line 1: Interpreter Magic Number line, or Shebang: `#!` ------------------------------------------------------- The very first line of every executable standard shell script in this course must be an interpreter magic number line, or *shebang* line (short for "shell bang" or "shell exclamation") with this exact text (12 characters plus newline): #!/bin/sh -u The first two characters `#!` need to be the first two characters in the file, because together they form a [**Magic Number**] that tells the kernel this is an executable script file with a special format. No additional shell comments (`#`) are allowed on the first line of the script, because this first *shebang* `#!` line must be executed by the Linux kernel, not by the shell, and the kernel does not understand shell comments. Use the exact line given above for all the scripts in this course. ### The shebang `#!` line tells the kernel which program to run The `#!` magic number at the start of the file must be followed by the absolute path of a binary executable program file that kernel will run when you execute this file. This file pathname on this first line is how the kernel chooses which program will process your script file. The program specified for a shell script is almost always the standard system shell `/bin/sh` or (less commonly) `/bin/bash`: #!/bin/sh -u If an executable file starts with `#!`, the kernel will execute the program name that follows the `#!` and hand the file name being executed to the program as an argument. A shell program (e.g. `/bin/sh`) will read and execute the lines in the file as commands. For example, if the first line of a script named `myscript.sh` is the standard *shebang* line `#!/bin/sh -u`, then the kernel will create and then execute this command line when it executes the script file: /bin/sh -u myscript.sh The `/bin/sh` program will read and execute commands from the `myscript.sh` file that is its argument, using the `-u` shell option. A limited number of option arguments can be supplied to the program on the *shebang* line, after the program name. The `-u` option argument to a Bourne shell program tells the shell to generate an error if the script tries to make use of a variable that's not set and has no value. We always use `-u` to force the shell to detect undefined variables: - Undefined variables should never happen if the script is well written and tested. - If the error does happen, it's better to have the shell stop processing than continue on using incorrect data. ### The shebang `#!` line is a comment when the shell reads it The *shebang* line deliberately begins with a hashtag shell comment character `#` and will be ignored as a shell comment when the shell program specified after the `#!` reads the script file. If, instead of executing the shell script, you want to pass the shell script as an argument to a shell program, remember to also pass the options given in the *shebang* line. For example, if the *shebang* line in the file `myscript.sh` is `#!/bin/sh -u`, you must type this command line to run it properly, remembering to include the `-u` option: $ /bin/sh -u myscript.sh When you call the shell directly and pass the script as an argument, the *shebang* line is treated as a comment line and is ignored by the shell. You have to remember to include the options on the *shebang* line yourself, as shown in the above command line. ### Any program name is possible after `#!` Any executable program name can be specified after the `#!` on the *shebang* line, and the kernel will run that program, passing to the program the name of the file being executed as an argument. If the *shebang* line in file `myscript.sh` is: #!/bin/sh -u ...then the kernel executes: /bin/sh -u myscript.sh #!/bin/bash -u ...then the kernel executes: /bin/bash -u myscript.sh #!/usr/bin/csh ...then the kernel executes: /usr/bin/csh myscript.sh Q: What would happen if you changed the first line of a script file to be: - `#!/bin/ls -l` - `#!/usr/bin/wc` - `#!/bin/cat` - `#!/bin/rm` Line 2: Set the shell search PATH --------------------------------- The second line of the **Standard Script Header** sets the search PATH for the script: PATH=/bin:/usr/bin ; export PATH # (2) PATH line - Set the `PATH` so that the script will run the standard commands from the standard locations. - You don't want to inherit a `PATH` that may not be correct for the commands used in the shell script. - You may need to add the administration and privileged commands directories to the `$PATH` if your script needs to run those commands: - `PATH=/bin:/usr/bin:/sbin:/usr/sbin ; export PATH` > Remember that, since the script is being executed by its own shell program, > nothing you set or change inside the script will affect any shell outside > the script. Settings in your login shell remain unchanged. Line 3: Set the shell umask --------------------------- The third line of the **Standard Script Header** sets the permissions `umask` for the script: umask 022 # (3) umask line - Any files the script creates will be created with known permissions, as allowed by the given `umask` value. - You don't want to inherit a `umask` value that may not be correct for the commands used in the shell script. - A script that wants to create files with more protection (no access for group and others) might use a `umask` of `077`: `umask 077` > Remember that, since the script is being executed by its own shell program, > nothing you set or change inside the script will affect any shell outside > the script. Settings in your login shell remain unchanged. The body of the script (after the script header) ================================================ The "body" of the script is the code that follows the script header. - We follow the standard script header lines with the list of commands, large or small, that we want the shell to read, find, and execute. - **Comments**, ignored by the shell when reading the script file, begin with the hashtag / number sign `#` and extend to the end of the line. - Use useful comments to document how your script works. - Note that the first `#!` line of the script is a comment line and is ignored when the shell reads the script file. `stdin stdout stderr` are unchanged ----------------------------------- - The stdin, stdout, stderr of the commands inside the script are the stdin, stdout, stderr of the script as it is run. - When a command in your script prints output to stdout, the output goes to stdout just as if the command were not in the script. - When a command in your script reads from stdin, the input comes from stdin just as if the command were not in the script. - Redirection applied to the script applies to every command that is run inside the script file: $ sh -u ./myscript.sh >out # all script stdout will go into file out $ ./myscript.sh >out # all script stdout will go into file out $ myscript >out # all script stdout will go into file out Arguments to the script on the command line =========================================== - We may supply arguments to our script on the command line, the same as we do when using any command we type: $ sh -u myscript.sh arg1 arg2 "arg 3" "arg 4" $ ./myscript.sh arg1 arg2 "arg 3" "arg 4" $ myscript arg1 arg2 "arg 3" "arg 4" - These command-line arguments are accessible inside the script using **positional parameter** variables such as `$1`, `$2`, etc. Positional Parameter variables: `$0`, `$1`, `$2`, `$#`, etc. ------------------------------------------------------------ - When our script is running, the command line arguments are available as Positional Parameter variables inside the script starting at 1: `$1`, `$2`, etc. - The script may access the argument values through these variable names. - The nine shell positional parameter variables `$1` through `$9` contain the first nine arguments passed to the script on the command line. - To access arguments past `$9`, surround the number with braces and use `${10}`, `${11}`, etc. - The variable `$#` holds the number of arguments used on the command line, not counting the command name (which is never an argument). - Shell variable `$0` is the pathname of the script itself, often used in script messages to tell you which script is running: echo "$0: Input file '$file' will be used" Aggregate Positional Parameter variables: `$*` and `$@` ------------------------------------------------------- Shell variables `$*` and `$@` both expand to be *all* of the arguments supplied on the command line. They behave differently when double quoted: - Double-quoted variable `"$*"` is one word (one argument; one token) with spaces between the contained command line argument values. - Double-quoted variable `"$@"` behaves unlike any other double-quoted string. It is a list of multiple words (multiple arguments; multiple tokens) where each command line argument is separately quoted. Using double-quoted `"$@"` inside a shell script creates *multiple arguments*, not a single argument as you would expect. Double-quoted `"$@"` is the only variable that creates multiple arguments when quoted. We will explore how these variables work in a separate document. Sample script One -- variables and positional parameters ======================================================== Below is a small sample shell script that demonstrates the use of various shell variables, including positional parameters. Put these lines below in a file named `positional.sh` and make it executable and then run it. The first three lines of the script are a copy of the standard script header: #!/bin/sh -u PATH=/bin:/usr/bin ; export PATH umask 022 # The lines below are the body of this shell script: # myvar="howdy doody" echo "The value of $myvar is: $myvar" # use backslash to hide first $ echo "The command name (the script name) is $0" echo "The number of command line arguments is: $#" echo "All the command line arguments are: $*" echo "The first argument is: $1" # fails if no arguments echo "The second argument is: $2" # fails if not two arguments echo "The third argument is: $3" # fails if not three arguments Because the script is running the shell with the `-u` option, the shell will issue an error message if any variables, including positional parameter variables, are undefined. To avoid these errors, make sure you execute the above script with at least three command line arguments: $ chmod ugo+x positional.sh # only needs to be done once $ ./positional.sh one two three # give at least three arguments The value of $myvar is: howdy doody The command name (the script name) is ./positional.sh The number of command line arguments is: 3 All the command line arguments are: one two three The first argument is: one The second argument is: two The third argument is: three You can also run any script file, even if it isn't executable, by passing its file name to the shell as a file name argument: $ sh -u positional.sh The *shebang* line is *not* used when you run a shell script the above way, which is why we must explicitly supply the `-u` option to check for undefined variables. Shifting positional parameters with `shift` =========================================== The built-in shell command `shift` will delete the first command line argument, causing all the positional parameter numbers to shift down one: $ cat myscript.sh #!/bin/sh -u echo "All arguments: $*" echo "First argument $1 is '$1' and second argument $2 is '$2'; count is $#" echo "Doing a shift" shift echo "All arguments: $*" echo "First argument $1 is '$1' and second argument $2 is '$2'; count is $#" echo "Doing a shift" shift echo "All arguments: $*" echo "First argument $1 is '$1' and second argument $2 is '$2'; count is $#" $ ./myscript.sh one two three four All arguments: one two three four First argument $1 is 'one' and second argument $2 is 'two'; count is 4 Doing a shift All arguments: two three four First argument $1 is 'two' and second argument $2 is 'three'; count is 3 Doing a shift All arguments: three four First argument $1 is 'three' and second argument $2 is 'four'; count is 2 As you can see, after a `shift`, all the arguments shift down one place. Argument `$2` becomes `$1` (because the first argument is gone); argument `$3` becomes argument `$2`, etc. The `shift` command is most useful inside a looping control structure such as a `while` or `for` loop. Comments in your own shell scripts -- no "Instructor-Type" comments =================================================================== Many of the comments in script file examples in this course are "Instructor-Type" comments and are *not* appropriate for real scripts that you write. (e.g. Comments such as `This is the body of the shell script` or `use backslash to hide first $`) "Instructor-Type" comments explain features about the syntax and structure of a shell script and are used to teach scripting to beginners. I put Instructor-Type comments in my examples because I am teaching you how to write scripts. You would not put Instructor-Type comments in your own scripts, because the scripts you write always assume that you and your script readers already know how to write shell scripts. Do not put "Instructor-Type" comments into the scripts that you submit for marking. Comments should address what the script is doing, not how ordinary script features work. Do not submit my Instructor-Type comments back to me again in scripts that you write. -- | Ian! D. Allen, BA, MMath - idallen@idallen.ca - Ottawa, Ontario, Canada | Home Page: http://idallen.com/ Contact Improv: http://contactimprov.ca/ | College professor (Free/Libre GNU+Linux) at: http://teaching.idallen.com/ | Defend digital freedom: http://eff.org/ and have fun: http://fools.ca/ [Plain Text] - plain text version of this page in [Pandoc Markdown] format [www.idallen.com]: http://www.idallen.com/ [Course Home Page]: .. [Course Outline]: course_outline.pdf [All Weeks]: indexcgi.cgi [Plain Text]: 700_shell_scripts.txt [Lynda.com]: http://lynda.com/ [Algonquin Lynda.com Link]: http://algonquincollege.com/onlineresources/mobileStudent/lynda.htm [**Magic Number**]: https://en.wikipedia.org/wiki/Magic_number_%28programming%29 [Pandoc Markdown]: http://johnmacfarlane.net/pandoc/