Shell Scripting 1 • A shell can be used in one of two ways: ◦ A command interpreter, used interactively ◦ A programming language, to write shell scripts (your own custom commands) 2 • If we have a set of commands that we want to run on a regular basis, we could write a script • A script acts as a Linux command, similarly to binary programs and shell built in commands • In fact, check out how many scripts are in /bin and /usr/bin ◦ file /bin/* | grep 'script' ◦ file /usr/bin/* | grep 'script' • As a system administrator, you can make your job easier by writing your own custom scripts to help automate tasks • Put your scripts in ~/bin, and they behave just like other commands (if your PATH contains ~/bin) 3 • As we've already discussed, it's good practice to use a standard header at the top of our scripts • You could put this in a file that you keep in a convenient place, and copy that file to be the beginnings of any new script you create • Or, copy an existing script that already has the header #!/bin/sh -u PATH=/bin:/usr/bin ; export PATH # add /sbin and /usr/sbin if needed umask 022 # use 077 for secure scripts 4 •The interpreter magic, or "shebang": #!/bin/sh –u ◦ #! need to be the first two characters in the file, because they form a magic number that tells the kernel this is a script ◦ #! is followed by the absolute path of the binary program that kernel will launch to interpret (that is, run) the script, /bin/sh in our case, and arguments can be supplied, –u in our case ◦ The –u flag tells the shell to generate an error if the script tries to make use of a variable that's not set ‧ That will never happen if the script is well written and tested ‧ If it does happen, it's better to stop processing than continue processing garbage. 5 • Set the PATH • The script will run the standard commands from the standard locations PATH=/bin:/usr/bin ; export PATH # add /sbin and /usr/sbin if needed • Set the umask • Any files the script creates should have sane permissions, and we lean to the secure side umask 022 # use 077 for secure scripts 6 • We then follow the header with commands like the ones we type at the shell prompt. • The stdin, stdout, stderr of the of the commands inside the script are the stdin, stdout, stderr of the script as it is run. • When a command in your script prints output to stdout, your script will print that output to its stdout • When a command in your script reads from stdin, your script reads from stdin 7 • Today we cover the following scripting topics • Running scripts ◦ arguments passed on the command line ◦ ways to invoke a script • Writing scripts ◦ examining exit status ◦ positional parameters and receiving arguments ◦ variables ◦ interacting with the user ◦ the test program for checking things ◦ control flow with if statements, looping, etc 8 • we supply arguments to our script on the command line (as with any command args) • command is executable and in PATH command arg1 arg2 arg3 • command.sh is executable and in PATH command.sh arg1 arg2 arg3 • command.sh is executable and not necessarily in PATH ./command.sh arg1 arg2 arg3 9 • We can also invoke the script interpreter directly, with its own arguments • We pass the file containing the script after the interpreter arguments • The shebang line mechanism is not being used in this form sh –u command.sh arg1 arg2 arg3 sh –u ./command.sh arg1 arg2 arg3 • The arguments seen by our script are arg1 arg2 arg3 10 command "a b c" ◦ 1 argument ‧ a b c command 'a b c"' "d 'e f" ◦ 2 arguments ‧ a b c" and d 'e f command 'a ' b '"def"' ◦ 3 arguments ‧ a and b and "def" command 'a b' "c 'd e' f" ◦ 2 arguments ‧ a b and c 'd e' f 11 • Each command finishes with an exit status • The exit status is left in the variable ? ($?) • A non-zero exit status normally means something went wrong (grep is an exception) • non-zero means "false" • A exit status of 0 normally means everything was OK • 0 means "true" • grep returns 0 if a match occurred, 1 if not, and 2 if there was an error 12 • On the command line, after running a command we can use echo $? immediately after a command runs to check the exit status of that command [wen99999@centOS65 ~]$ls accounts empty rpm test.sh [wen99999@centOS65 ~]$ echo $? 0 [wen99999@centOS65 ~]$ls nosuchfile ls: cannot access nosuchfile: No such file or directory [wen99999@centOS65 ~]$echo $? 2 13 • When our script is running, the command line arguments are available as Positional Parameters • The script accesses these through variables. • $# holds the number of arguments on the command line, not counting the command itself • $0 is the name of the script itself • $1 through $9 are the first nine arguments passed to the script on the command line • After $9, there's ${10}, ${11}, and so on 14 • $* and $@ both denote all of the arguments and they mean different things when double quoted: ◦ "$*" is one word with spaces between the arguments ◦ "$@" produces a list where each argument is a separate word 15 16 #!/bin/sh -u PATH=/bin:/usr/bin ; export PATH umask 022 # Body of script myvar="howdy doody" echo "The value of \$myvar is: $myvar" #notice backslash echo "The number of arguments is: $#" echo "The command name is $0" echo "The arguments are: $*" echo "The first argument is: $1" echo "The second argument is: $2" echo "The third argument is: $3" 17 • How to write a command to swap two files? $ cat swap #!/bin/sh mv “$1” /tmp/“$1” mv “$2” “$1” mv /tmp/“$1” “$2” $ cat it1 contents of file1 $ cat it2 contents of file2 $ swap it1 it2 $ cat it1 contents of file2 $ cat it2 contents of file1 $ 18 • The shift command promotes each command line argument by one (e.g., the value in $2 moves to $1, $3 moves to $2, etc.) $ cat shiftargs #!/bin/sh echo "The args are 0 = $0, 1 = $1, 2 = $2" shift echo "The args are 0 = $0, 1 = $1, 2 = $2" shift echo "The args are 0 = $0, 1 = $1, 2 = $2" shift $ shiftargs arg1 arg2 arg3 The args are 0 = shiftarg, 1 = arg1, 2 = arg2 The args are 0 = shiftarg, 1 = arg2, 2 = arg3 The args are 0 = shiftarg, 1 = arg3, 2 = • The previous $1 becomes inaccessible 19 How to write a general version of the swap command for two or more files? swap f1 f2 f3 ... fn_1 fn f1 <--- f2 f2 <--- f3 f3 <--- f4 ... fn_1 <--- fn fn <--- f1 20 • to get input from the user, we can use the read builtin • read returns an exit status of 0 if it successfully reads input, or non-zero if it reaches EOF • read with one variable argument reads a line from stdin into the variable • Example: #!/bin/sh -u read aline #script will stop, wait for user echo "you entered: $aline" 21 • Use the –p option to read to supply the user with a prompt • Example #!/bin/sh –u read –p "enter your string:" aline echo "You entered: $aline" 22 • read var1 puts the line the user types into the variable var1 • read var1 var2 var3 puts the first word of what the user types in to var1, the second word into var2, and the remaining words into var3 #!/bin/sh –u read var1 var2 var3 echo "First word: $var1" echo "Second word: $var2" echo "Remaining words: $var3" 23 • A command or pipeline surrounded by backquotes causes the shell to: ◦ Run the command/pipeline ◦ Substitute the output of the command/pipeline for everything inside the quotes • You can use backquotes anywhere: $ whoami wen99999 $ cat test7 #!/bin/sh user=`whoami` numusers=`who | wc -l` echo "Hi $user! There are $numusers users logged on." $ ./test7 Hi wen99999! There are 6 users logged on. 24 • In Linux shell scripting, the if statement is used as a form of decision-making • The syntax of the if statement follows: if condition then statements fi – If 0 (condition is true), then statements following the “then” execute 26 • Alternate method of if statement uses the else statement • The syntax: if condition then statements else statements fi – If 0 (condition is true), then statements following the “then” execute – If not 0 (condition is not true), then statements following the “else” execute 28 • The elif statement combines else and if to construct a nested set of if…then…else structure. if condition1 then statements elif condition2 then statements … else statements fi 29 30 • Numeric relational: -eq, -ne, -gt, -ge, -lt, -le • File operators: -f file True if file exists and is a regular file -d file True if file exists and is a directory -s file True if file exists and has a size > 0 • String operators: -z string True if the length of string is zero -n string True if the length of string is nonzero s1 = s2 True if s1 and s2 are the same s1 != s2 True if s1 and s2 are different s1 True if s1 is not the null string 31 • INTEGER1 -eq INTEGER2 INTEGER1 is equal to INTEGER2 • INTEGER1 -ge INTEGER2 INTEGER1 is greater than or equal to INTEGER2 • INTEGER1 -gt INTEGER2 INTEGER1 is greater than INTEGER2 • INTEGER1 -le INTEGER2 INTEGER1 is less than or equal to INTEGER2 • INTEGER1 -lt INTEGER2 INTEGER1 is less than INTEGER2 • INTEGER1 -ne INTEGER2 INTEGER1 is not equal to INTEGER2 32 • -n STRING the length of STRING is nonzero • STRING equivalent to -n STRING • -z STRING the length of STRING is zero • STRING1 = STRING2 the strings are equal • STRING1 != STRING2 the strings are not equal 33 • These are just a few of them See man test for more: • -d FILE FILE exists and is a directory • -e FILE FILE exists • -f FILE FILE exists and is a regular file • -r FILE FILE exists and read permission is granted • -w FILE FILE exists and write permission is granted • -x FILE FILE exists and execute (or search) permission is granted 34 • A common command to use in the test list of an if statement is the test command • man test • Examples: • test –e /etc/passwd • test "this" = "this" • test 0 –eq 0 • test 0 –ne 1 • test 0 –le 1 35 if test "$1" = "hello"; then echo "First arg is hello" fi if test "$2" = "hello"; then echo "Second arg is hello" else echo "Second arg is not hello" fi 36 [wen001:centOS65 ~]$ ls -li /usr/bin/test /usr/bin/[ 786463 -r-xr-xr-x 1 root root 34716 22 Nov 2013 /usr/bin/[ 786517 -r-xr-xr-x 1 root root 31124 22 Nov 2013 /usr/bin/test • notice that on OSX, [ is another name for the test program: if [ -e /etc/passwd ]; then echo "/etc/passwd exists" fi is the same as if test –e /etc/passwd; then echo "/etc/passwd exists" fi 37 $ [ 0 –eq 0 ] $ echo $? 0 $ [ "this" = "that" ] $ echo $? 1 $ [ "this" = "this" ] echo $? 0 $ ["this" = "this"] # forgot the space after [ -bash: [this: command not found $ [ "this" = "this"] # forgot the space before ] -bash: [: missing ']' 38 • ( EXPRESSION ) EXPRESSION is true • ! EXPRESSION EXPRESSION is false • EXPRESSION1 -a EXPRESSION2 both EXPRESSION1 and EXPRESSION2 are true • EXPRESSION1 -o EXPRESSION2 either EXPRESSION1 or EXPRESSION2 is true 39 • You can combine and negate expressions with: -a And -o Or ! Not $ cat test10 #!/bin/sh if [ `who | grep gates | wc -l` -ge 1 -a `whoami` != “gates" ] then echo “Bill is loading down the machine!” else echo “All is well!” fi $ test10 Bill is loading down the machine! 40 • test is a program we run just to find out its exit status • The arguments to the test command specify what we're testing • The spaces around the arguments are important because test will not separate arguments for you: ◦ "a" ="a" is the same as a =a which is two args and test wants three with the second one = • When trying out test examples, we can run test and find out the results by looking at $? immediately after the test command finishes 41 • Alternatively, we can try any example by putting it in an if-statement: if [ 0 –eq 1 ]; then echo that test is true else echo that test is false fi 42 • Is the value of myvar an empty (zero-length) string? [ -z "$myvar" ] • Is the value of myvar a non-empty string? [ -n "$myvar" ] or [ "$myvar" ] 43 • Is the value of myvar equal to the string "yes"? [ "$myvar" = "yes" ] or [ "$myvar" = yes ] or [ "yes" = "$myvar" ] or [ yes = "$myvar" ] 44 • Is the value of myvar NOT equal to the string "yes"? [ "$myvar" != "yes" ] or [ ! "$myvar" = yes ] or [ "yes" != "$myvar" ] or [ ! yes = "$myvar" ] 45 • Is the value of myvar a number equal to 4? [ "$myvar" -eq "4" ] or [ "$myvar" -eq 4 ] • Notice that double quotes around a number just means the shell will not honor special meaning, if any, of the characters inside • Digits like 4 have no special meaning in the first place, so double quotes do nothing 46 The double quotes are necessary to prevent a syntax error or logic error: $ a= $ b= $ test $a -ne $b && echo hi hi $ test "$a" -ne "$b" && echo hi sh: 10: test: Illegal number: Note: always double-quote variable expansions. 47 • Is the value of myvar a number NOT equal to 4? [ "$myvar" –ne 4 ] or [ ! 4 -eq "$myvar" ] or [ ! "$myvar" –eq 4 ] or [ "$myvar" -ne 4 ] 48 • Is 00 a number equal to 0? yes [ 00 –eq 0 ] • Is 004 a number equal to 4? yes [ 004 –eq 4 ] • Notice double quotes don't change anything • Is 00 equal to 0 as strings? no [ 00 = 0 ] • Is 0004 equal to 4 as strings? no [ 0004 = 4 ] 49 • Is abc a number equal to 0? error [ abc –eq 0 ] ERROR abc is not a number • The following is the same as [ 1 ] with stdin redirected from file named 2 [ 1 < 2 ] • Remember we can put redirection anywhere in the command we want: ls > myfile is the same as > myfile ls 50 • Does /etc/passwd exist? [ -e /etc/passwd ] • Does /etc exist? [ -e /etc ] • Does the file myvar exist? [ -e "$myvar" ] 51 • Is /etc/passwd readable? [ -r /etc/passwd ] • Is /etc readable? [ -r /etc ] • Is the value of myvar readable as a file or directory? [ -r "$myvar" ] • Not readable? [ ! –r "$myvar" ] 52 • If we need to check whether two files both exist, we check for each individually, and combine the tests with –a, meaning AND [ -e /etc/foo –a –e /etc/bar ] • Given a number in myvar we can check whether it's greater than or equal to 4 AND less than or equal to 10 [ "$myvar" –ge 4 –a "$myvar" –le 10 ] 53 • If we need to check whether at least one of two files exists, we check for each individually, and combine the tests with –o, meaning OR [ -e /etc/foo –o –e /etc/bar ] • Given a number in myvar we can check whether it's greater than or equal to 4 OR less than or equal to 10 [ "$myvar" –ge 4 –o "$myvar" –le 10 ] 54 • We can use ! to test if something is NOT true • Test whether /etc/passwd is NOT executable [ ! –e /etc/passwd ] 55 • Just like arithmetic, we use parenthesis to control the order of operations • Remember that ( and ) are special to the shell so they need to be escaped or quoted from the shell • Check whether file1 or file2 exists, and also check whether 1 is less than 2: [ \( -e file1 –o –e file2 \) –a 1 –lt 2 ] • Without parentheses we'd be testing whether file1 exists, or whether file2 exists and 1 is less than 2 56 • Like regular expressions, to get comfortable with the order of operations, we can borrow our comfort with arithmetic expressions test arithmetic alalog comment operation ( ) ( ) \( and \) or '(' and ')' to protect from shell ! - That's the arithmetic unary "oposite of" operator, as in -4 or –(2+2) -a multiplication -o addition 57 #!/bin/sh -u PATH=/bin:/usr/bin ; export PATH umask 022 echo "You passed $# arguments, and those are:$*:" if [ $# -eq 0 ]; then echo "You didn't give me much to work with" else echo -n "Here are the arguments capitalized:" echo "$*" | tr '[[:lower:]]' '[[:upper:]]' fi 58 • Often the purpose of a script is to produce useful output, like filenames, or maybe a list of student numbers ◦ this output should go to stdout ◦ it may be redirected to a file for storage ◦ we don't want prompts and error messages in there • There may also be other output, like warning messages, error messages, or prompts for the user, for example ◦ this output should go to stderr ◦ we don't want this type of output to be inseparable from the real goods the script produces 59 • Here is an example of a good error message echo 1>&2 "$0: Expecting 1 argument; found $# ($*)" • Why is it good? ◦ It redirects the message to stderr: 1>&2 ◦ It gives the user all the information they may need to see what is wrong ‧ $0 is the name used to invoke the script (remember, files can have more than one name so it shouldn't be hard-coded into the script) ‧ $# is the number of arguments the user passed ‧ $* shows the actual arguments, put in parenthesis so the user can see spaces, etc. 60 #!/bin/sh –u PATH=/bin:/usr/bin ; export PATH umask 022 if [ $# -ne 1 ]; then echo 1>&2 "$0: Expecting 1 argument; found $# ($*)" else read -p "Enter your string:" userString if [ "$userString" = "$1" ]; then echo "The string you entered is the same as the argument" else echo "The string you entered is not the same as the argument" fi fi 61