Week 4 Notes for DAT2330 - Ian Allen Supplement to Text, outlining the material done in class and lab. Remember - knowing how to find out the answer is more important than memorizing the answer. Learn to fish! RTFM![*] ([*] Read The Fine Manual) Look under the "Notes" button on the course web page for these study notes for the Linux textbook: chapter5.txt (the Shell) Complete these Floppix Labs on http://floppix.ccai.com/ Floppix Lab 14: redirection Floppix Lab 15: pipes and filters Floppix Lab 18: the search path Floppix Lab 20: configuring the bash shell Floppix Lab 27: superuser Note: An early version of bash is also available on ACADAIX. (The default shell on ACADAIX is the Korn shell - "ksh".) -------------- The Unix Shell - Chapter 5 -------------- What is a shell for? To find and run programs (also called commands or utilities)! (It also does programming kinds of things; but, all that is usually to aid in the finding and running of programs.) Most (but not all) commands take what as arguments? Pathnames, either file names or directory names. The shell has features to make matching pathnames easier. How does the shell help run commands? Shells look for commands in various places, using a list of directories stored in the $PATH environment variable. Shells provide aliases and variables to save typing the same things (commands or pathnames) over and over. Shells provide wildcards (glob patterns) to generate lists of pathnames as arguments for commands. Shells provide ways of recalling and editing the last commands you enter, to save retyping them. Shells provide ways of completing command and file names, to save typing. Where are wildcard characters (glob patterns) expanded? The shell does the wildcard (glob) expansion, NOT the commands. The wildcards are expanded before the shell looks for the command! Note how different commands treat the same set of arguments differently: $ echo * - interpreted as a list of text words $ ls -l * - interpreted as a list of pathnames (files or dirs) $ cat * - interpreted as a list of file names to open $ mail * - interpreted as a list of userids to email The shell does not know anything about the type of command being used when it prepares the argument list for a command. It is quite possible to prepare a list of arguments that don't make sense for the command being run, e.g. $ sleep * - this probably makes no sense Define: whitespace, arguments, wildcard, prompt ------------- Nested Shells ------------- Your default shell on ACADAIX is the Korn shell (ksh). Your default shell on Linux is the Bourne-Again shell (bash). You can call up other shells by name: csh, tcsh, ksh, bash, sh (Not all shells may be installed on your system.) Each new shell is another Unix process. To get a list of your processes, use: ps To exit from a shell: exit When you exit from your login shell, you log out from Unix. Another way to exit a shell is to type your EOF character. (Your EOF character is usualy CONTROL-D.) Some shells can be told to ignore EOF. Remember: all Unix programs (should) have manual pages! $ man sh $ man csh $ man bash $ man ksh ------------------ Output Redirection ------------------ Output redirection diverts (redirects) output that would normally appear on the screen to some other place, either into the input of another command (a pipe) or into a file. Output redirection into files: $ echo hello - to terminal $ echo hello >file - erase file; send to file $ echo hello more >>file - append to end of file Shells don't care where on the command line you do the redirection. All these do exactly the same thing: $ echo hi there mom >file $ echo hi there >file mom $ echo hi >file there mom $ echo >file hi there mom $ >file echo hi there mom Like wildcarding (called "globbing" on Unix), shells handle redirection. The command doesn't see any part of the redirection syntax. The redirection is done by the shell, then the redirection information is removed from the command line before the command is called. Redirection is never counted as arguments to a command. $ echo hello there - shell calls "echo" with two arguments ==> echo(hello,there) - "echo" echoes two arguments - output appears in default location (standard output is your screen) $ echo hello there >file - shell creates "file" and diverts standard output into it - shell calls "echo" with two arguments ==> echo(hello,there) (note NO CHANGE in arguments to "echo" from the previous example) - "echo" echoes two arguments - standard output is captured in output "file", NOT on your screen $ >file echo hello there - this is identical to the above example - standard output is captured in output "file", NOT on your screen - you can put the redirection anywhere in the command line! Unix Big Redirection Mistake #1: Don't do this: $ cat * >z - shell creates "z" and redirects all future standard output into it - shell expands wildcards; wildcard "*" includes file "z" that was just created by the shell - shell finds and calls cat command with all file names as arguments ==> e.g. cat(a,bcd,e,file1,file2,...etc...,z) - cat command processes each argument, opening each file and sending the output into file "z" - when cat opens file "z", it ends up reading from the top of file "z" and writing to the bottom of file "z" at the same time! - Result: an infinite loop that fills up the disk drive as "z" gets bigger and bigger Fix #1: $ cat * >.z - uses a hidden file name not matched by the shell "*" wildcard Fix #2 (two ways): $ cat * >../z $ cat * >/tmp/z - use a file that is not in the current directory Unix Big Redirection Mistake #2 Don't do this: $ cat a b >a - shell creates "a" and redirects command output into it - original contents of "a" are lost - truncated - GONE! - shell finds and calls cat command with two file name arguments ==> i.e. cat(a,b) - cat command processes contents of file "a" (now an empty file) - cat command processes contents of file "b" - output has been redirected by the shell to appear in file "a" - Result: file "a" gets a copy of "b"; original contents of "a" are lost Fix: $ cat b >>a - double-redirect syntax appends file "b" safely to the end of "a" ------------------ Input Redirection: ------------------ Most Unix commands read input from files, if file names are given on the command line, and from standard input if no file names are given. If a command reads from standard input, you can tell the shell to use input redirection to change from where it reads: $ cat food - reads from file "food" $ cat - reads from stdin (keyboard) $ cat myfile Why is the file "myfile" empty after this command is run? The following command line doesn't work either: $ tr 'a-z' 'A-Z' myfile >new Why does this generate an error message from "tr"? (RTFM) ------- Aliases ------- Watch out for "helpful" system admin that define aliases for your shells when you log in. (This is especially true on ACADAIX!) The aliases may mislead you about how Unix commands actually work. (For example, the "rm" command does *not* prompt you for confirmation. On some systems, "rm" is an alias for "rm -i", which *does* prompt.) To avoid pre-defined aliases, start up a fresh copy of the shell: $ alias [...many ACADAIX aliases print here...] $ bash bash$ alias [...no more aliases here...] To define your own aliases, look up "aliases" in the Linux Text index. You must put your aliases in a file to have them saved between sessions. ------------------------------------- Studies in Quoting Special Characters ------------------------------------- Understand how the shell handles quotes and blanks: $ echo hi there hi there $ echo "hi there" hi there $ echo 'hi there' hi there Explain the above three outputs. How many arguments are passed to the "echo" command in each case? The "touch" command creates empty files by name. Try this: $ touch "a b" $ ls a b $ rm a b Explain the error message that is output by the above "rm" command. How many arguments are passed to the "touch" and "rm" commands? Here are some more things to try, and to understand. $ echo "'hello'" $ echo '"hello"' $ touch a b c d $ echo ' * ' $ echo '" * "' $ echo '"' * '"' $ echo '"'" * "'"' $ echo ' * ' * " * " You should be able to predict the output of all of the above command lines without having to type them in to try them. -------------------------------------- More shell wildcards: Character Ranges -------------------------------------- Try these shell patterns in your home directory on ACADAIX: $ echo ../[a-c]* $ echo ../*[3-5] $ echo ../[r-t]*[13579] What would be different if you changed "echo" to "ls"? -------------------------------------------- Understanding the different types of "sort": -------------------------------------------- Explain the difference in output of these two "sort" pipelines: $ list="1 11 2 22 3 33 4 44 3 33 2 22 1 11" $ echo "$list" | tr ' ' '\n' | sort $ echo "$list" | tr ' ' '\n' | sort -n (The translate command "tr" is turning blanks into newlines so that the numbers appear on separate lines on input to sort.) Why is the sort order different in these two examples? ------------------------------------------------------------------- Using commands and pipes to "mine" and extract data from the system ------------------------------------------------------------------- Problem: "Print the fifth directory from your $PATH environment variable." Iterative solution built up slowly using simple commands: $ echo "$PATH" $ echo "$PATH" | tr ':' '\n' $ echo "$PATH" | tr ':' '\n' | head -5 $ echo "$PATH" | tr ':' '\n' | head -5 | tail -1 Problem: "Print the second-to-last directory from your $PATH environment variable." $ echo "$PATH" | tr ':' '\n' | tail -2 | head -1 Problem: "Sort the elements in the PATH variable in ascending order." $ echo "$PATH" $ echo "$PATH" | tr ':' '\n' $ echo "$PATH" | tr ':' '\n' | sort $ echo "$PATH" | tr ':' '\n' | sort | tr '\n' ':' Problem: "Keep only the first five elements of the PATH." $ echo "$PATH" | tr ':' '\n' | head -5 | tr '\n' ':' Problem: "How many unique shells are in the /etc/passwd file?" Build up the solution iteratively, starting with simple commands. The shell is the seventh colon-delimited field in the passwd file. Either the "awk" or "cut" commands can pick out a field from a file. We will use "cut" to pick out the 7th field delimited by a colon. We will start with the first 10 lines of the passwd file until we know we have the correct command line, then we will use the whole passwd file. $ head /etc/passwd $ head /etc/passwd | cut -d : -f 7 $ head /etc/passwd | cut -d : -f 7 | sort $ head /etc/passwd | cut -d : -f 7 | sort | uniq We have the correct command line. Now do the whole file: $ cat /etc/passwd | cut -d : -f 7 | sort | uniq - OR - $ cut -d : -f 7 /etc/passwd | sort | uniq - OR - $ cut -d : -f 7 /etc/passwd | sort -u Does this pipeline (the reverse of the above) give the same output? $ sort -u /etc/passwd | cut -d : -f 7 Note that many Unix commands can act as filters. With no file names on the command line, the comands read from standard input and write to standard output. (You can redirect both.) If file names are given on the command line, the commands usually ignore standard input and only operate on the file names. If a command does read from file names supplied on the command line, it is more efficient to let it open its own files than to use "cat" to open the files and feed the data to the command on standard input. (There is less data copying done!) Advice: Let commands open their own files; don't feed them with "cat". Do this: $ head /etc/passwd $ sort /etc/passwd Do not do this (wasteful): $ cat /etc/passwd | head # DO NOT DO THIS - INEFFICIENT $ cat /etc/passwd | sort # DO NOT DO THIS - INEFFICIENT Problem: "Now, count the number of each kind of shell in /etc/passwd." $ cut -d : -f 7 /etc/passwd | sort | uniq -c Problem: "Count the number of each kind of shell in /etc/passwd and display the results sorted in descending numeric order." $ cut -d : -f 7 /etc/passwd | sort | uniq -c | sort -nr