========================== Unix Shell I/O Redirection - Part II ========================== -IAN! idallen@idallen.ca Output redirection diverts (redirects) output that would normally appear on the screen to some other place, either into the input of another command (using a pipe metacharacter) or into a file (using a file redirect metacharacter). * redirection is done by the shell, first, before finding the command * you can only redirect the output that you can see * redirection can only go to *one* place ---------------------- Redirection into files ---------------------- The shell metacharacter ">" signals that the next word on the command line is an output file: date >outfile Spaces between ">" and the file name are optional: date > outfile * redirection is done by the shell, first, before finding the command - the shell creates a new empty file or truncates (empties) an existing file * you can only redirect the output that you can see - redirection does not invent new output - if you don't see any output from a command, adding redirection will simply have the shell create an empty file (no output) * redirection can only go to *one* place - the right-most file redirection wins (others create empty files) - redirection to a file wins over redirection into a pipe (if you redirect into a file, the pipe gets nothing) * the output file is emptied (truncated) unless you append via >> - the file is emptied *before* the shell looks for and runs the command * most commands have two separate output "streams" - stdout - unit 1 - Standard Output (output via "cout <<" in C++) - stderr - unit 2 - Standard Error Output (output via "cerr <<" in C++) In the examples below, I use the metacharacter ";" to put multiple commands on one shell command line: $ date ; who ; echo hi ; pwd These behave as if you had typed each of them on separate lines. - stdout and stderr mix on your screen (they look the same on the screen) $ date >out ; ls -l out nosuchfile - stderr (error messages) often appear first, before stdout - you can redirect stdout and stderr separately with the shell by putting the unit number before the ">" metacharacter: $ date >out # same as "1>out"; redirect stdout only $ ls -l out nosuchfile >foo # same as "1>foo"; redirect stdout only $ ls -l out nosuchfile >foo 2>errs # redirect both stdout and stderr $ ls -l out nosuchfile 2>errs # redirect stderr (unit 2) only - ">foo" (no preceding unit number) is a shell shorthand for "1>foo" ">foo" redirects the default unit 1 (stdout) only ">foo" and "1>foo" are identical - you need a special syntax "2>&1" to redirect both stdout and stderr together; read this as "send unit 2 to the same place as unit 1": $ date >out $ ls -l out nosuchfile >foo 2>foo # *** WRONG *** $ ls -l out nosuchfile >foo 2>&1 # correct - redirect stdout and stderr - the order of these two redirections matters! - the >foo stdout redirect must come first (to the left of stderr 2>&1) - all redirection (and file truncation) happens first (done by the shell) - the command executes second (the shell finds and executes it) - the output from the command (if any) happens third, and it goes into the indicated file last Examples: $ date >out ; sort out >out - first command (date): - shell truncates out - shell finds and runs date - output of date goes into out (1 line) - second command (sort): - shell truncates out (file out is now EMPTY) - shell finds and runs sort - sort opens its argument file "out" for reading (an empty file) - output from sort (empty) goes into out (empty) - out remains empty! $ date >out ; wc out >out - first command (date): - shell truncates out - shell finds and runs date - output of date goes into out (1 line) - second command (wc): - shell truncates out (file out is now EMPTY) - shell finds and runs wc - wc opens it argument file "out" for reading (an empty file) - output from wc (1 line: 0 0 0 out) goes into out - out now has one line: 0 0 0 out - you can only redirect the output that you can see - redirecting a command that produces no output creates an empty file $ date >a ; cp a b >out # creates an empty file named "out" $ touch a ; rm a >out # creates an empty file named "out" - you can only redirect to one place $ date >a >b >c # files "a" and "b" are truncated; date is in file "c" $ date >a | wc # date is in file "a"; no input is passed to "wc" -------------------- Throwing away output -------------------- There is a special file on every Unix system, into which you can redirect output that you don't want to keep or see: /dev/null This command generates some error output we don't like to see: $ cat * >/tmp/out cat: course_outlines: Is a directory cat: jclnotes: Is a directory cat: labs: Is a directory cat: notes: Is a directory We can throw away the errors (stderr, unit 2) into /dev/null: $ cat * >/tmp/out 2>/dev/null The file /dev/null never fills up; it just eats output. When used as an input pathname, it always appears to be empty: $ wc /dev/null 0 0 0 /dev/null ------------------------- Redirection into programs ------------------------- The shell metacharacter "|" ("pipe") signals the start of another command. The standard output (only stdout; not stderr) of the command on the left of the "|" is connected ("piped") to the standard input of the command on the right: date | wc You can approximate the behaviour of a pipe using a temporary file for intermediate storage: $ ls -l >out ; wc out | wc # nothing goes into the pipe; it's all in the file "out" * pipe redirection is done by the shell, first, before file redirection * you can only redirect into a pipe the standard output that you can see - if you want to redirect standard error into a pipe, you have to redirect standard error to go to the same place as standard output: 2>&1 * redirection can only go to *one* place (file redirection always wins) Examples: - you can only redirect into a pipe the standard output that you can see $ date >a ; rm a | wc # screen output: 0 0 0 - to redirect standard error into a pipe, make it appear on the same place as standard output (i.e. redirect unit 2 into the pipe too): $ ls nosuchfile 2>&1 | wc - you can only redirect to one place (files win over pipes) $ ls /bin >a | wc # screen output: 0 0 0 - combining head and tail to select any set of lines from a file: $ head /etc/passwd | tail -5 # last five lines of first ten: lines 6-10 $ tail /etc/passwd | head -5 # first five lines of last ten lines -------------------------------- Unix commands and standard input -------------------------------- - if a Unix command that opens and reads the contents of pathnames is not given any pathnames to open, it usually reads lines from standard input (stdin) instead: $ date | wc foo # wc opens and reads foo; wc completely ignores stdin $ date | wc # wc opens and reads standard input, counts date stdout $ date | sort foo # sort opens and reads foo; sort completely ignores stdin $ date | sort # sort opens and reads standard input, sorts date stdout $ date | head foo # head opens and reads foo; head completely ignores stdin $ date | head # head opens and reads standard input, heads date stdout - if there is no input redirection involved, commands that read standard input usually read your keyboard: $ wc foo # wc opens and reads foo; wc completely ignores stdin $ wc # wc opens and reads standard input = your keyboard $ cat foo # cat opens and reads foo; cat completely ignores stdin $ cat # cat opens and reads standard input = your keyboard $ tail foo # tail opens and reads foo; tail completely ignores stdin $ tail # tail opens and reads standard input = your keyboard To tell the command to stop reading your keyboard, send it an EOF (end-of-file) signal (usually ^D = control-D). If you interrupt the command, you may kill it and prevent it from producing any output. - commands that do not open and read files usually ignore standard input, no matter what silly things you try to send them on standard input: $ echo hi | ls # ls doesn't open files, it always ignores stdin $ echo hi | pwd # pwd doesn't open files, it always ignores stdin $ echo hi | cd # cd doesn't open files, it always ignores stdin $ echo hi | date # date doesn't open files, it always ignores stdin $ echo hi | chmod +x . # chmod doesn't open files, it always ignores stdin $ echo hi | rm foo # rm doesn't open files, it always ignores stdin $ echo hi | rmdir dir # rmdir doesn't open files, it always ignores stdin $ echo hi | echo me # echo doesn't open files, it always ignores stdin $ echo hi | mv a b # mv doesn't open files, it always ignores stdin $ echo hi | cp a b # cp always needs at least 2 names and ignores stdin Standard input is thrown away if it is sent to a command that ignores it. The shell *cannot* make a command read stdin; it's up to the command. - commands that are ignoring standard input (because they are opening and reading from pathnames on the command line) will always ignore standard input, no matter what silly things you try to send them on standard input: $ echo hi | head -1 /etc/passwd # head has a pathname; it ignores stdin $ echo hi | tail -9 /etc/group # tail has a pathname; it ignores stdin $ echo hi | wc -l .vimrc # wc has a pathname; it ignores stdin $ sort a | cat b # "cat" has a pathname; it ignores stdin $ cat a | sort b # "sort" has a pathname; it ignores stdin Standard input is thrown away if it is sent to a command that ignores it. The shell *cannot* make a command read stdin; it's up to the command. Examples: - the very long sequence of pipes below is pointless - the last (rightmost) command has a pathname and will open and read it, ignoring all the standard input coming from the pipes on the left: $ head /etc/passwd | sort | tail -3 | sort -r | head -1 /etc/passwd - the above mal-formed pipeline is equivalent to this (same output): $ head -1 /etc/passwd An exception: - the Unix "tr" command is one of the few (only?) Unix commands that reads standard input but does *not* allow any pathnames on the command line - you must *always* supply input to "tr" on standard input: $ tr 'a-z' 'A-Z' file1 file2 >out # *** WRONG *** $ cat file1 file2 | tr 'a-z' 'A-Z' >out # correct A warning: - keyboard interactive programs such as the VIM text editor do not behave nicely if you redirect their input and output - they really want to be talking to your keyboard and screen; don't redirect them