============================================ Unix Shell I/O Redirection (including Pipes) ============================================ -Ian! D. Allen - idallen@idallen.ca - www.idallen.com Contents of this file: 1. Output Redirection - standard output and standard error 1.1 Output redirection into files 1.2 Throwing away output via /dev/null 1.3 Output redirection mistakes to avoid 2. Input Redirection 3. Redirection into programs (Pipes) 4. Misuse of redirection into programs 5. Unique STDIN and STDOUT 6. Redirection Examples 7. Using commands as Filters In the examples below, I use the meta-character ";" to put multiple commands on one shell command line: $ date ; who ; echo hi ; pwd These behave as if you had typed each of them on separate lines. ========================================================== 1. Output Redirection - standard output and standard error ========================================================== In output redirection, the shell (not the command) diverts (redirects) most command output that would normally appear on the screen to some other place, either into the input of another command (using a pipe meta-character) or into a file (using a file redirect meta-character). * Redirection is done by the shell, first, before finding the command; the shell has no idea if the command exists or will produce any output. * You can only redirect the output that you can see (if there is no output without redirection, adding redirection won't create any). * Redirection can only go to *one* place; you can't use multiple redirections to send output to multiple places (see the "tee" command). * By default, error messages are not redirected; only "normal output" is redirected (but you can also redirect error output with more syntax). The normal (non-error) outputs on your screen come from the "standard output" ("stdout") of the command. Stdout is the output from "printf" and "cout" statements in C and C++ programs, and from "System.print" and "System.println" in Java. This is the expected, usual output. The error message outputs on your screen come from the "standard error output" ("stderr") of the command. Stderr is the output from "fprintf(stderr" and "cerr" statements in C and C++ programs, and from "System.err.print" and "System.err.println" in Java. Programs print on this output only for error messages. Normally, both stdout and stderr appear together on your terminal. The shell can redirect the two outputs individually or together. The default type of output redirection (whether redirecting to files or to programs using pipes) redirects *only* standard output and lets standard error go, untouched, to your terminal. Below are some examples, and we will cover each in detail in this document: $ ls /etc/passwd nosuchfile # no redirection used ls: nosuchfile: No such file or directory # this on screen from stderr /etc/passwd # this on screen from stdout $ ls /etc/passwd nosuchfile >out # shell redirects only stdout ls: nosuchfile: No such file or directory # only stderr appears on screen You can also tell the shell to redirect standard error, unit 2, to a file: $ ls /etc/passwd nosuchfile 2>out # shell redirects only stderr /etc/passwd # only stdout appears on screen $ ls /etc/passwd nosuchfile >out 2>errors # shell redirects each one $ # nothing appears on screen The BASH shell has a special syntax for redirecting both at once: $ ls /etc/passwd nosuchfile &>mixedout # shell redirects both together $ # nothing appears on screen --------------------------------- 1.1 Output redirection into files --------------------------------- The shell meta-character ">" signals that the next word on the command line is an output file (not a program) that should be truncated (set to empty) and made ready to receive the standard output of a command: $ date >outfile The spaces between ">" and the file name are optional: $ date > outfile The file is always truncated to empty *before* the shell finds and runs the command, unless you double up the character like this: $ date >> outfile # output is *appended* to outfile; no truncation An example of output redirection of stdout into a file: $ echo hello # stdout goes to terminal $ echo hello >file ; cat file # erase file; send stdout to file hello $ echo there >>file ; cat file # append stdout to end of file hello there It is the shell that creates or truncates the file and sets up the redirection, not the command being redirected. The command knows nothing about the redirection - the redirection is removed from the command line before the command is found and executed: $ echo one two three # echo has three arguments one two three $ echo one two three >out # echo still has three arguments Shells handle redirection before they go looking for the command name to run. Indeed, you can have redirection even if the command is not found or if there is no command at all: $ nosuchcommandxxx >out # file "out" is created empty sh: nosuchcommandxxx: command not found $ wc out 0 0 0 out # shell created an empty file $ >out # file "out" is created empty $ wc out 0 0 0 out # shell created an empty file The shell creates or truncates the file "out" empty, and then it tries to find and run the nonexistent command and fails. The empty file remains. Any existing file will have its contents removed: $ echo hello >out ; cat out hello $ nosuchcommandxxx >out sh: nosuchcommandxxx: command not found $ wc out 0 0 0 out # shell truncated the file Redirection is done by the shell *before* the command is run: $ mkdir empty $ cd empty $ ls -l total 0 # no files found $ ls -l >out # shell creates "out" first $ cat out # display output total 0 -rw-r--r-- 1 idallen idallen 0 Sep 21 06:02 out $ date >out $ ls -l total 4 -rw-r--r-- 1 idallen idallen 29 Sep 21 06:04 out $ ls -l >out # shell empties "out" first $ cat out # display output total 0 -rw-r--r-- 1 idallen idallen 0 Sep 21 06:06 out The shell creates or empties the file "out" before it runs the "ls" command. Explain this sequence of commands: $ mkdir empty $ cd empty $ cp a b cp: cannot stat `a': No such file or directory $ cp a b >a $ # why is there no error message? Shells don't care where on or in the command line you do the file redirection. The file redirection is done by the shell, then the redirection information is removed from the command line before the command is called. The command actually being run doesn't see any part of the redirection syntax; the number of arguments is not affected. All the command lines below are equivalent to the shell; in every case the echo command sees only three arguments and the three command line arguments "hi", "there", and "mom" are all redirected into "file": $ echo hi there mom >file # echo has three arguments $ echo hi there >file mom # echo has three arguments $ echo hi >file there mom # echo has three arguments $ echo >file hi there mom # echo has three arguments $ >file echo hi there mom # echo has three arguments Redirection is removed by the shell before the command runs; so, redirection syntax is never counted as arguments to a command. Examples: $ echo hello there - shell calls "echo" with two arguments ==> echo(hello,there) - "echo" echoes two arguments on standard output - output appears in default location (standard output is your screen) $ echo hello there >file - shell creates "file" and diverts standard output into it - shell removes ">file" from the command line - shell calls "echo" with two arguments ==> echo(hello,there) (note NO CHANGE in arguments to "echo" from the previous example) - "echo" echoes two arguments on standard output - standard output is captured in output "file", NOT on your screen $ >file echo hello there - this is identical to the above example (the shell does not care where in the command line you put the redirection) - standard output is captured in output "file", NOT on your screen - you can put the redirection anywhere in the command line! Most commands have two separate output "streams": 1. stdout - unit 1 - Standard Output (output via "cout <<" in C++) 2. stderr - unit 2 - Standard Error Output (output via "cerr <<" in C++) Redirection is done by the shell, first, before finding the command: - the shell creates a new empty file or truncates (empties) an existing file - after doing the redirection, and removing it from the command line, the shell finds and executes the command (if any): $ mkdir empty ; cd empty ; touch out ; rm out >out ; ls $ # no file found; rm removed it You can only redirect the output that you can see! *Only* what you see! - Redirection does not invent new output! *ONLY WHAT YOU SEE!* - If you don't see any output from a command, adding redirection will simply have the shell create an empty file (no output): Example: $ cp /etc/passwd x # no output on standard output $ cp /etc/passwd x >out # file "out" is created empty $ cd /tmp # no output on standard output $ cd /tmp >out # file "out" is created empty $ touch x ; rm x # no output from rm on standard output $ touch x ; rm x >out # file "out" is created empty Redirection can only go to *one* place: - the right-most file redirection wins (others create empty files) Example: $ date >a >b >c # output goes into file c; a and b are empty Redirection to a file wins over redirection into a pipe: - see the following section on redirection into programs using "|" pipes - if you redirect into a file and a pipe, the pipe gets nothing Example: $ date >a | cat # output goes into file "a"; cat shows nothing The redirection output file is emptied (truncated) unless you append via >> - the file is emptied *before* the shell looks for and runs the command - don't use output redirection files as input to the same command Bad Example: $ sort a >a # WRONG! file "a" is truncated to be empty The stdout and stderr mix on your screen (they look the same on the screen) so you can't tell by looking at your screen what comes on stdout and what comes on stderr. Example of stdout and stderr on your screen: $ date >out ; ls -l out nosuchfile ls: nosuchfile: No such file or directory # STDERR -rw-r--r-- 1 idallen idallen 29 Jan 16 06:52 out # STDOUT The stderr (error messages) output often appears first, before stdout, due to internal I/O buffers used by commands for stdout. You can redirect stdout and stderr separately using unit numbers: - stdout is always unit 1 and stderr is always unit 2 (stdin is unit 0) - put the unit number immediately (no blank) before the ">" meta-character: $ date >out # same as "1>out"; redirect stdout only $ ls -l out nosuchfile # no redirection used yet ls: nosuchfile: No such file or directory -rw-r--r-- 1 idallen idallen 29 Jan 17 13:01 $ ls -l out nosuchfile 1>foo # same as ">foo"; redirect stdout only ls: nosuchfile: No such file or directory $ ls -l out nosuchfile 2>errs # redirect stderr (unit 2) only -rw-r--r-- 1 idallen idallen 29 Jan 17 13:01 $ ls -l out nosuchfile >foo 2>errs # redirect both stdout and stderr $ cat foo -rw-r--r-- 1 idallen idallen 29 Jan 17 13:01 # was STDOUT $ cat errs ls: nosuchfile: No such file or directory # was STDERR ">foo" (no preceding unit number) is a shell shorthand for "1>foo" ">foo" redirects the default unit 1 (stdout) only ">foo" and "1>foo" are identical Until recently, you needed a special syntax "2>&1" to redirect both stdout and stderr safely together into a single file in the Bourne shells. Read the syntax "2>&1" as "send unit 2 to the same place as unit 1": $ date >out $ ls -l out nosuchfile >both 2>&1 # redirect BOTH stdout and stderr - the order of these two redirection syntaxes matters! - the ">both" stdout redirect must come first (to the left of) stderr "2>&1" because you must set where stdout (unit 1) goes *before* you send stderr (unit 2) to go "to the same place as unit 1". Don't reverse these! You must use ">both 2>&1". Don't use the following, which is not the same: $ ls -l out nosuchfile >both 2>both # WRONG! *** DO NOT DO THIS *** - this WRONG example will cause stderr and stdout to overwrite each other - the result is usually a mangled output file; don't do this The C Shells (csh, tcsh, itcsh) provide a simpler syntax to redirect both: $ ls -l out nosuchfile >& both # CSH only! redirect both STDOUT and STDERR Modern versions of the Bourne-Again shell BASH also have a simple syntax: $ ls -l out nosuchfile &> both # BASH only! redirect both STDOUT and STDERR Yes, the CSH and BASH syntaxes have the two characters reversed. BASH also accepts the CSH syntax but the man page says "don't use it". Summary: 1. First: All redirection (and file truncation) is done by the shell. The shell removes all the redirection syntax from the command line. This redirection and truncation happens even if no command executes. The command has no idea that its output is being redirected. 2. Second: The command (if any) executes and may produce output. The shell executes the command *after* doing all the redirection. 3. Third: The output from the command (if any) happens, and it goes into the indicated redirection output file. This happens last. ----------------------------------------- 1.2. Throwing away output using /dev/null ----------------------------------------- There is a special file on every Unix system, into which you can redirect output that you don't want to keep or see: /dev/null The following command generates some error output we don't like to see: $ cat * >/tmp/out cat: course_outlines: Is a directory # errors print on STDERR cat: jclnotes: Is a directory # errors print on STDERR cat: labs: Is a directory # errors print on STDERR cat: notes: Is a directory # errors print on STDERR We can throw away the errors (stderr, unit 2) into /dev/null: $ cat * >/tmp/out 2>/dev/null The file /dev/null never fills up; it just eats output. When used as an input pathname, it always appears to be empty: $ wc /dev/null 0 0 0 /dev/null ---------------------------------------- 1.3 Output redirection mistakes to avoid ---------------------------------------- Unix Big Redirection Mistake #1 ------------------------------- Do not use a redirection file as both output and input to a program or a pipeline! The sort command is used as the example program below - anything that reads files and produces output is at risk: $ sort a b >a # WRONG! - shell first truncates the file "a" - file is now empty - original contents of "a" are lost - truncated - GONE! - before the shell even goes looking for the "sort" command to run! - shell redirects standard output of sort (if any) into file "a" - shell finds and runs the "sort" command with two file name arguments ==> i.e. sort(a,b) - sort command opens the argument file "a" for reading (now an empty file) - sorting an empty file produces no output - sort command opens the argument file "b" for reading - the sorted contents of file "b" are sent to standard output by "sort" - standard output has been redirected by the shell to appear in file "a" - Result: file "a" gets a sorted copy of "b"; and the original contents of "a" are lost forever Here is another incorrect example. First, create a data file "out": $ cp /etc/passwd out $ wc out >out # WRONG! - shell first truncates the file "out" - file is now empty - original contents of "out" are lost - truncated - GONE! - before the shell even goes looking for the "wc" command to run! - shell finds and runs the "wc" command with one file name argument "out" ==> i.e. wc(out) - wc command opens the argument file "out" for reading (now an empty file) - counting an empty file produces 1 line "0 0 0 out" on standard output - standard output has been redirected by the shell to appear in file "out" - the one line "0 0 0 out" is placed into out - Result: out now has one line, a count of an empty file: 0 0 0 out The original contents of "out" were truncated away and never used. Here is another incorrect example. First, create a data file "out": $ date >out - shell first truncates the file "out" - file is now empty - shell redirects standard output of command "date" into file "out" - shell finds and runs the "date" command - standard output of the date command goes to standard output (1 line) - standard output has been redirected by the shell to appear in file "out" - file "out" contains one line $ tr ' ' '_' out # WRONG! - shell opens the file "out" as standard input to "tr" (due to "out") - original contents of "out" are lost - truncated - GONE! - before the shell even goes looking for the "tr" command to run! - shell finds and runs command "tr" with two string arguments - command "tr" reads from standard input - standard input is attached to "out", now an empty file - translating an empty input file will produce no output - output from command "tr" goes to standard output - standard output has been redirected by the shell to appear in file "out" - translating an empty input file produces no output in "out" - Result: file "out" gets a translated copy of an empty input file; the file "out" is now empty Work Around: Use a Temporary Third File $ sort a b >c $ mv c a - the third file safely receives the output of "a" and "b" Other examples that DO NOT WORK: $ head file >file # creates an EMPTY FILE $ tail file >file # creates an EMPTY FILE $ tr a-z A-Z file # creates an EMPTY FILE $ wc file >file # file will always contain: 0 0 0 ...etc... Never use the same file name for both input and output - the shell will truncate the file before the command reads it. Unix Big Redirection Mistake #2 ------------------------------- Do not use a wildcard/glob that picks up the name of the output redirection file and causes it to become an input file. Bourne shells (e.g. BASH) will do the wildcard expansion *before* the file creation; you can skip this section. C Shells do the file creation first; read on! The cat program is used as the example program here - anything that reads files and produces output is at risk: $ cat * >z - shell creates "z" and redirects all future standard output into it - shell expands wildcards; wildcard "*" includes file "z" that was just created by the shell - shell finds and calls cat command with all file names as arguments ==> e.g. cat(a,b,c,d,e,file1,file2,...etc...,z) - cat command processes each argument, opening each file and sending the output into file "z" - when cat opens file "z", it ends up reading from the top of file "z" and writing to the bottom of file "z" at the same time! - Result: an infinite loop that fills up the disk drive as "z" gets bigger and bigger Fix #1: Use a hidden file name $ cat * >.z - uses a hidden file name not matched by the shell "*" wildcard - the cat command is not given ".z" as an argument, so no loop occurs Fix #2 (two ways): Use a file in some other directory $ cat * >../z $ cat * >/tmp/z - redirect output into a file that is not in the current directory so that it is not read by the cat command and no loop occurs ==================== 2. Input Redirection ==================== Many Unix commands read input from files, if pathnames are given on the command line, and from standard input ("stdin", often your keyboard) if no file names are given. Not all commands read from standard input. Examples of common Unix commands that never read from standard input: ls, cp, mv, date, who, pwd, echo, cd, etc. # NEVER READ STDIN With the exception of "cp" above, all the above commands have in common the fact that they never open any files for reading. If a command never reads any data from any files, it can't read any data from standard input. Examples of commands that may read from pathnames or from standard input: less, more, cat, head, tail, sort, wc, etc. # MAY READ STDIN Commands such as the above may read standard input. They will read your keyboard only if there are no pathnames to read on the command line, and no input redirection is involved: $ wc foo # wc opens and reads file "foo"; wc completely ignores stdin $ wc # wc opens and reads standard input = your keyboard $ cat foo # cat opens and reads file "foo"; cat completely ignores stdin $ cat # cat opens and reads standard input = your keyboard $ tail foo # tail opens and reads "foo"; tail completely ignores stdin $ tail # tail opens and reads standard input = your keyboard [...etc. for all commands that can read from stdin...] To tell a command to stop reading your keyboard, send it an EOF (end-of-file) character (usually ^D = control-D). If you interrupt the command (e.g. ^C), you may kill it and fail to see any output. You can tell the shell to use input redirection to change from where standard input comes, so that it doesn't come from your keyboard. You can only redirect input from a single place, not multiple places. If (and only if!) a command reads from standard input, the redirected standard input will cause the program to read from whatever you attach to standard input. Here are examples of attaching files to standard input: $ cat food # reads from file "food" $ cat # reads from stdin (keyboard) $ cat file $ sleep file # create a data file for use as input $ wc file 1 1 3 file - wc was passed the file name "file"; it knows the name, so it prints it $ wc out # *** WRONG - ERROR *** tr: too many arguments $ cat file1 file2 | tr 'a-z' 'A-Z' >out # correct for multiple files $ tr 'a-z' 'A-Z' out # correct for a single file Note: System V versions of "tr" demand that character ranges appear inside square brackets, e.g.: tr '[a-z]' '[A-Z]' Berkeley Unix and Linux do not use the brackets. No version of "tr" accepts pathnames on the command line. All version of "tr" only read standard input. Do not redirect full-screen programs such as VIM ------------------------------------------------ Full-screen keyboard interactive programs such as the VIM text editor do not behave nicely if you redirect their input or output - they really want to be talking to your keyboard and screen; don't redirect them or try to run them in the background using "&". You can hang your terminal if you try. ==================================== 3. Redirection into programs (Pipes) ==================================== The shell meta-character "|" ("pipe") signals the start of another command on the command line. The standard output (only stdout; not stderr) of the command on the left of the "|" is attached/connected ("piped") to the standard input of the command on the right: $ date | wc 1 6 29 You can approximate some of the behaviour of a pipe using a temporary file for intermediate storage before using the second command: $ date >out ; wc out | wc # WRONG! 0 0 0 - shell first splits the line on the pipe, redirecting the output of the command on the left into the input of the command on the right, but: - then the shell processes the standard output file redirection on the "ls" on the left and changes the "ls" standard output into the file "out" - finally, the shell finds and runs both commands simultaneously - all the standard output from "ls" goes into the file "out"; nothing is available to go into the pipe - wc counts an empty input from the pipe and outputs: 0 0 0 As with file output redirection, you can only redirect into a pipe the standard output that you can see; redirection never creates output: Example: $ cp /etc/passwd x # no output on standard output $ cp /etc/passwd | cat # no output is passed to "cat" $ cd /tmp # no output on standard output $ cd /tmp | head # no output is passed to "head" $ touch x ; rm x # no output from rm on standard output $ touch x ; rm x | wc # no output is passed to "wc" 0 0 0 # wc counts an empty input from the pipe As with file redirection, you need a special syntax "2>&1" to redirect both stdout and stderr both into a pipe. Recall that "2>&1" means "redirect standard error to go to the same place as standard output": $ ls /etc/passwd nosuchfile # no redirection used ls: cannot access nosuchfile: No such file or directory # STDERR /etc/passwd # STDOUT $ ls /etc/passwd nosuchfile | wc # only stdin is redirected to "wc" ls: cannot access nosuchfile: No such file or directory # STDERR 1 1 12 # stdout went into the pipe to "wc" $ ls /etc/passwd nosuchfile 2>&1 | wc # both stdin and stderr redirected 2 10 68 # wc counts both lines from pipe How do you redirect *only* stderr into the pipe, and let stdout go to the terminal? This is tricky; on the left of the pipe you have to swap stdout (attached to the pipe) and stderr (attached to the terminal). You need a temporary output unit (use "3") to record and remember where the terminal is (redirect unit 3 to the same place as unit 2: "3>&2"), then redirect stderr into the pipe (redirect unit 2 to the same place as unit 1: "2>&1"), then redirect stdout to the terminal (redirect unit 1 to the same place as unit 3: "1>&3"): $ ls /etc/passwd nosuchfile 3>&2 2>&1 1>&3 | wc # switch STDOUT and STDERR /etc/passwd # STDOUT appears on terminal 1 9 56 # STDERR goes into the pipe Remember: Redirection can only go to *one* place, and file redirection always wins over pipes. Examples: - combining head and tail to select any set of lines from a file: $ head /etc/passwd | tail -5 # last five lines of first ten: lines 6-10 $ tail /etc/passwd | head -5 # first five lines of last ten lines - you can only redirect into a pipe the standard output that you can see $ date >a ; rm a # rm has no output if it works $ date >a ; rm a | wc # nothing to redirect - screen output: 0 0 0 - to redirect standard error into a pipe, make it appear on the same place as standard output (i.e. redirect unit 2 into the pipe too): $ ls nosuchfile 2>&1 | wc - you can only redirect to one place (files win over pipes) $ ls /bin >a | wc # WRONG! nothing in pipe - screen output: 0 0 0 ====================================== 4. Misuse of redirection into programs ====================================== If a Unix command that can open and read the contents of pathnames is not given any pathnames to open, it usually reads input lines from standard input (stdin) instead: $ wc /etc/passwd # wc reads /etc/passwd, ignores stdin and your keyboard $ wc # wc reads stdin, in this case, your keyboard If the command is given a pathname, it reads from the pathname and always ignores standard input, even if you try to send it something: $ date | wc # wc opens and reads standard input, counts date $ date | wc foo # WRONG! wc opens and reads foo; wc ignores stdin $ date | head # head opens and reads standard input, read date $ date | head foo # WRONG! head opens and reads foo; head ignores stdin $ date | less # less opens and reads standard input, less shows date $ date | less foo # WRONG! less opens and reads foo; less ignores stdin If you want a command to read stdin, you cannot give it any file name arguments. Commands with file name arguments ignore standard input. Commands that are ignoring standard input (because they are opening and reading from pathnames on the command line) will always ignore standard input, no matter what silly things you try to send them on standard input: $ echo hi | head /etc/passwd # WRONG: head has a pathname and ignores stdin $ echo hi | tail /etc/group # WRONG: tail has a pathname and ignores stdin $ echo hi | wc .vimrc # WRONG: wc has a pathname and ignores stdin $ sort a | cat b # WRONG: cat has a pathname and ignores stdin $ cat a | sort b # WRONG: sort has a pathname and ignores stdin Standard input is thrown away if it is sent to a command that ignores it. The shell *cannot* make a command read stdin; it's up to the command. The command must *want* to read standard input. Commands that do not open and process the contents of files usually ignore standard input, no matter what silly things you try to send them on standard input. All these commands will never read standard input: $ echo hi | ls # NO: ls doesn't open files - always ignores stdin $ echo hi | pwd # NO: pwd doesn't open files - always ignores stdin $ echo hi | cd # NO: cd doesn't open files - always ignores stdin $ echo hi | date # NO: date doesn't open files - always ignores stdin $ echo hi | chmod +x . # NO: chmod doesn't open files - always ignores stdin $ echo hi | rm foo # NO: rm doesn't open files - always ignores stdin $ echo hi | rmdir dir # NO: rmdir doesn't open files - always ignores stdin $ echo hi | echo me # NO: echo doesn't open files - always ignores stdin $ echo hi | mv a b # NO: mv doesn't open files - always ignores stdin $ echo hi | ln a b # NO: ln doesn't open files - always ignores stdin Some commands *only* operate on file name arguments and never read stdin: $ echo hi | cp a b # NO: cp opens arguments - always ignores stdin Standard input is thrown away if it is sent to a command that ignores it. The shell *cannot* make a command read stdin; it's up to the command. Commands that might read standard input will do so only if no file name arguments are given on the command line. The presence of any file arguments will cause the command to ignore standard input and process the file(s) instead. File name arguments always win over standard input. Example of mis-used redirection: -------------------------------- The very long sequence of pipes below is pointless - the last (rightmost) command ("head") has a pathname and will open and read it, ignoring all the standard input coming from the pipes on the left: $ head /etc/passwd | sort | tail -3 | sort -r | head -1 /etc/passwd The above mal-formed pipeline is equivalent to this (same output): $ head -1 /etc/passwd If you give a command a file to process, it will ignore standard input. ========================== 5. Unique STDIN and STDOUT ========================== There is only one standard input and one standard output for each command. Each can only be redirected to *one* other place. You cannot redirect standard input from two different places, nor can you redirect standard output into two different places. The Bourne shells (including bash) do not warn you that you are trying to redirect the input of a command from two or more different places (and that only one of the redirections will work - the others will be ignored): bash$ wc a >b >c >d >e - the "date" output goes into file "e"; the other four output files are each created and truncated by the shell but they are all left empty because only the final redirection into "e" wins bash$ date >out | wc 0 0 0 - the "date" output goes into file "out"; nothing goes into the pipe (file redirection always wins over pipe redirection) Some shells (including the "C" shells, but not the Bourne shells) will try to warn you about silly shell redirection mistakes: csh% date a >b >c Ambiguous output redirect. csh% date >a | wc Ambiguous output redirect. The C shells tell you that you can't redirect stdin or stdout to/from more than one place at the same time. Bourne shells do not tell you - they simply ignore the "extra" redirections and do only the last one of each. ======================= 6. Redirection Examples ======================= A command line to convert lower-case to upper-case from the "who" command: $ who | tr 'a-z' 'A-Z' Shell question: Are the single quotes required around the two arguments? (Are there any special characters in the arguments that need protection?) Using redirection, you can use a similar command to convert a lower-case file of text into upper-case. EXPERIMENT: Why doesn't this convert the file "myfile" to upper-case? $ date >myfile $ tr 'a-z' 'A-Z' myfile # WRONG! $ wc myfile 0 0 0 myfile # what happened? Why is the file "myfile" empty after this command is run? What about the following command lines - what is in "myfile" when the command finishes? $ cat myfile # WRONG! $ sort myfile # WRONG! $ head myfile # WRONG! Given the above, why is "myfile" not left empty in the following case? $ wc myfile # WRONG! The following command line doesn't work because the programmer doesn't understand the "tr" command syntax: $ tr 'a-z' 'A-Z' myfile >new # WRONG! Why does this generate an error message from "tr"? (The "tr" command is unusual in its handling of command line pathnames. RTFM) The following command line redirection is faulty (input file is also output file); however, it sometimes works for small files: $ cat foo bar | tr 'a' 'b' | grep "lala" | sort | head >foo # WRONG! There is a critical race between the first "cat" command trying to read the data out of "foo" before the shell truncates it to zero when launching the "head" command at the end of the pipeline. Depending on the system load and the size of the file, "cat" may or may not get out all the data before the "foo" file is truncated or altered by the shell in the redirection at the end of the pipeline. Don't depend on long pipelines saving you from bad redirection! Never redirect output into a file that is being used as input in the same command or anywhere in the command pipeline. ============================ 7. Using commands as Filters ============================ Note that many Unix commands can be made to act as filters - reading from stdin and writing to stdout. With no file names on the command line, the commands read from standard input and write to standard output. (You can redirect both.) $ grep "/bin/sh" /etc/passwd | sort | head -5 The "grep" command is reading from the filename /etc/passwd given on the command line. When reading from files, commands do not read from standard input. (Files take priority over standard input.) The "sort" and "head" commands have no file names to read; this means they read from standard input. Both "sort" and "head" are acting as filters; they are reading from stdin and writing to stdout. The "grep" command is technically not a filter - it is reading from the supplied argument pathname, not from stdin. If file names are given on the command line, the commands almost always ignore standard input and only operate on the file names: $ grep "/bin/sh" /etc/passwd | sort | head -5 /etc/passwd # WRONG! This is the same command line as the above example, except the "head" command is now ignoring standard input and is reading from its filename argument. The "grep" and "sort" commands are doing a lot of work for nothing, since "head" is not reading the output of sort, it is reading the file name instead. File names take precedence over standard input. *** Commands ignore standard input if they are given file names to read. *** If a command does read from file names supplied on the command line, it is more efficient to let it open its own file name than to use "cat" to open the file and feed the data to the command on standard input. (There is less data copying done!) Advice: Let commands open their own files; don't feed them with "cat". Do this: $ head /etc/passwd $ sort /etc/passwd Do not do this (wasteful of processes and I/O): $ cat /etc/passwd | head # <- DO NOT DO THIS - INEFFICIENT $ cat /etc/passwd | sort # <- DO NOT DO THIS - INEFFICIENT Examples using pipes and filters: Problem: "Count the number of each kind of shell in /etc/passwd." $ cut -d : -f 7 /etc/passwd | sort | uniq -c - the cut command picks out field 7 in the password file - the sort command puts all the shell names in order - the uniq command counts the adjacent names Problem: "Count the number of each kind of shell in /etc/passwd and display the results sorted in descending numeric order." $ cut -d : -f 7 /etc/passwd | sort | uniq -c | sort -nr - use the previous pipeline and add to it: - sort the above output numerically and in reverse Problem: "Count the number of each kind of shell in /etc/passwd and display the top two results sorted in descending numeric order." $ cut -d : -f 7 /etc/passwd | sort | uniq -c | sort -nr | head -2 - use the previous pipeline and add to it: - pick off only the top two lines of the above output -- | Ian! D. Allen - idallen@idallen.ca - Ottawa, Ontario, Canada | Home Page: http://idallen.com/ Contact Improv: http://contactimprov.ca/ | College professor (Free/Libre GNU+Linux) at: http://teaching.idallen.com/ | Defend digital freedom: http://eff.org/ and have fun: http://fools.ca/