-------------------------
Week 04 Notes for NET2003
-------------------------
-Ian! D. Allen - idallen@idallen.ca

Remember - knowing how to find out an answer is more important than
memorizing the answer.  Learn to fish!  RTFM!  (Read The Fine Manual)

Midterm test coming

Review:
 - you know what a shell is for
 - you can navigate the file system using . and ..
 - you can generate simple GLOB patterns for the shell
 - you know the basic function of 40 Unix/Linux commands
 - you can sort lines alphabetically or numerically
 - you can see the modification time of the contents of a directory or
   of the directory itself
 - you know the difference between Unix/Mac/Windows line endings

Q: give the absolute/relative paths from current dir to HOME, /home, ROOT, etc.

Motivations for learning shell script programming:

1) Six out of 9 third year co-op students in BIT/NET said they used scripting
   on the job (and needed to know more!).

2) See the shell script /etc/init.d/apache2 that starts the Apache
   web server.  Can you read and modify this script?

3) The senior network admin thinks someone is abusing their server
   login account.  Can you generate an instant sorted list of the top
   10 logins to the machine?  It's one line of script:

     $ last | awk '{print $1}' | sort | uniq -c | sort -nr | head

   (Generate a list of logins; print the userid field on each line; sort
   the userids; count adjacent lines; sort the count in reverse order;
   pick the top ten.)

Redirection of Input and Output by the shell
--------------------------------------------
  see the course Notes file:
  redirection.txt - Unix Shell I/O Redirection (including Pipes)

The "standard input" of most programs is usually your keyboard.
 - this is "unit 0" to the shell: cat 0<file
The "standard output" of most programs is usually your screen.
 - this is "unit 1" to the shell: date 1>file
The "standard error" output of most programs is usually your keyboard.
 - this is "unit 2" to the shell: ls * 2>file

You can easily tell the shell to redirect the standard input and/or output
of any command before it runs the command.

* Redirection to and from single files:

 - if a program normally produces output on your screen, the shell can
   redirect that output to go to a file instead
   - but the program has to be one that would have produced output!
 - simple output redirection to a file:  sort /etc/passwd >out

 - if a program normally reads your keyboard, the shell can also redirect 
   input to come from a single file instead
   - but the program has to be one that would have read the keyboard!
 - simple input redirection from a file:  sort </etc/passwd
  
 - both redirection (files):  sort </etc/passwd >out

 - shells are designed to issue prompts before reading your keyboard
   - most other programs do not prompt you

* Redirection between two or more programs (Unix pipes):
 - a pipe is shell redirecton from one program into another program
   without the use of an intervening temporary file:
   COMPARE:  date > out ; wc <out    -VS-    $ date | wc
   $ ls /usr/bin | wc
   $ ls /bin | sort -r
 - send standard output of one command to standard input of another
   - print line 20 of /etc/passwd:
     $ head -20 /etc/passwd | tail -1
 - may be chained: connecting simple commands together in chains
   - count the characters in line 20 of /etc/passwd:
     $ head -20 /etc/passwd | tail -1 | wc -c
   - chains can be as long as you want:
     $ head -15 /etc/passwd | tail -6 | sort | head -2 | tail -1
 - commands must be compatible - can't send output to just any command
   - the command must *want* to read standard input!
   - pipe examples that don't work (because command ignores stdin):
     $ date | ls   ;   date | rm x   ;   date | cp a b
     $ date | date ;   date | echo hi   ;   date | sleep 3
     $ date | who  ; date | hostname
 - commands that don't read stdin, don't read stdin
   - you cannot force a command to read stdin
   - commands that do read stdin, read stdin only if they have *NO* files
     to read - files on the command line take priority
 - watch out for line end characters on Unix/Linux:
   $ echo hi | wc -c   # prints 3 not 2 on Unix because of \n at end

Redirection is never counted as arguments to a command, since it is
done by the shell before the shell runs the command.
 - the shell does the redirection and removes the syntax
 - the command never sees any of the redirection syntax
 - the command cannot tell that the shell has redirected input or output

Output redirection:
 - you can only redirect output that you can see on your screen
 - if you see no command output, adding redirection will create an empty file!
    $ cp a b  ;  cp a b >out  ;  cp a b | wc

 - redirection is done first by the shell and is removed from command line
    $ echo a b c ; echo a b c | wc ; echo a b c >out
   - redirection is done by the shell *BEFORE* the command runs!
   - never redirect output into a file used as input!
    $ head /etc/passwd >out ; sort out
    $ head /etc/passwd >out ; sort out >out  # BAD! SHELL OVERWRITES OUT FIRST
    $ rm out >out   # no error message, even if out did not exist first
    $ date >out ; ls -l out >out  # BAD! SHELL OVERWRITES OUT FIRST

 - redirection is never passed to the command; it is never a command argument
 - never use redirection to direct output into an input file!
 - you can append to a file, instead of overwrite, using >>

The shell does not care where in the command line you put the file
redirection; it is always found, done, and removed before the command runs:
  $ date >wc  ;  >wc date  

Note the special syntax for making standard error go to "the same
place" as standard output:
 - the 2>&1 must appear anywhere to the right of the >out on the line
   $ sum *.c >out 2>&1
   $ sum >out *.c 2>&1
   $ >out sum 2>&1 *.c
   $ >out 2>&1 sum *.c

You can "throw away" unwanted output by redirecting it to /dev/null
  $ find / 2>/dev/null    # suppress only the error messages
  $ cat * >/dev/null      # only show errors; throw away standard output