------------------------
Week 5 Notes for NET2003
------------------------
-Ian! D. Allen - idallen@idallen.ca

Remember - knowing how to find out an answer is more important than
memorizing the answer.  Learn to fish!  RTFM!  (Read The Fine Manual)

Keep up on your readings (Course Outline: average 5 hours/week)

Standard Output and Standard Error
  Notes: redirection.txt
  Notes: stdxxx.c++.txt - sample C++ program using cout/cerr

  Two default kinds of output stream for Unix programs:

    - Unix "unit 1": same as cout/stdout in C programming
      $ command 1>file
      $ command  >file

    - Unix "unit 2": same as cerr/stderr in C programming
      $ command 2>file

   $ touch foo
   $ ls -l foo nosuchfile 1>out 2>errors

  - to redirect both units, must use:  2>&1  not  >out 2>out

  - scripts: issuing error messages: echo 1>&2 "$0: error msg"

Throwing away output to /dev/null
    $ ls -l date nosuchfile 2>/dev/null
    $ ls -l date nosuchfile  >/dev/null

Standard Unix command-line mail program:

  Without arguments, reads your Unix mail file and prompts for commands:
    $ mail
    & ?          # <- ? gives a short list of some useful commands
    & x          # <- quit without changing mailbox
    & q          # <- quit and empty mailbox into "$HOME/mbox"

  With one or more arguments, sends email to those userids by reading stdin:

    $ mail idallen@idallen.ca idallen@idallen.net
    Subject: this is a test
    This is what
    I want to say.
    ^D
    Cc: idallen@hotmail.com

  You can also use a single period instead of EOF to end the input.

  If mail is not reading a terminal, it doesn't prompt for a subject.
  (You can supply one using a command-line option.)

    $ echo "this is a one line email" | mail idallen@idallen.org
    $ mail idallen@idallen.ca <file.txt
    $ ( sleep 3600 ; echo "Go home" | mail -s Reminder idallen@idallen.net ) &

Text-mode Internet: lynx wget

  Command "lynx" is a text-only web browser:

    $ lynx http://idallen.com/

  You can use lynx to fetch a formatted web page and save the text locally:

    $ lynx -dump http://idallen.com/ >out
    $ less out

  Command "wget" fetches raw web pages and saves them locally (unformatted):

    $ wget -O out http://idallen.com/
    $ less out

Translating or deleting individual characters: tr

  "tr" is one of the very few Unix commands that *only* reads stdin
  (it cannot accept pathnames on the command line)

  To change words and strings, you need to learn to use "sed", not "tr".

  Translate all colon characters to blanks from the password file:

    $ wc -w </etc/passwd
    112
    $ tr ':' ' ' </etc/passwd | wc -w
    378

  Make a file UPPER-CASE:

    $ tr 'a-z' 'A-Z' </etc/passwd | head -1
    ROOT:X:0:0:ROOT:/ROOT:/BIN/BASH

  Delete all digits from a userid:

    $ echo abcd0123 | tr -d '0-9'
    abcd

  Delete all non-digits from a userid:

    $ echo abcd0123 | tr -c -d '0-9'
    0123

  Change spans of multiple blanks into one blank:

    $ echo "a b c  d      e" | tr -s ' '
    a b c d e

  Changing newlines ('\n') to spaces:

    $ touch a b c
    $ echo "Names are $(ls)"
    Names are a
    b
    c
    $ echo "Names are $(ls | tr '\n' ' ')"
    Names are a b c

Selecting fields (columns) from text input

  Two useful commands are "cut" and "awk".

  cut: treats every field separator as a different field number (annoying)

     $ echo "a b c  d  e" | cut -f 3 -d ' '
     c
     $ echo "a b c  d  e" | cut -f 4 -d ' '
     
     $ echo "a b c  d  e" | cut -f 5 -d ' '
     d

  awk: treats spans of blanks as one separator (like the shell)

     $ echo "a b c  d  e" | awk '{print $3}'
     c
     $ echo "a b c  d  e" | awk '{print $4}'
     d
     $ echo "a b c  d  e" | awk '{print $5}'
     e

Sometimes you can massage your data to have the delimiters you want:

   $ last | tr ' ' '\t' | cut -f 1 | less

   $ date
   Sun Jan 23 20:20:00 EST 2005
   $ date | tr ':' ' ' | awk '{print "The hour is:", $4}'
   The hour is: 20

Row selectors and Column selectors in text files:
 "print the third word on every line"
   - select column/field #3 using awk or cut

   $ awk '{print $3}' foo

 "print every word on the third line"
   - select row #3 using a combination of head and tail
 
   $ head -3 foo | tail -1

 "print every word on the lines containing BAR"
   - select lines using grep with a pattern

   $ grep 'BAR' foo

 "print the third word on the fifth line"
   - select row #5 using head/tail and then pipe to column #3 using awk,cut

   $ head -5 foo | tail -1 | awk '{print $3}'

   "awk" can also do this line selection directly with some added syntax:
   
   $ awk 'NR == 5 {print $3}' foo  #<- on record number 5, print field number 3

 "print the third word on lines containing BAR"
   - select lines by pattern using grep then pipe to column #3 using awk,cut

   $ grep 'BAR' foo | awk '{print $3}'

   "awk" can also do this pattern selection directly with some added syntax:

   $ awk '/BAR/ {print $3}' foo    #<- on lines containing BAR, print field 3 

Passwd File format (selecting fields):
  Unix Password file is seven fields, delimited by colon characters:

  userid:password:numeric_uid:numeric_gid:full_name:home_directory:shell

  Generate a list of userids (first field):
    
    $ awk -F: '{print $1}' /etc/passwd
    $ cut -d: -f1          /etc/passwd

  Selecting and displaying the shells (field 7) from the Unix password file:

    $ awk -F: '{print $7}' /etc/passwd
    $ cut -d: -f7          /etc/passwd

Examples of data mining (extracting lines/fields from text files):
  
  How many unique shells are being used?

    $ awk -F: '{print $7}' /etc/passwd | sort | uniq
    $ awk -F: '{print $7}' /etc/passwd | sort | uniq | wc -l

    Why doesn't this following command line produce the same answer?
    (You can't exchange awk and sort and get the same results!)

      $ sort /etc/passwd | awk -F: '{print $7}' | uniq

  Show the shells along with the counts of usage, sorted in reverse:

    $ awk -F: '{print $7}' /etc/passwd | sort | uniq -c | sort -nr

  How many unique directories hold those shells?
  (Split the shell pathnames on slashes to isolate the directory names.)
    
    $ awk -F: '{print $7}' /etc/passwd | awk -F/ '{print $2}' | sort | uniq

  Show the userids with the most logins to this computer, sorted in
  reverse with the count of logins at the beginning:

    $ last | awk '{print $1}' | sort | uniq -c | sort -nr | head

Readings:

  quick_tests.txt     Return Code, Exit Status, ||, &&, test, and if
  unix_command_list.txt     Basic Unix/Linux Command List
  script_style.txt     Shell Script Programming Conventions and Style
  script_checklist.txt Shell Script Checklist - a list of things to verify
  collate_order.txt     Collate order - making GLOB patterns and sorting work
  command_substitution.txt     Command Substitution - $(unix command)
  less_code.txt     Less Code is Better Code - stop cutting and pasting
  deep_nesting.txt     Avoiding deeply nested IF statements
  shell_read.txt     Reading stdin into shell variables
  man_page_RTFM.txt     Searching for items in the Unix manual pages (RTFM)
  shell_variables.txt     Shell Variables you should know (including $* and $@)
  exit_status.txt     Return Code, Exit Status, test, if, and while

Command return status
  Commands have a return status:
  $ true ; echo $?
  $ false ; echo $?
  $ grep foo /etc/passwd ; echo $?
  $ grep root /etc/passwd ; echo $?
  $ cmd1 || cmd2  # execute cmd2 only if cmd1 FAILS
  $ cmd1 && cmd2  # execute cmd2 only if cmd1 SUCCEEDS

  In your scripts, use a quick exit to read on EOF:
  
    read foo || exit 1  # quick exit on EOF

  New: true, false, test, if/then/elif/else/fi, while/do/done

Control Structures 
   if/then/elif/else/fi
   while/do/done

  C programs do arithmetic in their IF statements
   - shells find and run commands; don't do arithmetic
  shells IF statements test return codes, do not do arithmetic
  for shells, return code zero is good (TRUE); opposite of C programs

  example syntax of IF statement:
   if grep lakdsjfl /etc/passwd ; then echo found ; else echo not found ; fi
   - can put "then" on new line instead
   - make into script using $1, then fix as "$1"

  shells often want to do comparisons of strings or numbers
  must run a command to do comparisons:  "test"
  test str1 = str2  ; test str1 != str2
  different ways to compare integers and strings in C; shells also differ
  test int1 -eq int2 ; -ne -lt -gt -le -ge
  Understand: test 0 = 00 ; test 0 -eq 00

- test command string operators: -n -z
- test command file operators: -e -f -d -s -r -w -x
- test Boolean operators: -a and -o

- true and false commands

NET2003 script header
 - 10 parts to overall script layout for this course
 - follow script_style.txt and script_checklist.txt

  Getting input:
    - command line and/or read (and EOF and return status)
    - reading input into shell variables: shell_read.txt

  Basic script structure: Input, Validate, Process, Output (script_style.txt)

  Good error messages!

Arguments to shell scripts
  See shell_variables.txt

Command Substitution
  See command_substitution.txt

Sample Shell Scripts
  See Shell Scripts heading under Notes
  (these are for reading and reference, not for cut-and-paste!)

Readings for next lecture:
  umask.txt     Umask and Permissions
  permissions.txt Unix Permissions