------------------------
Week 3 Notes for NET2003
------------------------
-Ian! D. Allen - idallen@idallen.ca

Remember - knowing how to find out an answer is more important than
memorizing the answer.  Learn to fish!  RTFM!  (Read The Fine Manual)

Reminder: Midterm Test #1 begins Week #4 (see the course home page).

New and updated Notes files to read:

    week03notes.txt     Week 3 Notes for NET2003
    glob_patterns.txt     GLOB patterns (wildcard pathname matching)
    miscellaneous.txt     Miscellaneous Unix Facts
    quotes.txt     Unix/Linux Shell Command Line Quoting
    redirection.txt     Unix Shell I/O Redirection (including Pipes)
    shell_variables.txt    Shell Variables you should know
    unix_command_list.txt     Basic Unix/Linux Command List
    unix_shell.txt     The Unix/Linux Shell

    argv.sh.txt      This shell script displays the command line arguments.
    argv.c++.txt     Display the arguments on the command line.
    argv.c.txt       Display the arguments on the command line.

    practiceTest1.pdf     Practice Test #1 Questions    
    practiceTest1.txt     Practice Test #1 Questions
    practiceTest1.ps      Practice Test #1 Questions
    practiceCommands_1.txt     Practice Unix/Linux Questions #1

- keep up on your readings (Course Outline: average 5 hours/week)

- IBM LPI certification 101 (release 2) Part 1 on Course Home Page
  - covers material on basic Linux command line usage
  - good to review NET2003 lecture material

Usage of various Unix Commands:
 Notes: unix_command_list.txt     Basic Unix/Linux Command List

 ls command output changes into column format if output goes to a tty:
   $ ls
   a  b  c
   $ ls | cat
   a
   b
   c
   $ ls | wc -l
   3

 The sort command sorts all input together (alphabetically), sends it to stdout
  Notes: miscellaneous.txt see "Understanding the different types of sort"

  - original files are not changed  (always to stdout)
  - sort a ; sort b ; sort a b ; sort -r a b
  - to sort lines beginning with numbers, use option "sort -n"

 The "uniq" command removes adjacent duplicate lines (and writes stdout)
  - compare:
   $ sort a a b b               # duplicate lines
   $ sort a a b b | uniq        # no duplicate lines
     (sorting always makes duplicate lines adjacent; uniq removes them)
   $ cat a a b b | uniq         # still has duplicated lines
     (the lines must be adjacent for uniq to remove them)
  - WARNING: uniq takes only zero or one input file, second file is *output*!
    - what if I do:  uniq *     # <- very bad idea

  - useful option to count the duplicate lines: uniq -c
    $ sort /etc/passwd /etc/passwd /etc/passwd | uniq -c | head -2
    3 adm:x:3:4:adm:/var/adm:/bin/sh
    3 alleni99:x:780:780::/home/alleni99:/bin/bash

 The diff command compares two text files (text only)
  - diff a b
  - no output if files are identical
  - lines prefixed with < are different in the file on the left
  - lines prefixed with > are different in the file on the right
  - usually the more lines of output, the more different the files
  - "diff a b" gives different output than "diff b a"
    (at minimum, the < and > in the output point to the opposite files)

Hidden directory pathnames:
  - dot and dot dot are hidden names, not shown by ls or matched by GLOB
  - to see hidden pathnames, use "ls -a" and/or use a leading period in GLOB
    $ ls -a
    $ echo .*

Shell GLOB file name pattern/wildcard matching:
 Notes: glob_patterns.txt - GLOB patterns (wildcard pathname matching)

 - GLOB patterns do not match names that start with a period (hidden
   pathnames), unless you explicitly put the period into the file name
 - like ls, GLOB patterns do not match hidden names
   $ echo *      # <- does not match any hidden pathnames (leading dot)
 - if you want to match hidden *and* non-hidden names, you need at
   least two different GLOB patterns, e.g.
   $ echo * .*   # <- match both visibile and hidden pathnames
 - beware that . and .. are present in every Unix directory and that
   .* matches these two names!

Variables:
 Notes: shell_variables.txt - Shell Variables you should know

 Use variables to hold constants (like #define)
  - e.g. a temporary file name: tmp="/tmp/$USER.$$"
  - shell script maintenance: how many edits needed to change a constant?
 
 For a first reading in shell_variables.txt, know these variables:

   TERM HOME SHELL USER $$

 Variable names are case-sensitive.  $USER is not $User is not $user

 Creating files with unique names:

    $ date >"out$$"          # <- $$ gets replaced with PID of shell
    $ date >"out-$USER-$$"     # <- $USER and $$ get replaced by shell

  QUOTE ALL VARIABLES to protect contents from the shell!
  Quoted strings are always one single argument, except for "$@"

  Double quoting lets $ expand but stops GLOB wildcards
  Without quoting, GLOB inside variables do expand

  Order of processing: quoting/splitting, redirection, variables, globs

  Variables inside variables don't expand:
    $ x=x ; y='$x' ; echo $y

  GLOB inside GLOB don't expand:
    $ touch 'a * *' b c ; echo a*

  GLOB inside variables **DO** expand (unless quoted):
    $ x='*' ; echo $x  # BAD!  WRONG!
    $ msg="*** happy birthday ***" ; echo $msg   # BAD!  WRONG!
    $ file='my file' ; date >$file ; rm $file    # BAD!  WRONG!

  Global vs Local variables: "export" puts a local variable into the
  shell "environment" so that it can be inherited by future child
  procssses.

 Skip the section "Command line arguments" until next week.

Shell command line order of processing:
 Notes: unix_shell.txt - The Unix/Linux Shell

 Shell does expansions on your command line in a strict order:
    1) find quotes, blanks; split into commands on visible pipes and semicolons
    2) locate and remove redirection from each command line
    3) expand variables once on each command line (and in redirection)
    4) expand GLOB patterns (wildcards) once on each command line

 ORDER: quotes and blanks, redirection, variables (blanks again), GLOB
 
  $ x='date | echo hi'
  $ echo $x                # shell does not see any pipe character
  date | echo hi

 You must double-quote all your uses of variables, to prevent the
 contents of the variable from being split on blanks and having GLOB
 characters expanded:

   $ date >$x       # <- wrong if $x contains blanks
   $ date >"$x"     # <- always correct

Pathnames/inodes:
 Notes: pathnames.txt     Unix/Linux Pathnames
 Notes: file_system.txt     Unix/Linux File System
 Notes: links_and_inodes.html     Hard links and Unix file system nodes
 Notes: home_and_HOME.txt     Directories: current, HOME, and /home

  Rationale for multiple names for the same file:
  1. music collection, one file per CD
    - artist directory with all CD's
    - genre directory with all CD's
    - era directory with all CD's
  2. renamed web page - create page with two names for same content
  3. share one file between accounts

 - absolute path vs. relative path
 - explain options and output fields of ls -l -i -s -a
 - making and removing directories - mkdir rmdir
 - empty dir ; ls -a and hidden files (define hidden file) 
   - every empty directory has a link count of two (why?)
   - how many subdirectories are under a directory with a link count of 8?
 - the "rm" command removes a name; it does not delete a file!
   - when all the names are gone, the system removes the file data
 - what permissions do you need on a file to remove its name from a directory?
   - see file_system.txt
 - inode numbers are visible using ls -i
 - the "ln" command makes a new name (only for a file; cannot do dir)
 - hard links are only names; no name is "better" than any other
 - everything except name is stored with the distant inode

  $ date >a ; ln a b ; who >b ; cat a
  $ ls -il a b ; touch a ; ls -il a b
  $ ls -il a b ; chmod +x a ; ls -il a b
  $ who > a ; ln a b ; sort a >b
    - tricky! - never use same input file for output!

  A student was asked to demonstrate linked files;
  they handed in the following information in their answer file;
  what is wrong with the output?
 
    $ touch foo ; ln foo bar ; ls -il foo bar
    816 -rw-r--r-- 1 student student 0 Jan 24 01:03 foo
    816 -rw-r--r-- 1 student student 0 Jan 24 01:03 bar

  Why do empty directories have link counts of two?
  Why do subdirectories increase parent link counts (but not files)?

Man pages:
 See updated Notes file man_page_RTFM.txt

Set up your vi editor profile  (see vi_basics.txt)

Put stty erase in your .bash_profile (see startup_files.txt)

Saving a command-line terminal session log: script
  miscellaneous.txt - "Using script to create a session log"

Messages on your screen:  write mesg
  miscellaneous.txt - "Allowing messages to your Terminal"

Shell Input and Output Redirection:
 Notes: redirection.txt     Unix Shell I/O Redirection (including Pipes)

 - You can only redirect what you can see (no visible output = empty file)!
 - Output Redirection can only go to *one* place (files win over pipes)!

 - no visible output means redirection creates an empty file/pipe
   $ cp a b      # <- no visible output on screen
   $ cp a b >c   # <- file c is created empty (there is no output to redirect)
   $ cp a b | wc # <- nothing goes into the pipe to wc
   0 0 0

Standard Input - input redirection  (commands read stdin without file names)
  Notes: redirection.txt

  Input redirection "<file" replaces your keyboard (stdin) with a file

  Input redirection applies when commands would read keyboard (stdin):
    $ sort /etc/passwd    # <- sort opens and reads /etc/passwd
    $ sort </etc/passwd   # <- shell opens /etc/passwd; sort reads it via stdin

    $ wc /etc/passwd      # reads file
    $ wc                  # reads stdin (keyboard)
    $ wc </etc/passwd     # reads stdin (file opened by shell)
 
  Input Redirection can only come from *one* place (files win over pipes)!

  Cannot force input redirection by commands that ignore stdin!
  - if the command does not read stdin (your keyboard), it will not
    read from the redirected file name; the redirection is wasted

    $ date                # stdin ignored (no reading from keyboard)
    $ date </etc/passwd   # stdin ignored (no reading from file)
    $ echo                # stdin ignored (no reading from keyboard)
    $ echo </etc/passwd   # stdin ignored (no reading from file)
    $ ls                  # stdin ignored (no reading from keyboard)
    $ ls </etc/passwd     # stdin ignored (no reading from file)
    $ wc /etc/passwd      # stdin ignored (no reading from keyboard)
    $ wc /etc/passwd </etc/group  # stdin ignored (no reading from group file)

Shell Scripts:
  Like many Unix commands, if you give the bash command a file argument,
  bash reads input from the file instead of from stdin/keyboard.

  $ cat file     # cat reads file, ignores stdin/keyboard
  $ sort file    # sort reads file, ignores stdin/keyboard
  $ wc file      # wc reads file, ignores stdin/keyboard
  $ head file    # head reads file, ignores stdin/keyboard
  $ bash file    # bash reads file, ignores stdin/keyboard

  Each program does different things when reading the text of the file.
  The job of shells is to find and run programs, so that's what bash
  does when it reads lines from a text file - it treats each line as if
  it were a command line, finds the program, and runs it.  The output
  appears on standard output (your screen).

  Shells issue prompts before reading from a keyboard, other programs do not.
  Shells do not issue prompts if reading from pipes or files.

  A simple shell script of three lines, saved in a file named "scr"
  and given to the bash command as a file name argument:
    $ cat scr
    echo hi
    date
    echo done
    $ bash scr
    hi
    Tue Jan 25 11:28:43 EST 2005
    done

  Another way to feed the script to bash is to send the lines to bash on
  stdin, so that bash reads the lines from the pipe instead of reading
  your keyboard:
    $ cat scr | bash

  A third way to have the shell execute the commands in the file is to
  have your shell open and supply the file to bash on standard input:
    $ bash <scr
  
Making a shell script into an executable command

  By default, you can't directly execute a text file:
    $ ./scr
    bash: ./scr: Permission denied

  You have to enable execute permissions to execute a text file (a script):
    $ chmod +x scr
    $ ./scr
    hi 
    Tue Jan 25 11:29:33 EST 2005
    done

  Unfortunately, an executable script like this doesn't tell the Linux
  kernel what shell it should use to read the script file.  To be *sure*
  to get the correct shell, put a special line at the start of the script:
    $ cat scr
    #!/bin/sh -u
    echo hi
    date
    echo done

  The #! line tells the kernel which program to use to read the script.

  The line  #!/bin/sh -u  must be the VERY FIRST line of the script.
  No blank lines allowed.  The line must not have any blanks in front
  of it.  Unrecognized lines produce errors from the Linux kernel that
  the shell covers up for you by calling another shell to execute the
  script (which may be the wrong shell!).

  All scripts in this course must start with a correct #! line.

  The pathname after the #! must name an existing executable program.
  Usually this is /bin/sh; but, it can be any program:
    $ cat silly
    #!/usr/bin/wc
    echo hi
    date
    echo done
    $ ./silly
    4  6  37 ./silly

  Since on Linux /bin/sh is usually a link to /bin/bash, using /bin/sh is
  the same as using /bin/bash to execute the script.  If you use the name
  /bin/bash in the script, the script will only work on Linux computers.
  If you use the universal name /bin/sh, the script will work on every
  Linux/Unix computer in the world.  Always use /bin/sh as the shell to
  execute a Bourne shell (bash-style) script.

  The "-u" option passed to /bin/sh causes the shell that is reading
  the script file to check for undefined variables and tell you.

Debugging shell scripts
  Since you can't see the lines that bash is reading from the file as
  the script executes, you may not know from which line some output
  is generated.  You can make sh or bash show you the contents of the
  script as it is read using one of two options.

  Verbose (-v) shows every line of the script as it is being read
  and executed, before any variables or GLOB patterns are expanded:
    $ cat scr
    #!/bin/sh -u
    echo "Your shell is $SHELL"
    echo "A GLOB pattern /bin/?sh:"  /bin/?sh

    $ sh -v scr
    #!/bin/sh -u
    echo "Your shell is $SHELL"
    Your shell is /bin/tcsh
    echo "A GLOB pattern /bin/?sh:"  /bin/?sh
    A GLOB pattern /bin/?sh: /bin/csh /bin/ksh /bin/zsh

  The -x option shows only lines that execute (no blank lines or
  comments), and it shows the lines *after* variables and GLOB patterns
  have expanded (the executing lines are prefixed with a plus sign):

    $ sh -x scr
    + echo 'Your shell is /bin/tcsh'
    Your shell is /bin/tcsh
    + echo 'A GLOB pattern /bin/?sh:' /bin/csh /bin/ksh /bin/zsh
    A GLOB pattern /bin/?sh: /bin/csh /bin/ksh /bin/zsh

Review:
  You should know what the following shell meta-characters mean, and
  how to stop the shell from processing them when you don't want it:

    >  <  >>  |  ;  &
    *  ?  []  $
    "  '  \  #

  All of the above characters (in fact, any and all characters except
  NUL and forward slash) can appear in a Unix pathname.