------------------------- Week 05 Notes for NET2003 ------------------------- -Ian! D. Allen - idallen@idallen.ca - www.idallen.com Remember - knowing how to find out an answer is more important than memorizing the answer. Learn to fish! RTFM! (Read The Fine Manual) A 50-minute midterm test is at 10AM next Tuesday (Week 5). A 50-minute lecture follows the midterm at 11AM as usual. ---------------------------------------------------------------------------- Review: - you know how the shell splits lines to create comands with arguments - new commands: see Class Notes unix_command_list.txt - you can find files on the system using "find" and "slocate" - you can sort, forward or reverse, ASCII or numeric - you understand file and pipe redirection of input and output - Class Notes file: redirection.txt - you can write long pipelines readably in scripts by folding lines - you can select fields from lines using "cut" and/or "awk" ---------------------------------------------------------------------------- To compare files: diff - compare text files and show differences cmp - compare binary files and indicate the first different byte Q: How do I compare text files? Q: How do I compare binary files? ---------------------------------------------------------------------------- Commands for selecting and manipulating lines --------------------------------------------- see Notes: data_mining.txt - Using commands and pipes to "mine" and extract Many commands read standard input and write standard output, allowing them to be used in command pipelines to do data mining: Select lines from text streams: grep, awk, sed, head, tail, look, uniq, comm, diff Select fields in lines or parts of lines: awk, sed, cut Transform text (change characters or words in lines): awk, sed, tr Mining for content in the course notes using "grep -l": $ grep -l ssh ~idallen/public_html/teaching/net2003/08w/notes/*.txt /home/idallen/public_html/teaching/net2003/08w/notes/installing_linux.txt /home/idallen/public_html/teaching/net2003/08w/notes/lab01.txt /home/idallen/public_html/teaching/net2003/08w/notes/terminal.txt /home/idallen/public_html/teaching/net2003/08w/notes/unix_command_list.txt /home/idallen/public_html/teaching/net2003/08w/notes/vi_basics.txt /home/idallen/public_html/teaching/net2003/08w/notes/vpn.txt /home/idallen/public_html/teaching/net2003/08w/notes/week01notes.txt /home/idallen/public_html/teaching/net2003/08w/notes/week03notes.txt /home/idallen/public_html/teaching/net2003/08w/notes/week04notes.txt Q: Find all lines matching a given pattern in a file. Q: Output only line 7 from a file. Q: Output only the third blank-delimited field from line 7 from a file. Shell Quoting ------------- see Notes: quotes.txt - Unix/Linux Shell Command Line Quoting The problem - the shell splits on blanks: $ grep -l GLOB pattern ~idallen/public_html/teaching/net2003/08w/notes/*.txt grep: pattern: No such file or directory /home/idallen/public_html/teaching/net2003/08w/notes/finding_files.txt /home/idallen/public_html/teaching/net2003/08w/notes/glob_patterns.txt /home/idallen/public_html/teaching/net2003/08w/notes/shell_basics.txt /home/idallen/public_html/teaching/net2003/08w/notes/week03notes.txt /home/idallen/public_html/teaching/net2003/08w/notes/week04notes.txt The solution - hide the blank from the shell with quotes or a backslash: $ grep -l "GLOB pattern" ~idallen/public_html/teaching/net2003/08w/notes/*.txt /home/idallen/public_html/teaching/net2003/08w/notes/finding_files.txt /home/idallen/public_html/teaching/net2003/08w/notes/glob_patterns.txt /home/idallen/public_html/teaching/net2003/08w/notes/shell_basics.txt /home/idallen/public_html/teaching/net2003/08w/notes/week03notes.txt /home/idallen/public_html/teaching/net2003/08w/notes/week04notes.txt $ grep -l GLOB\ pattern ~idallen/public_html/teaching/net2003/08w/notes/*.txt ...same output... Quoting the metacharacters, to stop the shell from helping you: - quotes and backslashes hide things from the shell - quotes delimit each argument; they tell the shell where it starts/ends - quotes are not counted as part of the argument: $ echo hi | wc ; echo "hi" | wc ; echo 'hi' | wc 1 1 3 1 1 3 1 1 3 - two different strengths of quotes - single quotes hide all characters from the shell - double quotes hide everything *except* ", \, `, and $variable expansions - only single quotes stop backslash handling and $variable expansion - both kinds stop GLOB expansion and blanks - you can also use backslash to hide single characters from the shell - use argv.sh to see how shell creates arguments (can also use ls -d) (fetch argv.sh.txt from the course Notes area and make it executable) Notes: argv.sh.txt - Display the individual arguments on the command line. $ echo " abc "' def ' " ghi " ' jkl ' $ ./argv " abc "' def ' " ghi " ' jkl ' $ mkdir empty ; cd empty ; touch a b c d $ ./argv ' * ' $ ./argv '" * "' $ ./argv '"' * '"' $ ./argv '"'" * "'"' $ ./argv ' * ' * " * " $ ./argv \' * \' Q: How do you use quotes to prevent GLOB expansion by the shell? Q: How do you use quotes to prevent $variable expansion by the shell? Shell $variable substitutions ----------------------------- see Notes: shell_variables.txt - Variables you should know $!/bin/sh -u echo "first argument is $1" echo "all arguments are $*" echo "number of arguments is $#" Don't confuse the '$5' passed to awk with the '$5' used in shell scripts to select the 5th command-line argument. Q: How do you reference command line arguments inside a shell script? Q: How do set a shell variable? How do you expand it? Q: Why must you double-quote shell variables? Network Debugging Commands -------------------------- The output of these commands should be somewhat familiar to you: ping (doesn't work well at Algonquin College) traceroute ( -n ) ip route route The "netstat" command shows TCP (-t) or UDP (-u) connections to this machine: netstat ( -n -a -t -u -p )