% CST8207 Week 05 Notes -- I/O redirection % Ian! D. Allen -- -- [www.idallen.com] % Winter 2015 - January to Apil 2015 - Updated 2015-10-01 03:54 EDT - [Course Home Page] - [Course Outline] - [All Weeks] - [Plain Text] Readings, Assignments, Labs, Tests, and ToDo ============================================ - Read (at least) these things (All The Words): 1. [Week 05 Notes HTML] -- this file -- **Read All The Words** 2. [Unix Shell I/O Redirection (including Pipes)] 3. [List of Commands You Should Know] 4. [Video Tutorials on Lynda.com] - Create your [lynda.com] account and watch [Unix for Mac OS X Users] - **6. Directing Input and Output 20m 39s** - Standard input and standard output 1m 24s - Directing output to a file 4m 13s - Appending to a file 2m 44s - Directing input from a file 5m 28s - Piping output to input 4m 40s - Suppressing output 2m 10s - **7. Configuring Your Working Environment 41m 28s** - Setting command aliases 6m 59s - saving aliases in your `.bashrc` file Assignments this week --------------------- Check the due date for each assignment and put a reminder in your agenda, calendar, and digital assistant. - Read All The Words, Do, and then Submit via Blackboard: - [Assignment #03 HTML] -- working with files: `PS1, find, mkdir, mv` - Bonus (optional) [Assignment #04 HTML] -- the VIM text editor - [Assignment #05 HTML] -- using GLOB patterns and redirection - Really do **Read All The Words**. You don't get a second chance to get it right. ### Worksheets The worksheets are available in four formats: Open Office (ODT), PDF, HTML, and Text. Only the Open Office format allows you "fill in the blanks" in the worksheet. The PDF format looks good but doesn't allow you to type into the blanks in the worksheet. The HTML format is crude but useful for quick for viewing online. Do **NOT** open the ODT files using any Microsoft products; they will mangle the format and mis-number the questions. Use the free Libre Office or Open Office programs to open these ODT documents. On campus, you can [download Libre Office here]. - [Worksheet #04 ODT] -- shell GLOB patterns and Aliases - [Worksheet #04 PDF] - [Worksheet #04 HTML] - shell GLOB patterns (wildcards), Aliases: `alias, sum, unalias` - [Worksheet #05 ODT] -- shell I/O redirection (including pipes) - [Worksheet #05 PDF] - [Worksheet #05 HTML] - shell I/O redirection, `date, head, nl, tail, tr, wc` - [Worksheet #06 ODT] -- *Optional* Bonus VIM Text Editor Practice - [Worksheet #06 PDF] - [Worksheet #06 HTML] - This is an *optional* worksheet for a BONUS assignment using `vim` - Optional command-line VIM tutorial: the `vimtutor` program on the CLS. - Optional Reading: [The VI (VIM) Text Editor] - Bonus (optional) [Assignment #04 HTML] -- the VIM text editor Lab work this week ------------------ - Bonus (optional) [Assignment #04 HTML] is ready - [Assignment #05 HTML] is ready - Finish the above assignments, which involve work on the worksheets. Upcoming tests -------------- Read the [Test Instructions] (all the words) before your first midterm test. 1. First Midterm test: 45 minutes; in class 8am Thursday in Week 6 (February 12) 2. Second Midterm test: 45 minutes; in class 8am Thursday in Week 9 (March 12) Tests take place in your 8am lecture hour, not in your lab period. ### Midterm Test #1 - Midterm #1 takes place 8am Thursday February 12 (Week 6) in your scheduled lecture hour (not in your lab period). - For full marks, you must read the [Test Instructions] before the test for important directions on how to enter your answers, your lab (not lecture) section number, and the test version number on the question sheet and the mark-sense forms. - There may be more questions on the test than you can answer in the time allowed; answer the ones you know, first. - A set of practice questions and answers for the first midterm test is posted: [Practice Tests and Answers]. - Blackboard has some quizzes taken randomly from the practice test. See below. ### Quizzes: Midterm #1 Quiz This quiz is one of several quizzes in this course. Each midterm and final exam will have an associated quiz. See the course outline for the mark weight of all course quizzes, midterm tests, and exams. The quizzes are open-book, but the midterm tests and final exam are closed-book. This quiz is based on the [Midterm #1 Practice Test] questions that are posted in the [Class Notes][All Weeks]. The quiz is 10 questions long and you see the answers right after you submit the quiz. You can take the quiz as many times as you like. Every time you take the quiz, you get a random set of ten questions from the practice test. You will not see all the practice questions by doing quizzes; to see all the practice questions, you must do all the questions in the actual PDF practice test posted in the Course Notes. This quiz closes just before the Final Exam in this course; quizzes submitted after the Final Exam begins may not count toward your best score. Your mark for this quiz is the average of your five best quiz scores. Examples: - Your best scores: 10 10 10 10 10 10 10 9 8 7 7 4 - Your quiz mark: (10+10+10+10+10)/50 = 100% - Your best scores: 10 10 10 10 9 8 7 7 4 3 - Your quiz mark: (10+10+10+10+9)/50 = 98% - Your best scores: 10 10 10 9 8 7 7 4 3 2 - Your quiz mark: (10+10+10+9+8)/50 = 94% - Your best scores: 10 8 8 8 8 5 4 4 4 - Your quiz mark: (10+8+8+8+8)/50 = 84% - Your best scores: 10 10 8 (only three quizzes submitted) - Your quiz mark: (10+10+8+0+0)/50 = 56% Your score will be taken from the five best scores. The more times you do the quiz, the more likely you are to have a set of excellent best scores. You must have five perfect quiz scores to get a perfect averaged quiz mark. Missing quizzes (fewer than five) count as zeroes. Only the five best scores are averaged. (Blackboard averages all the scores and thus displays your score incorrectly.) This quiz closes just before the Final Exam in this course; quizzes submitted after the Final Exam begins may not count toward your best score. See the **Assignments and Quizzes** section in the Blackboard left side-bar for your course. **Note:** *Blackboard displays the quiz mark incorrectly, since it cannot calculate "best 5" and instead averages the marks of *all* your quiz attempts. Your quiz mark is actually the average of your five *best* attempts, not all the attempts as shown by Blackboard. Ignore the Blackboard mark; it is wrong.* From the Class Notes link on the Course Home Page ================================================= - [Course Home Page] - [All Weeks] - Review last week. Did you do everything assigned last week? From the Classroom Whiteboard/Chalkboard ======================================== - **Take notes in class!** Your in-class notes would go here. - [Do your class notes look like this?] - Practice questions and answers for the midterm test next week are posted; see above. - This week, we continue to learn about I/O redirection. - Are you making notes from the worksheets on how each command works? - What do the options used in the worksheets mean, for each command? - Don't copy and use options that you don't understand! Read All The Words ------------------ - When I say "read all the words in a question, especially the Hints" - When I say "don't use Word to open the worksheets" - When I say "do the worksheet to know how to do the assignment" - When I say "keep a notebook with descriptions of each command" - When I say "don't work on a problem longer than 15 minutes before trying something else, such as getting help" Useful Command to extract fields from lines: `awk` -------------------------------------------------- The oddly-named `awk` command can extract a field, by field number, from one or more input lines. The default is to find fields separated by any amount of *space* characters: $ echo one two three four five one two three four five $ echo one two three four five | awk '{ print $1 }' one $ echo one two three four five | awk '{ print $2 }' two $ echo one two three four five | awk '{ print $5 }' five You can also use the field number `NF` (Number of Fields) to extract just the *last* field from any input lines: $ echo one two three four five | awk '{ print $NF }' five $ echo one two three four | awk '{ print $NF }' four $ echo one two three | awk '{ print $NF }' three $ echo one two | awk '{ print $NF }' two $ echo one | awk '{ print $NF }' one The single command-line argument to `awk` must be single-quoted to hide the dollar character from the shell. Here is a common use of `fgrep` to select lines and `awk` to extract fields from a system log file and count the unique occurrences: $ fgrep 'refused connect' /var/log/auth.log \ | awk '{print $NF}' \ | sort | uniq -c | sort -nr | head Working Smart under Linux ------------------------- Growing up under Mac/Windows, you are accustomed to having to "go to" a folder to make any changes to files in that folder. This leads to some inefficient behaviour under Unix/Linux, because Unix/Linux can operate on any file in any folder without having to "go" there: The "Windows/Mac" way to create a file under `a/b/c/d/date.txt`: $ mkdir a $ cd a $ mkdir b $ cd b $ mkdir c $ cd c $ mkdir d $ cd d $ date >date.txt $ cd The Unix/Linux way: $ mkdir -p a/b/c/d $ date >a/b/c/d/date.txt The "Windows/Mac" way to rename a file under `a/b/c/d/date.txt` to be `old.txt`: $ cd a $ cd b $ cd c $ cd d $ mv date.txt old.txt $ cd The Unix/Linux way: $ mv a/b/c/d/date.txt a/b/c/d/old.txt The "Windows/Mac" way to delete `a/b/c/d/old.txt`: $ cd a $ cd b $ cd c $ cd d $ rm old.txt $ cd The Unix/Linux way: $ rm a/b/c/d/old.txt Using Unix/Linux pathnames, you can work faster, and all your commands are saved in your history for modification and re-use. Current Working Directory Stack ------------------------------- The shell built-in command `pushd` works like `cd` to change the current directory, but it saves the previous directory on a stack and lets you type `popd` to return to the previous directory. You may find this saves you some typing. See the shell man page or use the `help` built-in. Real Sysadmin Work ================== Locked out of Course Linux Server --------------------------------- I told you not to try to log in to the CLS with a blank userid. Now your home IP address is locked out from the CLS: Jan 30 14:25:52 Accepted password for XXXXXXXX from 74.14.198.226 Jan 30 17:35:15 Invalid user from 74.14.198.226 Jan 30 17:35:30 Failed password for invalid user from 74.14.198.226 Jan 30 17:35:59 Failed password for invalid user from 74.14.198.226 Jan 30 17:36:07 Failed password for invalid user from 74.14.198.226 Jan 30 17:37:25 Invalid user from 74.14.198.226 Jan 30 17:38:01 Invalid user from 74.14.198.226 Jan 30 17:38:30 Failed password for invalid user from 74.14.198.226 Jan 30 17:39:03 Failed password for invalid user from 74.14.198.226 Jan 30 17:41:51 refused connect from bas16-ottawa23-1242482402.dsl.bell.ca (74.14.198.226) Jan 30 17:43:03 refused connect from bas16-ottawa23-1242482402.dsl.bell.ca (74.14.198.226) Someone else did the same thing, and also got locked out: Feb 1 00:10:21 Accepted password for XXXXXXXX from 23.91.145.228 Feb 1 05:58:08 Invalid user from 23.91.145.228 Feb 1 05:58:31 Failed password for invalid user from 23.91.145.228 Feb 1 05:58:59 Failed password for invalid user from 23.91.145.228 Feb 1 05:59:21 Failed password for invalid user from 23.91.145.228 Feb 1 06:01:12 Invalid user from 23.91.145.228 Feb 1 06:01:51 Failed none for invalid user from 23.91.145.228 Added the following hosts to /etc/hosts.evil: 23-91-145-228.cpe.pppoe.ca Someone spelled their own userid incorrectly (seven characters) over and over, and got locked out: Jan 26 00:11:25 Accepted password for XXXXXXXX from 76.69.121.114 Feb 1 09:21:04 Invalid user XXXXXXX from 76.69.121.114 Feb 1 09:21:25 Failed password for invalid user XXXXXXX from 76.69.121.114 Feb 1 09:21:50 Failed password for invalid user XXXXXXX from 76.69.121.114 Feb 1 09:22:10 Failed password for invalid user XXXXXXX from 76.69.121.114 Feb 1 09:22:30 Failed password for invalid user XXXXXXX from 76.69.121.114 Added the following hosts to /etc/hosts.evil: otwaon0916w-lp140-02-1279621490.dsl.bell.ca Don't do that. Attacks on the Course Linux Server ---------------------------------- - Count the attacks: `fgrep -c "refused connect" /var/log/auth.log` - Another way to count the lines (less efficient): `fgrep "refused connect" /var/log/auth.log | wc` - Only show the last 10 lines: `fgrep 'refused connect' /var/log/auth.log | tail` - See the current list of *evil* host IPs: `less /etc/hosts.evil` Job Opening in Ottawa --------------------- From the local `linux-consult` mailing list: From: "David F. Skoll" To: linux-consult@list.flora.ca Date: Wed, 7 Jan 2015 14:51:49 -0500 Organization: Roaring Penguin Software Inc. Subject: [Linux-consult] Term position for technical support specialist at Roaring Penguin Please see below for an opening at Roaring Penguin. If you're interested in applying, please send your cover letter and resume (plain-text, HTML or PDF only, please) to jobs@roaringpenguin.com Info about our company is at https://www.roaringpenguin.com/company Regards, David. ========================================================================== Roaring Penguin Software seeks a Technical Support Specialist to provide customer support for its anti-spam products and services. This position is a full-time term position for six months. Depending on circumstances, it is possible that the position will become a permanent position. Requirements for the position are: o Extensive familiarity with Linux system administration. Familiarity with the Debian or Debian-derived distributions would be an asset. o Extensive familiarity with SMTP and related email standards such as DKIM, SPF, DMARC and MIME. o Understanding of the TCP/IP suite and related protocols including DNS. o A pleasant, efficient phone manner and excellent English verbal and written communication skills. o The willingness to be on-call for remote support between 9am and 9pm for one week out of four. The following are not requirements, but are highly desirable: o Experience in a technical support role. o Familiarity with Perl and PHP. o Familiarity with the PostgreSQL database system. o Familiarity with the Sendmail MTA. _______________________________________________ Linux-consult mailing list Linux-consult@list.flora.ca http://list.flora.ca/mailman/listinfo/linux-consult ----------------------------------------------------------------------------- ![Take Notes in Class] -- | Ian! D. Allen - idallen@idallen.ca - Ottawa, Ontario, Canada | Home Page: http://idallen.com/ Contact Improv: http://contactimprov.ca/ | College professor (Free/Libre GNU+Linux) at: http://teaching.idallen.com/ | Defend digital freedom: http://eff.org/ and have fun: http://fools.ca/ [Plain Text] - plain text version of this page in [Pandoc Markdown] format [www.idallen.com]: http://www.idallen.com/ [Course Home Page]: .. [Course Outline]: course_outline.pdf [All Weeks]: indexcgi.cgi [Plain Text]: week05notes.txt [Week 05 Notes HTML]: week05notes.html [Unix Shell I/O Redirection (including Pipes)]: 200_redirection.html [List of Commands You Should Know]: 900_unix_command_list.html [Video Tutorials on Lynda.com]: 910_lynda_index.html [lynda.com]: http://algonquincollege.com/onlineresources/mobileStudent/lynda.htm [Unix for Mac OS X Users]: http://www.lynda.com/Mac-OS-X-10-6-tutorials/Unix-for-Mac-OS-X-Users/78546-2.html [Assignment #03 HTML]: assignment03.html [Assignment #04 HTML]: assignment04.html [Assignment #05 HTML]: assignment05.html [download Libre Office here]: 050_course_introduction.html#install-libreoffice-or-openoffice-into-windows [Worksheet #04 ODT]: worksheet04.odt [Worksheet #04 PDF]: worksheet04.pdf [Worksheet #04 HTML]: worksheet04.html [Worksheet #05 ODT]: worksheet05.odt [Worksheet #05 PDF]: worksheet05.pdf [Worksheet #05 HTML]: worksheet05.html [Worksheet #06 ODT]: worksheet06.odt [Worksheet #06 PDF]: worksheet06.pdf [Worksheet #06 HTML]: worksheet06.html [The VI (VIM) Text Editor]: 300_vi_text_editor.html [Test Instructions]: 000_test_instructions.html [Practice Tests and Answers]: PRACTICE_TEST_README.html [Midterm #1 Practice Test]: practicetest1.pdf [Do your class notes look like this?]: data/kf_link_redirect.png [Take Notes in Class]: data/remember.jpg "Take Notes in Class" [Pandoc Markdown]: http://johnmacfarlane.net/pandoc/