% CST8207 Week 04 Notes -- I/O redirection, pipes, stdout, stderr, `2>&1` % Ian! D. Allen -- -- [www.idallen.com] % Winter 2017 - January to April 2017 - Updated 2017-03-13 09:11 EDT - [Course Home Page] - [Course Outline] - [All Weeks] - [Plain Text] Readings, Assignments, Labs, Tests, and ToDo ============================================ Read (at least) these things (All The Words) -------------------------------------------- 1. [Week 04 Notes HTML] -- this file -- **Read All The Words** 2. [Shell GLOB patterns (wildcard pathname matching)] 3. [Shell I/O Redirection] -- stdin, stdout, stderr, and Pipes 4. [Practice Tests and Answers] -- practice questions for your midterm test. 5. [List of Commands You Should Know] 6. [Video Tutorials on Lynda.com] -- tagged by week number 7. [Linux and Sysadmin News in the World] - Create your [lynda.com] account and watch these sections of [Unix for Mac OS X Users] - these videos are for *all* Unix/Linux students, not just Mac OSX users - **3. Working with Files and Directories** - Searching for files and directories 6m 32s - `find` using "wildcard" (GLOB) patterns - same as shell GLOB - **6. Directing Input and Output 20m 39s** - Standard input and standard output 1m 24s - Directing output to a file 4m 13s - Appending to a file 2m 44s - Directing input from a file 5m 28s - Piping output to input 4m 40s - Suppressing output 2m 10s Assignments this week --------------------- Check the due date for each assignment and put a reminder in your agenda, calendar, and digital assistant. Just like in the Real World, not all due dates are on the same days or at the same times. - Review last week. Did you do everything assigned last week? - Read All The Words, Do, and then Submit via Blackboard: - [Assignment #03 HTML] -- working with files and pathnames: `PS1 find mkdir mv cp rm touch file` - Bonus (optional) [Assignment #04 HTML] -- the VIM text editor - [Assignment #05 HTML] will be out soon -- using GLOB wildcard patterns, selecting files using GLOB, redirection and pipes - Really do **Read All The Words**. You don't get a second chance to get it right. Lab work this week ------------------ - Finish the readings, worksheets, and [Assignment #03 HTML] from last week. - Do the weekly readings on [Shell GLOB patterns][Shell GLOB patterns (wildcard pathname matching)] (wildcard pathname matching) and [Shell I/O Redirection] (including Pipes), also listed above. - Complete and save the ODT or PDF versions of [Worksheet #04 HTML] and [Worksheet #05 HTML] in preparation for your Midterm Test and for doing [Assignment #05 HTML] - Read [The VI (VIM) Text Editor] and optionally do the bonus [Assignment #04 HTML]. - Get ready for upcoming [Assignment #05 HTML] -- GLOB wildcard patterns, selecting files using GLOB, redirection and pipes - This assignment requires you to understand the material in the web pages and Worksheets 4 and 5. Worksheets ---------- Worksheets are preparation for your assignments. You can't do the assignments without having done the worksheets first, and you can't do the worksheets without having first read the Course Notes: 1. Read. 2. Worksheets. 3. Assignment. Form a small study group to do the worksheets. Each person tries the example given, and you make sure you all get the same answers. Worksheets are not for hand-in; they are not worth marks; the assignments test your knowledge of the lectures and worksheets. > The worksheets are available in four formats: Open Office (ODT), PDF, HTML, > and Text. Only the Open Office format allows you "fill in the blanks" in > the worksheet. The PDF format looks good but doesn't allow you to type into > the blanks in the worksheet. The HTML format is crude but useful for quick > for viewing online. Do **NOT** open the Worksheet ODT files using any Microsoft products; they will mangle the format and mis-number the questions. Use the free Libre Office or Open Office programs to open these ODT documents. On campus, you can [download Libre Office here]. These first two worksheets require you to have read [File System and Pathnames]: - [Worksheet #02 ODT] -- Using standard Linux commands I - [Worksheet #02 PDF] - [Worksheet #02 HTML] - `PS1, cd, find, less, ls, man, mkdir, passwd, pwd, rmdir` - [Worksheet #03 ODT] -- Using standard Linux commands II - [Worksheet #03 PDF] - [Worksheet #03 HTML] - `cat, clear, cp, find, fgrep, history, less, man, mv, rm, sleep, touch` - [Worksheet #04 ODT] -- shell GLOB patterns and Aliases - [Worksheet #04 PDF] - [Worksheet #04 HTML] - shell GLOB patterns (wildcards), Aliases: `alias, sum, unalias` - [Worksheet #05 ODT] -- shell I/O redirection (including pipes) - [Worksheet #05 PDF] - [Worksheet #05 HTML] - shell I/O redirection, `date, head, nl, tail, tr, wc` - [Worksheet #06 HTML] -- *Optional* Bonus VIM Text Editor Practice - This is an *optional* worksheet for a BONUS assignment using `vim` - Optional command-line VIM tutorial: the `vimtutor` program on the CLS. - Bonus (optional) [Assignment #04 HTML] -- the VIM text editor - Read [The VI (VIM) Text Editor] Worksheets prepare you for the upcoming assignments. Upcoming tests ============== For full marks, read the [Test Instructions] (all the words) before your midterm tests. Put these dates into your phone! Use the **Name Game** link to test your name before the test. I don't answer questions about the instructions during the test. 1. **First Midterm test:** 45 minutes; in lecture class 10am on Friday in Week 5 (Feb 10) 2. **Second Midterm test:** 45 minutes; in lecture class 10am on Friday in Week 9 (Mar 17) Tests take place at 10am in your 50-minute lecture class, not in your lab period. Each midterm test is 45 minutes long and contains approximately 45 multiple-choice questions similar to those found in [Practice Tests and Answers]. Midterm Test #1 -- Friday February 10 ------------------------------------- - Midterm #1 takes place at 10am on Friday, February 10 (Week 5) in your scheduled lecture class (not in your lab period). - For full marks, you must read the [Test Instructions] before the test for important directions on how to enter your name, student number, answers, your lab (not lecture) section number, and the test version code on the question sheet and the mark-sense forms. - There may be more questions on the test than you can answer in the time allowed; answer the ones you know, first. - A set of practice questions and answers for the first midterm test is posted: [Practice Tests and Answers]. - The tests use the semicolon `;` to separate multiple commands on the same line, to save space and paper, e.g. three separate commands can be written on one line like this: `date ; echo "hi" ; ls` - **Do not use semicolons as an interactive human user!** Type each command on its own line with the \[Enter\] key. - Blackboard has some quizzes taken randomly from the first practice test in [Practice Tests and Answers]. The full practice test itself is *not* part of your course grade, but these quizzes *are* part of your course grade. See below. ### Midterm #1 Topics - The Weekly Notes 1, 2, 3, and 4 (this week), including the readings. - Weekly web pages up to and including [Shell I/O Redirection]. - Worksheets 2, 3, 4, and 5. - Assignments 2 and 3. - The questions in the first practice test in [Practice Tests and Answers]. - Does *not* include anything about editors `vim` or `nano`. - Does *not* include anything about the [Course Introduction]. For full marks, you must read the [Test Instructions]. ### Midterm Test Quizzes Marking Scheme Three Blackboard quizzes are created from the Practice Tests made available before the two Midterm tests and Final Exam. Each Midterm and Final Exam will have an associated set of unmarked practice questions and a set of marked quizzes based on those practice questions. See the course outline for the mark weight of all course quizzes, midterm tests, and exams. The quizzes are open-book, but the Midterm Tests and Final Exam are closed-book (as will be your job interview). Each Midterm Test quiz is a small sample taken from the associated full Practice Test located in [Practice Tests and Answers]. Each quiz is 10 questions long and you see the answers right after you submit the quiz. You can take the quiz as many times as you like. Every time you take the quiz, you get a random set of ten questions from the practice test. You will not see all the practice questions by doing quizzes; to see all the practice questions, you must do all the questions in the actual PDF practice test posted in the Course Notes. *Review the full PDF file; don't rely on the quizzes to prepare you for the test!* Each quiz closes just before the Final Exam in this course; quizzes submitted after the Final Exam begins may not count toward your best score. Your mark for this quiz is the average of your **five** best quiz scores. Examples: - Your best scores: 10 10 10 10 10 10 10 9 8 7 7 4 - Your quiz mark: (10+10+10+10+10)/50 = 100% - Your best scores: 10 10 10 10 9 8 7 7 4 3 - Your quiz mark: (10+10+10+10+9)/50 = 98% - Your best scores: 10 10 10 9 8 7 7 4 3 2 - Your quiz mark: (10+10+10+9+8)/50 = 94% - Your best scores: 10 8 8 8 8 5 4 4 4 - Your quiz mark: (10+8+8+8+8)/50 = 84% - Your best scores: 10 10 8 (only three quizzes submitted) - Your quiz mark: (10+10+8+0+0)/50 = 56% Your score will be taken from the **five** best scores. The more times you do the quiz, the more likely you are to have a set of excellent best scores. You must have **five** perfect quiz scores to get a perfect averaged quiz mark. Missing quizzes (fewer than five) count as zeroes. Only the **five** best scores are averaged. (Blackboard averages all the scores and thus displays your score incorrectly. Ignore the mark on Blackboard.) This quiz closes just before the Final Exam in this course; quizzes submitted after the Final Exam begins may not count toward your best score. See the **Assignments and Quizzes** section in the Blackboard left side-bar for your course. **Note:** *Blackboard displays the quiz mark incorrectly, since it cannot calculate "best 5" and instead averages the marks of **all** your quiz attempts. Your quiz mark is actually the average of your five **best** attempts, not all the attempts as shown by Blackboard. Ignore the Blackboard mark; it is wrong.* Notes from the Classroom ======================== - **Take notes in class!** Keep a pad open on your desk. - This week, we finish with GLOB patterns and learn about redirection and pipes. Draw file system diagrams ------------------------- Draw File System Diagrams to understand where files are when you do the worksheets, the assignments, and the tests and exams! See Ian Martin's [File System Diagram Example] Did not change password -- CLS access revoked --------------------------------------------- Students will have their CLS accounts disabled if they have not changed their default password. (I do not want people breaking into my machine because of you.) See my online timetable for how to make an office appointment to see me if you want your account back. Commands Used ------------- > Keep a notebook with a [List of Commands][List of Commands You Should Know] > in it. > > - You need to write down yourself what each command *does*. > - Check the updated list of commands each week. > - I will check for this list in your lab periods. > - Bring your notes to class! Stop wasting time looking up commands. - Are you making notes from the worksheets on how each command works? - What do the options used in the worksheets mean, for each command? - Don't copy and use options that you don't understand! - The stupid `nano` text editor. (Learn `vim` instead!) - Using GLOB patterns as `-name` arguments to `find` - Remember to quote the `find` GLOB patterns to hide them from the shell! - GLOB patterns are *NOT* needed in Assignment 3. See Assignment 5. - Confused about redirection and the copy and move commands? These are wrong: - `cp cal.txt >calnew,txt # WRONG: error message from cp` - `mv cal.txt ../../ >cal.txt # WRONG: what is now in cal.txt ?` - No example in any of the notes or worksheets ever uses redirection and `cp` or `mv` together. These commands do not produce anything on standard output that is worth redirecting. Usually, the commands produce no output at all. Do not use redirection with `mv` or `cp`. - Confused about pathnames and blanks on the command line? This is wrong: - `cp file.txt ../../../ file.copy # WRONG` - Blanks separate arguments. This is covered in detail in [Command Arguments and Options]. You are giving the copy command three pathnames (three arguments), where you should have only two. The first argument is the source; the second is the destination: - `cp file.txt ../../../file.copy # RIGHT` Not Reading All The Words ------------------------- These do not produce the same output file: $ find . >foo ; mv foo dir/foo $ find . >dir/foo If the assignment asks you to redirect the output into a file in a specific directory, then only the second answer above is correct. The first answer, using `mv`, will have the wrong pathnames in it. Working hard, not working smart ------------------------------- - Stop using `cd` Windows People! Use pathnames! This is very inefficient: $ cd CST8207-17W $ cd Assignments $ cd assignment03 $ cd topdir $ cd other $ cat foo.txt This is all you need to do (using the TAB key to help you): $ cat CST8207-17W/Assignments/assignment03/topdir/other/foo.txt - Stop using `cd` Windows People! Use pathnames! This is very inefficient: $ cd dir1 $ cd subdir $ touch file $ cd .. $ cd .. $ cd dir2 $ cd subdir $ touch file $ cd .. $ cd .. [...etc...] This is all you need to do (using the TAB key to help you): $ touch dir1/subdir/file dir2/subdir/file [...etc...] - If a pathname doesn't work, use `ls` to find out why. Usually it's because you're spelling it wrong. Use the TAB key and the shell will complete the pathname for you, with no spelling errors. Not doing worksheets: working hard, not smart --------------------------------------------- This student below didn't do the worksheets and doesn't know about any options to the `mkdir` command. Instead of typing one command to make the directory tree, he has to use ten (!), and gets the name wrong, too. He also didn't hear me talk about not using `cd` all the time, so he marks himself as a Windows newbie. This won't look good in a job interview: $ mkdir 1ldIr $ cd 1ldIr $ mkdir one $ mkdir two $ mkdir one/0neOne $ cd one $ ls $ cd .. $ ls $ mkdir two/tw0two **Don't be this guy.** No employer wants to pay you to type ten commands when one will do the job. Work smarter, not harder. Do the worksheets! Fifteen minute rule: don't waste your time ========================================== See the [Course Introduction: fifteen minute rule] Attacks on the CLS ================== You are not paranoid if they really are out to get you. See the [Week 03 Notes HTML] Locked out of the CLS ===================== See the [Week 03 Notes HTML] Jan 17 20:26:05 Failed password for XXXXXXXX from 99.248.210.222 Jan 17 20:26:32 Failed password for XXXXXXXX from 99.248.210.222 Jan 17 20:27:50 Failed password for XXXXXXXX from 99.248.210.222 Jan 17 20:28:03 Failed password for XXXXXXXX from 99.248.210.222 Jan 17 20:28:19 Failed password for XXXXXXXX from 99.248.210.222 Jan 19 22:13:39 Invalid user XXXXXXXX@algonquinlive.com from 99.248.210.222 Jan 19 22:15:35 Invalid user XXXXXXXX@algonquinlive.com from 99.248.210.222 Jan 19 22:15:54 Failed password for invalid user XXXXXXXX@algonquinlive.com from 99.248.210.222 Jan 21 18:45:13 Accepted password for XXXXXXXX from 99.248.210.222 Jan 21 19:18:34 Accepted password for XXXXXXXX from 99.248.210.222 Jan 22 17:43:57 Accepted password for XXXXXXXX from 99.248.210.222 Jan 29 17:41:59 Invalid user XXXXXXXX from 99.248.210.222 Jan 29 17:42:14 Failed password for invalid user XXXXXXXX from 99.248.210.222 Jan 29 17:42:35 Failed password for invalid user XXXXXXXX from 99.248.210.222 ![Take Notes in Class] -- | Ian! D. Allen, BA, MMath - idallen@idallen.ca - Ottawa, Ontario, Canada | Home Page: http://idallen.com/ Contact Improv: http://contactimprov.ca/ | College professor (Free/Libre GNU+Linux) at: http://teaching.idallen.com/ | Defend digital freedom: http://eff.org/ and have fun: http://fools.ca/ [Plain Text] - plain text version of this page in [Pandoc Markdown] format [www.idallen.com]: http://www.idallen.com/ [Course Home Page]: .. [Course Outline]: course_outline.pdf [All Weeks]: indexcgi.cgi [Plain Text]: week04notes.txt [Week 04 Notes HTML]: week04notes.html [Shell GLOB patterns (wildcard pathname matching)]: 190_glob_patterns.html [Shell I/O Redirection]: 200_redirection.html [Practice Tests and Answers]: PRACTICE_TEST_README.html [List of Commands You Should Know]: 900_unix_command_list.html [Video Tutorials on Lynda.com]: 910_lynda_index.html [Linux and Sysadmin News in the World]: 950_linux_world.html [lynda.com]: http://algonquincollege.com/onlineresources/mobileStudent/lynda.htm [Unix for Mac OS X Users]: http://www.lynda.com/Mac-OS-X-10-6-tutorials/Unix-for-Mac-OS-X-Users/78546-2.html [Assignment #03 HTML]: assignment03.html [Assignment #04 HTML]: assignment04.html [Assignment #05 HTML]: assignment05.html [Worksheet #04 HTML]: worksheet04.html [Worksheet #05 HTML]: worksheet05.html [The VI (VIM) Text Editor]: 300_vi_text_editor.html [download Libre Office here]: 050_course_introduction.html#install-libreoffice-or-openoffice-into-windows [File System and Pathnames]: 160_pathnames.html [Worksheet #02 ODT]: worksheet02.odt [Worksheet #02 PDF]: worksheet02.pdf [Worksheet #02 HTML]: worksheet02.html [Worksheet #03 ODT]: worksheet03.odt [Worksheet #03 PDF]: worksheet03.pdf [Worksheet #03 HTML]: worksheet03.html [Worksheet #04 ODT]: worksheet04.odt [Worksheet #04 PDF]: worksheet04.pdf [Worksheet #05 ODT]: worksheet05.odt [Worksheet #05 PDF]: worksheet05.pdf [Worksheet #06 HTML]: worksheet06.html [Test Instructions]: 000_test_instructions.html [Course Introduction]: 050_course_introduction.html [File System Diagram Example]: data/filesystemdiagram.png [Command Arguments and Options]: 150_arguments_and_options.html [Course Introduction: fifteen minute rule]: 050_course_introduction.html#fifteen-minute-rule-dont-waste-your-time [Week 03 Notes HTML]: week03notes.html [Take Notes in Class]: data/remember.jpg "Take Notes in Class" [Pandoc Markdown]: http://johnmacfarlane.net/pandoc/