% CST8207 Week 03 Notes -- absolute and relative paths, ROOT and HOME, finding files % Ian! D. Allen -- -- [www.idallen.com] % Winter 2018 - January to April 2018 - Updated 2018-04-04 12:11 EDT - [Course Home Page] - [Course Outline] - [All Weeks] - [Plain Text] Readings, Assignments, Labs, Tests, and ToDo ============================================ Read (at least) these things (All The Words) -------------------------------------------- 1. [Week 03 Notes HTML] -- this file -- **Read All The Words** 2. [File System and Pathnames] -- ROOT, current directory, absolute, relative, `.` (dot), `..` (dot dot) 3. [Directories] -- ROOT, `/root`, HOME, `/home`, and the current directory 4. [Finding files] -- searching for files by name, size, use, modify time, etc. 5. [Shell GLOB patterns] -- wildcard pattern matching of pathnames 6. [List of Commands] -- Command names you should know, listed by week 7. [Video Tutorials on Lynda.com] -- tagged by week number 8. [Linux and Sysadmin News in the World] - Create your [lynda.com] account and watch these sections of [Unix for Mac OS X Users] - these videos are for *all* Unix/Linux students, not just Mac OSX users - **2. Filesystem Basics** - The working directory 2m 49s - `pwd` - Listing files and directories 3m 59s - `pwd`, `ls`, hidden files - Moving around the filesystem 4m 58s - `cd`, TAB filename completion, absolute/relative paths - Filesystem organization 4m 12s - ROOT, standard Unix folders - **3. Working with Files and Directories** - Naming files 5m 41s - Unix is case-sensitive, OSX is case-insensitive! - avoid special characters or blanks in Unix pathnames - Creating files 2m 19s - `touch` - Unix text editors 6m 39s - `vi/vim, nano` - `head, tail` - Reading files 5m 35s - `cat, more, less` - Reading portions of files 3m 27s - `head, tail` - Creating directories 2m 40s - `mkdir` - Moving and renaming files and directories 8m 32s - `mv` (including using `..`) - Copying files and directories 3m 7s - `cp` - Deleting files and directories 3m 38s - `rm, rmdir` - Searching for files and directories 6m 32s - `find` using "wildcard" (GLOB) patterns - same as shell GLOB - **5. Commands and Programs** - Command basics 4m 4s - commands are usually small programs; ways to exit from programs - Using the command history 8m 16s - shell command history is saved your `.bash_history` file Assignments this week --------------------- Check the due date for each assignment and put a reminder in your agenda, calendar, and digital assistant. Just like in the Real World, not all due dates are on the same days or at the same times. - Review last week. Did you do everything assigned last week? - Read All The Words, Do, and then Submit via Blackboard: - [Assignment #03 HTML] -- working with files and pathnames: `PS1 find mkdir mv cp rm touch file` - remember to set your `PS1` prompt every time you log in to the CLS - OPTIONAL: Read [The VI (VIM) Text Editor] and optionally do the bonus [Assignment #04 HTML] for extra credit. - [Assignment #05 HTML] will be out soon -- using GLOB wildcard patterns, selecting files using GLOB, redirection and pipes. This assignment requires reading [Shell GLOB patterns] and [Shell I/O Redirection] and then doing [Worksheet #04 HTML] and [Worksheet #05 HTML]. - Really do **Read All The Words**. You don't get a second chance to get it right. Lab work this week ------------------ - Last week you used some simple commands on the [Course Linux Server]. This week, you use the commands you leaned in the two Worksheets: [Worksheet #02 HTML] and [Worksheet #03 HTML] - First, do the weekly readings on [File System and Pathnames], [Directories], and [Finding Files], also listed above. - Then complete and save the ODT or PDF versions of [Worksheet #02 HTML] and [Worksheet #03 HTML] - These worksheets require you to understand the weekly readings, above. - After completing the readings and the two worksheets, do [Assignment #03 HTML] -- working with files and pathnames: `PS1 find mkdir mv cp touch file` - This assignment requires you to understand the material in the readings and the two worksheets. - Remember to set your `PS1` prompt every time you log in to the CLS. - Study the pathname questions in first [Practice Tests and Answers] for your upcoming Midterm Test #1 in Week 5. - OPTIONAL: Read [The VI (VIM) Text Editor] and optionally do the bonus [Assignment #04 HTML]. ### Worksheets Worksheets are preparation for your assignments. You can't do the assignments without having done the worksheets first, and you can't do the worksheets without having first read the Course Notes: 1. Read. 2. Worksheets. 3. Assignment. Make notes from the worksheets on *how each command works*. What do the options used in the worksheets mean, for each command? (See the weekly [List of Commands].) Form a small study group to do the worksheets. Each person tries the example given, and you make sure you all get the same answers. Worksheets are not for hand-in; they are not worth marks; the assignments test your knowledge of the lectures and worksheets. > The worksheets are available in four formats: Open Office (ODT), PDF, HTML, > and Text. Only the Open Office format allows you "fill in the blanks" in > the worksheet. The PDF format looks good but doesn't allow you to type into > the blanks in the worksheet. The HTML format is crude but useful for quick > for viewing online. Do **NOT** open the Worksheet ODT files using any Microsoft products; they will mangle the format and mis-number the questions. Use the free Libre Office or Open Office programs to open these ODT documents. On campus, you can get a copy here: [Course Introduction: Install Libre Office]. Worksheets #02 and #03 require you to have read [File System and Pathnames]: - [Worksheet #02 ODT] -- Using standard Linux commands I - [Worksheet #02 PDF] - [Worksheet #02 HTML] - `PS1, cd, find, less, ls, man, mkdir, passwd, pwd, rmdir` - remember to set your `PS1` prompt every time you log in to the CLS - [Worksheet #03 ODT] -- Using standard Linux commands II - [Worksheet #03 PDF] - [Worksheet #03 HTML] - `cat, clear, cp, find, fgrep, history, less, man, mv, rm, sleep, touch` Worksheet #06 is for the optional VIM Bonus Assignment (extra credit): - [Worksheet #06 HTML] -- *Optional* Bonus VIM Text Editor Practice - This is an *optional* worksheet for a BONUS assignment using `vim` - Optional command-line VIM tutorial: the `vimtutor` program on the CLS. - Bonus (optional) [Assignment #04 HTML] -- the VIM text editor - Read [The VI (VIM) Text Editor] Worksheets prepare you for the upcoming assignments. Upcoming tests -------------- **Midterm Test #1 updated February 7 to exclude GLOB and I/O redirection** > Note that your upcoming Midterm Test #1 is worth more than three > assignments put together. Don't spend all your time doing an assignment at > the expense of studying the [Practice Tests and Answers] for the Midterm > Test. For full marks, read the [Test Instructions] (all the words) before your midterm tests. Midterm tests and the final exam are closed-book, no aids (just like your job interview). You will be given scrap paper to write on, and you may write on the question sheet. Put these test dates into your phone! Use the [Algonquin Registered Name Game] link to test your name before the test. I don't answer questions about the instructions during the test. 1. First Midterm test: 45 minutes; in your one-hour lecture class at 3pm on Wednesday in Week 5 (February 14). 2. Second Midterm test: 45 minutes; in your one-hour lecture class at 3pm on Wednesday in Week 9 (March 21). Midterm tests take place in your 50-minute lecture class, not in your lab period. Each midterm test is 45 minutes long and contains approximately 45 multiple-choice questions similar to those found in the Practice Tests posted under [Practice Tests and Answers]. You must write the test in the lecture class in which you are registered. ### Midterm Test #1 -- Wednesday February 14 at 3pm in C346 **Midterm Test #1 updated February 7 to exclude GLOB and I/O redirection** - Midterm #1 takes place in Week 5 on Wednesday February 14 at 3pm in C346 in your scheduled lecture class (not in your lab period). - The tests and exams are all closed-book, no aids, just like your job interview. You will be given some scrap paper to write on. - For full marks, you must read the [Test Instructions] before the test for important directions on how to enter your name, student number, answers, your lab (not lecture) section number, and the test version code on the question sheet and the mark-sense forms. - There may be more questions on the test than you can answer in the time allowed; answer the ones you know, first. ### Midterm #1 Topics **Midterm Test #1 updated February 7 to exclude GLOB and I/O redirection** - Weekly Notes 1, 2, and part of Week 3, including the readings. - Weekly web pages up to and including [Finding Files]. - Worksheets 2 and 3 - Assignments 2 and 3. - The 150 questions in the (new) first practice test in [Practice Tests and Answers]. - This 150-question test PDF was updated on February 7. Refresh your browser to get the latest version, labelled CST8207 Winter 2018, containing 150 questions. Do not use the old 2017 test. - Does *not* include anything about editors `vim` or `nano`. - Does *not* include anything about the [Course Introduction]. For full marks, you must read and understand the [Test Instructions] before you write the test. I don't answer questions about the instructions during the test. ### Practice Test #1 and Blackboard Quiz #1 **Practice Test #1 updated February 7 to exclude GLOB and I/O redirection** - A new set of 150 practice questions, answers, and quizzes for the first midterm test is posted in [Practice Tests and Answers]. - This 150-question test PDF was updated on February 7. Refresh your browser to get the latest version, labelled CST8207 Winter 2018, containing 150 questions. Do not use the old 2017 test. - The tests use the semicolon `;` to separate multiple commands on the same line, to save space and paper, e.g. three separate commands can be written on one line like this: `date ; echo "hi" ; ls` - **Do not use semicolons as an interactive human user!** Type each command on its own line with the \[Enter\] key. **Blackboard Quiz #1 updated February 7 to exclude GLOB and I/O redirection** - If you submitted any of the old quizzes on Blackboard, you will need to re-do them using the new Quiz content; the old attempts don't count. (You can send me email if you want your old quiz score to be used instead of the new quiz score.) - Blackboard has some quizzes taken randomly from the first practice test in [Practice Tests and Answers]. - The practice test itself is *not* part of your course grade, but these quizzes *are* part of your course grade. Do the quizzes for marks. - See the **Blackboard Quizzes** section located in [Practice Tests and Answers][1]. Assignment #2: Read All The Words --------------------------------- **Please Read All The Words.** Notes from the Classroom ======================== - **Take notes in class!** Keep a pad open on your desk. - This week, we learn how to use absolute and relative pathnames and then learn how to find files. ![[Pathnames Diagram on Whiteboard]][2] Simplifying pathnames --------------------- 1. Rewrite as a simplified absolute path: `/home/me/../you/../../etc/../home/me/../you/../me/../foo` 2. Rewrite as a simplified absolute path: `/usr/./bin/../lib/../../etc/../usr/./lib/../bin/./bar` 3. Rewrite as a simplified absolute path: `/../../var/./a/../../var/b/../../etc/./bar/../foo` Draw File System Diagrams ------------------------- Draw File System Diagrams on paper to understand where files are when you do the worksheets, the assignments, and the tests and exams! See the note in [Assignment #03 HTML] about drawing file system diagrams on paper. Do it! - [File System Drawing V1] - [File System Drawing V2] - [File System Drawing V3] - [Pathnames Diagram on Whiteboard][3] Did not change password -- CLS access revoked --------------------------------------------- Students will have their CLS accounts disabled if they have not changed their default password. (I do not want people breaking into my machine because of you.) See my online timetable for how to make an office appointment to see me if you want your account back. Fifteen minute rule: don't waste your time ------------------------------------------ See the [Course Introduction: fifteen minute rule] Commands Used ------------- > Keep a notebook with a [List of Commands] in it. > > - You need to write down yourself what each command *does*. > - Check the updated list of commands each week. > - I will check for this list in your lab periods. > - Bring your notes to class! Stop wasting time looking up commands. - You can tell whether a file system name is a directory or a file by using the `file` command or by looking for a letter `d` on the left in the output of `ls -l`. Ordinary files are indicated by a dash `-` character on the left. The first two output lines, below, are files. The last three output lines are directories. $ ls -l / -rw-r--r-- 1 root root 22528 Sep 4 11:01 aquota.group -rw-r--r-- 1 root root 73728 Sep 4 11:01 aquota.user drwxr-xr-x 2 root root 4096 Sep 3 15:57 bin drwxr-xr-x 147 root root 12288 Sep 17 14:30 etc drwxr-xr-x 1586 root root 36864 Sep 15 12:23 home - The `rmdir` command will only remove an empty directory. - You know this from doing Section 5 of [Worksheet #02 HTML]. - How do you remove a directory *and* everything inside it? You know the answer from doing the Worksheets, especially Section 5 of [Worksheet #03 HTML]. - The copy command `cp` will not copy directories unless you use the right option. You know this option from doing Section 3 of [Worksheet #03 HTML]. - The copy command `cp` and rename/move command `mv` can both accept a directory as the destination, and will put all the source files into that destination directory using the source file name(s): - If `bar` is a directory, then `cp foo bar` creates file `bar/foo` - If `bar` is a directory, then `mv foo bar` creates file `bar/foo` (and removes file `foo`) - If `bar` is *not* a directory, then `cp foo bar` creates file `bar` - If `bar` is *not* a directory, then `mv foo bar` creates file `bar` (and removes file `foo`) You learn this in [Worksheet #03 HTML]. - You can use pathnames anywhere you can use a file name, even for output redirection, e.g.: $ date >date.txt # date.txt in the current directory $ date >../date.txt # date.txt in the parent directory $ mkdir foo $ date >foo/date.txt # date.txt in the foo directory $ mkdir foo/bar $ date >foo/bar/date.txt # date.txt in the foo/bar directory You will need to know this for [Assignment #03 HTML]. - Are you making notes from the worksheets on how each command works? - What do the options used in the worksheets mean, for each command? - Don't copy and use options that you don't understand! - Using GLOB patterns as `-name` arguments to `find` - Remember to quote the `find` GLOB patterns to hide them from the shell! - GLOB patterns are *NOT* needed in Assignment 3. See Assignment 5. - Confused about redirection and the copy and move commands? These are wrong: - `cp cal.txt >calnew,txt # WRONG: error message from cp` - `mv cal.txt ../../ >cal.txt # WRONG: what is now in cal.txt ?` - No example in any of the notes or worksheets ever uses redirection and `cp` or `mv` together. These commands do not produce anything on standard output that is worth redirecting. Usually, the commands produce no output at all. Do *not* use redirection with `mv` or `cp`. - Confused about pathnames and blanks on the command line? This is wrong: - `cp file.txt ../../../ file.copy # WRONG` - Blanks separate arguments. This is covered in detail in [Command Arguments and Options]. You are giving the copy command three pathnames (three arguments), where you should have only two. The first argument is the source; the second is the destination: - `cp file.txt ../../../file.copy # RIGHT` Not Reading All The Words ------------------------- These do not produce the same output file: $ find . >foo ; mv foo dir/foo $ find . >dir/foo If the assignment asks you to redirect the output into a file in a specific directory, then only the second answer above is correct. The first answer, using `mv`, will have the wrong pathnames in it. Working hard, not working smart ------------------------------- - Stop using `cd` Windows People! Use pathnames! This is very inefficient: $ cd CST8207-18W $ cd Assignments $ cd assignment03 $ cd topdir $ cd other $ cat foo.txt This is all you need to do (using the TAB key to help you): $ cat CST8207-18W/Assignments/assignment03/topdir/other/foo.txt No employer wants to pay you to type six commands when one will do the job. Work smarter, not harder. - Stop using `cd` Windows People! Use pathnames! This is very inefficient: $ cd dir1 $ cd subdir $ touch file $ cd .. $ cd .. $ cd dir2 $ cd subdir $ touch file $ cd .. $ cd .. [...etc...] This is all you need to do (using the TAB key to help you): $ touch dir1/subdir/file dir2/subdir/file [...etc...] No employer wants to pay you to type ten commands when one will do the job. Work smarter, not harder. - If a pathname doesn't work, use `ls` to find out why. Usually it's because you're spelling it wrong. Use the TAB key and the shell will complete the pathname for you, with no spelling errors. Not doing worksheets: working hard, not smart --------------------------------------------- This student below (from a previous term) didn't do the worksheets and doesn't know about any options to the `mkdir` command. Instead of typing one command to make the directory tree, he has to use ten (!), and gets the name wrong, too. He also didn't hear me talk about not using `cd` all the time, so he marks himself as a Windows newbie. This won't look good in a job interview: $ mkdir 1ldIr $ cd 1ldIr $ mkdir one $ mkdir two $ mkdir one/0neOne $ cd one $ ls $ cd .. $ ls $ mkdir two/tw0two **Don't be this guy.** No employer wants to pay you to type ten commands when one will do the job. Work smarter, not harder. Do the worksheets! Attacks on the CLS ================== Commands to show the number of locked out IP addresses and attempts to use locked-out IP addresses in `/etc/hosts.evil`: $ wc /etc/hosts.evil $ fgrep -c 'refused connect' /var/log/auth.log $ fgrep 'refused connect' /var/log/auth.log | less Count the unique locked-out addresses, and then show the top twenty: $ fgrep 'refused connect' /var/log/auth.log | awk '{print $NF}' | sort -u | wc $ fgrep 'refused connect' /var/log/auth.log | awk '{print $NF}' | sort | uniq -c | sort -nr | head -n 20 Password failures: $ fgrep -c 'Failed' /var/log/auth.log $ fgrep 'Failed' /var/log/auth.log | less Userid failures: $ fgrep -c 'input_userauth' /var/log/auth.log $ fgrep 'input_userauth' /var/log/auth.log | less You can use the `whois` command on Linux to identify which country is responsible for an IP address, or use a [Web Whois Lookup]. > The Linux `whois` command is blocked from use at Algonquin College, except > if you are on the CLS. You can't use the `whois` command in your own Linux > machines while you are on campus. Use `whois` on the CLS or do a lookup via > the web interface instead. HTTP attacks on Course Linux Server ----------------------------------- In addition to the SSH attacks, machines on the Internet receive attacks using bogus HTTP requests. These unsuccessful attacks below attempt to use a chain of parent directories to fetch a file out of the file system: [Dec 13 13:50:57 2015] [client 54.201.251.33] Invalid URI in request GET /../../ratings.html HTTP/1.1 [Dec 13 13:51:01 2015] [client 54.201.251.33] Invalid URI in request GET /../../index.html HTTP/1.1 [Apr 26 17:09:59 2016] [client 67.79.37.250] Invalid URI in request GET /../../../../../../../mnt/mtd/yqcP HTTP/1.1 [May 02 12:21:45 2016] [client 222.235.67.147] Invalid URI in request GET /../../../../../../../mnt/mtd/IJSj HTTP/1.1 [Jun 27 17:54:40 2016] [client 222.198.128.207] Invalid URI in request GET /../../../../../../../mnt/mtd/3qS9 HTTP/1.1 [Oct 07 09:34:11 2016] [client 64.137.234.42] Invalid URI in request GET /../../../../../../../mnt/mtd/ahMk HTTP/1.1 [Oct 15 06:49:57 2016] [client 177.142.99.103] Invalid URI in request GET /../../../../../../../mnt/mtd/qt HTTP/1.0 [Oct 19 20:49:33 2016] [client 189.100.188.31] Invalid URI in request GET /../../../../../../../mnt/mtd/qt HTTP/1.0 [Oct 23 00:43:39 2016] [client 191.189.55.9] Invalid URI in request GET /../../../../../../../mnt/mtd/qt HTTP/1.0 [Nov 02 21:35:09 2016] [client 201.37.223.72] Invalid URI in request GET /../../../../../../../mnt/mtd/qt HTTP/1.0 [Jan 07 20:51:33 2017] [client 201.17.154.181] Invalid URI in request GET /../../../../../../../mnt/mtd/qt HTTP/1.0 [May 10 21:22:29 2017] [client 187.67.167.31] Invalid URI in request GET /../../../../../../../mnt/mtd/qt HTTP/1.0 [28/Sep/2017:23:57:19 -0400] "GET /?i=../../../../../proc/self/environ HTTP/1.1" [10/Feb/2018:04:44:53 -0500] "GET /index.php?option=com_jimtawl&Itemid=12&task=../../../../../../../../../../../../../../../configuration.php HTTP/1.1" [01/Apr/2018:23:35:23 -0400] "GET /wp-content/plugins/wp-ecommerce-shop-styling/includes/download.php?filename=../../../../../../../../../etc/passwd HTTP/1.1" [02/Apr/2018:06:02:56 -0400] "GET /cst8207/17w/notes/data//wp-content/plugins/ibs-mappro/lib/download.php?file=../../../../wp-config.php HTTP/1.1" [02/Apr/2018:06:02:57 -0400] "GET /cst8207/17w/notes/data//wp-content/plugins/s3bubble-amazon-s3-html-5-video-with-adverts/assets/plugins/ultimate/content/downloader.php?path=../../../../../../../wp-config.php HTTP/1.1" The Apache web server blocked all the above attacks. Locked out of the CLS ===================== When you are locked out, [follow the directions] in the notes to get your IP address re-enabled. ![Take Notes in Class] ![Ian! and the Pathnames Diagram] -- | Ian! D. Allen, BA, MMath - idallen@idallen.ca - Ottawa, Ontario, Canada | Home Page: http://idallen.com/ Contact Improv: http://contactimprov.ca/ | College professor (Free/Libre GNU+Linux) at: http://teaching.idallen.com/ | Defend digital freedom: http://eff.org/ and have fun: http://fools.ca/ [Plain Text] - plain text version of this page in [Pandoc Markdown] format [www.idallen.com]: http://www.idallen.com/ [Course Home Page]: .. [Course Outline]: course_outline.pdf [All Weeks]: indexcgi.cgi [Plain Text]: week03notes.txt [Week 03 Notes HTML]: week03notes.html [File System and Pathnames]: 160_pathnames.html [Directories]: 170_home_and_HOME.html [Finding files]: 180_finding_files.html [Shell GLOB patterns]: 190_glob_patterns.html [List of Commands]: 900_unix_command_list.html [Video Tutorials on Lynda.com]: 910_lynda_index.html [Linux and Sysadmin News in the World]: 950_linux_world.html [lynda.com]: http://algonquincollege.com/onlineresources/mobileStudent/lynda.htm [Unix for Mac OS X Users]: http://www.lynda.com/Mac-OS-X-10-6-tutorials/Unix-for-Mac-OS-X-Users/78546-2.html [Assignment #03 HTML]: assignment03.html [The VI (VIM) Text Editor]: 300_vi_text_editor.html [Assignment #04 HTML]: assignment04.html [Assignment #05 HTML]: assignment05.html [Shell I/O Redirection]: 200_redirection.html [Worksheet #04 HTML]: worksheet04.html [Worksheet #05 HTML]: worksheet05.html [Course Linux Server]: 070_course_linux_server.html [Worksheet #02 HTML]: worksheet02.html [Worksheet #03 HTML]: worksheet03.html [Practice Tests and Answers]: PRACTICE_TEST_README.html [Course Introduction: Install Libre Office]: 050_course_introduction.html#install-libreoffice-or-openoffice-into-windows [Worksheet #02 ODT]: worksheet02.odt [Worksheet #02 PDF]: worksheet02.pdf [Worksheet #03 ODT]: worksheet03.odt [Worksheet #03 PDF]: worksheet03.pdf [Worksheet #06 HTML]: worksheet06.html [Test Instructions]: 000_test_instructions.html [Algonquin Registered Name Game]: http://cst8207.idallen.ca/~idallen/namegame.cgi [Course Introduction]: 050_course_introduction.html [1]: PRACTICE_TEST_README.html#blackboard-quizzes [Pathnames Diagram on Whiteboard]: data/cameron_whiteboard_big.jpg [2]: data/cameron_whiteboard_small.jpg "Pathnames Diagram on Whiteboard" [File System Drawing V1]: data/violett_file_system_drawing1.jpg [File System Drawing V2]: data/violett_file_system_drawing2.jpg [File System Drawing V3]: data/filesystemdiagram.png [3]: data/violett_pathnames.jpg [Course Introduction: fifteen minute rule]: 050_course_introduction.html#fifteen-minute-rule-dont-waste-your-time [Command Arguments and Options]: 150_arguments_and_options.html [Web Whois Lookup]: http://whois.domaintools.com/ [follow the directions]: 070_course_linux_server.html#getting-locked-out-of-the-server [Take Notes in Class]: data/remember.jpg "Take Notes in Class" [Ian! and the Pathnames Diagram]: data/gordon_pathnames.jpg "Ian! and the Pathnames Diagram" [Pandoc Markdown]: http://johnmacfarlane.net/pandoc/