% CST8207 Week 10 Notes -- Midterm 2 Analysis, Server Attacks, Shell Scripts % Ian! D. Allen -- -- [www.idallen.com] % Winter 2017 - January to April 2017 - Updated 2017-12-11 02:30 EST - [Course Home Page] - [Course Outline] - [All Weeks] - [Plain Text] Readings, Assignments, Labs, Tests, and ToDo ============================================ Read (at least) these things (All The Words) -------------------------------------------- 1. [Week 10 Notes HTML] -- this file -- **Read All The Words** 2. [Shell Scripts -- lists of commands, executable scripts, script header, command arguments and positional parameters] 3. [Shell command substituion -- interpolate stdout into a command line using $(...) or \`...\`] 4. [Shell Control Structures -- if, then, else, test, `[...]`][1] 5. [List of Commands You Should Know] 6. [Linux and Sysadmin News in the World] 7. [Video Tutorials on Lynda.com] -- tagged by week number Assignments and lab work this week ---------------------------------- > **Reminder:** There are now two quizzes that you need to complete on > Blackboard as part of your term Quiz mark. A third quiz will be posted > before the Final Exam. The Quizzes are *not* optional; see the [Course > Outline]. Check the due date for each assignment and put a reminder in your agenda, calendar, and digital assistant. Just like in the Real World, not all due dates are on the same days or at the same times. - Review last week. Did you do everything assigned last week? - Do [Worksheet #08 HTML] and then do [Assignment #08 HTML] -- setting permissions, mode, `umask` - This assignment requires [Permissions] and [Umask] - Do Bonus (optional) [Assignment #09 HTML] about Midterm #2 - There is a checking program available to check your file format for this bonus assignment, but only people who Read All These Words will know about it. Wrong format means no marks. *Do not redirect or submit the output of this checking program!* - Read [The VI (VIM) Text Editor] and optionally do the bonus [Assignment #04 HTML]. - **Reminder:** You must actually use the VIM editor during the term to get full marks for this bonus assignment. Read All The Words. - Coming soon: [Assignment #10 HTML] -- tar, processes, syslog, crontab, at, mail, shell script Worksheets ---------- Worksheets are preparation for your assignments. You can't do the assignments without having done the worksheets first, and you can't do the worksheets without having first read the Course Notes: 1. Read the web notes. (Please: **Read All The Words**) 2. Do the relevant Worksheet(s). 3. Do the relevant Assignment(s). Form a small study group to do the worksheets. Each person tries the example given, and you make sure you all get the same answers. Worksheets are not for hand-in; they are not worth marks; the assignments test your knowledge of the lectures and worksheets. > The worksheets are available in four formats: Open Office (ODT), PDF, HTML, > and Text. Only the Open Office format allows you "fill in the blanks" in > the worksheet. The PDF format looks good but doesn't allow you to type into > the blanks in the worksheet. The HTML format is crude but useful for quick > for viewing online. Do **NOT** open the Worksheet ODT files using any Microsoft products; they will mangle the format and mis-number the questions. Use the free Libre Office or Open Office programs to open these ODT documents. On campus, you can [download Libre Office here]. - [Worksheet #06 HTML] -- *Optional* Bonus VIM Text Editor Practice - This is an *optional* worksheet for a BONUS assignment using `vim` - Optional command-line VIM tutorial: the `vimtutor` program on the CLS. - Bonus (optional) [Assignment #04 HTML] -- the VIM text editor - Read [The VI (VIM) Text Editor] - [Worksheet #08 ODT] -- Linux file system permissions (modes) - This Worksheet is a prerequisite for [Assignment #08 HTML] - [Worksheet #08 PDF] -- PDF version - [Worksheet #08 HTML] -- HTML version - `chmod, ls -lid, umask` Worksheets prepare you for the upcoming assignments. Upcoming tests and exams ------------------------ This course has two midterm tests and one final exam. - Put these dates below into your phone! - Read the [Test Instructions] (all the words) before your tests and exam. - Use the **Name Game** link (in the [Test Instructions]) to test the spelling of your name before the test. - I don't answer questions about the instructions during the test. Ask me in a lab period before the test. ### The Final Exam -- 8am April 28 -- 40% The Final Exam is three hours long and contains approximately 180 multiple-choice questions similar to those found in the three preceding [Practice Tests and Answers]. Do *all three* practice tests before the Final Exam! - Time: 08h00-11h00 (8am to 11am) Friday April 28 (Week 15) - Location: CA-105 A,B,C (across the Woodroffe pedestrian overpass) - This exam has **assigned seating** -- see your email for your assigned seat. - 180 minutes (three hours) for 180 questions - Do *all three* practice tests before the Final Exam! - Three Hours! Take a bathroom break before you start the exam! All three practice tests will be posted under [Practice Tests and Answers]. The Final Exam is comprehensive of the whole course; you need to do *all three* practice tests for the Final Exam. Midterm Test #2 Analysis ======================== Here are the final statistics for the second midterm test: - 88 students are registered in the course. - Of the 88, 6 did not write the test, leaving 82 who did. - Of the 82, 14 did not enter a valid test version code. - Of the 82, 11 did not enter their own name correctly. - Of the 82, 2 did not enter their own student number correctly. - Of the 82, 2 did not put their name on their question sheet. - Of the 82, 1 used a pen instead of pencil and got zero marks. - Of the 82, 31 got question #45 wrong even though the answer was given in the **Test Instructions** printed at the start of the test. - The 82 class scores: 100 98 97.7 97.7 96.8 95.8 95.5 93.2 90.9 89.1 88.6 88.6 88.6 86.9 86.9 86.4 86.4 84.1 84.1 82.4 81 79.5 77.3 77.3 75 75 72.7 71.6 70.5 70.5 70.5 69 67.5 66.2 62.4 62.1 61.4 60.1 59.1 59.1 59.1 58.5 58.5 56.8 55.7 54 53.5 52.4 52.3 52.3 51.2 51.2 51.2 50 50 49.7 48.1 47.7 45.5 44.5 43.7 38.5 38.1 37.9 36.4 34.4 34.2 33.4 31.8 29.5 29.5 26.7 26.1 23.1 22.7 20 18 17.8 14.4 14.3 13.4 8 82: Pass 56 (68.3%) Fail 26 (31.7%) 22 A (26.8%) 5 A- 8 A 9 A+ 9 B (11.0%) 4 B- 3 B 2 B+ 7 C ( 8.5%) 4 C- 1 C 2 C+ 18 D (22.0%) 9 D- 3 D 6 D+ 26 F (31.7%) 21 F- 3 F 2 F+ 90% - 100% 9 ********* 80% - 90% 13 ************* 70% - 80% 9 ********* 60% - 70% 7 ******* 50% - 60% 18 ****************** 40% - 50% 5 ***** 30% - 40% 10 ********** 20% - 30% 5 ***** 10% - 20% 5 ***** 0% - 10% 1 * - You can see the class errors in [Midterm Test #2 PDF]. > I spent an hour correcting your errors on your mark-sense forms. Before > your next test, some of you need to re-read the [Test Instructions]. > Penalties go up again for making these errors on the final exam. Notes from the Classroom ======================== - **Take notes in class!** Keep a pad open on your desk. - Don't forget to finish your five Blackboard quizzes for each of Midterm 1 and Midterm 2. There will be a third quiz to prepare for the Final Exam. - Regarding world-writable files in your account: - Outside of the one directory in [Assignment #08][Assignment #08 HTML], you must not have any world-writable (writable by "other") files or directories in your account. - The checking program will deduct marks if it finds world-writable files. - You can find world-writable files by re-reading the "Examples of uses" in [Finding Files] from the Week 3 notes. Commands Used ------------- - Keep a notebook with a [List of Commands][List of Commands You Should Know] in it. - You need to write down yourself what each command *does*. - Check the updated list of commands each week. - Bring your notes to class! Stop wasting time looking up commands. - I will check for this list in your lab periods. - Are you making notes from the worksheets on how each command works? - What do the options used in the worksheets mean, for each command? - Don't copy and use options that you don't understand! Case Study: enabling blocked IP addresses ----------------------------------------- This case study needs [Command Substitution][Shell command substituion -- interpolate stdout into a command line using $(...) or \`...\`] and [Control Statements][1] and super-user (`root`) permission. The [Course Linux Server] runs the [Denyhosts] intrusion detection package (`man denyhosts`). Blocked IP addresses are automatically added to the file `/etc/hosts.evil` that is included by `/etc/hosts.allow` to block access to the machine: $ wc -l /etc/hosts.evil 7908 /etc/hosts.evil Visual inspection of `/etc/hosts.evil` suggests that some of the recent blocked IP addresses are people on the local Rogers cable network: $ whois 99.224.86.21 [...] NetRange: 99.224.86.0 - 99.224.87.255 CIDR: 99.224.86.0/23 Parent: ROGERS-COM-HSD (NET-99-224-0-0-1) The sysadmin wants to find and unblock all these IP addresses. 1. Find some obvious Rogers IP addresses in the file: $ fgrep ' 99.2' /etc/hosts.evil sshd: 99.224.86.21 sshd: 99.245.238.68 sshd: 99.246.18.16 sshd: 99.254.149.12 sshd: 99.246.3.39 sshd: 99.239.40.207 2. Isolate the just IP addresses on each line: $ fgrep ' 99.2' /etc/hosts.evil | awk '{print $NF}' 99.224.86.21 99.245.238.68 99.246.18.16 99.254.149.12 99.246.3.39 99.239.40.207 3. Write a debugging FOR loop that uses the IP addresses via command substitution and echoes them to the screen: $ for ip in $( fgrep ' 99.2' /etc/hosts.evil | awk '{print $NF}' ) ; do echo "IP is $ip" ; done IP is 99.224.86.21 IP is 99.245.238.68 IP is 99.246.18.16 IP is 99.254.149.12 IP is 99.246.3.39 IP is 99.239.40.207 4. Replace the debugging `echo` with the real unblocking command (requires privilege to work): $ for ip in $( fgrep ' 99.2' /etc/hosts.evil | awk '{print $NF}' ) ; do sudo /usr/share/denyhosts/DenyHosts/dh_reenable "$ip" ; done Done! Please restart denyhosts Done! Please restart denyhosts Done! Please restart denyhosts Done! Please restart denyhosts Done! Please restart denyhosts Done! Please restart denyhosts Done! Please restart denyhosts 5. Verify that it worked: $ fgrep ' 99.2' /etc/hosts.evil $ # no output - all addresses were removed 6. Restart the Denyhosts package (requires privilege to work): $ sudo service denyhosts restart * Stopping DenyHosts denyhosts ...done. * Starting DenyHosts denyhosts ...done. Done. Finding world-writable files in your account -------------------------------------------- The assignment **Checking Programs** may issue this message: Number of world-writable pathnames in abcd0001 account: 1 ERROR: Sysadmin do not create files that anyone can overwrite. ERROR: See "Examples of uses of find" to find these files. ERROR(-1): Fix the permissions on these files Don't create files or directories that anyone ("other") can write, except the few required ones in the one [Assignment #08 HTML] `head` directory! You must look at all the files in your account to try to find these files or directories that you have created with "other" write permissions. You can find the files the hard way, using `cd` and `ls`, or you can do it the easy way using a recursive command: - To learn how to `find` these world-writable files in your account, first use a command to search for the files that contain the text `world-writable` in all the course notes. The course notes have an example showing what command to use to find world-writable files. - To search the course notes for a text string, re-read section 4.5 of [Assignment #05 HTML], especially item #8 about searching for the text string `Filezilla` in all the course notes. - You will be able to identify which course notes web page you should read to find the example showing how to find world-writable files. - Fix the permissions on these files. Remove write permission for "other". Marking Scheme for Bonus Assignment #09 ======================================= Bonus (optional) [Assignment #09 HTML] may be done now that you have your marks back for Midterm #2 by email. (Your marks were sent to you late Friday afternoon, March 17.) You can use this bonus assignment to make up for lost marks on your second midterm test. See the assignment for the exact Marking Scheme. There is a checking program available to check your file format for this bonus assignment, but only people who Read All These Words will know about it. Wrong format means no marks. Tutoring ======== Many students find that hiring a personal tutor helps them get through the first term. Financial assistance is available. See the **Tutoring** heading in the [Course Introduction]. Fifteen minute rule: don't waste your time ========================================== See the [Course Introduction: fifteen minute rule] Attacks on the HTTP port of the Course Linux Sever ================================================== People are using `../..` paths to try to trick the Apache Web server into revealing files: [CLS Apache Web Logs] Look at the IP addresses of the attacking machines. Do you notice something interesting about the attacks on January 31 and February 2? Attacks on the SSH port on three of my servers ============================================== Up to Sun Mar 19 23:11 EDT 2017. I did some **whois** lookups on a few of the IP addresses and added the network owners as comments (all from China). Course Linux Server ------------------- # Since: Jan 1 07:51:01 $ fgrep 'refused connect' /var/log/auth.log \ | awk '{print $NF}' | sort | uniq -c | sort -nr | head 33409 (116.31.116.25) # CHINANET Guangdong province network 10498 (153.99.182.35) # China Unicom Jiangsu province network 10041 (218.65.30.46) # CHINANET jiangxi province network 9955 (182.100.67.76) 5071 (122.194.229.16) 3990 (218.65.30.251) 3232 (218.65.30.80) 3104 (61.177.172.60) 2387 (153.99.182.11) 2148 (116.31.116.23) Home machine (ISPs: TekSavvy and Distributel) --------------------------------------------- # Since Jan 1 07:36:15 $ zfgrep 'refused connect' /var/log/auth.log{,.{?,10,11}.gz} \ | awk '{print $NF}' | sort | uniq -c | sort -nr | head 66204 (116.31.116.53) # CHINANET Guangdong province network 24451 (153.99.182.10) # China Unicom Jiangsu province network 22199 (153.99.182.26) # China Unicom Jiangsu province network 21173 (123.183.209.139) 20876 (116.31.116.36) 15789 (218.65.30.46) 13893 (58.218.200.37) 13621 (116.31.116.24) 12596 (153.99.182.39) 11666 (153.99.182.13) Algonquin T313 Office machine (same network as CLS) --------------------------------------------------- # Since: Feb 13 07:37:01 $ zfgrep 'refused connect' /var/log/auth.log* \ | awk '{print $NF}' | sort | uniq -c | sort -nr | head 28235 (116.31.116.25) # CHINANET Guangdong province network 12751 (61.177.172.60) # CHINANET jiangsu province network 10859 (153.99.182.35) # China Unicom Jiangsu province network 5112 (122.194.229.16) 4347 (153.99.182.36) 2841 (218.65.30.251) 1672 (218.65.30.210) 1508 (219.153.15.82) 1498 (209.159.145.140) 1339 (116.31.116.53) *You're not paranoid if they really **are** out to get you!* Career with Communications Security Establishment ================================================= Do you think you need Linux skills for this job? ![[sudo stop all hackers]][2] ![Take Notes in Class] -- | Ian! D. Allen, BA, MMath - idallen@idallen.ca - Ottawa, Ontario, Canada | Home Page: http://idallen.com/ Contact Improv: http://contactimprov.ca/ | College professor (Free/Libre GNU+Linux) at: http://teaching.idallen.com/ | Defend digital freedom: http://eff.org/ and have fun: http://fools.ca/ [Plain Text] - plain text version of this page in [Pandoc Markdown] format [www.idallen.com]: http://www.idallen.com/ [Course Home Page]: .. [Course Outline]: course_outline.pdf [All Weeks]: indexcgi.cgi [Plain Text]: week10notes.txt [Week 10 Notes HTML]: week10notes.html [Shell Scripts -- lists of commands, executable scripts, script header, command arguments and positional parameters]: 700_shell_scripts.html [Shell command substituion -- interpolate stdout into a command line using $(...) or \`...\`]: 710_command_substitution.html [1]: 730_control_statements.html [List of Commands You Should Know]: 900_unix_command_list.html [Linux and Sysadmin News in the World]: 950_linux_world.html [Video Tutorials on Lynda.com]: 910_lynda_index.html [Worksheet #08 HTML]: worksheet08.html [Assignment #08 HTML]: assignment08.html [Permissions]: 500_permissions.html [Umask]: 510_umask.html [Assignment #09 HTML]: assignment09.html [The VI (VIM) Text Editor]: 300_vi_text_editor.html [Assignment #04 HTML]: assignment04.html [Assignment #10 HTML]: assignment10.html [download Libre Office here]: 050_course_introduction.html#install-libreoffice-or-openoffice-into-windows [Worksheet #06 HTML]: worksheet06.html [Worksheet #08 ODT]: worksheet08.odt [Worksheet #08 PDF]: worksheet08.pdf [Test Instructions]: 000_test_instructions.html [Practice Tests and Answers]: PRACTICE_TEST_README.html [Midterm Test #2 PDF]: midterm2_17w.pdf [Finding Files]: 180_finding_files.html [Course Linux Server]: 070_course_linux_server.html [Denyhosts]: http://denyhosts.sourceforge.net/ [Assignment #05 HTML]: assignment05.html [Course Introduction]: 050_course_introduction.html [Course Introduction: fifteen minute rule]: 050_course_introduction.html#fifteen-minute-rule-dont-waste-your-time [CLS Apache Web Logs]: data/web_server_attacks_17w.txt [sudo stop all hackers]: http://cse-cst.gc.ca/careers [2]: data/cse-cst-careers.jpg "sudo stop all hackers" [Take Notes in Class]: data/remember.jpg "Take Notes in Class" [Pandoc Markdown]: http://johnmacfarlane.net/pandoc/