% CST8207 Assignment 03 - simple commands, copy, GLOB and redirection % Ian! D. Allen - - [www.idallen.com] % Fall 2013 - September to December 2013 - Updated 2017-05-31 12:15 EDT - [Course Home Page] - [Course Outline] - [All Weeks] - [Plain Text] Due Date and Deliverables ========================= > **Do not print this assignment on paper!** > > - On paper, you will miss updates, corrections, and hints added to the > online version. > - On paper, you cannot follow any of the [hyperlink URLs] that lead you > to hints and course notes relevant to answering a question. > - On paper, scrolling text boxes will be cut off and not print properly. - **Due Date**: `15h00 (3pm) Tuesday October 15, 2013 (start of Week 7)` - You have three weeks to do this assignment, but your next assignment will be available next week and will overlap this assignment by a week. Don't delay! - Late assignments or wrong file names may not be marked. Be accurate. - **Available online**: - Version 1 -- noon Tuesday September 24, 2013 (preliminary) - Version 2 -- 01:35am Friday September 27, 2013 (finished) - Version 3 -- 13:10pm Thursday October 3, 2013 (new due date) - Version 4 -- 02:30am Sunday October 13, 2013 (another new due date) - **Prerequisites**: - All [Class Notes] since the beginning of term. - All your previous [Assignments]. - [Worksheets 2, 3, 4, and 5]. - An ability to **READ ALL THE WORDS** to work effectively - **Deliverables**: 1. One text file uploaded to Blackboard according to the steps in the [Checking Program] section below. 2. Directory structure created and left for marking on the [Course Linux Server] (**CLS**).\ **Do not delete any assignment work from the CLS until after the term is over!** **WARNING:** Some inattentive students upload Assignment #3 into the Assignment #2 upload area. Don't make that mistake! Be exact. Purpose of this Assignment ========================== This assignment is based on your weekly [Class Notes][All Weeks]. 1. Complete [Lab Worksheet #04 ODT]. 2. Complete [Lab Worksheet #05 ODT]. 3. Execute some simple Linux commands and generate some Linux output. 4. Find some files. Move some files. Copy some files. 5. Practice with GLOB and I/O redirection. 6. Use `fgrep` and `grep` patterns to search the course notes for keywords. 7. Copy files based on a complex GLOB pattern. 8. Transfer and submit a file to Blackboard. Remember to **READ ALL THE WORDS** to work effectively and not waste time. Introduction and Overview ========================= This is an overview of how you are expected to complete this assignment. Read all the words before you start working. 1. Complete the **Tasks** listed below. 2. Verify your own work before running the [Checking Program]. 3. Run the [Checking Program] to help you find errors. 4. Submit the output of the [Checking Program] to Blackboard before the due date. 5. **READ ALL THE WORDS** to work effectively and not waste time. You will create file system structure in your CLS home directory containing various directories and files. You can use the [Checking Program] to check your work as you do the tasks. You can check your work with the checking program as often as you like before you submit your final mark. **Some task sections below require you to finish the whole section before running the checking program; you may not always be able to run the checking program successfully after every single task step.** When you are finished the tasks, leave these files, directories, and links in place on the CLS as part of your deliverables. **Do not delete any assignment work from the CLS until after the term is over!** Assignments may be re-marked at any time on the CLS; you must have your term work available on the CLS right until term end. > Since I also do manual marking of student assignments, your final mark may > not be the same as the mark submitted using the current version of the > [Checking Program]. I do not guarantee that any version of the [Checking > Program] will find all the errors in your work. Complete your assignments > according to the specifications, not according to the incomplete set of the > mistakes detected by the [Checking Program]. The Source Directory -------------------- All references to the "Source Directory" below are to the CLS directory `~idallen/cst8207/13f/assignment03/` and that name starts with a *tilde* character `~` followed by a userid with no intervening slash. The leading tilde indicates to the shell that the pathname starts with the HOME directory of the account `idallen` (seven letters). Tasks ===== - Do the following tasks in order, from top to bottom. - These tasks must be done in your account on the [Course Linux Server]. - **READ ALL THE WORDS!** and do not skip steps. - Your instructor will mark on the due date the work you do in your account on the CLS. Leave all your work on the CLS and do not modify it. - **Do not delete any assignment work from the CLS until after the course is over.** Worksheet and Prompt -------------------- These worksheets prepare you to do the numbered tasks listed below by executing commands via [Remote Login] to the [Course Linux Server]. You can download Libre Office (or Open Office) for Windows to edit the Worksheet `*.odt` files and save your answers. (Or you can print the PDF and write your answers on paper.) Do *not* use MSWord. Record and save all your worksheet answers for study and quizzes! See [Assignment #02] for how best to fill in the worksheets. 1. Do the worksheets [Lab Worksheet #04 ODT] and [Lab Worksheet #05 ODT] before this assignment. Record and save all your worksheet answers for study and quizzes! 2. Set your prompt every time you log in to the CLS, so that the prompt changes to tell you you the **basename** of your current working directory, as you did the worksheets and the previous assignment. Hint: You can use your saved shell history to set the prompt very quickly at login time, as demonstrated in lectures. Working with directories and files ---------------------------------- 3. Create the following directory structure in your CLS HOME directory and record (for study purposes) the series of Unix commands you used to create it. Spelling and capitalization must be exactly as shown: CST8207-13F `-- Assignments `-- assignment03 `-- top1 |-- dir1 | `-- dir1-1 `-- dir2 `-- dir2-1 Note: You can create the entire directory tree above using one single command with one option and two pathname arguments. System administrators like to work efficiently -- they learn how to do things quickly. 4. Make a directory named `warez` under your `assignment03` directory. You will put files here later. 5. Make a directory named `osfiles` under your `assignment03` directory. You will put files here later. 6. Make the `top1` directory your current directory and then use a command to recursively generate a list of all pathnames under your *current directory*. The recursive output of all pathnames under your *current directory* will be exactly five lines long and will contain this **exact** line in the output somewhere (along with another four lines): `./dir2/dir2-1` Note that there is no punctuation at the end of any pathname. 7. When the recursive output above on your screen looks correct, redirect the five lines of output of the command into the file named `top1find.txt` in the `assignment03` directory (not in the current directory). The word count of this file must be: `5 5 44`. If the word count is wrong, read all the words in the previous step. 8. RTFM and find the single-letter option to the copy program that preserves mode, ownership, and timestamps when you copy. 9. Under `assignment03` create another directory `top2` that contains a copy of everything you created under `top1`, with the timestamps preserved. Hint: The fastest way to do this is with a single recursive copy command, as you did in [Lab Worksheet #03 HTML], but add the option to preserve timestamps. 10. Create an empty file named `MT` in each of the five directories from `top2` down. (You can do this with one command name and five relative pathnames.) 11. Make the `top2` directory your current directory and then use a command to recursively generate a list of all pathnames under your current directory. The recursive output of all pathnames under your current directory will be exactly ten lines long and will contain this **exact** line in the output somewhere (along with another nine lines): `./dir2/dir2-1/MT` Note that there is no punctuation at the end of any pathname. 12. When the recursive output above on your screen looks correct, redirect the ten lines of output of the command into the file named `top2find.txt` in the `assignment03` directory. The word count of this file must be: `10 10 103`. If the word count is wrong, read all the words in the previous step. 13. Under `assignment03` create another directory `top3` that contains a copy of everything that you created under `top2`, with timestamps preserved. Hint: See the previous Hint! 14. Under `top3` rename each of the `MT` files to have the number of the parent directory that contains it, e.g. rename `./MT` to be `./MT3` and `dir2-1/MT` to be `dir2-1/MT2-1`, etc. (You don't know enough scripting yet to do this with a script; do each file manually.) 15. Make the `top3` directory your current directory and then use a command to recursively generate a list of all pathnames under your current directory. The recursive output of all pathnames under your current directory will be exactly ten lines long and will contain this **exact** line in the output somewhere (along with another nine lines): `./dir2/dir2-1/MT2-1` 16. When the recursive output above on your screen looks correct, redirect the ten lines of output of the command into the file named `top3find.txt` in the `assignment03` directory. The word count of this file must be: `10 10 112`. If the word count is wrong, read all the words in the previous steps. Run the [Checking Program] to verify your work so far. Finding multiple files in a maze using a GLOB pattern ----------------------------------------------------- 17. Under the [Source Directory] there is a name `maze` that is a *symbolic link* that leads to the same directory maze used in a previous assignment: [Assignment #2][Assignment #02]. Use the absolute pathname of the `maze` name in the [Source Directory] as an argument to `ls` along with an option that shows the long information about the pathname. You probably have the right option if the first word of the output is `lrwxrwxrwx`, indicating that `maze` is a symbolic link. Save the output (one line) into file `mazeinfo.txt` under your `assignment03` directory. 18. Again, you must find files in this maze, but the symbolic link requires some special handling because the command that finds files does not follow symbolic link arguments on the command line without using some options: a. You must choose one of these approaches to reach the maze: i. Use the maze pathname from the *previous* assignment, not the pathname in the [Source Directory] that is the symbolic link. **OR** ii. Use an option to the finding command that makes it follow symbolic links *while processing the command line arguments*. Hint: RTFM and do **not** use the `-L` option. **OR** iii. Make the `maze` your current directory and then recursively search the current directory. (A current directory can never be a symbolic link -- it must be a real directory.) b. Important: Look for the string `quote` in the course notes web page on [Searching for and finding files by name, size, use, modify time, etc.] Read the paragraphs containing this word and remember the importance of quoting. You will need this information for this next question: c. In this maze, use a single command (not a pipeline) to recursively find all pathnames with a **basename** that contains your eight-character userid *anywhere* in the name. For example, if your userid were `abcd0001` then you might output pathnames containing basenames such as `abcd0001`, `abcd0001YYY`, `XXXabcd0001`, and `XXXabcd0001YYY` where `XXX` and `YYY` can be anything (zero or more characters). You must use a single command that finds files by using a pattern **basename** to find these pathnames. Do not try to use `cd` and `ls` to find the names in the maze; the maze is very big. (You have previously used this command many times without a pattern for a **basename**. You should find approximately 26 pathnames.) d. Modify the same single command you used to find the names above and add the option that makes the command show the full information about the names (including permissions, owner, size, date, etc.) instead of just the pathname. (You know this option from your answers in Worksheets 2 and 3 and from reading the *More features* section of the [Searching for and finding files by name][Searching for and finding files by name, size, use, modify time, etc.] notes.) You will know you have the right option if the output of the command shows approximately 26 lines and 291 words. e. Pipe the pathname output of the above command (26 lines of output) into a sorting program and put the sorted output into file `mazefound.txt` under your `assignment03` directory. The sorted file will still contain approximately 26 lines and 291 words. Run the [Checking Program] to verify your work so far. Using shell GLOB patterns to select names ----------------------------------------- 19. In your HOME directory, create two symbolic links to the old and new course notes for CST8207 using the method described in [Copies of the CST8207 Course Notes]. (The old notes must be term `13w` and the new notes must be term `13f` in the pathnames you use.) In the notes, see the example use of `grep` and `fgrep` with GLOB patterns to match `*.txt` files in these old and new notes directories. These text-searching commands `grep` and `fgrep` are case-sensitive by default -- searching for `abc` won't find `ABC` unless you use an option to *ignore case distinctions*. (What option? RTFM) 20. In your HOME directory, use the `echo` command with a shell GLOB pattern to match all pathnames under `oldnotes/` that end in `.txt` and display all the names on your screen. The shell will find 88 pathnames ending in `.txt`, and the `echo` command will display those 88 names on your screen. When the `echo` output on your screen is correct (88 names), redirect the output into file `textfound.txt` under your `assignment03` directory. The file must contain 88 names on one line. Hints: Use a command to count the words to be sure you have 88. The example in the notes already uses the given GLOB pattern to generate pathnames for the `fgrep` command. Use `echo` not `fgrep` to display the pathnames on your screen and then redirect the names into the output file. 21. In your HOME directory, use the `echo` command with a shell GLOB pattern to match all pathnames under `oldnotes/` that contain the word `symbolic` anywhere in the file name and display the names on your screen. The shell will find two pathnames, one ending in `.html` and the other in `.txt`, and the `echo` command will display those two names on your screen. When the `echo` output on your screen is correct (two names), redirect the output into file `symfound.txt` under your `assignment03` directory. The file must contain two names on one line. 22. In your HOME directory, use the `echo` command with a shell GLOB pattern to match pathnames under `oldnotes/` that contain the word `vi` anywhere in the file name and end in the extension `.pdf` at the end. The shell will find two pathnames, each ending in `.pdf` at the end, and the `echo` command will display those two names on your screen. When the `echo` output on your screen is correct (two names), change the command name from `echo` to `ls` and add an option to show the full, long information about the pathnames. You should see two long lines on your screen, showing the full file information for each of the two files. Now redirect the long output into file `vifound.txt` under your `assignment03` directory. The file must contain two lines. Run the [Checking Program] to verify your work so far. Searching for text inside files (e.g. course notes) --------------------------------------------------- As mentioned in [Lab Worksheet #03 HTML], choose which text search command you use depending on whether special characters are being used in the search string. Always verify that the correct output appears on your screen *before* you redirect the output into a file. You can only redirect what you can see. 23. Search for lines containing your login userid in the password file and redirect the output into file `mypasswd.txt` under `assignment03`. You should find exactly one line. 24. Search for lines containing the *special character* asterisk (`*`) in the file `bash.bashrc` in the [Source Directory] and redirect the output into file `star.txt` under `assignment03`. The word count of the output (and the file) should be: `2 6 36`. 25. In your HOME directory, create two symbolic links to the old and new course notes for CST8207 using the method described in [Copies of the CST8207 Course Notes], unless you have already created these links earlier in this assignment. 26. In your HOME directory, use the `fgrep` command with a shell GLOB pattern to match all pathnames under `oldnotes/` that end in `.txt` and search for the word `Filezilla` in those files. One line of text from inside one of the files will display. 27. Repeat the above, but add the `fgrep` option that ignores case distinctions. Now, nine lines are found in three different files. Redirect the the nine lines of output into a file named `filezilla.txt` under your `assignment03` directory. The word count of the output (and the file contents) should be: `9 72 816`. Run the [Checking Program] to verify your work so far. The cracker WAREZ 100 files --------------------------- The "story" here is that a cracker has hidden a bunch of WAREZ files in a directory on the server. (See .) Your job is to take a copy of these files, and only these files, for use in a court case. You must not touch or copy any other files, only the WAREZ files. 28. Under the [Source Directory] there is a directory named `start`. Hidden deeper under this directory is one single directory containing about **158,730** files. Be careful about typing `ls` in this directory without using any output pagination pipe -- the amount of output may flood your terminal window for some time and even `^C` may take a minute or two to interrupt the command! Make this (huge) directory your current directory. Exactly 100 files in this one (huge) directory have names that contain your userid (lower-case) followed somewhere later by the string *warez*, where *warez* may appear in any combination of upper- and lower-case letters, e.g. `warez`,`Warez`,`wArez`,`waREz`, etc. Any amount of text may appear before your userid, between your userid and the *warez*, and after the *warez*. Some sample file names for userid `abcd0001` might look like these (note that the *warez* word *must* always follow the userid in all the required file names): - `HhUtfgYtyGhjJADGekCAkgtZEKsTGKdYZZabcd0001ADGekCwaREZZaFSrXJnxGex` - `zynabcd0001uKVUFOsCXaGFWZPECbYWVFKzynuKWaREZv` Using one single copy command and a GLOB expression, copy all 100 (exactly 100) of these cracker files (and no others) into your own directory named `warez` that you created above. Make sure you preserve the modify times of the copied files, as you did in a previous lab. (In this simulation, all the files are empty.) Hints: Use a shell GLOB pattern to match the right file names. Use `echo` and word count to see if your file names are correct before you try to use the copy command. Do not use a pipe or `find` to select the file names. The shell can do it all with one copy command using the right GLOB pattern. When you succeed, you can check you work using: `$ \ls warez | wc` which should show a count of 100 lines. (The backslash before `ls` turns off any alias you might have defined for `ls` that would affect the output.) All the files should have their original modify dates preserved -- verify this. Three different O/S Text Files ------------------------------ 29. Somewhere under that same `start` directory are exactly three non-empty files whose names contain your userid (lower-case) somewhere (anywhere) in the name. Find these three non-empty (size larger than zero) files and copy them (preserving modify times) to the `osfiles` directory that you created above. You will find your userid mentioned inside each file, but because the files are not all Unix/Linux text files, some of the text content may not display correctly on your terminal screen. Hints: What command finds files based on size and a **basename** that can be a GLOB-style pattern? You have used this command many times this term. See the end of [Worksheet #2 HTML]. The `less` command is better than `cat` when displaying files containing strange (e.g. unprintable) characters. Since there are only three file names, you can use your mouse to copy-and-paste the three file names you need to copy. You can also read this optional material on a better way to [Use find -exec and xargs] (optional). 30. In your `osfiles` directory, determine which operating system created each of the three non-empty files. Rename the Unix/Linux file to be `unix`, the Windows file to be `windows` and the Macintosh file to be `macintosh`. Hints: In [Assignment #1] you used a command that can determine file type to identify the text inside a `file.txt` created with Windows Notepad. You will also find this command listed under Week 01 in the [List of Commands] in your notebook. Use this command and the notes on [Text File Line End Differences] to identify the special line endings of the Windows and Macintosh files. Your instructor will mark the `assignment03` directory in your account on the due date. Leave them there on the CLS. Do not delete anything. Run the [Checking Program] to verify your work so far. Appending to files ------------------ 31. Count the lines, words, and characters in the file `services` under the `/etc` directory and put the count in file `wc.txt` under your `assignment03` directory. (Use the absolute pathname of the `services` file when you count and do not use any pipes.) The file `wc.txt` should contain one line. 32. Extract just the first line of the same `services` file and append this one line to the end of the `wc.txt` file, so that the file `wc.txt` now has two lines in it (the word count and the first line of `services`). Hint: You know a command that shows lines at the start of a file. 33. Append the count of the lines, words, and characters in the file `protocols` under the `/etc` directory to the end of file `wc.txt`, so that the `wc.txt` file now has three lines in it. (Use the absolute pathname of the `protocols` file when you count and do not use any pipes.) 34. Extract just the last line of the same `protocols` file and append this one line to the end of the `wc.txt` file, so that the file `wc.txt` now has four lines in it. Hint: You know a command that shows lines at the end of a file. Confirm that the word count of the file gives `4 20 140`. If you see the right number of lines but the other values differ, go back and re-read the parenthesized sentences that start with "Use the". Run the [Checking Program] to verify your work so far. That is all the tasks you need to do. Verify your word counts and then mark your work: Checking, Marking, and Submitting your Work =========================================== **Summary:** Do some tasks, then run the checking program to verify your work as you go. You can run the checking program as often as you want. When you have the best mark, upload the marks file to Blackboard. 1. There is a [Checking Program] named `assignment03check` in the [Source Directory] on the CLS. You can execute this program by typing its (long) pathname into the shell: `$ ~idallen/cst8207/13f/assignment03/assignment03check` 2. Execute the above "check" program. This program will check your work, assign you a mark, and display the output on your screen. (You may want to paginate the long output so you can read all of it.) You may run the "check" program as many times as you wish, to correct mistakes and get the best mark. **Some task sections require you to finish the whole section before running the checking program at the end; you may not always be able to run the checking program successfully after every single task step.** 3. When you are done with checking this assignment, and you like what you see on your screen, redirect the output of the [Checking Program] into the text file `assignment03.txt` under your `assignment03` directory on the CLS. Use the *exact* name `assignment03.txt` in your `assignment03` directory. Case (upper/lower case letters) matters. Be absolutely accurate, as if your marks depended on it. Do not edit the file. Make sure the file actually contains the output of the checking program! 4. Transfer the above `assignment03.txt` file from the CLS to your local computer and verify that the file still contains all the output from the checking program. Do not edit this file! No empty files, please! Edited or damaged files will not be marked. You may want to refer to your [File Transfer] notes. 5. Submit the `assignment03.txt` file under the correct Assignment area on Blackboard (with the exact name) before the due date. Upload the file via the **assignment03** "Upload Assignment" facility in Blackboard: click on the underlined **assignment03** link in Blackboard. Use "**Attach File**" and "**Submit**" to upload your plain text file. No word-processor documents. Do not send email. Use only "Attach File". Do not enter any text into the **Submission** or **Comments** boxes on Blackboard; I do not read them. Use only the "**Attach File**" section followed by the **Submit** button. (If you want to send me comments about your assignment, use email.) 6. Your instructor may also mark the `assignment03` directory in your CLS account after the due date. Leave everything there on the CLS. **Do not delete any assignment work from the CLS until after the term is over!** Use the *exact* file name given above. Upload only one single file of plain text, not HTML, not MSWord. No fonts, no word-processing. Plain text only. Did I mention that the format is plain text (suitable for VIM/Nano/Pico/Gedit or Notepad)? **NO EMAIL, WORD PROCESSOR, PDF, RTF, or HTML DOCUMENTS ACCEPTED.** No marks are awarded for submitting under the wrong assignment number or for using the wrong file name. Use the exact name given above. WARNING: Some inattentive students don't read all these words. Don't make that mistake! Be exact. **READ ALL THE WORDS. OH PLEASE, PLEASE, PLEASE READ ALL THE WORDS!** -- | Ian! D. Allen - idallen@idallen.ca - Ottawa, Ontario, Canada | Home Page: http://idallen.com/ Contact Improv: http://contactimprov.ca/ | College professor (Free/Libre GNU+Linux) at: http://teaching.idallen.com/ | Defend digital freedom: http://eff.org/ and have fun: http://fools.ca/ [Plain Text] - plain text version of this page in [Pandoc Markdown] format [www.idallen.com]: http://www.idallen.com/ [Course Home Page]: .. [Course Outline]: course_outline.pdf [All Weeks]: indexcgi.cgi [Plain Text]: assignment03.txt [hyperlink URLs]: indexcgi.cgi#Important_Notes__alphabetical_order_ [Class Notes]: indexcgi.cgi#XImportant_Notes__alphabetical_order_ [Assignments]: indexcgi.cgi#XAssignments [Worksheets 2, 3, 4, and 5]: indexcgi.cgi#XWorksheets__not_for_hand_in_ [Checking Program]: #checking-marking-and-submitting-your-work [Course Linux Server]: 070_course_linux_server.html [Lab Worksheet #04 ODT]: worksheet04.odt [Lab Worksheet #05 ODT]: worksheet05.odt [Remote Login]: 110_remote_login.html [Assignment #02]: assignment02.html [Lab Worksheet #03 HTML]: worksheet03.html [Source Directory]: #the-source-directory [Searching for and finding files by name, size, use, modify time, etc.]: 180_finding_files.html [Copies of the CST8207 Course Notes]: 070_course_linux_server.html#copies-of-the-cst8207-course-notes [Worksheet #2 HTML]: worksheet02.html [Use find -exec and xargs]: 185_find_and_xargs.html [Assignment #1]: assignment01.html#part-v-remote-network-log-in-to-the-course-linux-server [List of Commands]: 900_unix_command_list.html [Text File Line End Differences]: 015_file_transfer.html#text-file-line-end-differences [File Transfer]: 015_file_transfer.html [Pandoc Markdown]: http://johnmacfarlane.net/pandoc/