oldnotes newnotes

2. In your HOME directory, create two symbolic links to the old and new course notes for CST8207 using the ln -s command and option and the method described in Copies of the CST8207 Course Notes. (The old notes must be term 16w and the new notes must be term 16f in the pathnames you use.)

3. Do a long listing of the new oldnotes symlink and verify that it looks similar to this (but the userid and time will differ):

   lrwxrwxrwx 1 abcd0001 abcd0001 52 Oct  4 00:00 oldnotes -> /home/idallen/public_html/teaching/cst8207/16w/notes

   You should be able to do ls oldnotes | less and see all the course notes file names from last term (16w). If not, remove and redo the symlink.

4. In your HOME directory, use the ls command with no options and a single shell GLOB pattern to match all pathnames under the symbolic link oldnotes/ that end in .txt and display all the names on your screen. The shell will find 89 pathnames ending in .txt, and the ls command will display those 89 names on your screen. One of the last names on your screen should look exactly like this:

   oldnotes/worksheet08.txt

   Make sure you see 89 pathnames. (You can use a command pipeline to count the lines and words to be sure you have 89.)

   Hints: No pipeline or find commmand is required to generate the 89 pathnames, just use the ls command with no options and one single GLOB pattern argument starting with the symlink oldnotes/. This use of a GLOB pattern on a command line is illustrated in Copies of the CST8207 Course Notes. The example in the notes uses the given GLOB pattern to generate pathnames to the ls command and count them. Follow the example and display the 89 pathnames on your screen instead of counting them. (Don’t use any redirection.)

oldfound.txt

5. When the ls output on your screen is correct (89 names), redirect the output 89 names into file oldfound.txt under your Base Directory (not under your current HOME directory). The file must contain 89 names, one per line.

   Note: The ls command will put each name on a separate line when output is not being sent to your screen.

6. Still in your HOME directory, use the echo command with a shell GLOB pattern to match all pathnames under oldnotes/ that contain the acronym RTFM anywhere in the file name and display the names on your screen. The shell will find two pathnames, one ending in .html and the other in .txt, and the echo command will display those two names on your screen on one line.

   Hints: See the previous Hint. Use only a GLOB pattern.

foundman.txt

7. When the echo output on your screen is correct (two names on one line), redirect the output into file foundman.txt under your Base Directory (not under your current HOME directory). The file must contain two names on one line.

8. Again in your HOME directory, use the echo command with a shell GLOB pattern to match pathnames under oldnotes/ that contain the digit 2 anywhere in the file name and end in the extension .pdf at the end. The shell will find five pathnames, each ending in .pdf at the end, and the echo command will display those five names on your screen on one (long) line.

9. When the echo output on your screen is correct (five names on one long line), change the command name from echo to ls and add an option to show the full, long information about the pathnames. You should see five lines on your screen, showing the full file information for each of the five files. One of the lines should look like this:

   -rw-r--r-- 2 idallen idallen  25791 Jul  7  2015 oldnotes/2015-2016_CST8207.pdf

foundpdf.txt

10. Now redirect the five lines of long output on your screen into file foundpdf.txt under your Base Directory (not under your current HOME directory). The file must contain five lines and approximately 45 words.

Run the Checking Program to verify your work so far.

myaccount.txt

1. Search for lines containing your login userid in the password file. You should find exactly one line. (For an explanation of what the seven fields are in this line, see man 5 passwd.

2. When the output is correct (one line) then redirect the output into file myaccount.txt in your Base Directory. The file should contain one line.

3. Search for lines containing a period (dot) character (.) in the file special.txt in the Source Directory.

   Hint: A period can be a special character. Choose the right text searching command, as described at the start of this section. The word count of the six lines of correct output should be: 6 44 241

dotlines.txt

4. When you have the correct output on your screen, redirect that output into file dotlines.txt under your Base Directory. The word count of the file should be the same as above (six lines).

5. Search for lines containing two adjacent asterisk characters (**) in the file special.txt in the Source Directory.

   Hint: An asterisk is a special character to the shell. Hide the asterisks so that the shell does not GLOB expand them. Also choose the right text searching command, as described at the start of this section. The word count of the four lines of correct output should be: 4 37 209

starlines.txt

6. When you have the correct output on your screen, redirect that output into file starlines.txt under your Base Directory. The word count of the file should be the same as above (four lines).

7. In your Base Directory, create two more symbolic links to the old and new course notes for CST8207, as you did inside your HOME directory earlier in this assignment.

In Copies of the CST8207 Course Notes, see the example use of fgrep with shell GLOB patterns to match *.txt files in these oldnotes and newnotes directories. The GLOB pattern easily generates a huge list of file names for fgrep to search inside.

8. In the old course notes from last year, use one command to search inside all the .txt files for the word Filezilla (spelled exactly as shown, case-sensitive). Only three lines of text should display, from three files. Each line of text should be preceded by the file name in which it was found. (The word count of the output must be 3 39 323.)

   Hint: You will need to use the same GLOB pattern you used earlier in this assginment to match all the .txt files under oldnotes. This time, use the GLOB pattern to make the shell give all the file names to the command that searches for text inside all those files. No pipes are needed to find these lines; use just one command with no options and a single GLOB pattern. If you see more than three lines of output, you are likely using options that make the search case-insensitive. If your word count is wrong because the file names are missing, you are likely using unnecessary pipes. No pipes are needed to find the three lines.

9. Repeat the above on all the *.txt files, but add the searching option that ignores case distinctions when matching lines in the files (RTFM). Now, 12 lines are found in five different files and the word count should be 12 118 1165.

   Hint: These text-searching commands are case-sensitive by default – searching inside files for lines containing abc won’t find any lines containing ABC unless you use an option to ignore case distinctions during the search. (What option? RTFM)

zillalines.txt

10. Redirect the above lines of output into a file named zillalines.txt under your Base Directory. The word count of this file should match the one above.

Run the Checking Program to verify your work so far.

warez

2. Using one single copy command and a single shell GLOB pattern, copy all 100 (exactly 100) of these cracker files (and no others) into a new directory named warez that you must create in your own Base Directory. Make sure you preserve the modify times of the copied files, as you did in a previous lab. (In this simulation, all the files are empty.)

   Hints: Before you try to copy any files, use echo with the GLOB pattern into word count to verify your GLOB patterns before using them. The echo with GLOB pattern should produce exactly 100 pathnames.

   The shell must correctly expand the GLOB pattern argument to echo before you try to use the same GLOB pattern in a copy command. Once you can echo the 100 pathnames correctly, use exactly the same GLOB pattern to generate the source pathnames in a copy command.

   Do not use a pipe or find or fgrep to select the file names. Use only the copy command with a GLOB pattern for the source files, as you did in section 4.1 of Worksheet #04 HTML. The shell can do all the file name matching using the right GLOB pattern for the source files.

   Do not quote the shell GLOB patten. Quoting hides metacharacters and turns off shell GLOB patterns. You want the shell to expand the GLOB pattern for this task! (If you were passing a GLOB pattern as an expression in a find command, you would quote it so that the shell didn’t expand it. That is not what you are doing here.)

warezcopy.sh

3. Put the copy command line that you used into file warezcopy.sh in your Base Directory. Pay attention to the file name extension in this file name.

   Hints: Make sure that the content of the file is exactly the same as the copy command you typed, with no special characters expanded. The number of blank-separated words in the file should be about four.

   Hints: The best way to put this command line in the file is to use a Linux text editor, or you can use the cat keyboard and EOF method from section 5.5a in Worksheet #05 HTML. Read this Warning:

Warning: It is tricky to use echo with redirection to put this command line into the file because the line contains shell metacharacters. You can’t just stick echo on the front of a command line that contains shell metacharacters such as GLOB patterns; the shell will expand all those metacharacters before the echo command runs. You will need special Quoting to make it work. You will need to hide all the shell metacharacters in the command line from the shell. Make sure the command line echoes correctly to the screen before you try to redirect it into the file. You can only redirect what you can see! Use a text editor instead!

4. You can check your work by doing a listing of your warez directory and counting the number of names that were copied.

   All the files should have their original modify dates preserved – verify this.

Run the Checking Program to verify your work so far.

abcd0001.txt

1. Under the Source Directory there is a name maze (four letters). What is the absolute path of this maze under that directory? Put the absolute pathname of this maze in that directory into a file in your Base Directory with a basename similar to abcd0001.txt, but use the basename that starts with your own Blackboard userid, not the fake userid abcd0001. Use your own userid in the file name.

   Save the actual absolute pathname, not a shell tilde short-cut for an absolute pathname. (Do not start the name with a tilde.) The file basename must be exactly 12 characters long. The absolute pathname of the maze itself is over 40 characters long.

   You will need this maze absolute pathname in several places, below.

2. Use the ls command and a single shell GLOB pattern to display on your screen on separate lines the absolute paths of all names under the above maze directory that begin with your userid.

   Hints: Use the ls command (no options) with a single absolute path shell GLOB pattern as an argument, in a manner similar to how you displayed all the tty names in section 4.1 of Worksheet #04 HTML. Use the actual absolute pathname, not a shell tilde short-cut for an absolute pathname. Do not start the name with a tilde.

   Hints: You should see six absolute pathnames. One of the six absolute pathnames will end in abcd0001.txt where abcd0001 is your own userid. Each of the six absolute pathnames should contain seven forward slashes. The word count of the six output lines should be 6 6 349.

FirstMaze.txt

3. When you have the correct ls command that generates six lines of absolute pathname output, redirect and save the six lines into file FirstMaze.txt in your Base Directory.

   Hints: The word count of the file should also be 6 6 349.

FirstMaze.sh

4. Save the exact ls command line with GLOB pattern that you used in item 2 above into file FirstMaze.sh in your Base Directory. Pay attention to the file name extension in this file name.

   Hints: The file should contain only two blank-separated words: the command name and a single GLOB pattern. Do not save the redirection that you added in item 3 above. The word count of the file should show 1 2 57.

   Hints: Make sure that the content of the file is exactly the same as the ls command you typed in item 2, with no special characters expanded. The number of blank-separated words in the file should be exactly two: the command name and a single GLOB pattern argument. Running the command by typing sh -u FirstMaze.sh should print the six pathnames on your screen.

   Hints: The best way to put this command line in the file is to use a Linux text editor, or you can use the cat keyboard and EOF method from section 5.5a in Worksheet #05 HTML. Read this Warning:

Warning: It is tricky to use echo with redirection to put this command line into the file because the line contains shell metacharacters. You can’t just stick echo on the front of a command line that contains shell metacharacters such as GLOB patterns; the shell will expand the GLOB patterns before the echo command runs. You will need special Quoting to make it work. You will need to hide all the shell metacharacters in the command line from the shell. Make sure the command line echoes correctly to the screen before you try to redirect it into the file. You can only redirect what you can see! Use a text editor instead!

Run the Checking Program to verify your work so far.

These six pathnames are only six of the many file names in the maze that start with your userid. We need to find them all, in all the sub-directories, too.

howquote.txt

2. In the first paragraph you found, above, put the example command line (showing the use of quotes around the *.txt argument that contains a GLOB character) into file howquote.txt in your Base Directory. The file must contain just the example command line text after the e.g. and it will be one line, three words, 19 characters, according to wc.

   If the count is wrong, look in the file to see what is wrong with the text. Does the file contain exactly the same text as the course notes? If not, edit the file and fix it.

   Hints: The best way to put this example line in the file is to use a Linux text editor, or you can use the cat keyboard and EOF method from section 5.5a in Worksheet #05 HTML. Read this Warning:

Warning: It is tricky to use echo with redirection to put this command line into the file because the line contains shell metacharacters. You can’t just stick echo on the front of a command line that contains shell metacharacters such as quotes; the shell will remove the quotes before the echo command runs. You will need special Quoting to make it work. You will need to hide all the shell metacharacters in the command line from the shell. Make sure the command line echoes correctly to the screen before you try to redirect it into the file. You can only redirect what you can see! Use a text editor instead!

Run the Checking Program to verify your work so far.

lscmd.sh

2. Save the full and exact ls command line you just used into file lscmd.sh in your Base Directory. Pay attention to the file name extension in this file.

   Running the command by typing sh -u lscmd.sh should print the long information about the maze symlink.

infomaze.txt

3. When you have the correct ls command line that generates one long line of output, redirect and save the output (one line) into file infomaze.txt under your Base Directory.

Run the Checking Program to verify your work so far.

foundmaze1.txt

3. Pipe the 23 lines of pathname output of the above command into a sorting program and put the sorted output into file foundmaze1.txt under your Base Directory. The sorted file will still contain exactly the same number of lines and words as you counted, above.

findcmd1.sh

4. Put the entire above two-command pipeline with redirection that you just used, into file findcmd1.sh in your Base Directory. Pay attention to the file name extension of this file name.

   Hint: Make sure the command you save in the file includes both the pipeline and the output redirection. If you run the command file using sh -u findcmd1.sh you should see no errors and no output on your screen. (All the output should go into the foundmaze1.txt output redirection file.)

   Hint: The best way to put this command line in the file is to use a Linux text editor, or you can use the cat keyboard and EOF method from section 5.5a in Worksheet #05 HTML. Read this Warning:

Warning: It is tricky to use echo with redirection to put this command line into the file because the line contains shell metacharacters. You can’t just stick echo on the front of a command line that contains shell metacharacters such pipes; the shell will execute the pipes before the echo command runs. You will need special Quoting to make it work. You will need to hide all the shell metacharacters in the command line from the shell. Make sure the command line echoes correctly to the screen before you try to redirect it into the file. You can only redirect what you can see! Use a text editor instead!

Run the Checking Program to verify your work so far.

foundmaze2.txt

3. Pipe the 47 lines of pathname output of the above command into a sorting program and put the reverse-sorted output into file foundmaze2.txt under your Base Directory. The reverse-sorted file will still contain exactly the same number of lines and words as you counted, above.

findcmd2.sh

4. Put the entire above two-command pipeline with redirection that you just used, into file findcmd2.sh in your Base Directory.

   Hint: See the hints for the previous section. Make sure the command you save in the file includes both the pipeline and the output redirection. Run the file to make sure the saved command works.

Run the Checking Program to verify your work so far.

3filesOS

2. When you know the three pathnames, manually copy each of these files (preserving modify times) to a new directory named 3filesOS that you must create in your Base Directory.

   Since there are only three file names, you can use your mouse to copy-and-paste the three long file names you need to copy, once you know their names. Be careful to use quoting to hide any blanks in the names from the shell.

   (Optional advanced use: You can also read this optional material on a better way to use find -exec and xargs.)

Unix Windows Macintosh

3. In your 3filesOS directory, determine which operating system created each of the three non-empty files. Rename the Unix/Linux file to be Unix, the Windows file to be Windows and the Macintosh file to be Macintosh.

   Hints: In Assignment #02 HTML you used a command that can determine file type to identify the text inside a date.txt file. You will also find this command listed under Week 02 in the List of Commands in your notebook. Use this command and the notes on Text File Line End Differences to identify the special line endings of the Windows and Macintosh files.

Run the Checking Program to verify your work so far.

appendfile

1. Count the lines, words, and characters in the file services under the /etc directory and put the count in file appendfile under your Base Directory. (Use the absolute pathname of the services file when you count and do not use any pipes.) The file appendfile should contain one line containing three numbers and an absolute pathname at the end. There is no file extension on this file; Linux doesn’t care.

2. Extract just the last line of the same services file and append this one line to the end of the appendfile file, so that the file appendfile now has two lines in it (the word count line, and the last line of services).

   Hint: You know a command that shows any number of lines at the end of a file. Review your work in Worksheet #05 HTML and the notes on Redirection and Pipes.

3. Append the count of the lines, words, and characters in the file protocols in the /etc directory to the end of file appendfile, so that the appendfile file now has three lines in it. (Use the absolute pathname of the protocols file when you count and do not use any pipes.)

4. Extract just the first line of the same protocols file and append just this one line to the end of the appendfile file, so that the file appendfile now has four lines in it.

   Hint: You know a command that shows any number of lines at the start of a file. Review your work in Worksheet #05 HTML and the notes on Redirection and Pipes.

Confirm that the word count of the appendfile file gives 4 15 105. If you see the right number of lines but the other values differ, go back and re-read all the words in the sentences above, especially the sentences that start with the words “Use the”.

Run the Checking Program to verify your work so far.

refusedpipe.sh

2. When the output is correct, put the new seven-command pipline you used into file refusedpipe.sh in the Base Directory. You can put it on separate lines with backslashes at the end of each line, as shown in the notes, or you can remove the backslashes and put it all on one long line.

   Typing sh -u refusedpipe.sh should print the six most active attack IP addresses for the one month on your screen. If it doesn’t do this, you haven’t copied the command line correctly. Check it!

You can debug your script file by running it like this: bash -ux refusedpipe.sh and making sure you see seven commands execute before the six lines of output appears.

3. Edit the refusedpipe.sh file and add to the end of the file, underneath your seven-command pipeline, exactly seven numbered shell comments that explain briefly and in your own words the meaning of each of the seven commands used in the pipeline, using the comment format described below.

   Shell script comments start with the number-sign (or hash-tag) character # and extend to the end of the line. The seven numbered comment lines must have a syntax similar to this (though this is the wrong pipeline and wrong comments to use for this task):

   last idallen | awk '{ print $3 }' | grep '^[0-9]' | sort | uniq | wc -l
   # 1. last idallen: show last login lines only for user idallen
   # 2. awk '{ print $3 }': display only third field (IP address)
   # 3. grep '^[0-9]': select only lines starting with a digit
   # 4. sort: put IP addresses into sorted order
   # 5. uniq: throw away duplicate adjacent IP addresses, leaving only unique
   # 6. wc -l: count the number of unique IP addresses (number of lines)

   Comment Format: Since there are seven commands in your script pipeline, you will need to write exactly seven numbered comment lines to explain them. As you see in the above example, each of the seven comment lines starts at the left margin with the # comment character (no spaces in front), followed by a space, number, a period, space, the pipeline command name and options to which the comment refers, and then your own comment text written in your own words. Each comment text is written in your own words to explain what the command does in the pipeline. Do not copy words; write your own.

   Follow the syntax shown in the above example, and use your own words (don’t copy mine). Including the seven comment lines, your refusedpipe.sh file will be at least eight (or more) lines long.

Run the Checking Program to verify your work so far.

denycom1.sh

2. When the output is correct, put the command line you used to generate the number 2154 into file denycom1.sh in the Base Directory.

   Typing sh -u denycom1.sh should print the number 2154 on your screen. If it doesn’t do this, you haven’t copied the command line correctly. Check it!

You can debug your script file by running it like this: bash -ux denycom1.sh and making sure you see the correct command execute before the output appears.

3. Write a command pipeline (using pipes) to count the number of lines containing the string new denied hosts in only September 2015 in the denyhosts-2015 log file on the CLS. You should find 177 matching lines to count and the output should be the number 177.

   Hints: The Example 1 given in Using successive filters in pipes explains how you might find some lines in the auth.log file that were created in January. Apply what you learn there to solve this problem. Before you try, look at the denyhosts-2015 file and find out what format it uses to represent the date “September 2015”. You can’t just look for the text “September 2015” in the file; it’s not there. Look into the file to see the actual date format and create a filter command to search for that date format and count the lines. My solution used two command names with one pipe between. The second command used an option that counted the number of matching lines, as shown in the weekly course notes.

denycom2.sh

4. When the command pipeline is correct, put the command pipeline you used to generate the number 177 into file denycom2.sh in the Base Directory.

   Typing sh -u denycom2.sh should print the number 177 on your screen. If it doesn’t do this, you haven’t copied the command line correctly. Check it!

You can debug your script file by running it like this: bash -ux denycom2.sh and making sure you see the correct commands in the pipeline execute before the output appears.

5. Using your shell history and the command you used in the previous item, modify and redo the command a few times to manually find the number of denied hosts in each month in 2015. Use this to determine the month with the largest number of denied hosts (390).

   Hint: It’s one of the months before June.

denyhosts.txt

6. When you find the month with the largest number of denied hosts, Put the first five lines and the last five lines of log entries for this month into file denyhosts.txt in the Base Directory.

   Hint: Use a command pipeline to generate the first five lines of log output for this month and save them, then modify the command pipeline slightly to generate the last five lines of log output for this month and append them to the file containing the first five lines. That is your answer. The first five lines should be from the start of the month and the last five lines should be from the end of the month. The word count of this ten-line file should be: 10 100 854 and the sum should be 28039.

Run the Checking Program to verify your work so far.