3.1 The Source Directory

All references to the “Source Directory” below are to the CLS directory ~idallen/cst8207/14f/assignment04/ and that name starts with a tilde character ~ followed by a userid with no intervening slash. The leading tilde indicates to the shell that the pathname starts with the HOME directory of the account idallen (seven letters).

You do not have permission to list the names of all the files in the Source Directory, but you can access any files whose names you already know.

4.1 Worksheets and the PS1 Shell Prompt

These worksheets prepare you to do the numbered tasks listed below by executing commands via Remote Login to the Course Linux Server.

You can download Libre Office (or Open Office) for Windows to edit the Worksheet *.odt files and save your answers. (Or you can print the PDF and write your answers on paper.) Do not use MSWord; it will mangle the format of the worksheet.

Record and save all your worksheet answers for study and quizzes!

See Assignment #02 for how best to fill in the worksheets.

1. Use LibreOffice to complete Worksheet #04 ODT. (View online: Worksheet #04 HTML) Record and save all your worksheet answers for study and quizzes!
2. Use LibreOffice to complete Worksheet #05 ODT. (View online: Worksheet #05 HTML) Record and save all your worksheet answers for study and quizzes!

3. Set your prompt every time you log in to the CLS, so that the prompt changes to tell you you the basename of your current working directory, as you did in the worksheets and the previous assignment. After you create your shell start-up files in this assignment, you won’t have to do this ever again.

   Hint: You can use ^R in your saved shell history to set the prompt very quickly at login time, as demonstrated in lectures.

4.2 Creating Shell Start-Up Files

1. Review the Class Notes:
   • Shell Variables
   • Start-Up Files

2. Use a Linux text editor to create your .bash_profile and .bashrc files with the minimum suggested content described in the Summary section of page Start-Up Files. These files do not exist yet; you must create them. Your .bash_profile must contain only one line. Your .bashrc must contain at least two lines (not including comment lines).

   Do not set any options or aliases in your .bashrc that you do not understand! If you don’t know the meaning of a setting, don’t use it. You can RTFM in the bash man page for all BASH settings, and RTFM in command man pages to learn about options to commands.

   Using the PS1 variable from Worksheet #2 HTML, set your prompt to include your user name, your computer name, and the basename of your current working directory. Put this setting (or one like it) in your .bashrc file.

I will be spot-checking your knowledge of your aliases and shell options. Students using aliases they don’t understand will experience much confusion trying to do future assignments. Only use aliases you understand.

3. Verify that nothing prints on your screen after you enter your password when you run the non-interactive shell connection using ssh localhost true (as described in the section on Non-interactive shells and PS1:

   $ ssh localhost true
   *** COURSE LINUX SERVER ***
   user@localhost's password:
   $

   For non-interactive commands to work properly, there must be no output on your screen after you enter your password using the above non-interactive command line using the true command.

Your instructor will mark the .bashrc and .bash_profile files in your account after the assignment due date. Do not upload them to Blackboard. Leave them there on the CLS. Do not delete anything.

Run the Checking Program to verify your work so far.

4.3 Set Up – The Base Directory on the CLS

1. Create the following directory structure in your CLS HOME directory and record (for study purposes) the series of Unix commands you used to create it. (You do not have to create any directories that you have already created in a previous assignment.) Spelling and capitalization must be exactly as shown:
   1. Create the CST8207-14F directory in your CLS HOME directory.
   2. Create the Assignments directory in the CST8207-14F directory.
   3. Create the assignment04 directory in the Assignments directory.

Hint: You can create the entire directory tree above using one single command with one option and one pathname argument, as you did at the end of Section 4 in Worksheet #02 HTML. System administrators like to work efficiently – they learn how to do things quickly.

The correct directory tree will look similar to the ASCII tree diagram below. The spelling and capitalization must be exactly as shown.

CST8207-14F
`-- Assignments
    `-- assignment04

This assignment04 directory is the Base Directory for most pathnames in this assignment. Store your files and answers in this Base Directory.

Run the Checking Program to verify your work so far.

4.4 Finding multiple files in a maze using GLOB patterns

You need to understand Shell GLOB Patterns to do this task. The shell will not be expanding the GLOB patterns, since you will be quoting them and passing the GLOB patterns to another command for evaluation, but the GLOB pattern metacharacters work the same way.

1. Under the Source Directory there is a name maze (four letters) that is a symbolic link that leads to the same directory maze used in a previous assignment; refer to that assignment for details on the size of this maze.

   Use the absolute pathname of the maze name in the Source Directory as an argument to ls along with an option that shows the long information about the pathname. You probably have the right option if the first word of the output is lrwxrwxrwx, indicating that maze is a symbolic link. Save the output (one line) into file mazeinfo.txt under your Base Directory.

   Hint: If you get a directory listing full of files instead of one line starting with lrwxrwxrwx, make sure you are using the Source Directory path from this assignment and not any previous assignment.

   1. Again, in a manner similar to your previous assignments, you must find files in this maze, but the symbolic link requires some special handling because the command that recursively finds files does not follow symbolic link arguments on the command line without using some options. You must choose one of these approaches to reach the maze:

      1. Use the maze directory pathname from the previous assignment, not the pathname in the Source Directory that is the symbolic link, OR
      2. Use an option to the finding command that makes it follow symbolic links while processing the command line arguments. Hint: RTFM and do not use the -L option, OR
      3. Make the maze your current directory and then recursively search the current directory. (A current directory can never be a symbolic link – it must be a real directory.)

      Choose one of the previous three starting directory options to reach the maze when you start searching, below.

   2. Using the search tools in your web browser (not on the CLS), look for the string quote in the course notes web page on Searching for and finding files by name, size, use, modify time, etc. Read all the paragraphs containing this word (search multiple times) and remember the importance of quoting. You will need to know how to do this quoting when you start the searching work on the CLS, below.

2. In this maze, use a single command (not a pipeline) to recursively find all pathnames with a basename that begins with your eight-character userid at the start of the name. For example, if your userid were abcd0001 then you might output pathnames containing basenames such as abcd0001 and abcd0001YYY but not XXXabcd0001 or XXXabcd0001YYY or abcdYYYY where XXX and YYY can be any non-empty strings of characters. (You have previously used this command many times without a pattern for a basename; this task requires you to use a quoted GLOB pattern that matches your userid followed by zero or more characters.) The command you use should recursively find exactly 19 pathnames.

   Modify the same single command you used to find the names above and add the option that makes the command show the detailed attribute information about the names (including permissions, owner, size, date, etc.) instead of just the pathname. (You know this option from your answers in Worksheets 2 and 3 and from reading the detailed attribute information paragraph in the Searching for and finding files by name notes.) You will know you have the right option if the output of the command shows 19 lines and approximately 212 words.

   Pipe the pathname output of the above command (19 lines of output) into a sorting program and put the sorted output into file mazefound1.txt under your Base Directory. The sorted file will still contain exactly the same number of lines and words.

3. Put the above two-command pipeline with redirection that you just used, into file findcmd1.txt in your Base Directory. You can use a text editor (easy) or you can use the echo command (tricky) to do this, as you did in a previous assignment.

   Hint: You will need to hide all the shell meta-characters in the command line from the shell if you use echo; make sure the command line echoes correctly to the screen before you try to redirect it into the file.

4. In this maze, use a single command (not a pipeline) to recursively find all pathnames with a basename that contains your eight-character userid anywhere in the name. For example, if your userid were abcd0001 then you might output pathnames containing basenames such as abcd0001, abcd0001YYY, XXXabcd0001, and XXXabcd0001YYY where XXX and YYY can be anything (zero or more characters). You must use a single command that finds files by using a pattern basename to find these pathnames. (You have previously used this command many times without a pattern for a basename.) You should find exactly 40 pathnames.

   Modify the same single command you used to find the names above and add the option that makes the command show the detailed attribute information about the names (including permissions, owner, size, date, etc.) instead of just the pathname. (You know this option from your answers in Worksheets 2 and 3 and from reading the detailed attribute information paragraph in the Searching for and finding files by name notes.) You will know you have the right option if the output of the command shows 40 lines and approximately 455 words.

   Pipe the pathname output of the above command (40 lines of output) into a sorting program and put the reverse-sorted output into file mazefound2.txt under your Base Directory. The reverse-sorted file will still contain exactly the same number of lines and words.

5. Put the above two-command pipeline with redirection that you just used, into file findcmd2.txt in your Base Directory.

   Hint: See the previous hint before you try this.

Run the Checking Program to verify your work so far.

4.5 Using shell GLOB patterns to select names

You need to understand Shell GLOB Patterns to do this task.

1. Make your HOME directory your current directory.

2. In your HOME directory, create two symbolic links to the old and new course notes for CST8207 using the ln -s command and option and the method described in Copies of the CST8207 Course Notes. (The old notes must be term 14w and the new notes must be term 14f in the pathnames you use.)

3. Do a long listing of the new oldnotes symlink and verify that it looks similar to this (but the userid and time will differ):

   lrwxrwxrwx 1 abcd0001 abcd0001 52 Oct  2 07:37 oldnotes -> /home/idallen/public_html/teaching/cst8207/14w/notes

   You should be able to do ls oldnotes and see all the course notes file names from last term (14w). If not, remove and redo the symlink.

4. In your HOME directory, use the echo command with a single shell GLOB pattern to match all pathnames under the symbolic link oldnotes/ that end in .txt and display all the names on your screen. The shell will find 87 pathnames ending in .txt, and the echo command will display those 87 names on your screen in one long line. One of the last names on your screen should look exactly like this:

   oldnotes/worksheet08.txt

   Make sure you see 87 pathnames. (You can use a command to count the words to be sure you have 87.)

   Hints: No pipeline is required to generate the 87 pathnames, just use the echo command and one GLOB pattern argument containing the symlink oldnotes. This use of a GLOB pattern on a command line is illustrated in Copies of the CST8207 Course Notes. The example in the notes uses the given GLOB pattern to generate pathnames to the ls command and count them. Follow the example but use echo instead of ls, and display the pathnames on your screen instead of counting them (don’t use any pipes).

5. When the echo output on your screen is correct (87 names), redirect the output 87 names into file textfound.txt under your Base Directory. The file must contain 87 names on one line.

6. Again in your HOME directory, use the echo command with a shell GLOB pattern to match all pathnames under oldnotes/ that contain the word symbolic anywhere in the file name and display the names on your screen. The shell will find two pathnames, one ending in .html and the other in .txt, and the echo command will display those two names on your screen.

7. When the echo output on your screen is correct (two names), redirect the output into file symfound.txt under your Base Directory. The file must contain two names on one line.

8. Again in your HOME directory, use the echo command with a shell GLOB pattern to match pathnames under oldnotes/ that contain the word vi anywhere in the file name and end in the extension .pdf at the end. The shell will find two pathnames, each ending in .pdf at the end, and the echo command will display those two names on your screen.

9. When the echo output on your screen is correct (two names), change the command name from echo to ls and add an option to show the full, long information about the pathnames. You should see two long lines on your screen, showing the full file information for each of the two files.

10. Now redirect the long output on your screen into file vifound.txt under your Base Directory. The file must contain two lines.

Run the Checking Program to verify your work so far.

4.6 Searching for text inside files (e.g. course notes)

As mentioned in Worksheet #03 HTML, choose which text search command you use depending on whether special characters are being used in the search string. We almost always use the fixed-string fgrep command in this introductory course. You will learn regular expressions and the grep command in later terms.

Always verify that the correct output appears on your screen before you redirect the output into a file. You can only redirect what you can see.

1. Search for lines containing your login userid in the password file and redirect the output into file mypasswd.txt in your Base Directory. You should find exactly one line.

2. Search for lines containing a period (dot) character (.) in the file special.txt in the Source Directory. (The word count of the correct output should be: 6 32 167)

   Hint: A period is a special character. Choose the right command.

3. When you have the correct output on your screen, redirect that output into file periods.txt under your Base Directory. The word count of the file should be the same as above.

4. Search for lines containing two asterisk characters (**) in the file special.txt in the Source Directory. (The word count of the correct output should be: 3 28 159)

   Hint: An asterisk is a special character. Choose the right command.

5. When you have the correct output on your screen, redirect that output into file asterisks.txt under your Base Directory. The word count of the file should be the same as above.

6. In your HOME directory, create two symbolic links to the old and new course notes for CST8207 using the method described in Copies of the CST8207 Course Notes, unless you have already created these links earlier in this assignment.

   In the same notes section, see the example use of fgrep with shell GLOB patterns to match *.txt files in these oldnotes and newnotes directories. The GLOB pattern easily generates a huge list of file names for fgrep to search inside.

7. In the course notes from last year, search inside all the .txt files for the word Filezilla (spelled exactly as shown, case-sensitive). Only three lines of text should display, from three files.

   Hint: You will need to use the same GLOB pattern you used earlier to match all the .txt files under oldnotes. This time, use the GLOB pattern to give file names to the command that searches inside all those files. If you see more than three lines of output, you are likely using options that make the search case-insensitive. Don’t do that.

8. Repeat the above on all the *.txt files, but add the searching option that ignores case distinctions when matching lines in the files (RTFM). Now, 11 lines are found in five different files.

   Hint: These text-searching commands are case-sensitive by default – searching inside files for lines containing abc won’t find any lines containing ABC unless you use an option to ignore case distinctions during the search. (What option? RTFM)

9. Redirect the 11 lines of output into a file named filezilla.txt under your Base Directory.

Run the Checking Program to verify your work so far.

4.7 The cracker WAREZ 100 files

You need to understand Shell GLOB Patterns to do this task.

The “story” here is that a malicious cracker has dumped a bunch of WAREZ files in a directory on the server and has hidden them among thousands of other files. (See https://en.wikipedia.org/wiki/Warez.) Your job is to take a copy of the WAREZ files, and only the WAREZ files, for use in a court case. You must not touch or copy any other files, only the WAREZ files.

1. Under the Source Directory there is a directory named start. Hidden (really hidden) deeper under this directory is one single directory containing over 185,000 files. Be careful about typing ls in this directory without using any output pagination pipe – the amount of output may flood your terminal window for some time and even a ^C interrupt may take a minute or two to interrupt the command! One way to avoid flooding your screen is by using ls | wc to count how many pathnames would be output on your screen before you do just ls.

   Find this huge directory and make this huge directory your current directory, so that you can experiment with the GLOB pattern you will need in the following questions.

   Hints: This isn’t a maze. There is only one path down to the huge directory inside start, though the way is hidden. Remember not to type ls in this large directory, when you find it, because the output is very large!

2. Exactly 100 files in this one (huge) directory have names that contain your userid (lower-case) followed somewhere later by the string warez, where warez may appear in any combination of upper- and lower-case letters, e.g. warez,Warez,wArez,waREz, etc. Any amount of text may appear before your userid, between your userid and the warez, and after the warez.

   Some sample file names for userid abcd0001 might look like these (note that the warez word must always follow the userid in all the required file names):

   • HhUtfgYtyGhjJADGekCAkgtZEKsTGKdYZZabcd0001ADGekCwaREZZaFSrXJnxGex
   • zynabcd0001uKVUFOsCXaGFWZPECbYWVFKzynuKWaREZv

   Using one single copy command and a single shell GLOB expression, copy all 100 (exactly 100) of these cracker files (and no others) into a new directory named warez that you must create in your Base Directory. Make sure you preserve the modify times of the copied files, as you did in a previous lab. (In this simulation, all the files are empty.)

   Hints: Use one shell GLOB pattern to match the 100 file names. The shell can do it all with one copy command using the right GLOB pattern for the source files, as you did in section 4.1 of Worksheet #04 HTML. Always use echo with the GLOB pattern into word count to see if your file names are correct before you try to use the GLOB expression in a copy command.

   Do not use a pipe or find to select the file names. Use only the copy command with a GLOB pattern for the source files.

   Do not quote the shell GLOB patten. Quoting turns off shell GLOB patterns. You want the shell to expand the GLOB pattern for this task! (If you were passing a GLOB pattern as an expression in a find command, you would quote it so that the shell didn’t expand it. That is not what you are doing here.)

3. Put the copy command line that you used into file copycmd.txt in your Base Directory. You can use a text editor or you can use the echo command to do this, as you did in a previous assignment.

   Hint: If you use echo to echo and redirect the line into the file, make sure you quote all the shell metacharacters that might expand. Make sure that the content of the file is exactly the same as the command you typed, with no special characters expanded. The number of words in the file should be less than a dozen.

4. You can check your work by doing a recursive listing of your warez directory and counting the number of names that were copied.

   All the files should have their original modify dates preserved – verify this.

Run the Checking Program to verify your work so far.

4.8 Three different O/S Text Files

1. Somewhere under that same start directory are exactly three non-empty files whose names contain your userid (lower-case) somewhere (anywhere) in the name. (All the other files whose names contain your userid, except those three, are empty.)

   Find these three non-empty (size larger than zero) files with your userid in the name and copy them (preserving modify times) to a new directory named OSfiles that you must create in your Base Directory.

   Since there are only three file names, you can use your mouse to copy-and-paste the three file names you need to copy, once you know their names. You can also read this optional material on a better way to use find -exec and xargs (advanced use, optional).

   Hints: You will also find your userid mentioned inside each file, but because the files are not all Unix/Linux text files, some of the text content may not display correctly on your terminal screen. The less command is better than cat when displaying files containing strange (e.g. unprintable) characters, but see also the “show-nonprinting” option to cat.

   Hints: What command finds files based on expressions that can include both size and a basename that can be a GLOB-style pattern? You have used this command many times this term. See the end of Worksheet #02 HTML and the notes on Finding Files.

2. In your OSfiles directory, determine which operating system created each of the three non-empty files. Rename the Unix/Linux file to be unix, the Windows file to be windows and the Macintosh file to be macintosh.

   Hints: In Assignment #1 you used a command that can determine file type to identify the text inside a cal.txt file. You will also find this command listed under Week 01 in the List of Commands in your notebook. Use this command and the notes on Text File Line End Differences to identify the special line endings of the Windows and Macintosh files.

Your instructor will also mark the Base Directory in your account on the due date. Leave everything there on the CLS. Do not delete anything.

Run the Checking Program to verify your work so far.

4.9 Appending to files

1. Count the lines, words, and characters in the file services under the /etc directory and put the count in file wc under your Base Directory. (Use the absolute pathname of the services file when you count and do not use any pipes.) The file wc should contain one line containing three numbers and an absolute pathname at the end.

2. Extract just the first line of the same services file and append this one line to the end of the wc file, so that the file wc now has two lines in it (the word count line and the first line of services).

   Hint: You know a command that shows lines at the start of a file. Review your work in Worksheet #05 HTML and the notes on Redirection and Pipes.

3. Append the count of the lines, words, and characters in the file protocols in the /etc directory to the end of file wc, so that the wc file now has three lines in it. (Use the absolute pathname of the protocols file when you count and do not use any pipes.)

4. Extract just the last line of the same protocols file and append just this one line to the end of the wc file, so that the file wc now has four lines in it.

   Hint: You know a command that shows lines at the end of a file. Review your work in Worksheet #05 HTML and the notes on Redirection and Pipes.

Confirm that the word count of the wc file gives 4 20 140. If you see the right number of lines but the other values differ, go back and re-read all the words in the sentences above, especially the sentences that start with the words “Use the”.

Run the Checking Program to verify your work so far.

4.10 Scanning the Log File

1. Use a two-command pipeline to find every line containing the string new denied hosts that appears in September 2014 in the denyhosts log file. (This log file is in the same directory as the auth.log file used in class.) You should find 129 matching lines in the file.

   When you have the right 129 lines on your screen, redo the command and redirect them into file deny1.txt in your Base Directory. The file must have 129 lines in it.

   Hint: See your class notes from Week 05 Notes. That explains how you might find only lines in the auth.log file that were created in January 2014. Read the denyhosts file and find out what format it uses to represent the date “September 2014”. (Note: You can’t look for the text “September 2014” in the file; it’s not there. Read the file to see the actual date format and look for that.)

2. Remove the redirection and add one more command to the two-command pipeline (making a total of three commands) to extract only the blank-separated field containing the IP address from each of the above 129 lines and show only the 129 IP addresses on your screen. The first line (of 129) will be ['122.225.109.101'] and the last line (of 129) will be ['222.186.34.25'].

   When you have the right 129 IP addresses on your screen, redo the command and redirect them into file deny2.txt in your Base Directory. The file must have 129 IP addresses in it.

   Hint: See your class notes from Week 05 Notes. That explains how you night extract any blank-delimited field from a line. (As shown in the weekly notes, do not use the cut command for this. Use the other command.)

3. Remove the redirection and put a copy of the three-command pipeline you just created into file denygrep.sh in the Base Directory. (Put the command pipeline in the file, not the output of the command.)

   If you execute sh ./denygrep.sh the output on your screen should be the same 129 IP addresses that you put in deny2.txt. (If you word cout the script output, it will be 129 lines, 129 words, and about 2362 characters, same as deny2.txt.)

Run the Checking Program to verify your work so far.

4.11 Using shell environment variables

Follow correct double-quoting procedures when expanding variables, as noted in Shell Variables.

1. Use the echo command to display the value of the environment variable containing your userid. Put the command you use to do this into file enviro.sh in the Base Directory. (Put in the command, not the output.) The file will contain one command line.

2. Use the echo command to display the value of the environment variable containing your home directory. Append the command you use to do this to file enviro.sh in the Base Directory. (Put in the command, not the output.) The file will contain two command lines.

3. Use the echo command to display the value of the environment variable containing your assigned shell. Append the command you use to do this to file enviro.sh in the Base Directory. (Put in the command, not the output.) The file will contain three command lines.

4. Use the echo command to display the value of the environment variable containing your shell search path. Append the command you use to do this to file enviro.sh in the Base Directory. (Put in the command, not the output.) The file will contain four command lines.

5. Run sh ./enviro.sh and the values of all four environment variables should display on your screen.

6. To verify that you have used proper procedures for expanding variables inside your script, set the USER variable to be a GLOB character temporarily and then run the script, like this:

   $ USER='*' sh ./enviro.sh

   Make sure that the first line of script output is the single GLOB character and not a list of file names. If you see a list of file names, re-read the first sentence of this task, above.

Run the Checking Program to verify your work so far.

4.12 When you are done

That is all the tasks you need to do.

Check your work a final time using the Checking Program and save the output as described below. Submit your mark following the directions below.