----------------------- Exercise #4 for NET2003 due February 3, 2005 ----------------------- -Ian! D. Allen - idallen@idallen.ca Remember - knowing how to find out an answer is more important than memorizing the answer. Learn to fish! RTFM! (Read The Fine Manual) Global weight: 3% of your total mark this term Due date: 15h00 Thursday February 3, 2005. The deliverables for this exercise are to be submitted online on the Course Linux Server using the "datsubmit" method described in the exercise description, below. No paper; no email; no FTP. Late-submission date: I will accept without penalty exercises that are submitted late but before 10h00 on Saturday, February 5. After that late-submission date, the exercise is worth zero marks. Exercises submitted by the *due date* will be marked online and your marks will be sent to you by email after the late-submission date. This exercise is due 15h00 Thursday February 3, 2005. Exercise Synopsis: Marks: 3% Shell scripts continued: On the Course Linux Server, create an executable bash shell script that does "data mining" on the output of Unix commands. Write another script that fetches information from the Internet and displays part of it. Where to work: Do your Unix command line work on the Course Linux Server. The files you work on will remain on the server even after you log off. Do not erase your files after submission; always keep a spare copy of your exercises. WARNING: Do not attempt this exercise on a Windows machine - the text file format is different. You must connect to and work on Unix/Linux. Note that you may connect to the Course Linux Server *from* a Windows machine (using PuTTY); however, you may not use the Windows machine itself to do your work. Use the vim editor on the Course Linux Server. Location of the course notes on the Course Linux Server: You can find a copy of all the course Notes files on the Linux Server under directory: ~idallen/public_html/teaching/net2003/05w/notes/ You can copy files from this directory to your own account for modification or study, if you like. (To avoid plagiarism charges, you must credit any material that you copy and submit unchanged.) Exercise Preparation: Notes: see notes and readings in week04notes.txt A. Finish your Notes Readings (see the weekly Notes files). Any questions? See me in a lab or post questions to the Discussion news group (on the top left of the Course Home Page). B. Review the use of Unix pipes. Do *not* create temporary files for any output in this week's exercise. Use Unix pipes only. --------------------------------------------- Exercise Details (on the Course Linux Server) --------------------------------------------- 0. Have you done all the preparation steps? If not, go back and do them. You may find it useful to create separate directories in your account to store the files for each exercise. Part I - myscript4.sh --------------------- 1. Using VI/VIM, edit a new executable script file named myscript4.sh on the Course Linux Server. The spelling of the file name must be exact, othewise it won't be marked. The spelling must be exact. Exact! The contents of the file will be modelled on the argv.sh script: a) The first line of the file must be a valid shell interpreter. b) The second line of the file is a one-line (less than 80 characters) description of what this script does. c) The "Syntax" section for this script should say "$0 (no arguments)", since there will be no command-line arguments passed to the script. d) The Purpose is a few lines that describes what the script does. (You will know what that is when you are done the exercise!) e) Next is your Assignment Label for this exercise. f) Next are the three lines setting PATH, umask, and LC_COLLATE. The above are standard parts of every NET2003 shell script. Make sure the script file is executable by you. 2. At the bottom of your script file, add two test command lines. One command should be found by the shell in the /bin directory, the other in the /usr/bin directory. (Hint: check out the locations of the "date" and "wc" commands.) Execute your script with its two test command lines and verify that the script can find and run both commands without any errors. When this is true, insert shell comment characters in front of the two lines (octothorpe: "#"). Do not remove the lines. 3. At the botton of your script file, add a line that uses an *undefined* shell variable name (you pick the name). Run your script and verify that the shell gives you an error for using an undefined name. When this is true, comment out the line. Do not remove it. 4. Add another command line to your script file, below the lines you have already added. The output of this new command line must be the current working directory in which the script is running. Save and test your script after each line that you add. Many people find it useful to have one window open with the script in the editor and a second window open at a command prompt when you can to run the script and verify the output. 5. Add another command line to your script file, below the lines you have already added. The output of this new command line must be the letters of the current date in UPPER CASE, e.g.: TUE JAN 27 03:36:07 EST 2005 Hint: See the week05notes.txt section on "Translation" and the "tr" examples in Notes file "redirection.txt". 6. Add another command line to your script file, below the lines you have already added. The output of this new command line must be just the "time" portion (fourth field) of the date command: 03:46:07 Hint: See the the examples on selecting fields in week05notes.txt Note that on February 1, there will be two blanks between the month and the digit 1. Make sure your solution to extracting the time works no matter how many blanks separate the fields. (You cannot easily use the "cut" command to do this.) 7. Add another command line to your script file, below the lines you have already added. The output of this new command line must be a count of the total number of userids currently logged in to this machine (one number). Hint: Count just the lines of output of a common Unix command. Your count of lines may differ, depending on who is online when you run your script. 8. Add another command line to your script file, below the lines you have already added. The output of this new command line must be the checksum of the last five lines of the sorted Unix password file (two numbers will be output from the command doing the checksum). Note that "last five lines of a sorted file" is not the same as "last five lines of a file, sorted". Order matters. Use the checksum command mentioned in the unix_commands.txt file. 9. Add another command line to your script file, below the lines you have already added. Search the list of the last userids to login to this machine and find and output only the lines that contain the pattern "reboot". The lines look similar to this: reboot system boot Tue Jan 6 03:17 (21+00:42) 2.4.21-0.13mdk 10. Add another command line to your script file, below the lines you have already added. Preparation: The command that shows the listing of last logged in users has an option to display the host name in the last column; use this option for this next command pipeline. (RTFM) Search the list of the last userids to login to this machine and select only the lines that do *NOT* contain the pattern "reboot". In those lines (that do not contain "reboot"), select just the last field in each line (the host name) and display the top 20 most frequent login host names prefixed with the count of times that host name appears, sorted in descending order. The output will resemble this: 1531 pnat-atc.algonquincollege.com 581 aip65-50.algonquincollege.com 117 aip65-77.algonquincollege.com 106 pnat-rs.algonquincollege.com 102 mailhost.home.idallen.ca [...15 more lines...] Your output will differ from the above, depending on who actually logs in to the machine. To get the 20 lines of host name output requires a command pipeline containing about seven Unix commands. Build the pipeline slowly, verifying the output of each added command. Hint: See the "data mining" commands in Notes file week05notes.txt. Part II - weather.sh -------------------- 11. Preparation: ---------- Using lynx ---------- You can find the current weather for Ottawa at this URL: http://weatheroffice.ec.gc.ca/city/pages/on-118_metric_e.html There is a text-mode web browser available on Linux named "lynx" that can fetch any URL and display the formatted text results on standard output. It can be used interactively and also in scripts. Our use of the "lynx" command requires exactly two arguments. The first argument must be "-dump" to send output to standard output. The second argument is the URL we want to fetch, e.g. the URL for the weather for Ontario city code "on-118" (Ottawa). Try using the "lynx" command and above arguments to fetch the current weather page onto your screen to see what you get. You will need to pipe the output of lynx into a program that shows you the output of lynx one page at a time. Identify the line containing the current Ottawa temperature. (It is above the line saying "Pressure:".) ---------- Using grep ---------- With appropriate options, the grep command can print leading and trailing context lines that appear both before and after the lines found by the grep pattern. For example, we can look for the string "games" in the password file and output that line plus one line after it: $ grep --after-context=1 'games' /etc/passwd games:x:12:100:games:/usr/games:/bin/sh nobody:x:65534:65534:Nobody:/:/bin/sh You will need this option to output the Ottawa temperature value. 12. Create a second script file named "weather.sh". Set up the beginning of the script in a manner similar to your work in Steps 1, 2, and 3 of Part I, above. Make it executable. 13. Construct a command pipeline that will fetch, extract, and display the current temperature lines from the Ottawa weather page fetched by lynx. Put this command pipeline into the bottom of your new "weather.sh" shell script file. (My pipeline only needed two different commands - yours may differ.) Sample output from your working "weather.sh" script: $ ./weather.sh Temperature -14 °C Only the two related temperature lines must be output, nothing else. You may wish to experiment with the weather in other cities to make sure your script works for all of them. (Change the URL in your script to try any one of the cities on-1 through on-174, etc.) 14. Bonus: Modify your command pipelie to re-format the output so that it has no extra blanks and fits nicely on one single output line: $ ./weather.sh Temperature -14 °C (Hint: The versatile "tr" command also has options to delete characters and remove duplicate repeated characters. Although you can only do one or the other, not both, in a single invocation of tr, you can always chain together multiple tr commands using pipes.) Documentation ------------- On the lines above each of the steps in your scripts, add a few lines of comments (each line less than 80 characters!) explaining in your own words what the step that follows the comment does and what the expected output is. Shell script comments start with "#" and extend to the end of the line. For readability, the lines must not be longer than 80 characters. (Avoid putting comments to the right of commands.) Scripts without added comment lines will not be marked. Submission ---------- Reference: datsubmit.txt - Using the datsubmit command Submit the finished and labelled script files for marking as Exercise 04 on the Course Linux Server, using the following datsubmit command line: $ datsubmit 04 myscript4.sh weather.sh This "datsubmit" program will copy the files to me for marking. Always submit both files at the same time.