----------------------- Lab #03 for CST8165 due February 5, 2007 (*NEW DATE*) ----------------------- -Ian! D. Allen - idallen@idallen.ca Remember - knowing how to find out an answer is more important than memorizing the answer. Learn to fish! RTFM! (Read The Fine Manual) Global weight: 5% of your total mark this term. Due date: before 10h00 AM Monday February 5 (*NEW*) Interim submission: at the end of your Week 5 lab period. (*NEW*) Bonus: a bonus will be granted for submissions arriving by 10AM Jan 29. (*NEW*) Interim submissions in Week 5: (*NEW*) You will submit whatever progress you have made on this assignment before the end of your weekly lab period this week (Week 5). The deliverables for this exercise are to be submitted online in the Linux Lab T127 using the "cstsubmit" method described in the exercise description, below. No paper; no email; no FTP. Late-submission date: I will accept without penalty exercises that are submitted late but before 12h00 (noon) on Tuesday, February 6. (*NEW*) After that late-submission date, the exercise is worth zero marks. Exercises submitted by the *due date* will be marked online and your marks will be sent to you by email after the late-submission date. Exercise Synopsis: Enhance your Lab #2 server code. Test it. Enhance an existing TCP/IP client. Test it. Where to work: Submissions must make, compile, and run cleanly in the T127 Linux Lab, though you are free to work on them anywhere you like. Helpful code: ------------ a. printf size You can printf exactly 9 bytes from a buffer (no \0 needed) using: printf("%.9s",buf); // print only 9 bytes from buf AND it gets better if you use '*' instead of 9 (more useful in this case): n = read(fd,buf, .... ); ... printf("%.*s",n,buf); // the "*" means pick up the current value of "n" This kind of printf can be useful for buffers that don't have \0 in them. You can also output n bytes in a buffer directly using write(fd,buf,n). - Standard input (usually your keyboard) is Unix fd 0. - Standard output (usually your screen) is Unix fd 1. - Standard error (usually your screen) is Unix fd 2. b. buffer size Never do this: char buf[256]; ... read(fd,buf,256); Do this: char buf[256]; ... read(fd,buf,sizeof(buf)) Buffer sizes must be set in only *one* place for easy maintenance. c. equivalence of read() and recv(), write() and send() for sockets If you don't set any special TCP/IP flags in recv() or send(), the system calls recv() and read() are the same/equivalent for accessing sockets, as are the syscalls send() and write(). You can't use the socket syscalls recv() or send() on file descriptors that are *not* sockets (even if the TCP/IP flags are zero); using read() and write() works for both sockets and ordinary files. All of the code in this lab uses plain read() and write(). See also: "man 2 recv" and "man 2 send" Coding the TCP/IP server: ------------------------ Reference: http://www.cs.rpi.edu/courses/sysprog/sockets/sock.html The server2.c code is explained line-by-line in the above web page. 1) Fix your Lab 2 server2.c to accept only *exactly* one port argument, not more, not less. Never ignore user input. Do not run the program if too many or too few arguments are given. 2) Fix your Lab 2 server2.c to convert INADDR_ANY to network byte order. Reference: http://beej.us/guide/bgnet/output/htmlsingle/bgnet.html#bind Read the highlighted paragraphs above and below the one containing: "you might have seen that I didn't put INADDR_ANY into Network Byte Order! Naughty me." Note the change in the line "// use my IP address". 3) Read the Notes file programming_style.txt 4) Fix the inconsistent indentation in all your source files. One quick way to do this is the command "indent -kr -i8 *.c"; but, you may want to add more options to treat comments differently. Review: http://lxr.linux.no/source/Documentation/CodingStyle 5) Using the given examples, add internal comments and proper file and function header comments to all the source files and functions. (The Assignment Label is not a programming header comment.) You must fix or add comments to all the code, even if you didn't write the code yourself. (If you use code from other people, you have to bring the coding style up to the standard of your company.) The function headers are suggested minimums. If you have a header that provides more detail, you may use it. 6) Improve the error messages to have the four-part format described in Notes file programming_style.txt. *Every* error message must be output on standard error and be prefixed by the actual name of the program. Do not hard-code the program name into the code; use the actual name. If you rename the binary, the error message name must also change. (Recall that the program name is given to main() in argv[0]. How can you make this string globally available to all your code, including myerror()? Yes, you might reasonably use a global variable for this, since the value will be set once at program start and never changed.) If you want to use [f]printf() before a call to perror(), read the NOTES section of "man 3 errno" first! Rather than using [f]printf(), you could use snprintf() to construct the string you wish to pass to perror(). (Why is using [f]printf() before perror() not allowed; but, using snprintf() before perror() is allowed?) Do you know how to write va_list (variadic, varargs) functions? You might find such a function useful for printing error messages; since, it avoids the need for buffers and snprintf(). http://www.gnu.org/software/libc/manual/html_node/Variadic-Functions.html 7) Replace the deprecated bzero() function everywhere. (See "man bzero".) 8) Rewrite the dostuff() function incrementally as follows: a. Recode dostuff() to eliminate and remove the need for bzero/memset() Zeroing out the entire buffer is unnecessary and wasteful. Recode it. (Hint: use the printf and sizeof code from above) Test your code using netcat ("nc") to make sure it works. b. Replace the 18-byte write() statement by writing all the bytes received from the client back to the client (to the socket). Only write the bytes that were actually read from the client. Test your code to make sure it works - one line in, one line out. c. Recode dostuff() to loop reading/writing lines from/to a connected client until EOF (zero bytes read) or error. Test your code to make sure it works. d. Issue the following message on standard error when EOF is seen: DEBUG %s EOF reading from unit %d where %s is the program name (which main() can get from argv[0]), and %d is the I/O descriptor number on which EOF is detected e. Change the "Here is the message" printf statement into a DEBUG statement that prints (only) the bytes that were read. Don't print zero bytes or negative bytes read. Make sure the code continues to test for errors on all system calls. 9) When writing to a network socket, the write() may write fewer bytes than you ask it to write. Reference: http://beej.us/guide/bgnet/output/htmlsingle/bgnet.html#sendrecv In server2.c, replace the write() with a function sendall() that loops sending bytes until either all the bytes are sent, or error. You may want to start with this code: http://beej.us/guide/bgnet/output/htmlsingle/bgnet.html#sendall If you copy (with credit) that sendall() function, fix the initialization bug that happens if *len is zero. If you want to use sendall() to write to a file descriptor that is not a socket (e.g. your screen), you must ensure that sendall() uses write() and not send(). Both send() and recv() only work on sockets; write() and read() work on both sockets and ordinary file descriptors. Put sendall() in its own file sendall.c and create sendall.h 10) Update your Makefile to include sendall.c and sendall.h Make sure that your server2 is rebuilt if any of its source files change. Coding a TCP/IP client ---------------------- Reference: http://www.cs.rpi.edu/courses/sysprog/sockets/sock.html The client.c code is explained line-by-line in the above web page. 11) Using wget, download this TCP/IP client code named "client.c": http://www.cs.rpi.edu/courses/sysprog/sockets/client.c 12) Add the appropriate target "client" to your Makefile. Make sure you can rebuild the client with all these required CFLAGS: CFLAGS = -g -O -Wall -Wextra -Wshadow -Wstrict-prototypes \ -Wmissing-prototypes -Wmissing-declarations \ -Wdeclaration-after-statement \ -Wmissing-field-initializers -Wredundant-decls -Wunreachable-code 13) Just as you did with the server, fix the compilation errors and warnings. You will need to add one missing function argument cast. Also: Replace the deprecated bzero() and bcopy() functions. 14) Fix the code to accept only *exactly* one host and one port argument, not more, not less. Never ignore user input. Do not run the program if too many or too few arguments are given. Also: The client.c code intermixes argument parsing and handling with the network code (e.g. socket() is called before validating the argv[1] server name with gethostbyname(), and the argv[2] port argument isn't validated at all). The code would have better structure if all the argument parsing and validation were done before using the arguments, as much as possible. Any errors in arguments should result in a non-zero exit() code. 15) Upgrade the error messages to the standard four-part format documented in Notes file programming_style.txt. Make sure that the error messages in code that uses command line input contain what the user typed. Error messages should echo what the user entered, if what the user entered is relevant to the error. (An error in calling socket() does not depend on anything entered by the user.) Note: For extra arguments present on the command line, rather than echoing them all to the user in an error message (which would require coding a loop), compromise and simply tell the user how many incorrect arguments were given. 16) Modify the code to use your own external myerror() function. (Remove the existing error() function from the client.) Make sure that your client is rebuilt by the Makefile if any of its source files change. 17) Test the client against your server. You can also test your client against a netcat ("nc") server that you start up like this: $ nc -v -l localhost 55555 # server on Red Hat systems $ nc -v -l -q 9 -p 55555 localhost # server on Debian systems In another window, connect your client to the netcat server: $ ./client localhost 55555 Make sure the client can send one line to the server. 18) Recode the client to remove the need to bzero/memset() the buffers used for read/write I/O. Zeroing the entire buffer is wasteful and not necessary if we know how much of the buffer is in use. Since fgets() doesn't tell you how many characters were read, get rid of fgets() and use a plain "read()" system call to read from standard input (from Unix fd unit 0). (Recall: The server dostuff() function also uses a plain read() syscall to read from a socket.) 19) Fix the prompt. Make sure the prompt for input appears on standard error, not standard output. (Prompts should always be sent to standard error so that they don't get redirected into output files.) You may need to call fflush() with the correct stdio output descriptor to flush any buffered stdio prompt output before you call read() to read from standard input; otherwise, the prompt for input may not appear. (If you use the syscall write() to print the prompt, this won't be a problem. Buffering only happens using stdio.) Use isatty() ("man 3 isatty") to avoid printing a prompt for input if standard input is not a terminal (is not a keyboard). Prompts should only be printed if a human is required to enter input. 20) This client.c as written fails to detect EOF or error on standard input. Fix it. Make sure both EOF (zero bytes from read()) and error are handled by the code: On EOF from read(), print this debug message on standard error: DEBUG %s EOF reading from standard input where %s is the program name (which main() can get from argv[0]). On error from read() on standard input, print a useful error message. 21) When the client is done writing the server socket, it needs to signal EOF on the socket to the server. It can't close() the socket, since closing it would prevent the client from reading the server's response from the socket later. Also, in a multi-process forking client, having only one process close the socket would still leave the socket open in the other process and the server would not see EOF until *both* processes closed the socket. The client needs a way to close only the *writing* part of the socket, signalling EOF to the server. To do this, use the special syscall "shutdown(sockfd,SHUT_WR)". The time to call shutdown() is when the client is done writing to the socket. That happens on EOF or error on standard input (no bytes are available to write to the socket), or after writing the one line to the socket. Use shutdown() to close the writing half of the connected server socket after EOF or error on standard input, or after writing the line to the socket. The shutdown() will cause the server to see EOF (zero bytes read) from the client. The shutdown() function is explained in Notes file eof_handling.txt Make sure that your server gets EOF from the client call to shutdown() *before* the client exits. Test this! For debugging, you can put a sleep(30) after shutdown(), to make sure that the server sees EOF (and prints a DEBUG message saying it sees EOF) as soon as the shutdown() happens. Note: Since you must now always call shutdown() even after an EOF or error, you can't continue to use myerror() to handle I/O errors, since myerror() exits the program and doesn't call shutdown(). You must handle the errors using another function that calls perror() without exiting; or, you must recode myerror() not to exit (and then you must add exit() statements after all the places in the code where myerror() should exit). 22) If the client read any bytes from standard input, write those bytes to the open server socket. (Error tests should already be done as specified above.) 23) Writing to a network socket may not write all the bytes. Convert the write() that sends bytes to the server socket to use your sendall() function, just as you did in the server. Make sure that your client is rebuilt by the Makefile if any of its source files change. 24) The client currently reads the server socket and checks for errors. If the client sees EOF on the server socket (zero bytes read), print this debug message on standard error: DEBUG %s EOF reading from server socket %d where %s is the program name and %d is the socket fd unit number. 25) If the client read any bytes from the server socket, write those bytes onto standard output. Don't write zero or negative bytes! 26) Add a fork() to create two separate processes to handle simultaneously reading standard input and reading the socket. The parent process should have the prompt and all the code that reads standard input and writes the socket. The child process should have all the code that reads the socket and writes standard output. At the normal exit of the parent process, write this debug message: DEBUG %s parent normal exit where %s is the program name. Write a similar debug message before the normal exit of the child process. Debug messages should always be on standard error. DEBUG %s child normal exit 27) Test your processes: Test #1: Start your server. Start your client. Kill the server. The client child process should say: DEBUG ./client EOF reading from server socket 3 DEBUG ./client child normal exit The client parent process should still be reading (one line) from the keyboard. The parent will exit when you push RETURN, possibly printing an error message about "broken pipe" writing to the nonexistent server process. (Depending on how signals are being handled in the client, the error caused by writing to the nonexistent server process may trigger a SIGPIPE signal that kills the client process rather than returning an EPIPE syscall error.) Test #2: Start your server. Start your client. Enter one line of input to the client. The line should go to the server and back to the client, appearing on your screen. Both the client parent and child processes will exit with debug messages: DEBUG ./client parent normal exit DEBUG ./client child normal exit Test #3: Start your server. Start your client. Signal EOF from the keyboard. You will see these messages from the client (perhaps not in this order): DEBUG ./client EOF reading from standard input DEBUG ./client parent normal exit DEBUG ./client EOF reading from server socket 3 DEBUG ./client child normal exit 28) Make each of the two processes loop reading/writing until EOF or error: The client parent process reading standard input should stop looping and exit on EOF or error from standard input, or if writing the server socket fails. The client child process reading the server socket should stop looping and exit on EOF or error from the server socket, or if writing standard output fails. The client parent process reading standard input and the client child process reading the server socket are largely independent. The exception to the independence of the two processes is this: 29) Before the client child process (reading the server socket) exits for any reason, it should kill (with SIGHUP) the client parent process that is reading standard input. The reason is: if the client child process is exiting and will not be printing anything more from the server, then nothing more should be sent to the server either. The client parent process won't be able to send any more input to the server, and both client processes should exit; so, the child has to kill() the parent that is blocked reading standard input. See: "man 2 kill" You can get the pid (process ID) of the current (parent) process before you fork() using getpid(). See: "man 2 getpid" Note: Since your client child process must now always call kill() before it exits, even after an error, you can't continue to use myerror() to handle I/O errors, since myerror() exits the program and doesn't call kill(). You must handle the errors using another function that calls perror() without exiting; or, you must recode myerror() not to exit (and then you must add exit() statements after all the places in the code where myerror() should exit). Challenge: Write your code so that the parent and child status of the two processes are interchangeable; that is, after fork() it doesn't matter which process is coded to be the child/parent. You will need to add a little bit of logic before the kill() so that kill() always kills the "other" process, no matter whether it is the parent or the child process. (Hint: The process doing the kill() has to kill the child pid if it is coded to be the parent process, or kill the parent pid if it is coded to be the child process.) 30) Test that the kill with SIGHUP works before the client child exits: Test #4: Start your server. Start your client. Kill the server. The client should say: DEBUG ./client EOF reading from server socket 3 DEBUG ./client child normal exit and the shell should report a "Hangup" signal from the client as the child process calls kill() on the parent and exits, and your shell prompt should apper. Both client processes should be gone. Note: You can't continue to use myerror() to handle errors after forking in either the child or the parent processes, since myerror() exits the program. Both the client child and parent have things that must be done before exiting. You must handle the errors using another function that calls perror() without exiting; or, you must recode myerror() not to exit (and then you must add exit() statements after all the places in the code where myerror() should exit). 31) The client as written uses a buffered printf() from stdio to print what it reads from the socket onto standard output, with no error checking. Replace the printf() with your sendall() function, and add error checking after sendall() to make sure it worked. Remember: You cannot immediatly exit the program on error in either the parent or child process. You must print the error, break the loop, and do some cleanup before the process exits. Reminder: standard input is fd unit 0 for the Unix I/O syscalls. Reminder: standard output is fd unit 1 for the Unix I/O syscalls. If you want to use sendall() to write to a file descriptor that is not a socket, you must ensure that sendall() uses write() and not send(). 32) Notice that your parent and child processes contain almost exactly the same code. They both loop reading from one fd and writing to another fd. They both check for EOF and errors. The only differences in the code are (1) whether or not to prompt for input and (2) whether to call shutdown() or kill() before exiting. Can you combine most of the parent and child code into one single looping function, where you pass in the input fd and the output fd? (Hints: Most of the code for child and parent is the same and can be parametrized to read from one fd and write to another. You only prompt for input if the fd you read from is a terminal; this code is harmless if applied to a socket that is not standard input. In your common function, you will need to change the error message text to refer to file descriptor numbers and not to "standard input".) Documentation ------------- 33) Document the code. Did you bring all source files up to programming and comment standards? Are comments relevant? Is indentation consistent? Reference: Notes file programming_style.txt Note: I haven't asked for any separate User Manual for using the client or server2 programs. Include brief syntax/usage information (how to run these programs) in the comments at the top of server2.c and client.c source files. Official Testing ---------------- These are the official tests for handing in. Make sure your script session contains *only* the required commands below and their output - do not flood the script session with VIM sessions or other unneccessary interactions. Remember that you can throw away unwanted DEBUG output by shell redirection of standard output and/or standard error to /dev/null. The script command has an option to append to a script file, if you want to run the testing below in separate sections. 34) Start a "script" session (man script) with output file "testing.txt". 35) Test the Makefile. Run a full clean and recompile test: $ make clean $ make clean # second time should not produce any errors $ make server2 client $ make server2 client # second time should say everything is up to date Use the exact four tests above, in the above order. Make sure you rebuild with all these required CFLAGS: CFLAGS = -g -O -Wall -Wextra -Wshadow -Wstrict-prototypes \ -Wmissing-prototypes -Wmissing-declarations \ -Wdeclaration-after-statement \ -Wmissing-field-initializers -Wredundant-decls -Wunreachable-code 36) Verify that your server2 works using the standard TCP client netcat: Note: If your server produces many lines of debugging output, you must turn that off (or redirect server output to /dev/null) before running this test, so that you don't flood the script session with hundreds of lines of unnecessary DEBUG output. Redirect the input of netcat from a large text file, and write the standard output to a file. Compare the input and output to make sure they are identical (after having passed through the server): $ ./server2 55555 & # WARNING! don't produce high-volume DEBUG output $ nc localhost 55555 out # double I/O redirection $ diff /etc/termcap out # should be identical $ sum /etc/termcap out 63422 786 /etc/termcap 63422 786 out $ killall server2 37) Verify that your client works using the standard TCP server netcat. Since your client is two processes, test each direction separately: a) Send data (one-way) from client to fake netcat server: Start a netcat server and redirect its output into a file. Start and redirect the input of your client from a large text file. Compare the input and output to make sure they are identical: $ nc -v -l localhost 55555 >out & # server on Red Hat systems $ nc -v -l -q 9 -p 55555 localhost >out & # server on Debian systems $ ./client localhost 55555 out $ diff /etc/termcap out # should be identical $ sum /etc/termcap out 63422 786 /etc/termcap 63422 786 out b) Send data (one-way) from fake netcat server to client: Start a netcat server and redirect its intput from a large text file. Start and redirect the output of your client into a file. Compare the input and output to make sure they are identical: $ nc -v -l localhost 55555 out Please enter the message: DEBUG ./client EOF reading from unit 3 Hangup DEBUG ./client child normal exit [1]+ Done nc -v -l localhost 55555 out # double I/O redirection DEBUG ./client EOF reading from unit 0 DEBUG ./client parent normal exit DEBUG ./client EOF reading from unit 3 DEBUG ./client child normal exit $ diff /etc/termcap out # should be identical $ sum /etc/termcap out 63422 786 /etc/termcap 63422 786 out $ killall server2 39) Verify your client against your server using a binary file: Note1: If your server produces many lines of debugging output, you must turn that off (or redirect server output to /dev/null) before running this test, so that you don't flood the script session with hundreds of lines of unprintable binary DEBUG output. Note2: To handle binary data, your client and server must read and write data using either the basic Unix syscalls read()/write(); or, you must be using the binary I/O functions from standard I/O named fread() and fwrite(). You can *not* use fgets(), fputs(), fprintf() or printf() with binary data, since they stop at NUL bytes. Start your server. Run your client with input redirected from a large binary file and output redirected into a file. Compare the files: $ ./server2 55555 & # WARNING! don't produce high-volume DEBUG output $ ./client localhost 55555 out # double I/O redirection DEBUG ./client EOF reading from unit 0 DEBUG ./client parent normal exit DEBUG ./client EOF reading from unit 3 DEBUG ./client child normal exit $ cmp /bin/bash out # use cmp for binary files $ sum /bin/bash out 09626 704 /bin/bash 09626 704 out $ killall server2 40) Exit the script session subshell. Script will tell you the name of your testing.txt script output file. Verify that your output file contains *only* the above commands. No VIM sessions. Not hundreds of lines of DEBUG output. 41) In a new file named "README.txt", summarize briefly the results of your testing. For each of the 40 points in this assignment, state whether you succeeded or failed, e.g. you might write this: 1-12. done 13. could not get rid of htonl and htons warnings (harmless) 14-34. done 35. "make clean" still has errors 36-40. all tests passed, script output is accurate 42) In the README.txt file, under a heading: Feedback to instructor: Tell me how easy/difficult this assignment was. How long did it take to complete? 43) Add assignment headers to the top of all your files and submit the files (see below). Add a header to *every* file you submit. Verify that your testing.txt file contains *only* the above commands. No VIM sessions. Not hundreds of lines of DEBUG output. Submission ---------- Note: The Assignment Label is not a substitute for a proper program file header giving the purpose of this program code. Add a proper file header to all program source code submitted. A. At the top of each and every submitted file, as comments, create an Exterior Assignment Submission label following the directions from last week's lab. Add this header to *every* file you submit. B. For material you copy from other sources, credit the author and source, following the directions from last week's lab. C. Submit these text files for marking as Exercise 03 using the following exact and *single* cstsubmit command line: $ ~alleni/bin/cstsubmit 03 Makefile README.txt testing.txt \ server2.c client.c myerror.c myerror.h sendall.c sendall.h See last week's lab for details on using cstsubmit. All file names must be spelled *exactly* as given above. Incorrect submissions are worth zero marks. If you use any automated test scripts, you may submit those on the same command line. (You can safely submit extra files, as long as you always submit the required files at the same time.) P.S. Did you spell all the assignment label fields and file names correctly?