------------------------- Week 05 Notes for CST8165 ------------------------- -Ian! D. Allen - idallen@idallen.ca Remember - knowing how to find out an answer is more important than memorizing the answer. Learn to fish! RTFM! (Read The Fine Manual) ------------------- INDEX to this file: - current and previous assignments - review layers, UDP, TCP, encapsulation - control of the Internet: IP addresses, ports, and options - special IP address blocks - three ranges of TCP/UDP port numbers - MTU and "fragmentation considered harmful" - determining PMTU - improving congestion control - access to TCP/IP socket options from Unix/Linux - Application Protocols: RFC1123 "explains" earlier RFCs - SMTP RFC2821 and Internet Message Format RFC2822 - extending the HELO to EHLO - a basic SMTP session using telnet or nc - (aside) choosing the best buffer size in an Internet server ------------------- Current assignment: - lab1 programming comments: http://teaching.idallen.com/cst8165/06f/notes/lab01-comments.txt - lab3 readall_poor() vs. readall() - two server types: fork()ing server vs. concurrent via select() - readall() has internal state that only works for a single fd (i.e. it works in a fork()ing server but not a select() server) Review: - four layers Application, Transport, Network, Physical - TCP and UDP on "top" of IP (means packets go *inside* IP packets) - IP RFC791 is 45 pages - UDP RFC768 is 3 more pages on top of IP - TCP RFC793 is 95 more pages on top of IP - DCCP RFC4340 is 125 pages on top of IP Q: T/F packets get larger as they move down the protocol stack from Layer 4 (Application) down to the Physical media. - note the peculiar TCP/UDP pseudo-header for checksums - checksum includes the source and destination IP addresses! Q: T/F TCP and UDP include the IP packet source and destination addresses in their checksum calculations. - be familar with interpreting a TCP state diagram in RFC793 - three-way handshake for an asymmetric (non-simultaneous) open - trace a simultaneous open in RFC793 - the corrections suggested by RFC1122 break the simultaneous open! - RFC1122 section 4.2.2.10 says: "It sometimes surprises implementors that if two applications attempt to simultaneously connect to each other, only one connection is generated instead of two. This was an intentional design decision; don't try to "fix" it." Q: T/F When two systems attempt simultaneous connections with each other, you end up with two separate TCP streams. - TCP windowing - http://www.tcpipguide.com/free/t_TCPNonContiguousAcknowledgmentHandlingandSelective.htm - ACK bit says highest byte received is in the 32-bit ACK field - cumulative ACK says all previous bytes received OK - basic TCP/IP cannot issue out-of-sequence or selective ACKs (ACK ranges) - you can't say you got packets 1, 2, and 5 - "selective ACK (SACK)" capability was added later as a TCP Option - RFC1072/RFC2018 describe the TCP SACK option - TCP buffering is possible; use PSH to "push" data out at either end - interactive programs need to do this to get good response times Q: The TCP ACK field was sufficent to implement the "selective ACK" enhancment - No, the TCP ACK field can only acknowledge a single byte value; you need multiple byte ranges to implement SACK - these were added as TCP options New Material: ------------- * Who controls handing out the IP numbers and port numbers? - the Internet Corporation for Assigned Names and Numbers (ICANN) through its operating unit the Internet Assigned Numbers Authority (IANA) "Dedicated to preserving the central coordinating functions of the global Internet for the public good." ICANN: http://www.icann.org/ IANA: http://www.iana.org/ - IANA delegates to a few Regional Internet Registries (RIRs) to distribute the large blocks of IP addresses http://www.iana.org/ipaddress/ip-addresses.htm http://www.iana.org/assignments/ipv4-address-space - e.g. ARIN IP address list http://www.arin.net/ - special addresses (historical and current) http://www.rfc-editor.org/rfc/rfc3330.txt - note: RFC1918 private address space 10.0.0.0 - 10.255.255.255 (10/8 prefix) 172.16.0.0 - 172.31.255.255 (172.16/12 prefix) 192.168.0.0 - 192.168.255.255 (192.168/16 prefix) "the Internet does not inherently protect against abuse of these addresses; if you expect (for instance) that all packets from the 10.0.0.0/8 block originate within your subnet, all border routers should filter such packets that originate from elsewhere. Attacks have been mounted that depend on the unexpected use of some of these addresses." - IANA TCP/UDP port list (see RFC4340 for the three big divisions) http://www.iana.org/assignments/port-numbers - a shorter Unix/Linux specific copy of this file is kept in /etc/services Q: Who is the ultimate authority on IP addresses and ports? Q: Who is delegated to manage IP addresses in North America? Q: Give any two of the three RFC1918 private address space blocks and masks Q: T/F Special address block 0.0.0.0 is reserved for hosts on your local network. [RFC3330] Q: T/F IP address 0.0.0.0 is not a valid address. Q: What Unix/Linux file is used to turn "smtp" into "25" when you do $ nc -v localhost smtp * How does TCP send large amounts of data, if the wires won't? Linux command: ifconfig - shows MTU (Maximum Transmission Unit) size for each interface - raw Ethernet shows limit of 1,500 bytes MTU - other protocols will show other limits (e.g. PPP, PPPoE) Q: Give the MTU for your ethernet card (eth0) and the loopback (lo) - The IP Layer can split packets into "fragments" to pass them through routers that can't handle large packets. - IP packets also have a "Don't Fragment" bit that prevents fragmentation Fragmentation Considered Harmful - Google for this: "fragmentation considered harmful" - www.acm.org/sigs/sigcomm/ccr/archive/1995/jan95/ccr-9501-mogulf1.pdf - SIGCOMM October 1987 - 1. inefficient use of resources "Consider a TCP process that tries to send 1024 data bytes across a route that includes the ARPAnet, which has an MTU of 1006 bytes. The IP and TCP headers are at least 40 bytes long, leading to a total unfragmented IP datagram 1064 bytes in length. To cross the ARPAnet, this will be broken into a 1006 byte fragment, followed by a 78 byte fragment. These short fragments amortize the fixed overhead per ARPAnet packet over very few bytes of data, and the total packet count is much higher than needed. If the sending TCP instead chooses segments that fit in a 1006 byte ARPAnet packet, the total packet count is minimized, and the total overhead is as low as possible." - 2. degraded performance (in reassembly, fragment loss) "When segments are sent that are large enough to require fragmentation, the loss of any fragment requires the entire segment to be retransmitted. This can lead to poorer performance than would have been achieved by originally sending segments that didn't require fragmentation." - 3. lack of efficient reassembly - TCP windowing communicates well the size of receive queue/buffer - but IP has no indication of how many IP fragments are coming! - TCP can ACK the bytes received so far and ship the data up the stack - but TCP works on the *packet* level, not the *fragment* level - not possible to partially ACK an initial sequence of fragments - applications must cooperate with the IP layer in minimizing fragmentation Q: Why should IP fragmentation be avoided? Q: Of the four layers, which common layer is best suited to handling fragmentation avoidance? Q: T/F TCP can ACK each fragment of a packet as it arrives ---- First Midterm Test is on material up to this point ---- Path Maximum Transmission Unit discovery PMTU RFC1191 - November 1990 - 19 pages - ftp://ftp.rfc-editor.org/in-notes/rfc1191.txt "This memo describes a technique for dynamically discovering the maximum transmission unit (MTU) of an arbitrary internet path. It specifies a small change to the way routers generate one type of ICMP message. For a path that passes through a router that has not been so changed, this technique might not discover the correct Path MTU, but it will always choose a Path MTU as accurate as, and in many cases more accurate than, the Path MTU that would be chosen by current practice." "In this memo, we describe a technique for using the Don't Fragment (DF) bit in the IP header to dynamically discover the PMTU of a path. The basic idea is that a source host initially assumes that the PMTU of a path is the (known) MTU of its first hop, and sends all datagrams on that path with the DF bit set. If any of the datagrams are too large to be forwarded without fragmentation by some router along the path, that router will discard them and return ICMP Destination Unreachable messages with a code meaning "fragmentation needed and DF set" [7]. Upon receipt of such a message (henceforth called a "Datagram Too Big" message), the source host reduces its assumed PMTU for the path." "Unfortunately, the Datagram Too Big message, as currently specified, does not report the MTU of the hop for which the rejected datagram was too big, so the source host cannot tell exactly how much to reduce its assumed PMTU. To remedy this, we propose that a currently unused header field in the Datagram Too Big message be used to report the MTU of the constricting hop. This is the only change specified for routers in support of PMTU Discovery." Q: How does IP PMTU discovery work? Q: What changes were made to the ICMP "Datagram Too Big" message to accommodate PMTU? * Congestion Control Even if packets aren't fragmented, routers can be come congested if too many packets arrive to process. When TCP originated, the only indication that a router is overloaded came when packets started to drop - you couldn't get any advance warning. The Addition of Explicit Congestion Notification (ECN) to IP RFC3168 - September 2001 - 63 pages - ftp://ftp.rfc-editor.org/in-notes/rfc3168.txt - the Introduction paragraphs (Section 1.) are important "Since TCP determines the appropriate congestion window to use by gradually increasing the window size until it experiences a dropped packet, this causes the queues at the bottleneck router to build up. With most packet drop policies at the router that are not sensitive to the load placed by each individual flow (e.g., tail-drop on queue overflow), this means that some of the packets of latency-sensitive flows may be dropped. In addition, such drop policies lead to synchronization of loss across multiple flows." - vanilla TCP minimizes effect of congestion on *throughput*, not *latency* - but, the mechanism for detecting congestion is lost packets - no mechanism for avoiding lost packets in the first place "Active queue management mechanisms detect congestion before the queue overflows, and provide an indication of this congestion to the end nodes. Thus, active queue management can reduce unnecessary queuing delay for all traffic sharing that queue." Q: T/F Traditional "drop packet" TCP congestion control mechanisms are designed to keep overall throughput high Q: T/F Traditional "drop packet" TCP congestion control mechanisms also keep packet latency to a minimum Q: What advantage does ECN have over traditional "drop-packet" methods for detecting and avoiding congestion? Datagram Congestion Control Protocol DCCP RFC4340 - NEW! March 2006 - 125 pages "The Datagram Congestion Control Protocol (DCCP) is a transport protocol that implements bidirectional, unicast connections of congestion-controlled, unreliable datagrams." - this RFC also contains this important port allocation information: "Port numbers are divided into three ranges. The Well Known Ports are those from 0 through 1023, the Registered Ports are those from 1024 through 49151, and the Dynamic and/or Private Ports are those from 49152 through 65535. Well Known and Registered Ports are intended for use by server applications that desire a default contact point on a system. On most systems, Well Known Ports can only be used by system (or root) processes or by programs executed by privileged users, while Registered Ports can be used by ordinary user processes or programs executed by ordinary users. Dynamic and/or Private Ports are intended for temporary use, including client-side ports, out-of- band negotiated ports, and application testing prior to registration of a dedicated port; they MUST NOT be registered." Q: What range of ports should your experimental application use? * What control do you have over the IP layer from Unix/Linux? - "man 7 socket" setsockopt(2) and getsockopt(2) - SO_KEEPALIVE - SO_RCVTIMEO SO_SNDTIMEO - SO_BINDTODEVICE - SO_REUSEADDR - SO_DONTROUTE - SO_BROADCAST - SO_LINGER - SO_PRIORITY Q: What function calls are available to C programmers to set options on sockets? Give two examples of the kind of options you can set. Interpreting the RFC documents and the raw protocols ---------------------------------------------------- * The "Requirements for Internet Hosts" documents: RFC 1122 and 1123 - 1122 and 1123 are clarifications and examples of how the RFCs work * The overview discussion document: RFC1127 "This group of people struggled with a broad range of issues in host implementations of the Internet protocols, attempting to reconcile theoretical and architectural concerns with the sometimes conflicting imperatives of the real world. The present RFC recaps the results of this struggle, with the issues that were settled and those that remain for future work." "Indeed, many of these are simply restatements or reinforcement of requirements that are already explicit or implicit in the original standards RFC's. Some more cynical members of the working group refer to these as "Read The Manual" provisions. However, they were included in the HR RFCs because at least one implementation has failed to abide by these requirements. In addition, many provisions of the HR RFCs are simply applications of Jon Postel's Robustness Principle [1.2.2 in either RFC]." Q: T/F The "Requirements for Internet Hosts" documents were written to extend the existing RFCs with new features. Application Protocols --------------------- * RFC1123 - Requirements for Internet Hosts - Application and Support - ftp://ftp.rfc-editor.org/in-notes/rfc1123.txt - RFC1123 reviews and clarifies many major protocols and standards: - TELNET, FTP, TFTP, SMTP, RFC822 (message format), DNS "This RFC enumerates standard protocols that a host connected to the Internet must use, and it incorporates by reference the RFCs and other documents describing the current specifications for these protocols. It corrects errors in the referenced documents and adds additional discussion and guidance for an implementor." "A good-faith implementation of the protocols that was produced after careful reading of the RFC's and with some interaction with the Internet technical community, and that followed good communications software engineering practices, should differ from the requirements of this document in only minor ways. Thus, in many cases, the "requirements" in this RFC are already stated or implied in the standard protocol documents, so that their inclusion here is, in a sense, redundant. However, they were included because some past implementation has made the wrong choice, causing problems of interoperability, performance, and/or robustness." * Sending electronic mail: SMTP - Note: The protocol and ports used to send email (SMTP) are completely separate from the ports and protocols used to fetch email (POP3, IMAP)! SMTP - Simple Mail Transfer Protocol - RFC821 -> RFC2821 - April 2001 - 79 pages on top of TCP (95 pages) on top of IP (45 pages) - a "PUSH" protocol - sender initiates (HTTP is "PULL" protocol) - ftp://ftp.rfc-editor.org/in-notes/rfc2821.txt "This document is a self-contained specification of the basic protocol for the Internet electronic mail transport. It consolidates, updates and clarifies, but doesn't add new or change existing functionality of the following: RFC822, DNS, RFC1123" - did not add or change RFC821; dropped obsolete items Q: T/F RFC2821 replaced RFC821 and added new SMTP functionality - the SMTP *protocol* does not define the format of the *message* - the *message* delivered by the *protocol* has its own description: RFC822 -> RFC2822 "Internet Message Format" (51 pages) ftp://ftp.rfc-editor.org/in-notes/rfc2822.txt - the content of the message (including To/From message header lines) is independent of the To/From used in the SMTP protocol! Q: T/F The SMTP protocol RFC defines the format and headers of an email message * Extending the original SMTP protocol "HELO" with "EHLO" - http://www.rfc-editor.org/rfc/rfc1869.txt - orignal SMTP had no protocol version number - no way to negotiate options or features - standard HELO vs. extended EHLO (awkward way to do protocol versioning) - SMTP extensions (must be registered with IANA) * SMTP is a readable ASCII protocol on top of TCP - not binary! - you can run it using "nc" or telnet to port 25 - but you can't do it here at Algonquin College! - port 25 blocked leaving the College (must use College servers) - College servers implement long wait times before answering - to discourage spam programs that don't wait as long - SMTP wait times are documented in ftp://ftp.rfc-editor.org/in-notes/rfc1122.txt "Timeouts are an essential feature of an SMTP implementation. If the timeouts are too long (or worse, there are no timeouts), Internet communication failures or software bugs in receiver-SMTP programs can tie up SMTP processes indefinitely. If the timeouts are too short, resources will be wasted with attempts that time out part way through message delivery." * a sample SMTP session: https://helpdesk.islandnet.com/pep/smtp.php - this uses obsolete HELO greeting - should use EHLO $ nc -v mail.islandnet.com 25 (or use telnet) 220 Islandnet.com ESMTP server ready helo a.b.c 250 mail.islandnet.com Hello x [YOUR_IP_ADDRESS] mail from: 250 is syntactically correct rcpt to: 250 verified data 354 Enter message, ending with "." on a line by itself From: Bugs Bunny To: Daffy Duck Subject: Loony Toons! Hi there! . 250 OK id=1778te-0009TT-00 quit 221 mail.islandnet.com closing connection - HELO vs. EHLO: ftp://ftp.rfc-editor.org/in-notes/rfc1122.txt "Contemporary SMTP implementations MUST support the basic extension mechanisms. For instance, servers MUST support the EHLO command even if they do not implement any specific extensions and clients SHOULD preferentially utilize EHLO rather than HELO. " - response to EHLO: http://www.rfc-editor.org/rfc/rfc1869.txt "Normally, this response will be a multiline reply. Each line of the response contains a keyword and, optionally, one or more parameters." - every line but the last of a multi-line response contains a "-" immediately following the response number, e.g. $ nc -v localhost smtp localhost.home.idallen.ca [127.0.0.1] 25 (smtp) open 220 elm.home.idallen.ca ESMTP Postfix (idallen@idallen.ca) EHLO idallen.ca 250-elm.home.idallen.ca 250-PIPELINING 250-SIZE 10240000 250-VRFY 250-ETRN 250-STARTTLS 250 8BITMIME - the response indicates what optional features this email server offers Aside: On choosing buffer sizes ------------------------------- When deciding how much buffer space an Internet server should allow for incoming request lines, you have to weigh memory use against functionality. Here's an excerpt from an RFC extending the SMTP protocol, which originally specified a maximum buffer of just 512 bytes: http://www.rfc-editor.org/rfc/rfc1869.txt 4.1.2. Maximum command line length This specification extends the SMTP MAIL FROM and RCPT TO to allow additional parameters and parameter values. It is possible that the MAIL FROM and RCPT TO lines that result will exceed the 512 character limit on command line length imposed by RFC 821. This limit is hereby amended to only apply to command lines without any parameters. Each specification that defines new MAIL FROM or RCPT TO parameters must also specify maximum parameter value lengths for each parameter so that implementors of some set of extensions know how much buffer space must be allocated. The maximum command length that must be supported by an SMTP implementation with extensions is 512 plus the sum of all the maximum parameter lengths for all the extensions supported.