[Beowulf] Measurement of TCP traffic between nodes
Many of your questions may have already been answered in earlier discussions or in the FAQ. The search results page will indicate current discussions as well as past list serves, articles, and papers.
Robert G. Brown rgb at phy.duke.eduWed Mar 23 12:09:01 PST 2005
- Previous message: [Beowulf] Measurement of TCP traffic between nodes
- Next message: [Beowulf] Re: Grid Engine, Parallel Environment, Scheduling, Myrinet, and MPICH
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
On Wed, 23 Mar 2005, Mikhail Kuzminsky wrote: > > Dear colleagues, > > I want to measure message sizes (and "frequencies" of messages > exchanges) between cluster nodes under parallelization of > some applications. These applications use parallelization tools > working over TCP (Linda, DDI), so I need to know the size of TCP > packages and the number of TCP packages per second. > > 1) There is no other traffic on the corresponding interfaces, > and (at least for 2-nodes "subcluster"; it's used for simplicity) I > may use simple ifconfig output (in a loop). Do I understand correctly, > that > the number of bytes transmitted at "application level" is > something about (for example for transmission) > Tx_bytes - TX_packets*40 ? > (i.e. a) there is no bytes from ethernet headers - in ifconfig output > b) the number of application transmitted bytes per TCP packet = > > size_of_packet - 40 (in average)) I'm not sure that this is a safe assumption. Let me try to analyze this briefly, although other folks will need to check me as it is easy to get confused here. * Don't use ifconfig in a loop. ifconfig's data comes from /proc/net/dev in columns. Easier to parse, and lower overhead: rgb at ganesh|B:1007>cat /proc/net/dev > /tmp/dev (contents of /tmp/dev) Inter-| Receive | Transmit face |bytes packets errs drop fifo frame compressed multicast|bytes packets errs drop fifo colls carrier compressed lo: 2487503 8119 0 0 0 0 0 0 2487503 8119 0 0 0 0 0 0 sit0: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 usb0: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 eth0:2244262754 11251382 0 0 0 0 0 0 1079809175 2321952 0 0 0 0 0 0 eth1: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 * xmlsysd will return this information wrapped in xml tags if you'd rather parse it that way. * From RFC 791, the bitwise layout of an IP header is: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |Version| IHL |Type of Service| Total Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Identification |Flags| Fragment Offset | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Time to Live | Protocol | Header Checksum | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Source Address | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Destination Address | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Options | Padding | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ or 24 bytes. * From RFC 768, bitwise layout of UDP header is: 0 7 8 15 16 23 24 31 +--------+--------+--------+--------+ | Source | Destination | | Port | Port | +--------+--------+--------+--------+ | | | | Length | Checksum | +--------+--------+--------+--------+ | | data octets ... +---------------- ... or 8 more bytes. * From RFC 793, the bitwise layout of a TCP header is: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Source Port | Destination Port | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Sequence Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Acknowledgment Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Data | |U|A|P|R|S|F| | | Offset| Reserved |R|C|S|S|Y|I| Window | | | |G|K|H|T|N|N| | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Checksum | Urgent Pointer | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Options | Padding | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | data | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ or 24 bytes. So there are 48 bytes in IP+TCP plus 14 in the ethernet header (plus 8 in the pre-header plus 4 in the trailing CRC that are usually ignored). Thus the smallest non-empty TCP message is 48+14 = 62+1 = 63 bytes, IIRC. For single messages that aren't part of a stream, this header cost will be preserved per message, so you'll want to count 48 or 62 bytes. For streaming messages the second and subsequent packages have abbreviated TCP headers but I cannot remember their length and don't have time to look it up (but the reference RFCs are indicated, I think). > 2) Are the tools like tcpdump better for this task (pls > take into account that I need only timestamps and data sizes) ? tcpdump would let you see each packet's headers, which contain everything you need to know, per packet, if you can figure out how to extract it, so I'd say yes. The downside is that it generates a lot of data from a short session of monitoring. rgb > > > Yours > Mikhail Kuzminsky > Zelinsky Institute of Organic Chemistry > Moscow > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu
- Previous message: [Beowulf] Measurement of TCP traffic between nodes
- Next message: [Beowulf] Re: Grid Engine, Parallel Environment, Scheduling, Myrinet, and MPICH
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
More information about the Beowulf mailing list