[Beowulf] Measurement of TCP traffic between nodes
Robert G. Brown
rgb at phy.duke.edu
Wed Mar 23 12:09:01 PST 2005
On Wed, 23 Mar 2005, Mikhail Kuzminsky wrote:
>
> Dear colleagues,
>
> I want to measure message sizes (and "frequencies" of messages
> exchanges) between cluster nodes under parallelization of
> some applications. These applications use parallelization tools
> working over TCP (Linda, DDI), so I need to know the size of TCP
> packages and the number of TCP packages per second.
>
> 1) There is no other traffic on the corresponding interfaces,
> and (at least for 2-nodes "subcluster"; it's used for simplicity) I
> may use simple ifconfig output (in a loop). Do I understand correctly,
> that
> the number of bytes transmitted at "application level" is
> something about (for example for transmission)
> Tx_bytes - TX_packets*40 ?
> (i.e. a) there is no bytes from ethernet headers - in ifconfig output
> b) the number of application transmitted bytes per TCP packet =
>
> size_of_packet - 40 (in average))
I'm not sure that this is a safe assumption. Let me try to analyze this
briefly, although other folks will need to check me as it is easy to get
confused here.
* Don't use ifconfig in a loop. ifconfig's data comes from
/proc/net/dev in columns. Easier to parse, and lower overhead:
rgb at ganesh|B:1007>cat /proc/net/dev > /tmp/dev
(contents of /tmp/dev)
Inter-| Receive | Transmit
face |bytes packets errs drop fifo frame compressed multicast|bytes packets errs drop fifo colls carrier compressed
lo: 2487503 8119 0 0 0 0 0 0 2487503 8119 0 0 0 0 0 0
sit0: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
usb0: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
eth0:2244262754 11251382 0 0 0 0 0 0 1079809175 2321952 0 0 0 0 0 0
eth1: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
* xmlsysd will return this information wrapped in xml tags if you'd
rather parse it that way.
* From RFC 791, the bitwise layout of an IP header is:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Version| IHL |Type of Service| Total Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Identification |Flags| Fragment Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Time to Live | Protocol | Header Checksum |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Source Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Destination Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Options | Padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
or 24 bytes.
* From RFC 768, bitwise layout of UDP header is:
0 7 8 15 16 23 24 31
+--------+--------+--------+--------+
| Source | Destination |
| Port | Port |
+--------+--------+--------+--------+
| | |
| Length | Checksum |
+--------+--------+--------+--------+
|
| data octets ...
+---------------- ...
or 8 more bytes.
* From RFC 793, the bitwise layout of a TCP header is:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Source Port | Destination Port |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Acknowledgment Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Data | |U|A|P|R|S|F| |
| Offset| Reserved |R|C|S|S|Y|I| Window |
| | |G|K|H|T|N|N| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Checksum | Urgent Pointer |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Options | Padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| data |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
or 24 bytes.
So there are 48 bytes in IP+TCP plus 14 in the ethernet header (plus 8
in the pre-header plus 4 in the trailing CRC that are usually ignored).
Thus the smallest non-empty TCP message is 48+14 = 62+1 = 63 bytes,
IIRC.
For single messages that aren't part of a stream, this header cost will
be preserved per message, so you'll want to count 48 or 62 bytes. For
streaming messages the second and subsequent packages have abbreviated
TCP headers but I cannot remember their length and don't have time to
look it up (but the reference RFCs are indicated, I think).
> 2) Are the tools like tcpdump better for this task (pls
> take into account that I need only timestamps and data sizes) ?
tcpdump would let you see each packet's headers, which contain
everything you need to know, per packet, if you can figure out how to
extract it, so I'd say yes. The downside is that it generates a lot of
data from a short session of monitoring.
rgb
>
>
> Yours
> Mikhail Kuzminsky
> Zelinsky Institute of Organic Chemistry
> Moscow
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
>
--
Robert G. Brown http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu
More information about the Beowulf
mailing list