[Beowulf] IEEE 1588 (PTP) - a better cluster clock?
patrick.ohly at intel.com
Wed Jul 25 03:36:39 PDT 2007
On Tue, 2007-07-24 at 16:51 -0600, Andrew Shewmaker wrote:
> On 7/18/07, Patrick Ohly <patrick.ohly at intel.com> wrote:
> > * Have you heard of PTP and considered to use it in clusters?
> > * How would applications or clusters benefit from a better
> > cluster-wide clock?
> > * What obstacles did or could prevent using PTP(d) for that
> > purpose?
> I wasn't aware of PTPd, and neither was my team leader Josip Loncaric.
> Now that I am, I'll try it out and compare it to a solution Josip came up
> with a while ago. He was satisfied with NTP at one time, but started
> writing BTime (http://btime.sf.net) in 2004.
Thanks for pointing that out. In general BTime looks similar in spirit
to PTP; let me point out similarities and differences below.
> BTime synchronizes client clocks to server broadcasts (not multicast),
> and uses a kernel module to provide more precise time-relevant data.
PTPd at some point also had a kernel patch to obtain time stamps for
packets, but replaced that with the SO_TIMESTAMP + loop back device
mechanism to simplify the installation.
> TUNING: BTime assumes that a certain fraction of timestamps will make it with
> minimal delays, and that those minimal delays are exponentially
> distributed. Over
> a high performance local network using UDP protocol, this
> characteristic noise is
> empirically about 3 microseconds (it would be about 10 microseconds
> for TCP), but
> if the network path has several hops, timing noise could be higher
> (e.g. 25 us UDP).
> BTW, BTime adaptively estimates the probability of receiving timestamps without
> extra delays; but it currently requires a fixed timing noise estimate.
Is this something that has to be specified when installing/starting
PTPd adapts to noisy measurements automatically; the author describes
that in this paper:
Manual tuning might be necessary to compensate for a constant offset
between client and server, which can be caused by different delays for
both directions, but I haven't seen that happen yet.
> TO DO: broadcast delay compensation... for now, BTime synchronizes
> all clients to server time minus uncompensated broadcast delay B.
PTP determines that delay by also measuring the delay of client->server
messages. So client and server are also truly in sync, not just the
clients among themselves.
These client->server messages are a scalability problem. I wonder if
assuming a zero client->server delay (and thus a constant offset between
clients and server) would be acceptable - in that case PTP clients could
operate without ever sending any packets. That might even be
Senior Software Engineer
Software & Solutions Group
Hermuelheimer Strasse 8a Phone: +49-2232-2090-30
50321 Bruehl Fax: +49-2232-2090-29
Intel GmbH, Dornacher Strasse 1, 85622 Feldkirchen/Muenchen Germany
Sitz der Gesellschaft: Feldkirchen bei Muenchen
Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer
Registergericht: Muenchen HRB 47456
Ust.- IdNr./VAT Registration No.: DE129385895
Citibank Frankfurt (BLZ 502 109 00) 600119052
More information about the Beowulf