[Beowulf] IEEE 1588 (PTP) - a better cluster clock?

[Patrick Ohly]

On Tue, 2007-07-24 at 16:51 -0600, Andrew Shewmaker wrote:
> On 7/18/07, Patrick Ohly <patrick.ohly at intel.com> wrote:
> >       * Have you heard of PTP and considered to use it in clusters?
> >       * How would applications or clusters benefit from a better
> >         cluster-wide clock?
> >       * What obstacles did or could prevent using PTP(d) for that
> >         purpose?
> 
> I wasn't aware of PTPd, and neither was my team leader Josip Loncaric.
> Now that I am, I'll try it out and compare it to a solution Josip came up
> with a while ago.  He was satisfied with NTP at one time, but started
> writing BTime (http://btime.sf.net) in 2004. 

Thanks for pointing that out. In general BTime looks similar in spirit
to PTP; let me point out similarities and differences below.

[...]
> BTime synchronizes client clocks to server broadcasts (not multicast),
> and uses a kernel module to provide more precise time-relevant data.

PTPd at some point also had a kernel patch to obtain time stamps for
packets, but replaced that with the SO_TIMESTAMP + loop back device
mechanism to simplify the installation.

> TUNING: BTime assumes that a certain fraction of timestamps will make it with
> minimal delays, and that those minimal delays are exponentially
> distributed.  Over
> a high performance local network using UDP protocol, this
> characteristic noise is
> empirically about 3 microseconds (it would be about 10 microseconds
> for TCP), but
> if the network path has several hops, timing noise could be higher
> (e.g. 25 us UDP).
> BTW, BTime adaptively estimates the probability of receiving timestamps without
> extra delays; but it currently requires a fixed timing noise estimate.

Is this something that has to be specified when installing/starting
BTime?

PTPd adapts to noisy measurements automatically; the author describes
that in this paper:
http://ptpd.sourceforge.net/ptpd_2005_1588_conference_paper.pdf

Manual tuning might be necessary to compensate for a constant offset
between client and server, which can be caused by different delays for
both directions, but I haven't seen that happen yet.

> TO DO: broadcast delay compensation...  for now, BTime synchronizes
> all clients to server time minus uncompensated broadcast delay B.

PTP determines that delay by also measuring the delay of client->server
messages. So client and server are also truly in sync, not just the
clients among themselves.

These client->server messages are a scalability problem. I wonder if
assuming a zero client->server delay (and thus a constant offset between
clients and server) would be acceptable - in that case PTP clients could
operate without ever sending any packets. That might even be
standard-compliant...

-- 
Best Regards

Patrick Ohly
Senior Software Engineer

Intel GmbH
Software & Solutions Group                
Hermuelheimer Strasse 8a                  Phone: +49-2232-2090-30
50321 Bruehl                              Fax: +49-2232-2090-29
Germany

Intel GmbH, Dornacher Strasse 1, 85622 Feldkirchen/Muenchen Germany
Sitz der Gesellschaft: Feldkirchen bei Muenchen
Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer
Registergericht: Muenchen HRB 47456
Ust.- IdNr./VAT Registration No.: DE129385895
Citibank Frankfurt (BLZ 502 109 00) 600119052