[Beowulf] IEEE 1588 (PTP) - a better cluster clock?

[Jim Lux]

At 01:07 PM 7/20/2007, Patrick Geoffray wrote:
>Hi Patrick,
>
>Patrick Ohly wrote:
>>That's all for now (and probably enough stuff, too ... although perhaps
>>you prefer detailed emails over bullet items on a PowerPoint
>>presentation). So what do you think?
>
>Since you are asking, here is my personal opinion: I don't think 
>there is a need for a system-wide PTP service. NTP's precision is 
>good enough for comparing log and other administration needs. The 
>only requirement for a high precision/high accuracy global clock 
>reference that I would consider legitimate is for tracing tools 
>(that's where you come from). In this case, why not running NTP as 
>part of the tracing environment ?


There are tons of reasons why one might want more precision than NTP 
can provide, leaving aside the obvious applications for things like 
controlling test equipment.

Consider if one is using distributed processing of high rate sensor 
data (radar, ultrasound, seismic).. 1ms sorts of accuracies (which is 
doing quite well with most NTP implementations) aren't going to hack 
it for doing the time alignment.  Sure, one can run discrete 
synchronization interfaces, etc, but it's truly a pain.

There are also cases where one might want to perform various things 
in synchronism (e.g. driving a video wall?) where the synchronization 
requirement is microseconds. Say you're generating subframes of 
video, and the processor speeds are all slightly different. You 
effectively need some form of barrier synchronization to 
synchronously update all the images.

A similar application might be real time panoramic stitching and 
resampling of multiple video streams.

Sure, there are application specific approaches to these things (e.g. 
genlock in the video world, various timecodes in the video, audio, 
and instrumentation worlds) but they require different hardware for 
different applications, provide varying levels of performance, and 
aren't necessarily well integrated with other things (e.g. the IRIG 
time codes derive from the days of analog instrumentation tape 
recorders to get microsecond timing precision)

1588 is a nice start on doing this in a more consistent and scalable 
way than current hacks like putting GPS receivers in each computer, etc.


NTP also has some problems with synchronization in relativistic 
environments and ones where the delays are rapidly changing over 
large magnitudes (consider trying to use NTP to synchronize a clock 
orbiting the moon).  While there are techniques currently used for 
time synchronization over large distances in these situations, they 
tend to be oriented towards determining sync for a single endpoint 
(i.e. a deep space probe) and also tend to involve post 
processing.  They're not as appropriate for things like a 
constellation of platforms (imagine a bunch of airplanes flying 
around, as an example).

There's great value in any sort of IP (internet protocol, not 
intellectual property) origin solution (even if it requires special 
PHY layers), because IP links are becoming ubiquitous.

>You could say that it would be useful to factorize this 
>functionality so that it can be shared by several tracing libraries. 
>I don't think it's worth the effort, as one rarely uses two 
>different tracing tool at the same time.
>
>I don't know enough about PTP to comment on the implementation, 
>except that relying on multicast is a bad, bad idea. It does not 
>scale and it's very unreliable under load (induces contention if reliable).

I'd have to go back and review the 1588 specs, but my impression is 
that one can implement a PTP system in a way that is scalable to very 
large sizes, otherwise, it wouldn't be much use.  Indeed, such an 
implementation might not coexist with other needs for the network, so 
you'd be back to running parallel networks, but at least they are 
more scalable than distribution amps and GPS receivers.


>Patrick
>_______________________________________________
>Beowulf mailing list, Beowulf at beowulf.org
>To change your subscription (digest mode or unsubscribe) visit 
>http://www.beowulf.org/mailman/listinfo/beowulf

James Lux, P.E.
Spacecraft Radio Frequency Subsystems Group
Flight Communications Systems Section
Jet Propulsion Laboratory, Mail Stop 161-213
4800 Oak Grove Drive
Pasadena CA 91109
tel: (818)354-2075
fax: (818)393-6875