[Beowulf] Large amounts of data to store and process

[Joe Landman]

On 3/4/19 8:00 PM, Lux, Jim (337K) via Beowulf wrote:
> I'm munging through not very much satellite telemetry (a few GByte), using sqlite3..
> Here's some general observations:
> 1) if the data is recorded by multiple sensor systems, the clocks will *not* align - sure they may run NTP, but....
> 2) Typically there's some sort of raw clock being recorded with the data (in ticks of some oscillator, typically) - that's what you can use to put data from a particular batch of sources into a time order.  And then you have the problem of reconciling the different clocks.
> 3) Watch out for leap seconds in time stamps - some systems have them (UTC), some do not (GPS, TAI) - a time of 23:59:60 may be legal.
> 4) you need to have a way to deal with "missing" data, whether it's time tags, or actual measurements - as well as "gaps in the record"
> 5) Be aware of the need to de-dupe data - same telemetry records from multiple sources.

Being satellite data, I am assuming you have relativistic corrections to 
the time, depending upon orbit, accuracy of the clock, data analysis 
needs, etc. . [1][2]

Missing data, of various types may be handled in data frame packages.  
R, Julia, and I think Python can all handle this without too much pain.

[1] https://gssc.esa.int/navipedia/index.php/Relativistic_Clock_Correction

[2] http://www.astronomy.ohio-state.edu/~pogge/Ast162/Unit5/gps.html

-- 
Joe Landman
e: joe.landman at gmail.com
t: @hpcjoe
w: https://scalability.org
g: https://github.com/joelandman
l: https://www.linkedin.com/in/joelandman