[Beowulf] Large amounts of data to store and process
joe.landman at gmail.com
Tue Mar 5 06:32:04 PST 2019
On 3/4/19 8:00 PM, Lux, Jim (337K) via Beowulf wrote:
> I'm munging through not very much satellite telemetry (a few GByte), using sqlite3..
> Here's some general observations:
> 1) if the data is recorded by multiple sensor systems, the clocks will *not* align - sure they may run NTP, but....
> 2) Typically there's some sort of raw clock being recorded with the data (in ticks of some oscillator, typically) - that's what you can use to put data from a particular batch of sources into a time order. And then you have the problem of reconciling the different clocks.
> 3) Watch out for leap seconds in time stamps - some systems have them (UTC), some do not (GPS, TAI) - a time of 23:59:60 may be legal.
> 4) you need to have a way to deal with "missing" data, whether it's time tags, or actual measurements - as well as "gaps in the record"
> 5) Be aware of the need to de-dupe data - same telemetry records from multiple sources.
Being satellite data, I am assuming you have relativistic corrections to
the time, depending upon orbit, accuracy of the clock, data analysis
needs, etc. . 
Missing data, of various types may be handled in data frame packages.
R, Julia, and I think Python can all handle this without too much pain.
e: joe.landman at gmail.com
More information about the Beowulf