<div dir="ltr"><div dir="ltr"><div>Talking about missing values... Joe Landman is sure to school me again for this one (owwwccchhh)</div><div><a href="https://docs.julialang.org/en/v1/manual/missing/index.html">https://docs.julialang.org/en/v1/manual/missing/index.html</a></div><div><br></div><div>Going back to the hardware, a 250Gbyte data size is not too large to hold in RAM.</div><div>This might be a good use case for Intel Optane persistent memory - I dont know exactly how this works when used in a memory mode as opposed to a block device mode.</div><div>The Diablo memory was supposed to migrate cold pages down to the lower, slower memory.</div><div>Does Optane function similarly?</div><div><br></div><div><br></div><div><br></div><div><br></div></div></div><br><div class="gmail_quote"><div class="gmail_attr" dir="ltr">On Tue, 5 Mar 2019 at 01:02, Lux, Jim (337K) via Beowulf <<a href="mailto:beowulf@beowulf.org">beowulf@beowulf.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">I'm munging through not very much satellite telemetry (a few GByte), using sqlite3..<br>
Here's some general observations:<br>
1) if the data is recorded by multiple sensor systems, the clocks will *not* align - sure they may run NTP, but.... <br>
2) Typically there's some sort of raw clock being recorded with the data (in ticks of some oscillator, typically) - that's what you can use to put data from a particular batch of sources into a time order. And then you have the problem of reconciling the different clocks.<br>
3) Watch out for leap seconds in time stamps - some systems have them (UTC), some do not (GPS, TAI) - a time of 23:59:60 may be legal.<br>
4) you need to have a way to deal with "missing" data, whether it's time tags, or actual measurements - as well as "gaps in the record"<br>
5) Be aware of the need to de-dupe data - same telemetry records from multiple sources.<br>
<br>
<br>
Jim Lux<br>
(818)354-2075 (office)<br>
(818)395-2714 (cell)<br>
<br>
<br>
-----Original Message-----<br>
From: Beowulf [mailto:<a href="mailto:beowulf-bounces@beowulf.org" target="_blank">beowulf-bounces@beowulf.org</a>] On Behalf Of Jonathan Aquilina<br>
Sent: Monday, March 04, 2019 1:24 AM<br>
To: Fred Youhanaie <<a href="mailto:fly@anydata.co.uk" target="_blank">fly@anydata.co.uk</a>>; <a href="mailto:beowulf@beowulf.org" target="_blank">beowulf@beowulf.org</a><br>
Subject: Re: [Beowulf] Large amounts of data to store and process<br>
<br>
Hi Fred,<br>
<br>
I and my colleague had done some research and found an extension for postgresql called timescaleDB, but then upon further research postgres on its own is good for such data as well. The thing is these are not going to be given to use as the data is coming in but in bulk at the end from the parent company.<br>
<br>
Have you used postgresql for such type's of data and how has it performed?<br>
<br>
Regards,<br>
Jonathan<br>
<br>
On 04/03/2019, 10:19, "Beowulf on behalf of Fred Youhanaie" <<a href="mailto:beowulf-bounces@beowulf.org" target="_blank">beowulf-bounces@beowulf.org</a> on behalf of <a href="mailto:fly@anydata.co.uk" target="_blank">fly@anydata.co.uk</a>> wrote:<br>
<br>
Hi Jonathan,<br>
<br>
It seems you're collecting metrics and time series data. Perhaps a time series database (TSDB) is an option for you. There are a few of these out there, but I don't have any personal recommendation.<br>
<br>
Cheers,<br>
Fred<br>
<br>
On 04/03/2019 07:04, Jonathan Aquilina wrote:<br>
> These would be numerical data such as integers or floating point numbers.<br>
> <br>
> -----Original Message-----<br>
> From: Tony Brian Albers <<a href="mailto:tba@kb.dk" target="_blank">tba@kb.dk</a>><br>
> Sent: 04 March 2019 08:04<br>
> To: <a href="mailto:beowulf@beowulf.org" target="_blank">beowulf@beowulf.org</a>; Jonathan Aquilina <<a href="mailto:jaquilina@eagleeyet.net" target="_blank">jaquilina@eagleeyet.net</a>><br>
> Subject: Re: [Beowulf] Large amounts of data to store and process<br>
> <br>
> Hi Jonathan,<br>
> <br>
> From my limited knowledge of the technologies, I would say that HBase with file pointers to the files placed on HDFS would suit you well.<br>
> <br>
> But if the files are log files, consider some tools that are suited for analyzing those like Kibana.<br>
> <br>
> /tony<br>
> <br>
> <br>
> On Mon, 2019-03-04 at 06:55 +0000, Jonathan Aquilina wrote:<br>
>> Hi Tony,<br>
>><br>
>> Sadly I cant go into much detail due to me being under an NDA. At this<br>
>> point with the prototype we have around 250gb of sample data but again<br>
>> this data is dependent on the type of air craft. Larger aircraft and<br>
>> longer flights will generate a lot more data as they have more<br>
>> sensors and will log more data than the sample data that I have. The<br>
>> sample data is 250gb for 35 aircraft of the same type.<br>
>><br>
>> Regards,<br>
>> Jonathan<br>
>><br>
>> -----Original Message-----<br>
>> From: Tony Brian Albers <<a href="mailto:tba@kb.dk" target="_blank">tba@kb.dk</a>><br>
>> Sent: 04 March 2019 07:48<br>
>> To: <a href="mailto:beowulf@beowulf.org" target="_blank">beowulf@beowulf.org</a>; Jonathan Aquilina <<a href="mailto:jaquilina@eagleeyet.net" target="_blank">jaquilina@eagleeyet.net</a>><br>
>> Subject: Re: [Beowulf] Large amounts of data to store and process<br>
>><br>
>> On Mon, 2019-03-04 at 06:38 +0000, Jonathan Aquilina wrote:<br>
>>> Good Morning all,<br>
>>><br>
>>> I am working on a project that I sadly cant go into much detail but<br>
>>> there will be quite large amounts of data that will be ingested by<br>
>>> this system and would need to be efficiently returned as output to<br>
>>> the end user in around 10 min or so. I am in discussions with<br>
>>> another partner involved in this project about the best way forward<br>
>>> on this.<br>
>>><br>
>>> For me given the amount of data (and it is a huge amount of data)<br>
>>> that an RDBMS such as postgresql would be a major bottle neck.<br>
>>> Another thing that was considered flat files, and I think the best<br>
>>> for that would be a Hadoop cluster with HDFS. But in the case of HPC<br>
>>> how can such an environment help in terms of ingesting and analytics<br>
>>> of large amounts of data? Would said flat files of data be put on a<br>
>>> SAN/NAS or something and through an NFS share accessed that way for<br>
>>> computational purposes?<br>
>>><br>
>>> Regards,<br>
>>> Jonathan<br>
>>> _______________________________________________<br>
>>> Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" target="_blank">Beowulf@beowulf.org</a> sponsored by Penguin<br>
>>> Computing To change your subscription (digest mode or unsubscribe)<br>
>>> visit http:/ /<a href="http://www.beowulf.org/mailman/listinfo/beowulf" target="_blank" rel="noreferrer">www.beowulf.org/mailman/listinfo/beowulf</a><br>
>><br>
>> Good morning,<br>
>><br>
>> Around here, we're using HBase for similar purposes. We have a bunch<br>
>> of smaller nodes storing the data and all the management nodes(ambari,<br>
>> HDFS namenodes etc.) are vm's.<br>
>><br>
>> Our nodes are configured so that we have a maximum of 2 cores per disk<br>
>> spindle and 4G of memory for each core. This seems to do the trick and<br>
>> is pretty responsive.<br>
>><br>
>> But to be able to provide better advice, you will probably need to go<br>
>> into a bit more detail about what types of data you will be storing<br>
>> and which kind of calculations you want to perform.<br>
>><br>
>> /tony<br>
>><br>
>><br>
>> --<br>
>> Tony Albers - Systems Architect - IT Development Royal Danish Library,<br>
>> Victor Albecks Vej 1, 8000 Aarhus C, Denmark<br>
>> Tel: +45 2566 2383 - CVR/SE: 2898 8842 - EAN: 5798000792142<br>
> <br>
> --<br>
> Tony Albers - Systems Architect - IT Development Royal Danish Library, Victor Albecks Vej 1, 8000 Aarhus C, Denmark<br>
> Tel: +45 2566 2383 - CVR/SE: 2898 8842 - EAN: 5798000792142<br>
> _______________________________________________<br>
> Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" target="_blank">Beowulf@beowulf.org</a> sponsored by Penguin Computing<br>
> To change your subscription (digest mode or unsubscribe) visit <a href="http://www.beowulf.org/mailman/listinfo/beowulf" target="_blank" rel="noreferrer">http://www.beowulf.org/mailman/listinfo/beowulf</a><br>
> <br>
_______________________________________________<br>
Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" target="_blank">Beowulf@beowulf.org</a> sponsored by Penguin Computing<br>
To change your subscription (digest mode or unsubscribe) visit <a href="http://www.beowulf.org/mailman/listinfo/beowulf" target="_blank" rel="noreferrer">http://www.beowulf.org/mailman/listinfo/beowulf</a><br>
<br>
<br>
_______________________________________________<br>
Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" target="_blank">Beowulf@beowulf.org</a> sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit <a href="http://www.beowulf.org/mailman/listinfo/beowulf" target="_blank" rel="noreferrer">http://www.beowulf.org/mailman/listinfo/beowulf</a><br>
_______________________________________________<br>
Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" target="_blank">Beowulf@beowulf.org</a> sponsored by Penguin Computing<br>
To change your subscription (digest mode or unsubscribe) visit <a href="http://www.beowulf.org/mailman/listinfo/beowulf" target="_blank" rel="noreferrer">http://www.beowulf.org/mailman/listinfo/beowulf</a><br>
</blockquote></div>