[Beowulf] Interesting

[Robert G. Brown]

On Wed, 27 Oct 2010, Sarma Tangirala wrote:

> The recent digests that I am getting are quite interesting (bad google) and I have a question.
>
> What I'd like to know is, is it possible to have a our history captured in its entirety so that none of the future generations have to run around (like Hari Seldon) because information from waaaay back is corrupt and not take care of?
>
> Do you guys know of any existing sources that you can point me to?
>
> Is this under distributed systems or under compression algorithms?

Sure.  It's very simple.  Obtain a very large store of platinum plates
(or maybe gold -- one of the more or less corrosion-proof, durable
metals at any rate).  Engrave our history, in pictures that require no
language to understand or encoded in such a way that the documentation
contains a universal rosetta stone allowing language to be bootstrapped
(there are theories for how to go about doing this; they were used to
engrave messages on the Voyager and some of the other spacecraft we shot
out of the solar system IIRC).  Go to the moon.  Dig a really big hole
at one of the poles (to avoid thermal extremes) and build a bunker out
of fused glass three meters thick and one kilometer underground.  Place
the individually shock wrapped and thermally isolated plates in neat
racks.  Seal it hermetically (melt the glass back over the hole you put
them in through).  Backfill the hole, fusing multiple sets of three
meter thick layers of steel reinforced glass separated by ten or so
meters of dirt across an area many times larger than the vault itself.
Cap the whole thing with a large basin of "soft" sand 100 meters thick.

With luck, it will then last until the sun reaches the point where it
starts to turn into a red giant and engulfs the earth and the moon,
unless of course one is unlucky and an asteroid big enough to overwhelm
the shock absorber strikes dead on.

To outlast the sun, consider duplicating this process on several of the
planetoids out in the Oort cloud or just shooting mid-sized asteroids
containing vaults out of the solar system entirely.  But out there one
has to worry about gravitational resonance and so on -- the inner
planets (including the moon) are actually a lot more stable and safer at
this point individually.

Beyond that, the only solution is not to rigidly resist entropy, but to
flow with, to adapt to, the process of entropy.  Copy, copy, and copy
again.  Copy in many formats, in many locations.  Avoid Library of
Alexandria solutions as they are not robust to events like fire and
political turmoil -- put the Library of Alexandria in the hands of
>>every citizen<< and constantly update them to keep them in sync with a
widely dispersed set of master copies, all nicely checksummed and so on.
Surf the technology as the waves dictate, rather than being crushed by
the tides of change.

No matter what you face thermal chaos, the inexorable force of entropy.
One day the hot winds will blow over the ashes of our species, our very
civilization, and we will become mere dust in the wind once again, and
very probably, long before that time arrives, all that we now know and
say and do will have been discovered and lost over and over again.  The
second law of thermodynamics has no mercy, and entropy will eventually
prevail.

    rgb

>
> Any other two cents on this is welcome!
> Sent from my BlackBerry
>
> -----Original Message-----
> From: beowulf-request at beowulf.org
> Sender: beowulf-bounces at beowulf.org
> Date: Wed, 27 Oct 2010 09:36:13
> To: <beowulf at beowulf.org>
> Reply-To: beowulf at beowulf.org
> Subject: Beowulf Digest, Vol 80, Issue 22
>
> Send Beowulf mailing list submissions to
> 	beowulf at beowulf.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> 	http://www.beowulf.org/mailman/listinfo/beowulf
> or, via email, send a message with subject or body 'help' to
> 	beowulf-request at beowulf.org
>
> You can reach the person managing the list at
> 	beowulf-owner at beowulf.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Beowulf digest..."
>
>
> Today's Topics:
>
>   1. RE: how Google warps your brain (Bill Rankin)
>   2. RE: how Google warps your brain (Douglas Eadline)
>   3. RE: Anybody using Redhat HPC Solution in their Beowulf
>      (Hearns, John)
>   4. Re: Anybody using Redhat HPC Solution in their Beowulf
>      (Ellis H. Wilson III)
>   5. Re: Anybody using Redhat HPC Solution in their Beowulf
>      (Kilian CAVALOTTI)
>   6. RE: Anybody using Redhat HPC Solution in their Beowulf
>      (Lux, Jim (337C))
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 26 Oct 2010 14:54:43 +0000
> From: Bill Rankin <Bill.Rankin at sas.com>
> Subject: RE: [Beowulf] how Google warps your brain
> To: Beowulf Mailing List <beowulf at beowulf.org>
> Cc: "Robert G. Brown" <rgb at phy.duke.edu>
> Message-ID:
> 	<76097BB0C025054786EFAB631C4A2E3C0948F542 at MERCMBX03D.na.SAS.com>
> Content-Type: text/plain; charset="us-ascii"
>
> Heading completely off-topic now, but the area of digital media and long-term archival/retrieval is something that I find very interesting.  I'll leave it to Rob to somehow eventually tie this back into a discussion of COTs technology and HPC.
>
>
>>> It's interesting: I just got an iPad a few weeks ago, mostly as a
>>> reader/web-browser device, and I've been reading a variety of
>>> out-of-copyright works: H. Rider Haggard, Joseph Conrad, Mark Twain.
>> Thank
>>> you Gutenberg Project!
>>
>> It is awesome, isn't it?
>
> Amazon also carries many of the out-of-copyright works in their Kindle store for $0 (and gives credit to Gutenburg to a small extent).  It was nice to be able to go pickup things like the Sherlock Holmes series, Homer's Illiad and some of Einstein's works (which I don't pretend to understand) and have them downloaded via 3G on Amazon's dime.
>
> I will say that because of this I tend to overlook their rather high (IMHO) price on current digital content and have probably purchased more e-books overall as a result.
>
>
>>> And, since I am sitting/lying here with a very sore back from moving boxes
>>> of books around this weekend looking for that book that I *know* is in there
>>> somewhere, the prospect of some magic box that would scan all my books into
>>> a format usable into eternity would be quite nice.  I might even think that
>>> a personal "print on demand" would be nice that could generate a cheap/quick
>>> copy for reading in bed(yes, the iPad and Kindle, etc., are nice, but
>>> there's affordances provided by the paper edition that is nice.. But I don't
>>> need hardcover or, even, any cover..)
>
> There is just *something* about paper, isn't there?  And while I don't have a library to the extent of RGBs or others, I do like having some books around (glancing at the two bookshelves in my office).  On the other hand, I still have boxes of books sitting around unopened since we moved house 4-5 years ago.  I certainly need a purge, lest I end up on one of those "hoarding" shows that seem to be popular as of late.
>
> At some point, I have to ask myself if I really *need* to have a old beat-up, falling apart copy of "Voyage of the Space Beagle" laying around.
>
>
>>> (or, even better, a service that has scanned all the books for me, e.g.
>>> Google, and that upon receiving some proof of ownership of the physical
>>> book, lets me have an electronic copy of the same...  I'd gladly pay some
>>> nominal fee for such a thing, providing it wasn't for some horrible locked,
>>> time limited format which depends on the original vendor being in business
>>> 20 years from now.  I also recognize the concern about how "once in digital
>>> form, copying becomes very cheap" which I think is valid.
>
> A scanning service would be wonderful for a lot of the books I have, mainly those I view as reference-type material.  For current reference material, Safari Books Online has a reasonable usage model that allows for making hardcopy of their online content.  Now if there was only a simple way to transcribe the same content for download to my Kindle I would be set (something beyond the OCR+PDF approach, which is awkward and inconsistent).
>
>
>> What a killer idea.  Acceptable use, doggone it!  I'd ship them books
>> by the boxful in exchange for a movable (even DRM controlled) image, a la
>> Ipod music.  I just don't want to rebuy them, like I've now bought most
>> of my music collection TWICE (vinyl and CD).
>
> [let's not get started about vinyl collections - that's a whole 'nother set of unopened boxes]
>
> The problem is that many of the media houses are still waging an underground war on Fair Use, despite the legal decisions handed down by the courts.  As an example, I recently had a email exchange with one of the customer service people at a major network.  I was trying to locate additional interview footage from when my brother-in-law was on a certain hour-long Sunday evening news show.  This person informed me that I did not have their "permission" to recorded the over-the-air broadcast of the show and burn it on a DVD to give to my sister, so what I was doing was not legal.
>
> This was news to me, since this usage model was clearly defined as permissible by the Supreme Court many years ago in the Sony v. Universal "Betamax Case".
>
> While the market for online music, video and written works have forced the various publishers to acknowledge to the need to provide content in digital form, to a great extent they had to be dragged kicking and screaming into the 21st century.  A lot of progress has been made but there is still a lot of resistance towards efforts to open up availability and access even further.
>
>
> I would like see a service where I could take bins of old books to a used book store and somehow get credits towards the purchase of e-books online.  I think that could break me of my paperback hoarding habit pretty quickly.
>
>
> -bill
>
>
>
>
> ------------------------------
>
> Message: 2
> Date: Tue, 26 Oct 2010 10:59:25 -0400 (EDT)
> From: "Douglas Eadline" <deadline at eadline.org>
> Subject: RE: [Beowulf] how Google warps your brain
> To: "Hearns, John" <john.hearns at mclaren.com>
> Cc: beowulf at beowulf.org, "Robert G. Brown" <rgb at phy.duke.edu>
> Message-ID:
> 	<49886.192.168.93.213.1288105165.squirrel at mail.eadline.org>
> Content-Type: text/plain;charset=iso-8859-1
>
> <Seinfeld>
> Not that there is anything wrong with that.
> </Seinfeld >
>
>>
>> As usual, a highly insightful post from RGB.
>>
>>
>>
>>>  a) Multiple copies.  Passenger pigeons may be robust, but once the
>> number of copies drops below a critical point, they are gone.  E. Coli
>> we will always have
>>> with us (possibly in a constantly changing form) because there are so
>> very many copies, so very widely spread.
>>
>> I probably shouldn't mention Wikileaks here...
>>
>>>
>>> At the moment, the internet has if anything VASTLY INCREASED a, b and
>> c
>>> for every single document in the public domain that has been ported
>> to,
>>> e.g. Project Gutenberg.
>>>
>>> Right now, I'm sitting on a cache of "Saint" books, by Leslie
>> Charteris
>>> (who was a great favorite of mine growing up and still is).
>>>
>>> Nobody is going to reprint the Saint stories.  They are a gay fantasy
>>> from another time,
>>
>> Simon Templar? Gay? Cough.
>>
>> Next you will be telling me that there are gay undertones in Top Gun,
>> the film with the sexiest astrophysicist ever.
>>
>>
>>> might well last to the end of civilization.  Replicate them a few
>>> million times, PERPETUATE them from generation to generation by
>>> renewing
>>> the copies, and backing them up, and recopying them in formats where
>>> they are still useful.
>>
>> The cloud backup providers will be keeping copies of data on
>> geographically spread sites.
>> However, we should at this stage be asking what are the mechanisms for
>> cloud storage companies
>> for
>> *) living wills - what happens when the company goes bust
>>
>> *) what are the strategies for migrating the data onto new storage
>> formats
>>
>>
>>>
>>> Or, to put it differently, suppose every single human on the planet
>> had
>>> access to the modern equivalent of Diophantus's Arithmetica on their
>>> computer, their Kindle, their Ipad
>> I believe that was the original intent for the Web. Still under
>> development!
>>
>>
>> The contents of this email are confidential and for the exclusive use of
>> the intended recipient.  If you receive this email in error you should not
>> copy it, retransmit it, use it or disclose its contents but should return
>> it to the sender immediately and delete your copy.
>>
>> _______________________________________________
>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
>> To change your subscription (digest mode or unsubscribe) visit
>> http://www.beowulf.org/mailman/listinfo/beowulf
>>
>> --
>> This message has been scanned for viruses and
>> dangerous content by MailScanner, and is
>> believed to be clean.
>>
>
>
> -- 
> Doug
>
> -- 
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
>
>
>
> ------------------------------
>
> Message: 3
> Date: Tue, 26 Oct 2010 09:16:47 +0100
> From: "Hearns, John" <john.hearns at mclaren.com>
> Subject: RE: [Beowulf] Anybody using Redhat HPC Solution in their
> 	Beowulf
> To: "Ellis H. Wilson III" <ellis at runnersroll.com>,
> 	<beowulf at beowulf.org>
> Message-ID:
> 	<68A57CCFD4005646957BD2D18E60667B12154E23 at milexchmb1.mil.tagmclarengroup.com>
>
> Content-Type: text/plain; charset="us-ascii"
>
>> I don't think you could find a statement more orthogonal to the spirit
>> of the Beowulf list than, "Please, please don't "roll your own"
>> system..."  Isn't Beowulfery about the drawing together of inexpensive
>> components in an intelligent fashion suited just for your particular
>> application while using standardized (and thereby cheap by the law of
>> scale) hardware?  I'm not suggesting Richard build his own NIC - but
>> there is nothing wrong with using even a distribution of Linux not
>> intended for HPC (so long as you're smart about it) and picking and
>> choosing the software (queuing managers, tracers, etc) he finds works
>> best.
>>
>> Also, I would argue if a company is selling you an HPC solution, it's
>> either:
>> 1. A true Beowulf in terms of using COTS hardware, in which case you
>> are
>> likely getting less than your money is worth or
>
>
> Ellis, I am going to politely disagree with you - now there's a
> surprise!
>
> I have worked as an engineer for two HPC companies - Clustervision and
> Streamline.
> My slogan phrase on this issue is "Any fool can go down PC World and buy
> a bunch of PCs"
> By that I mean that CPU is cheap these days, but all you will get is a
> bunch of boxes
> on your loading bay. As you say, and you are right, you then have the
> option of installing
> Linux plus a cluster management stack and getting a cluster up and
> running.
>
> However, as regards price, I would say that actually you will be paying
> very, very little premium
> for getting a supported, tested and pre-assembled cluster from a vendor.
> Academic margins are razor thin - the companies are not growing fat over
> academic deals.
> They also can get special pricing from Intel/AMD if the project can be
> justified - probably ending
> up at a price per box near to what you pay at PC World.
>
> Or take (say) rack top switches. Do you want to have a situation where
> the company which supports your cluster
> has switches sitting on a shelf, so when a switch fails someone (me!) is
> sent out the next morning to deliver
> a new switch in a box, cable it in and get you running?
> Or do you want to deal direct with the returns department at $switch
> vendor, or even (shudder) take the route
> of using the same switches as the campus network - so you don't get to
> choose on the basis of performance or
> suitability, but just depend on the warm and fuzzies your campus IT
> people have.
>
>
> We then come to support - say you buy that heap of boxes from a Tier 1 -
> say it is the same company your
> campus IT folks have a campus wide deal with. You'll get the same type
> of support you get for general
> servers running Windows - and you'll deal with first line support staff
> on the phone every time.
> Me, I've been there, seen there, done it with tier 1 support like that.
> As a for instance, HPC workloads tend to stress the RAM in a system, and
> you get frequent ECC errors on
> a young system as it is bedding in. Try phoning support every time a
> light comes on, and get talked through
> the "have you run XXX diagnostic", it soon gets wearing.
> Before Tier 1 companies cry foul, of course both the above companies and
> all other cluster companies integrate
> Tier 1 servers - but that is a different scenario from getting boxes
> delivered through your campus agreement with
> $Tier1.
>
>
>
>
>
>
>
>
>
>
>
>
> The contents of this email are confidential and for the exclusive use of the intended recipient.  If you receive this email in error you should not copy it, retransmit it, use it or disclose its contents but should return it to the sender immediately and delete your copy.
>
>
>
> ------------------------------
>
> Message: 4
> Date: Tue, 26 Oct 2010 12:09:12 -0400
> From: "Ellis H. Wilson III" <ellis at runnersroll.com>
> Subject: Re: [Beowulf] Anybody using Redhat HPC Solution in their
> 	Beowulf
> To: "Hearns, John" <john.hearns at mclaren.com>
> Cc: beowulf at beowulf.org
> Message-ID: <4CC6FD28.1050303 at runnersroll.com>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> On 10/26/10 04:16, Hearns, John wrote:
>> I have worked as an engineer for two HPC companies - Clustervision and
>> Streamline.
>> My slogan phrase on this issue is "Any fool can go down PC World and buy
>> a bunch of PCs"
>
> Well if you are buying PCs in bulk at retail pricing, you are a fool
> anyway.  Plus most PC World PCs won't have ECC RAM so I wasn't really
> referring to those as few of us tolerate random bit flips.
>
>> However, as regards price, I would say that actually you will be paying
>> very, very little premium
>> for getting a supported, tested and pre-assembled cluster from a vendor.
>> Academic margins are razor thin - the companies are not growing fat over
>> academic deals.
>> They also can get special pricing from Intel/AMD if the project can be
>> justified - probably ending
>> up at a price per box near to what you pay at PC World.
>
> Again, not comparing PC World to Tier 1 bulk purchases.  I'm comparing
> Tier 1 bulk purchases w/o an OS (so you can DIY) with specialized HPC
> vendor purchases where you don't have to DIY.  Even then, perhaps it
> breaks even the first year if you get a very, very good deal from the
> HPC vendor.  However, to get the deal you are probably contracted into
> four or five years of support and when considering HPC, involving more
> humans are the fastest way to get a really inefficient and expensive
> cluster.  After the first year and up until the lifetime of the cluster
> involving human support annually will add a large cost overhead you have
> to account for at the beginning (and probably buy less hardware because
> of which).
>
>> Or take (say) rack top switches. Do you want to have a situation where
>> the company which supports your cluster
>> has switches sitting on a shelf, so when a switch fails someone (me!) is
>> sent out the next morning to deliver
>> a new switch in a box, cable it in and get you running?
>
> That's probably a hell of a lot faster than waiting on a vendor to get
> you a new switch through some RMA process.  Plus you know the cabling is
> done right :).
>
> Optimally IMHO, in university setups physical scientists create the need
> for HPC.  These types shouldn't (as Kilian mentions) need to inherit all
> of the responsibilities and overheads of cluster management to use one
> (or pay cluster vendors annually for support).  They should simply walk
> over to the CS department, find system guys (who would probably drool
> over the potential of administering a reasonably sized cluster) and work
> out an agreement where the physical science types can "just use it" and
> the systems/CS guys administer it and can once in a while trace
> workloads, test new load balancing mechanisms, try different kernel
> settings for performance, etc.  This way the physical scientists get
> their work done on a well supported HPC system for no extra cash and
> computer scientists get great, non-toy traces and workloads to further
> their own research.  Both parties win.
>
> Now in organizations that don't have a CS department I agree that HPC
> vendors are the way to go.
>
> ellis
>
>
> ------------------------------
>
> Message: 5
> Date: Tue, 26 Oct 2010 11:18:56 +0200
> From: Kilian CAVALOTTI <kilian.cavalotti.work at gmail.com>
> Subject: Re: [Beowulf] Anybody using Redhat HPC Solution in their
> 	Beowulf
> To: "Ellis H. Wilson III" <ellis at runnersroll.com>
> Cc: beowulf at beowulf.org
> Message-ID:
> 	<AANLkTimORzXXM=69Lq3KLNuanO30v6k2+qHy7Vs-6e-_ at mail.gmail.com>
> Content-Type: text/plain; charset=UTF-8
>
> Hi,
>
> On Tue, Oct 26, 2010 at 1:00 AM, Ellis H. Wilson III
> <ellis at runnersroll.com> wrote:
>> Also, I would argue if a company is selling you an HPC solution, it's
>> either:
>> 1. A true Beowulf in terms of using COTS hardware, in which case you are
>> likely getting less than your money is worth or
>
> Well, depends on how you value your time and the required expertise to
> put all those COTS and OSS pieces together to make them run smoothly
> and efficiently.
> Most scientists and HPC systems users are not professional sysadmins
> (which is good, they have a job to do), and the value of trained,
> experienced, skilled individuals who can put together a reliable and
> useful HPC system is sometimes overlooked (ie. undervalued).
>
> I agree with your later statement, though:
>
>> I personally don't think the "market for cluster vendors" is [...]
>> the Beowulf list.
>
> Cheers,
> -- 
> Kilian
>
>
> ------------------------------
>
> Message: 6
> Date: Wed, 27 Oct 2010 09:32:43 -0700
> From: "Lux, Jim (337C)" <james.p.lux at jpl.nasa.gov>
> Subject: RE: [Beowulf] Anybody using Redhat HPC Solution in their
> 	Beowulf
> To: "Ellis H. Wilson III" <ellis at runnersroll.com>, "Hearns, John"
> 	<john.hearns at mclaren.com>
> Cc: "beowulf at beowulf.org" <beowulf at beowulf.org>
> Message-ID:
> 	<ECE7A93BD093E1439C20020FBE87C47FEDD29F961A at ALTPHYEMBEVSP20.RES.AD.JPL>
>
> Content-Type: text/plain; charset="us-ascii"
>
>
>>
>> Optimally IMHO, in university setups physical scientists create the need
>> for HPC.  These types shouldn't (as Kilian mentions) need to inherit all
>> of the responsibilities and overheads of cluster management to use one
>> (or pay cluster vendors annually for support).  They should simply walk
>> over to the CS department, find system guys (who would probably drool
>> over the potential of administering a reasonably sized cluster) and work
>> out an agreement where the physical science types can "just use it" and
>> the systems/CS guys administer it and can once in a while trace
>> workloads, test new load balancing mechanisms, try different kernel
>> settings for performance, etc.  This way the physical scientists get
>> their work done on a well supported HPC system for no extra cash and
>> computer scientists get great, non-toy traces and workloads to further
>> their own research.  Both parties win.
>>
>
>
> I don't know about this model.
> This is like developing software on prototype hardware.  The hardware guys and gals keep wanting to change the hardware, and the software developers complain that their software keeps breaking, or that the hardware is buggy (and it is).
>
> The computational physics and computational biology guys get to work on cool, nifty stuff to push their dissertation forward by using a hopefully stable computational platform.
> But I don't think the CS guys would drool over the possibility of administering a cluster. The CS guys get to be sysadmin/maintenance types...not very fun for them, and not the kind of work that would work for their dissertation.
>
> Now, if the two groups were doing research on new computational methods (what's the best way to simulate X) perhaps you'd get a collaboration.
>
>
>
>
> ------------------------------
>
> _______________________________________________
> Beowulf mailing list
> Beowulf at beowulf.org
> http://www.beowulf.org/mailman/listinfo/beowulf
>
>
> End of Beowulf Digest, Vol 80, Issue 22
> ***************************************
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
>

Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu