[Beowulf] why we need cheap, open learning clusters

[Joe Landman]

On 05/12/2013 01:55 PM, Lux, Jim (337C) wrote:
>
>
> I just ran across an interesting anecdote (in Malcolm Gladwell's
> "Outliers").  It's in the context of Bill Joy, who commented that using
> timesharing and interactive systems compared to traditional batch/card
> deck submission was like speed chess vs chess by mail.  That
> interactivity facilitated his spending thousands of hours working with
> software.
>
> I see parallels to the cluster world.  The original Beowulfs were
> essentially owned by one person, who could do what they wanted with
> them, when they wanted to. If they wanted to reload all the software
> with a new version and try something, they could do that. If they wanted
> to rearrange the network switches, they could.  It's very interactive.
>
> Compare to the current large clusters. When you have 1000 nodes, you're
> not going to say "hmm, if we rearrange the interconnect, I wonder what
> happens". And that million dollar machine is going to have a batch
> queue, a scheduler, etc. . I am amused to read all the stuff on this
> list over the years, as we've moved from "bunch of boxes on shelves in
> my office" to things very reminiscent of my early days in big iron.
>   Sure, the modern user of a cluster doesn't have to punch a deck of
> cards and hike down to the computer center (or, if lucky, to the RJE
> station in their building); they can submit the job by a few keystrokes
> online.  But it's still "submit and wait", as opposed to "type line,
> press enter, and get results immediately".
>
> Gladwell and Joy talk about this interactivity in the context of the
> famous 10,000 hour thing. (it takes 10,000 hours of doing something to
> become proficient).  If your "cycle time" for a job is an hour, it takes
> a LONG time to accumulate the 10,000 hours (especially, if most of the
> time is spent doing things like punching cards or reading greenbar
> output listings.. That's not part of the "learning computer" stuff.  On
> the other hand, if you can make a change in a few lines with a text
> editor (SOS on a DECWriter, I came to love you after I cast off the
> shackles of an 029), run the program, and see what happens, proficiency
> come that much faster.
>
> This is why I think things like ArduWulf or, more particularly LittleFE,
> are valuable.  And it's also why nobody should start packaging LittleFE
> clusters in an enclosure. Once all those mobos are in a box with walls,
> it starts to discourage random and rapid experimentation. If you put a
> littleFE in a sealed box with an inventory tag and a "breaking this seal
> voids warranty" and the only interface is the network jack or
> keyboard/monitor,  you might as well put a modern multicore mobo in
> there and spin up VM instances.  In this case, it's the very "assembled
> in a garage" kind of look that prompts the willingness of someone to go
> in and make some unauthorized changes, from which comes learning.
>
> The learning cluster has to be cheap enough (and, I think physically
> portable) to be "owned" by a single person. Otherwise, it starts to be
> "community, shared property", and subject to access restrictions.  It
> starts to look like significant capital equipment, with only authorized
> service, compliance with corporate/institutional IT security rules: Do
> you have all your patches up to date? Are you running the institutional
> virus checker?.   Do you have full disk encryption?

I had forgotten to plug this, but have a gander here: 
http://limulus.basement-supercomputing.com/

Cluster in a box, by one of our (Beowulf's) own.

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics, Inc.
email: landman at scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/siflash
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615