[Beowulf] High Performance SSH/SCP

[Robert G. Brown]

On Fri, 15 Feb 2008, Mark Kosmowski wrote:

>>
>>
>>
>> Robert G. Brown wrote:
>>>  Rsh "and" anything else is difficulty squared,
>>> and kerberos isn't the universally implemented tool it was a decade ago,
>>> largely superceded by ssh and/or ssl connections.  So finding experts to
>>> help you make it work if you're a newbie isn't going to be that easy.
>>
>> I don't think there's anything difficult about setting up rsh, ssh or
>> kerberos for anyone who know how to read a manual. A newbie shouldn't be
>> setting up a cluster in the first place. That's advanced kung-fu best
>> left to the black belts. Letting a neophyte build and run an HPC cluster
>> is some kind of oxymoron.
>>
>> Yes, I know that professors usually tell some green graduate student to
>> go build a cluster for the dept, but that's a completely different topic
>> outside the scope of this list...
>
>
>
> I'm either not as much of a newbie / neophyte as I think I am or I missed
> the memo about this list being for pro's only.

I'm not certain I agree with the principle premise anyway, so don't
worry about it.  I myself am a mere amateur by this standard, as almost
every cluster I ever heard of was built (originally) by a neophyte.
Very few places, almost none at Universities, hire a "cluster expert" to
come build a cluster.

Although that this could be said at all is a testament to the immense
success of the COTS cluster design and this list.  After all, I'd guess
that 90% or so of all "cluster professionals" in the universe got their
start, as newbies, right here.  There are at this point a tiny handful
of schools that have cluster computing programs, but a whole lot of pros
are created through a sort of "dynamic interactive apprenticeship" on
this list.  Some of them jumped in with some sort of sysadmin
experience, many came in from the research side in various sciences,
driven by the universal Hunger for Cycles that drives us all...;-)

At any given time, I usually have anywhere between two and five
"students" around the world who are communicating with me offlist while
trying to set up their own first cluster, so I actually have some reason
for making these statements.  I have three right now who are high school
students, seriously.  Bright ones.  One west coast and two east coast.
As well as a couple in India where I'm not so sure what kind of program
they are in.  Some I communicate with just once or twice and they're off
and running, others I talk with over months, explaining this, helping
them solve that, getting them to where they can actually install,
manage, and run jobs in parallel on a tiny cluster (and then they're
usually on their own and flying free).

Obviously the high school students aren't professional sysadmins (or
professional anything).  Sometimes they are learning linux from scratch.
They learn fast.

> I think it would be a disservice to the community to turn away cluster
> newbies from this list.  At the very least encouragement and resource links
> should be provided.  Appropriately experienced list members with a bit of
> time are also free to take discussions off to private email if that is more
> appropriate than the list in general.  After all, the world is replete with
> examples of complete newbie's coming up with ideas to revolutionize the
> fields to which they are new.

Don't sweat it.  Newbies are all welcome, and we (most of us) started
out as newbies once upon a time and were made welcome ourselves.  I
couldn't begin to count the number of times I've answered the "how do I
get started" question on this list.  And I'm hardly alone -- there are
at least thirty other people on this list who chime right in, and I'd
bet money that several of them help out random students offlist as well.

I'm just as happy to answer questions on or off list, and they don't
have to be about ubertech stuff.  You never know when you're going to
learn something useful.  Even the "useless" stuff on Tesla coils is
relevant to someone wondering why equipment racks sometimes come with
metal grids on all sides, and why those racks (that are typically a lot
more expensive) are still purchased.  Is it vanity?  Do we like our
cluster to look cool?  Well, maybe a little.  But a GOOD reason to do it
is if you're in an electrically noisy environment, which can happen in
both industry and really, really easily in a physics department...

I personally learn important EE things from Jim Lux every time he opens
his digital mouth, and I already know a LOT about E&M (and usually
"should" know the things he kindly corrects me on:-).

    rgb

>
> Mark Kosmowski
>

-- 
Robert G. Brown                            Phone(cell): 1-919-280-8443
Duke University Physics Dept, Box 90305
Durham, N.C. 27708-0305
Web: http://www.phy.duke.edu/~rgb
Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php
Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977