[Beowulf] massive parallel processing application required

[Jim Lux]

At 03:25 PM 2/1/2007, Mark Hahn wrote:
>>>the internet bubble.  in those days, it was popular to claim that 
>>>the network
>>>was becoming truely ubiquitous and incomprehensibly fast.  for instance:
>>
>>In the long run, ubiquitous and fast IS going to be true (however, latency is
>
>in the long run, everything is true ;)
>
>>gross oversupply of fiber across the Atlantic.  Hence the 
>>availability of cheap flat rate long distance (5c a minute 
>>anywhere, anytime).. the bulk of the system is no longer capacity limited.
>
>interesting - I assumed that long-distance became cheap not due to 
>oversupply of fiber and bandwidth, but rather transition away from 
>old-fashioned circuit switching (ie, towards digital compressed voice
>over packets.)

Not much compression going on for voice traffic.  It's carried as 64 
kbps data at 8 ksamples/second, pretty much.  There is some 
statistical multiplexing possible (TASI) because people don't talk at 
100% duty cycle, but not a huge amount.


>I know that buying fiber/lambdas/bandwidth is still very much not 
>what I'd call cheap, though I have no doubt it's much better/cheaper
>than in the past.

In 1993, the capital cost of "one voice channel" worth (64 kbps) of 
capacity across the atlantic was less than $10, as I recall.

Compare that to leased line T-1 rates back then of many dollars per 
eighth of a mile per month, and that's before you bought the CSU/DSU 
to connect to the copper.

Mind you, the ATT guy thought that it would be 155 Mbps ATM to the 
desktop, and we see where that went.



>>>I don't know about you, but in the 6 years since then, my home net 
>>>connection has stayed the same speed, possibly a bit more expensive.
>>
>>Interestingly, they've just rolled out FiOS (fiber to the home) in 
>>my area, which is a HUGE jump in potential bandwidth from the 
>>existing DSL or Cable Modem delivery methods.  And, moderately 
>>competitive in price (5 Mbps is $40/month, including the bundled 
>>ISP kinds of features).  What's fascinating is the faster tiers.. 
>>you can get 15 Mbps down/2 up for $50/mo and 30 M down/5 up for $180
>
>seems strange to me - what kind of residential customer would pay 
>for that kind of thing (and remain free of the RIAA/MPAA)?

An interesting question.. I think the upper tiers are there to 
complement similar offerings in the commercial/business market.  Or, 
there IS a burgeoning market for live video feeds from adult 
entertainment providers.  Without them, the VCR market would never 
have taken off.  Contemplate Youtube type applications, but in HD.. 
20 Mbps is the basic rate for HD.   I might be interested, for 
instance, in seeing the Mentos and Pepsi artists in HD, rather than 
lowfi 15 fps QCIF.

But, also, consider something like streaming audio at CD quality (not 
MP3 compressed).. A stereo 44.1ksps 16 bit stream is about 1.5 Mbps, 
and say I, my wife, and my daughters all want to listen to different 
programs at the same time.

There will also be video that is not afflicted by MPAA.  NasaTV is 
free to all and streamed over the network as well as being shoved out 
over C-band transponders.

I can see using 15M+ sorts of rates in bursts for myself (downloading 
the aforementioned climate databases, for instance...)


>some smart form of wireless seems like an obvious good solution for 
>residential last-mile.  maybe that's a disruptive innovation that will
>finally put the telco/cableco's out of their misery.

Nobody has come up with a *good* wireless solution that is as cheap 
and reliable as pulling a physical media.  There's a raft of spectrum 
occupancy issues, etc.

Let's assume you've got a neighborhood with 400 houses in it at a 
density of, say, 500 square meters/house (roughly 8 
houses/acre).  It's perhaps 10-20 meters between houses on 
average.  Say each house needs 50 Mbps of bandwidth (e.g. two cable 
channels worth).  If you use a short range wireless scheme (notional 
range of 50 m) a given transmitter is going to cover half a dozen 
houses, so each transmitter would need a bandwidth of about 300 Mbps 
(which is fairly hefty, but not out of the question).  AND there 
would need to be some smart switching in the system that feeds that 
transmitter the correct subset of the Terabits/second 
available...  And, some way to cleverly do spectrum reuse (so that if 
you have houses A, B,C, D, and E lined up, A can use channel 1, B can 
use channel 6, C can use channel 11, and by house D, the signal for 
Channel 1 going to house A has faded enough that we can reuse it for 
D, 6 for E, etc.)  This is highly nontrivial, and nobody has come up 
with a automagic way to do it that is efficient and self organizing.

Right now, though, the Cable TV folks feed 1 GHz of bandwidth to you 
and YOU do the channel selection, which reduces their physical plant 
cost... all they need is power distribution with no intelligence, 
just management of SNR.  (this breaks down in the upstream case, 
which is a fundamental problem with Cable Modems)


>>Likewise, a small business with half a dozen or a dozen desktops 
>>and a couple servers isn't going to see a huge benefit from faster 
>>networking, because they're throttled by the server's disk speed, 
>>more than anything else.
>
>if their servers disks are only 100bT speed, they're broken.  it may well
>be that most SMB servers are that crappy, in spite of the fact that 
>a recycled linux box and one disk will deliver 40 MB/s...

Not so much a limitation as that, as the 10 desktops aren't all going 
to be hitting the server at exactly the same time, most of the 
time.  Relatively few business desktops are doing things like 
streaming video.  They're just moving documents to and from the 
server, and that's a sort of bursty traffic, so its not a big 
deal.     And 40 MB/s implies 13 Megatransfers/second across a 32 bit 
bus, with a 33 MHz bus, a transfer from the disk and a transfer to 
the NIC doesn't leave a whole lot of time for fetching instructions 
from RAM, etc.


now, if your office is comprised of diskless clients....that's another story.


>>So, you're looking at GigE making a difference in two 
>>areas:  replacing cable TV (all those 20 Mbps HDTV streams)
>
>how many 20Mb streams does a typical endpoint need?  either residential
>or commercial?


I can see at least 3 streams for residential.  1 for live viewing, 1 
for recording on the TIVO, 1 for the second TV.


>>and in big companies.  But even in big companies, GigE to the 
>>desktop doesn't necessarily buy you much, if you're all competing 
>>for the same server resources.
>
>wow, dim view of the competence of server admins, but you may be right...

No.. it's that network traffic from desktop to server just isn't all 
that high in most environments.  For instance, I consume almost NO 
network bandwidth most of the time at work, because most of what I 
work with is on the local machine.

Even in a high transaction rate call center, there's just not that 
many bytes flying back and forth.  "yes, Mr. Lux, and your account 
number is? ...." blurp there's 100 bytes to the server in a SQL 
query, and maybe a kilobyte coming back.

10 seconds pass,
"And you'd like the bullion delivered where?"
blurp.. after 30 seconds, the operator sends the delivery address 
with a few hundred bytes to the transaction processor.
blurp...100 bytes come back
"your confirmation number is 2.71828, Thank you for calling"

Then, that triggers a few more kbytes of traffic to the vault and the 
delivery truck company, etc.

But overall, that's what, an average of 1 kb per second, at most?  So 
the call center has 1000 people.. we're up to only 1 Megabit/second.

Even if they do complete screen paints at every step over the 
network, it's still not that much traffic.

Some sort of call center where they look at scanned images might be 
an example of a bigger volume user.  "Yes.. I'm looking at your 
bearer bonds now, and we'll be able to execute that sell order for 
100,000 shares MSFT."  or, more realistically, "I'm looking through 
your loan application now and on page 32, there's a problem with the 
property description you submitted three years ago." or "Yes, Mr. 
Lux, that IS a big dent that we need to fix in your bumper"  But even 
then, a full screen image is only a few megabytes, at most, unless 
you're totally profligate with uncompressed 24 bit TIFF images.

The big advantage of GigE to the desktop is that when you do send big 
files (say a full screen image), it takes less time.  But the average 
rate is still low.
>regards, mark hahn.

James Lux, P.E.
Spacecraft Radio Frequency Subsystems Group
Flight Communications Systems Section
Jet Propulsion Laboratory, Mail Stop 161-213
4800 Oak Grove Drive
Pasadena CA 91109
tel: (818)354-2075
fax: (818)393-6875