[Beowulf] Broadcast - not for HPC - or is it?

[Lux, Jim (337C)]

Wireless for broadcast is generically a good idea (heck, I make my living doing such stuff) but tricky.. Latency is a challenge if you want off-the-shelf hardware.

The problem is that readily available wireless protocols (e.g. 802.11, 802.14, 802.16) are packet oriented, and more to the point, optimized for things other than precision timing and low latency across a small area. 802.11, for instance, is basically half duplex (e.g. Node doesn't transmit and receive at the same time) but is bidirectional, so that means you have to synchronize in each packet (just like ethernet.. There's a sync pattern at the beginning of each packet), and packets are fairly long, so that the sync is a small overall fraction: optimized for overall throughput.    Most wireless protocols choose a packet length that is convenient for the data rate, and the expected propagation delay, and the bandwidth of the transmitter/receivers.

If you want low latency, precision timing, you need a more classical "broadcast" scheme.. Keep one transmitter on all the time, so that the receivers have plenty of time to synchronize to it, and then transmit your data at whatever rate you want (1 Gbps is easy).  If you're interested in low latency, you're going to want a high symbol rate. Contrast to high bit rate.. Most high speed wireless links encode more than one bit per symbol: 64QAM, for instance, encodes 6 bits in every symbol, so the symbol rate is 1/6th that of the bit rate.    1 Gsymbol/second will, to a first order, need a GHz or two of bandwidth, which isn't a challenge if you're operating at some reasonable microwave frequency (5GHz, for instance).

You'll also need to deal with the decidedly bizarre propagation (multipath, etc.), and unfortunately, you can't rely on adaptive equalization and such, because you're interested in low latency, so you want to reject "late arriving echos".

Various UltraWideBand (UWB) might be a viable way to do this.. Think of them as a sort of "radar" pulse, and every receiver just listens for the pulse.  You could transmit 1 of N different kinds of pulses (say, different frequencies) and transmit log2(N) bits for each pulse.

You're stuck with the "nanosecond/foot" free space propagation, still.. So if you're radiating over your cluster of 1000 processors, physical size becomes an issue.
(this is a fundamental physics limit on computational speed, if the problem can't be pipelined or done in a systolic array.. Hence ideas of highly integrated multiple processors immersed in liquid coolant... Get lots of computation in a small volume)..

The people doing free space optical interconnects have a lot of clever ideas here.  Imagine a sphere with processing nodes on the surface, and an optical terminal sticking into the sphere. The terminal has many transmitters and receivers, with microlenses integrated, so you essentially have a point to point link between each possible pair of nodes.


On 10/8/10 5:40 AM, "Matt Hurd" <matthurd at acm.org> wrote:

>>
> So if you want another offbeat idea on how to do broadcast on a large
> cluster  - what about wireless ?
> (although again, you can't get realistically reliable transport with
> checking acks )
>
> Daniel
>
Nice idea.  Perhaps not so offbeat.  Though bandwidth and radio don't
go too well together normally.

Know some guys here in Australia that are doing extremely accurate
timing with wireless, not for timing's sake but to measure movement in
dam walls and other infrastructure.  Kind of like a localised GPS.
They have some very neat and accurate stuff.

Had a play with some radio foo and managed to get about 880ns + air
time on bit to bit from tx to rx but haven't quite figured out how to
get super low latency yet.  I think there may be a product in there
for wireless PPS dissemination for accurate timing to a cluster like
the guys do with the dam walls but I'm not sure if people really need
much more than what ptp can already do.

--Matt.

_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20101008/84f98603/attachment.html>