[Beowulf] Advice on 4 CPU node configurations

[Craig Tierney]

On 2/8/11 11:54 AM, Gus Correa wrote:
> How about the wiring?
> Since it is switchless, how much cabling would it require?
> The cables don't seem to be fiber.
> How do you implement the torus (or other) topology?
> With daisy-chains?
> A 2D/3D (wraparound) mesh?
> Other?
> I could only find a datasheet for the card and cables
> on their site, no pictures of a cluster or servers
> connected.
>

I believe the cables are still copper.  The connectors
looked similar at SC10 when I tried out Dolphinics
circa 2004.

For a 3D card there are 6 ports, so you get two connections
in the X,Y, and Z directions.  So the 3D torus is just basically
wiring the nodes serially along each axis.  I don't recall
if any of the dimensions wrap around.   But if you use
36 nodes (3x3x4) you can easily build a 864 core/1.7TB shared memory
node in a rack (24 cores/node, 48GB/node).

How it performs is the real question.

Craig



> Gus Correa
>
> Douglas Eadline wrote:
>> Although I wrote about the SMP and the Numascale hardware,
>> I have not yet had a chance to use it.
>>
>> To me there are two issues worth considering. First,
>> if you need a big SMP this might be a low
>> cost solution. Second, should your application
>> not scale best in a node-based ccNuma system (ScaleMP
>> or Numascale), MPI is still an option. Indeed, no need
>> to rewrite the codes. And, management is probably
>> much easier.
>>
>> Of course for large systems, clusters work best.
>>
>> --
>> Doug
>>
>>
>>> On Thu, 2011-02-03 at 09:45 +0000, Hearns, John wrote:
>>>> Also look at ScaleMP and Numascale
>>>> Here's a damn good article from Doug Eadline:
>>>> http://www.linux-mag.com/id/7947
>>>>
>>>> I must admit though I don't know how far a budget of 30K takes you
>>>> there!
>>> My understanding is that the Numascale cards cost "a bit more" than IB
>>> cards but you don't need a switch (it's a 3D taurus I think) so it would
>>> be worth looking into.
>>>
>>> Rob
>>>
>>> _______________________________________________
>>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
>>> To change your subscription (digest mode or unsubscribe) visit
>>> http://www.beowulf.org/mailman/listinfo/beowulf
>>>
>>> --
>>> This message has been scanned for viruses and
>>> dangerous content by MailScanner, and is
>>> believed to be clean.
>>>
>>
>>
>> --
>> Doug
>>
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf