[Beowulf] Advice on 4 CPU node configurations

[Craig Tierney]

On 2/9/11 1:21 PM, xingqiu yuan wrote:
>
>
> On Wed, Feb 9, 2011 at 2:37 PM, Prentice Bisbal <prentice at ias.edu
> <mailto:prentice at ias.edu>> wrote:
>
>     Craig Tierney wrote:
>     >  On 2/9/11 7:38 AM, Prentice Bisbal wrote:
>     > > I don't think Numascale/ScaleMP has much of a cost advantage anymore.
>     > >
>     > > About 6 months ago, I purchased a couple of Dell PowerEdge R815s with
>     > > 128 GB of RAM and 32 cores. We looked at similar RAM configurations a
>     > > couple of years ago. and the cost premium for that much RAM was
>     > > prohibitive (the price for additional RAM seemed to go up
>     exponentially
>     > > with the amount of RAM) so we stayed at 32 GB.  This time around, the
>     > > premium for the additional RAM seemed marginal. Not sure how
>     large you
>     > > can go and keep that "marginal" relationship, but it's much
>     larger than
>     > > it was a couple of years ago.
>     > >
>     >
>     >  There is only so much memory you can put in a box, regardless of the
>     >  cost.  ScaleMP/Numascale let you get around that issue.  What if I
>     need
>     >  a TB of RAM?  Yes, I might be able to convert my algorithms, but
>     if I can
>     >  add $3k per node to get cache-coherent memory, why not go that route?
>
>     >Agreed. I was going to add the caveat that there are limits to how much
>     >RAM you can add to a single box, or there eventually be a point the
>     >amount of RAM in a single box vs. price is no longer nonlinear, but then
>     >I got lazy. ;)
>
>
>
> Don't understand why you need TB of RAM? Don't forget the fact that $3K
> can almost buy one-node!!!
>
>

Reasons for a TB of RAM include databases, a lot of users that need 10's of
GB, but also having the additional cores provides some flexibility.

I can't say for certain a setup like this would be beneficial, but at $3k per
node, I would be willing to try it out!

You are right about the node price.  Well, I would put the cost much higher.
You could put 48GB in a node without much of a price penalty, and then you
are probably paying $5-6K per node.  Also, the $3k/node is list and who
ever pays list? :-)

Craig




>
>     >
>     >
>     > > And don't forget to figure in the cost of the interconnect. I
>     don't know
>     > > what Numascale requires, but ScaleMP require IB, which can add
>     > > significant costs if you don't have an existing IB fabric to use.
>     > >
>     >
>     >  ScaleMP is a software solution, Numascale is a cache-coherent
>     hardware solution.
>     >  The list cost of the 3D torus version Numascale card with 4GB Ram
>     (highest end card).
>     >  Cables are similar to IB, about $120 each.
>     >
>     >  It would be really cool to build SMP system to sit next to my
>     cluster.  When users
>     >  need large memory, or want to do things with UPC or other similar
>     languages,
>     >  an SMP system like this would be handy.
>     >
>     >  - I am not advocating for Numascale, or any other products here.
>       I am just
>     >  trying to provide information to the community.
>     >
>     >  Craig
>     >
>     >
>     >
>     > > Prentice
>     > >
>     > >
>     > > Douglas Eadline wrote:
>     > >> Although I wrote about the SMP and the Numascale hardware,
>     > >> I have not yet had a chance to use it.
>     > >>
>     > >> To me there are two issues worth considering. First,
>     > >> if you need a big SMP this might be a low
>     > >> cost solution. Second, should your application
>     > >> not scale best in a node-based ccNuma system (ScaleMP
>     > >> or Numascale), MPI is still an option. Indeed, no need
>     > >> to rewrite the codes. And, management is probably
>     > >> much easier.
>     > >>
>     > >> Of course for large systems, clusters work best.
>     > >>
>     > >> --
>     > >> Doug
>     > >>
>     > >>
>     > >>> On Thu, 2011-02-03 at 09:45 +0000, Hearns, John wrote:
>     > >>>> Also look at ScaleMP and Numascale
>     > >>>> Here's a damn good article from Doug Eadline:
>     > >>>> http://www.linux-mag.com/id/7947
>     > >>>>
>     > >>>> I must admit though I don't know how far a budget of 30K takes you
>     > >>>> there!
>     > >>> My understanding is that the Numascale cards cost "a bit more"
>     than IB
>     > >>> cards but you don't need a switch (it's a 3D taurus I think) so
>     it would
>     > >>> be worth looking into.
>     > >>>
>     > >>> Rob
>     > >>>
>     > > _______________________________________________
>     > > Beowulf mailing list, Beowulf at beowulf.org
>     <mailto:Beowulf at beowulf.org> sponsored by Penguin Computing
>     > > To change your subscription (digest mode or unsubscribe) visit
>     http://www.beowulf.org/mailman/listinfo/beowulf
>     >  _______________________________________________
>     >  Beowulf mailing list, Beowulf at beowulf.org
>     <mailto:Beowulf at beowulf.org> sponsored by Penguin Computing
>     >  To change your subscription (digest mode or unsubscribe) visit
>     http://www.beowulf.org/mailman/listinfo/beowulf
>     >
>
>     --
>     Prentice Bisbal
>     Linux Software Support Specialist/System Administrator
>     School of Natural Sciences
>     Institute for Advanced Study
>     Princeton, NJ
>     _______________________________________________
>     Beowulf mailing list, Beowulf at beowulf.org
>     <mailto:Beowulf at beowulf.org> sponsored by Penguin Computing
>     To change your subscription (digest mode or unsubscribe) visit
>     http://www.beowulf.org/mailman/listinfo/beowulf
>
>
>
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf