[Beowulf] File server dual opteron suggestions?

[Gerry Creager N5JXS]

Mark Hahn wrote:
>>> We'll probably go with midrange Ultra320 SCSI internal disks again
>>> (10K 74Gb drives from Maxtor or Seagate) since the high end SATA
>>> drives cost nearly as much for the same capacity.  For this particular
> 
> 
> but why are "boutique" SATA drives the appropriate comparison?
> compare instead to 5-year waranteed "raid edition" drives.
> 
>> Hmmm.... I would argue that 7200 RPM disks make more sense for a 
>> number of reasons.
> 
> 
> me too.  there are very few places where higher RPM is justified: it 
> gives you a lower latency to write-commit.  doesn't give a higher rate 
> of write commits (since more, slower spindles do that).  doesn't
> give a higher bandwidth, either.

We've got over 30TB spinning on SATA disks.  I just don't see us going 
to SCSI anytime soon (or, for that matter, anytime).

>>> socket 940   (or is that a mistake these days?)
>>
>>
>> Not really.  Socket F stuff is on pricewatch and other places.
> 
> 
> I certainly wouldn't fear buying "old" s940 stuff.  sure, it's becoming
> obsolete, but not quickly, and besides, why does that matter?  it's 
> become pretty uncommon to upgrade CPUs (and ddr ram will be around for 
> at least a year or two longer.  actually, I wouldn't be surprised if ddr2
> had a shorter total lifespan...)
> 
>>> 512M-1Gb ECC memory per socket (enough for a file server, no
>>>    serious computing on this node.)
>>
>>
>> I would recommend upping the memory.  Computing or not, large buffer 
>> caches on file servers are with very rare exception, a preferred config.
> 
> 
> unclear.  the FS's memory does act as an excellent cache, but then again,
> the client memory does too.  do you have a pattern of file accesses in 
> which
> the same files are frequently re-read and would fit in memory?  the servers
> I've looked at closely have had mostly write and attribute activity,
> since the client's own cache already has a high hit-rate.  for writes, of
> course, more FS memory is not important unless you have extremely high 
> bandwidth net and disks.  in fact, I've been using the following 
> sysctl.conf
> entries:
> 
> # delay writing dirty blocks hoping to collect further writes (default 30s)
> vm.dirty_expire_centisecs = 1000
> # try writing back every 1s (default 500=5s)
> vm.dirty_writeback_centisecs = 100
> 
> in short, don't bother working at write caching much.  with a lot of 
> memory,
> an untuned machine will exhibit unpleasant oscillations of delaying writes
> then frantically flushing.
> 
>> 2Gb/socket minimum.  Nothing serves files faster than having them 
>> already sitting in ram.
> 
> 
> true, but is that actually your working set size?  it would be rather 
> embarassing if 3 of the 4 GB were files read once a month...
> 
>>> 4 x 74 Gb disks Ultra320 (or make an argument for a particular SATA)
> 
> 
> SATA disks are SATA disks, of course.  dumb controllers are all pretty
> similar as well (cheap, fast, not-cpu-consuming).  if you have your
> heart set on HW raid, at least get a 3ware 9550, which is quite fast.
> (most other HW raid are surprisingly bad.)

I strongly concur here.  And, get the enterprise SATA.  We've gone thru 
a lot of consumer-grade SATA over the last year, while the enterprise, 
purchased a little later, after we started seeing some issues, have just 
kept ticking along.

>>> dual 10/100/1000 ethernet on the mobo
>>
>>
>> Careful on this... we and our customers have been badly bitten by tg3 
>> and broadcom NICs.  If the MB doesn't have Intel NICs, get an Intel 
>> 1000/MT dual gigabit card.  You won't regret that, and it is money 
>> well spent.
> 
> 
> that's odd; I have quite a few of both tg3 and bcm nics, and can't say 
> I've had any complaints.  what are the problems?

I haven't tuned much on tg3 or broadcom NICs, but we also see better 
performance with the Intel server-grade dual GBE NICs.

>>> case - 2U (big enough for adequate ventilation, right?)
>>
>>
>> Yeah, just make sure you have good airflow.
> 
> 
> 2U still requires a custom PS, doesn't it?  it's kind of nice to be able 
> to put in an ATX-ish PS.  and is 2U tall enough for stock/standard
> heatsink/fans?

I prefer a 4u for file servers, and fans... lots of fans.

If you end up with 10k or faster drives, or if you go with 500GB or 
larger SATA, the drives will get hot.  Fans and good forced air or 
appropriate ambient temperatures at the case inlet are your friend.

gerry
-- 
Gerry Creager -- gerry.creager at tamu.edu
Texas Mesonet -- AATLT, Texas A&M University	
Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.862.3983
Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843