[Beowulf] Geriatric computer does not stay up

[Gus Correa]

Hi David

Some of the built-in 3Com Ethernet 100 interfaces on
Tyan S2466[-4M] motherboards we have here became flaky/failed
after many years of use.
Those are main boards in in several standalone workstations/PCs.
I don't administer those systems, but I believe the symptoms
were somewhat random, as those you describe.

Disabling the onboard Ethernet (by jumper), and replacing them by
PCI Ethernet 100 cards, gave those systems additional lifetime.
Would this be the case of your cluster node?

Interesting that I also posted today a note asking for help with
Gigabit Ethernet on these very same motherboards!
We also have them in an old workhorse cluster.

Gus Correa
---------------------------------------------------------------------
Gustavo Correa
Lamont-Doherty Earth Observatory - Columbia University
Palisades, NY, 10964-8000 - USA
---------------------------------------------------------------------


Gerald Creager wrote:
> David Mathog wrote:
>> So we have a cluster of Tyan S2466 nodes and one of them has failed in
>> an odd way. (Yes, these are very old, and they would be gone if we had a
>> replacment.)  On applying power the system boots normally and gets far
>> into the boot sequence, sometimes to the login prompt, then it locks up.
>>  If booted failsafe it will stay up for tens of minutes before locking.
>>  It locked once on "man smartctl" and once on "service network start". 
>> However, on the next reboot, it didn't lock with another "man smartctl",
>> so it isn't like it hit a bad part of the disk and died.  Smartctl test
>> has not been run, but "smartctl -a /dev/hda" on the one disk shows it as
>> healthy with no blocks swapped out.  Power stays on when it locks, and
>> the display remains as it was just before the lock.  When it locks it
>> will not respond to either the keyboard or the network.  (The network
>> interface light still flashes.)  There is nothing in any of the logs to
>> indicate the nature of the problem.
>>
>> The odd thing is that the system is remarkably stable in some ways.  For
>> instance, the PS tests good and heat isn't the issue: after running
>> sensors in a tight loop to a log file, waiting for it to lock up, then
>> looking at the log on the next failsafe boot, there were negligible
>> fluctuation on any of the voltages, fan speeds, or temperatures.  It
>> will happily sit for 30 minutes in the BIOS, or hours running memtest86
>> (without errors).  The motherboard battery is good, and the inside of
>> the case is very clean, with no dust visible at all.  Reset the BIOS but
>> it didn't change anything.
>>
>> Here are my current hypotheses for what's wrong with this beast:
>>
>> 1. The drive is failing electrically, puts voltage spikes out on some
>> operations, and these crash the system.
>> 2. The motherboard capacitors are failing and letting too much noise in.
>>  The noise which is fatal is only seen on an active system, so sitting
>> in the BIOS or in Memtest86 does not do it. (But the caps all look good,
>> no swelling, no leaks.)  It will run memtest86 overnight though, just in
>> case.
>> 3. The PS capacitors are failing, so that when loaded there is enough
>> voltage fluctuation to crash the system.  (Does not agree very well with
>> the sensors measurements, but it could be really high frequency noise
>> superimposed on a steady base voltage.)
>> 4. Evil Djinn ;-(
>>
>> Any thoughts on what else this might be? 
> 
> 
> I'd also be suspicious of memory failures.  We have had DIMM failures 
> that were unseen on repeated MemTest86 runs until they failed hard, 
> hard, HARD.  While they were still trying to decide, they'd pass MemTest 
> and we'd try using them.
> 
> Capacitor failures are a potential problem but if the systems have been 
> in a stable environment and not subject to a lot of thermal stressors, 
> they should be fine.  Especially the power supply caps shouldn't decide 
> to get old and fail (I'm assuming you're talking electrolytics).  The 
> old paper electrolytics might have exhibited this behavior, but not even 
> tantalums will do this.  And, if tantalum caps go, they tend to be more 
> spectacular and take lots of other parts with them.
> 
> More to the point, (ceramic) chip caps that haven't been in a 
> wet/moist/temp-varying, humid environment shouldn't crack and fail.
> 
> Option 4 has potential, though.
> 
> gc
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit 
> http://www.beowulf.org/mailman/listinfo/beowulf