[Beowulf] NFS Read Errors

[Joe Landman]

Hi Michael:

Michael H. Frese wrote:
> We were having trouble restarting from our homegrown parallel 
> magnetohydrodynamic code's checkpoint files.  The files could be read, 
> but funny things happened in the run afterward.  Eventually we figured 
> out that the restarted parallel run differed from the serial restarted 
> run from the same checkpoint.
> 
> After much gnashing of teeth and rending of apparel, we found that the 
> checkpoint files were being read incorrectly across NFS.  That let us 
> simplify our search for the problem.  We first found that the local md5 
> digest [openssl dgst -md5 (file...)] on an NFS cp'ed version of the file 

	md5sum filename

does the same thing with a slightly simpler syntax.  There is mounting 
evidence that you should use sha1sum rather than md5sum.

> was different from that produced on the original file.  What was 
> interesting was that the copy either took forEVER -- like 10 minutes or 
> 20 minutes for a 1 GB file -- when the final result was bad or it took 
> about a minute when the file was perfect.  I'm guessing that whatever 
> error checking that gets done on the packets was rejecting so many it 
> finally got a bad packet it couldn't tell was bad.

Sounds a great deal like a bad disk/disk system or something mucking 
with your connection to the data.  1 GB file, even at 1 MB/s is 1000 
seconds, or 16 minutes.  If you have a disk which keeps timing out, or 
has bad blocks, and keeps retrying, well, stuff like this can happen, 
especially on old kernels (and old hardware).

Could also be a RAM error.

> 
> When we found that doing the md5 digest on a remote file produced a 
> different result than doing it on the processor on which the disk was 
> mounted, our tests got simpler.  And shorter, still, after we found that 
> we could get fairly frequent failures with 10 MB files or smaller.  
> Clearly we had an NFS failure, probably associated with hardware.

Yes.  I would venture a guess that you are seeing *lots* of errors in 
your /var/log/syslog or /var/log/messages files.


> This was all between two specific nodes of our small cluster.  [Old 
> hardware generally: AMD Athlon 32-bit single (MSI KT4V) and dual 
> (Tyan...) chip motherboards both running Redhat 9 one with the 2.4.20-8 
> kernels, though one is the smp version; NetGear GA311 NICs; and a 

Owie...

> NetGear GS108 8 port Copper 1 GB/s switch.  The single processor 
> motherboards have 32-bit PCI slots so their network speeds are limited 
> to 300 kbps as shown by netpipe.  All of the LEDs at the ends of the 
> cables show 1000Mb connections.]

300 kbps?  thats 300 kilo bits per second (abbreviations are *very* 
important to get right, kB/s is not the same as kb/s).  300 kbps is 
usually read as 300 kilo bits per second.  Or about about 37.5 kB/s. 
Which is about the average speed of various DSL lines.

I hope you mean 30 MB/s (or 240 Mb/s).

> 
> Then we started checking other pairs.  Some were fine.  Some were bad in 
> the same way.  So we replaced the switch, changing to a 16 port NetGear 
> GS216.  That seemed to cure most of the problem.  But we continued to 

We have seen bad switches a few times.

> have problems copying a file on one particular single processor machine 
> from the others.
> 
> That's where we are now.  The md5 digest run on that machine 
> consistently shows the same result, whereas the digest for that file 
> produced on a remote machine will be almost stochastic.  In some cases 
> it will eventually settle in to the right answer, and then the speed 
> goes WAY up.  I suppose that happens because the file request can be 
> served from the local machine's cache.  But why doesn't it happen after 
> it received bad blocks?

I am guessing you are using TCP NFS mounts as well?  TCP forces retries 
in the event of bad packets.  UDP doesn't force this, but the NFS 
protocol will try.  Ram errors, bad cables, burnt switches, and machines 
with interrupt problems (old machines often shared interrupts without 
being able to do a very good job of it).

> Most, if not all of the original network cards in those machines went 
> bad and have been replaced in the last few years, so I decided to try a 
> brand new GA311.  No joy there.  It still gives out the wrong info.  I 
> guess the motherboard PCI bus controller is hinky, but I'm far from sure.

Did you try a new cable?  Had a few cables go bad, usually they are 
marginal to begin with.

> 
> We are in the process of upgrading and thus replacing all the machines 
> we have of that configuration due to space limitations and their age, 
> but I'm still curious what the problem could be.

There are quite a few possibilities unfortunately.  Unless you plan to 
use these existing machines for quite a while longer, it might be less 
painful to shut off the malfunctioning node.

> 
> Suggestions?  Comments?

2.4.20?  Athlons?  I would say a serious hardware/OS refresh is in order :)



-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web  : http://www.scalableinformatics.com
        http://jackrabbit.scalableinformatics.com
phone: +1 734 786 8423
fax  : +1 866 888 3112
cell : +1 734 612 4615