<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 TRANSITIONAL//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; CHARSET=UTF-8">
<META NAME="GENERATOR" CONTENT="GtkHTML/1.0.2">
</HEAD>
<BODY>
On Wed, 2002-04-03 at 13:04, Cris Rhea wrote:
<BLOCKQUOTE>
<PRE><FONT COLOR="#737373"><FONT SIZE="3"><I>What are folks doing about keeping hardware running on large clusters?</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I></FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>Right now, I'm running 10 Racksaver RS-1200's (for a total of 20 nodes)...</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I></FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>Sure seems like every week or two, I notice dead fans (each RS-1200</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>has 6 case fans in addition to the 2 CPU fans and 2 power supply fans).</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I></FONT></FONT></I></PRE>
</BLOCKQUOTE>
<FONT SIZE="3">You running lm_sensors on your nodes? That's a handy tool for paying attention to things like that. We use ours in combination with ganglia and pump it to a web page and to big brother to see when a cpu might be getting hot, or a fan might be too slow. We actually saved a dozen machines that way...we have 32 4 processor racksaver boxes in a rack, and they rack was not designed to handle racksaver's fan system. That is to say, there was a solid sidewall on the rack, and it kept in heat. I set up lm_sensors on all the nodes (homogenous, so configured on one and pushed it out to all), then pumped the data into ganglia (ganglia.sourceforge.net) and then to a web page. I noticed that the temp on a dozen of the machines was extremely high. So, I took off the side panel of the rack. The temp dropped by 15 C on all the nodes, and everything was within normal parameters again.</FONT>
<BR>
<FONT SIZE="3"></FONT>
<BLOCKQUOTE>
<PRE><FONT COLOR="#737373"><FONT SIZE="3"><I>My last fan failure was a CPU fan that toasted the CPU and motherboard.</FONT></FONT></I></PRE>
</BLOCKQUOTE>
<FONT SIZE="3"></FONT>
<BR>
<FONT SIZE="3">Ya, we would have seen this on ours earlier...excellent tool</FONT>
<BLOCKQUOTE>
<PRE><FONT COLOR="#737373"><FONT SIZE="3"><I></FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>How are folks with significantly more nodes than mine dealing with constant</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>maintenance on their nodes? Do you have whole spare nodes sitting around-</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>ready to be installed if something fails, or do you have a pile of</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>spare parts?</FONT></FONT></I></PRE>
</BLOCKQUOTE>
<FONT SIZE="3"></FONT>
<BR>
<FONT SIZE="3">No, we don't actually, but we've talked about it</FONT>
<BR>
<FONT SIZE="3"></FONT>
<BLOCKQUOTE>
<PRE><FONT COLOR="#737373"><FONT SIZE="3"><I> Did you get the vendor (if you purchased prebuilt systems)</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>to supply a stockpile of warranty parts?</FONT></FONT></I></PRE>
</BLOCKQUOTE>
<FONT SIZE="3"></FONT>
<BR>
<FONT SIZE="3">we use racksaver as well, so our experience is similar. Probably should talk to our people about getting some spare nodes</FONT>
<BLOCKQUOTE>
<PRE><FONT COLOR="#737373"><FONT SIZE="3"><I></FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>One of the problems I'm facing is that every time something croaks, </FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>Racksaver is very good about replacing it under warranty, but getting</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>the new parts delivered usually takes several days.</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I></FONT></FONT></I></PRE>
</BLOCKQUOTE>
<FONT SIZE="3">Ya...this is another area where just monitoring the data can be helpful...if a fan is failing, you can see it coming (temperature slowly rises) and you can order it before hand and schedule downtime.</FONT>
<BR>
<FONT SIZE="3"></FONT>
<BLOCKQUOTE>
<PRE><FONT COLOR="#737373"><FONT SIZE="3"><I></FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>----</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I> Cristopher J. Rhea Mayo Foundation</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I> Research Computing Facility Pavilion 2-25</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I> crhea@Mayo.EDU Rochester, MN 55905</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I> Fax: (507) 266-4486 (507) 284-0587</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>_______________________________________________</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>Beowulf mailing list, Beowulf@beowulf.org</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>To change your subscription (digest mode or unsubscribe) visit </FONT></FONT></I><A HREF="http://www.beowulf.org/mailman/listinfo/beowulf"><FONT SIZE="3"><I>http://www.beowulf.org/mailman/listinfo/beowulf</FONT></I></A></PRE>
</BLOCKQUOTE>
<TABLE CELLSPACING="0" CELLPADDING="0" WIDTH="100%">
<TR>
<TD>
<PRE>--
Douglas J Nordwall http://rex.nmhu.edu/~musashi
System Administrator Pacific Northwest National Labs</PRE>
</TD>
</TR>
</TABLE>
</BODY>
</HTML>