<div dir="ltr"><div>Regarding failed disks, you of course have done the correct thing in sending it to a data recovery company.</div><div><br></div><div>However, I would like to heartily recommend sysrescuecd as a first line tool in these situations:</div><div><a href="http://www.system-rescue-cd.org/">http://www.system-rescue-cd.org/</a></div><div><br></div><div>this is a live CD, which has a utility for scanning for and rebuilding 'lost' partitions.</div><div>In your case this probably would not have helped.</div><div>However I carry a sysrescue image around with me on a USB stick on my lanyard,</div><div>so I am poised and ready to leap into the nearest telephone box and emerge as a superhero,</div><div>should enyone ever cry out "Help - my Linux system has gone down"</div><div>Hasn't happened yet, but I can dream.</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On 19 May 2017 at 09:39, John Hanks <span dir="ltr"><<a href="mailto:griznog@gmail.com" target="_blank">griznog@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thanks Arif, I'm signed up there now. <div><br></div><div>As a general update, the most recently failed disk of the pair is at a data recovery company who thinks they can recover a workable image from it. We should have that back in two or three weeks and will try to use it to recover the filesystem.</div><div><br></div><div>jbh</div></div><div class="HOEnZb"><div class="h5"><br><div class="gmail_quote"><div dir="ltr">On Thu, May 18, 2017 at 5:21 PM Arif Ali <<a href="mailto:mail@arif-ali.co.uk" target="_blank">mail@arif-ali.co.uk</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">
<div bgcolor="#FFFFFF" text="#000000">
Hi John,<br>
<br>
I would recommend joining up at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a> mailing list,
where you will find very good experts from the HPC industry who know
GPFS well, including, Vendors, users and integrators. More
specifically, you'll you'll find gpfs developers on there. Maybe
someone on that list can help out<br>
<br>
More direct link to the mailing list, here,
<a class="m_921538342554566001m_-1841834953486676765moz-txt-link-freetext" href="https://www.spectrumscale.org:10000/virtualmin-mailman/unauthenticated/listinfo.cgi/gpfsug-discuss/" target="_blank">https://www.spectrumscale.org:<wbr>10000/virtualmin-mailman/<wbr>unauthenticated/listinfo.cgi/<wbr>gpfsug-discuss/</a></div><div bgcolor="#FFFFFF" text="#000000"><br>
<br>
<div class="m_921538342554566001m_-1841834953486676765moz-cite-prefix">On 29/04/2017 08:00, John Hanks wrote:<br>
</div>
</div><div bgcolor="#FFFFFF" text="#000000"><blockquote type="cite">
<div dir="ltr">Hi,
<div><br>
</div>
<div>I'm not getting much useful vendor information so I thought
I'd ask here in the hopes that a GPFS expert can offer some
advice. We have a GPFS system which has the following disk
config:</div>
<div><br>
</div>
<div>
<div>[root@grsnas01 ~]# mmlsdisk grsnas_data</div>
<div>disk driver sector failure holds holds
storage</div>
<div>name type size group metadata data
status availability pool</div>
<div>------------ -------- ------ ----------- -------- -----
------------- ------------ ------------</div>
<div>SAS_NSD_00 nsd 512 100 No Yes
ready up system</div>
<div>SAS_NSD_01 nsd 512 100 No Yes
ready up system</div>
<div>SAS_NSD_02 nsd 512 100 No Yes
ready up system</div>
<div>SAS_NSD_03 nsd 512 100 No Yes
ready up system</div>
<div>SAS_NSD_04 nsd 512 100 No Yes
ready up system</div>
<div>SAS_NSD_05 nsd 512 100 No Yes
ready up system</div>
<div>SAS_NSD_06 nsd 512 100 No Yes
ready up system</div>
<div>SAS_NSD_07 nsd 512 100 No Yes
ready up system</div>
<div>SAS_NSD_08 nsd 512 100 No Yes
ready up system</div>
<div>SAS_NSD_09 nsd 512 100 No Yes
ready up system</div>
<div>SAS_NSD_10 nsd 512 100 No Yes
ready up system</div>
<div>SAS_NSD_11 nsd 512 100 No Yes
ready up system</div>
<div>SAS_NSD_12 nsd 512 100 No Yes
ready up system</div>
<div>SAS_NSD_13 nsd 512 100 No Yes
ready up system</div>
<div>SAS_NSD_14 nsd 512 100 No Yes
ready up system</div>
<div>SAS_NSD_15 nsd 512 100 No Yes
ready up system</div>
<div>SAS_NSD_16 nsd 512 100 No Yes
ready up system</div>
<div>SAS_NSD_17 nsd 512 100 No Yes
ready up system</div>
<div>SAS_NSD_18 nsd 512 100 No Yes
ready up system</div>
<div>SAS_NSD_19 nsd 512 100 No Yes
ready up system</div>
<div>SAS_NSD_20 nsd 512 100 No Yes
ready up system</div>
<div>SAS_NSD_21 nsd 512 100 No Yes
ready up system</div>
<div>SSD_NSD_23 nsd 512 200 Yes No
ready up system</div>
<div>SSD_NSD_24 nsd 512 200 Yes No
ready up system</div>
<div>SSD_NSD_25 nsd 512 200 Yes No
to be emptied down system</div>
<div>SSD_NSD_26 nsd 512 200 Yes No
ready up system</div>
<div><br>
</div>
</div>
<div>SSD_NSD_25 is a mirror in which both drives have failed due
to a series of unfortunate events and will not be coming back.
From the GPFS troubleshooting guide it appears that my only
alternative is to run </div>
<div>
<p class="m_921538342554566001m_-1841834953486676765inbox-inbox-p1">mmdeldisk grsnas_data SSD_NSD_25 -p</p>
<p class="m_921538342554566001m_-1841834953486676765inbox-inbox-p1">around which the documentation also
warns is irreversible, the sky is likely to fall, dogs and
cats sleeping together, etc. But at this point I'm already
in an irreversible situation. Of course this is a scratch
filesystem, of course people were warned repeatedly about
the risk of using a scratch filesystem that is not backed up
and of course many ignored that. I'd like to recover as much
as possible here. Can anyone confirm/reject that deleting
this disk is the best way forward or if there are other
alternatives to recovering data from GPFS in this situation?</p>
<p class="m_921538342554566001m_-1841834953486676765inbox-inbox-p1">Any input is appreciated. Adding
salt to the wound is that until a few months ago I had a
complete copy of this filesystem that I had made onto some
new storage as a burn-in test but then removed as that
storage was consumed... As they say, sometimes you eat the
bear, and sometimes, well, the bear eats you.</p>
<p class="m_921538342554566001m_-1841834953486676765inbox-inbox-p1">Thanks,</p>
<p class="m_921538342554566001m_-1841834953486676765inbox-inbox-p1">jbh</p>
<p class="m_921538342554566001m_-1841834953486676765inbox-inbox-p1">(Naively calculated probability of
these two disks failing close together in this array:
0.00001758. I never get this lucky when buying lottery
tickets.)</p>
</div>
</div>
<div dir="ltr">-- <br>
</div>
<div data-smartmail="gmail_signature">
<div dir="ltr">
<div>‘[A] talent for following the ways of yesterday, is not
sufficient to improve the world of today.’</div>
<div> - King Wu-Ling, ruler of the Zhao state in northern
China, 307 BC</div>
</div>
</div>
<br>
<fieldset class="m_921538342554566001m_-1841834953486676765mimeAttachmentHeader"></fieldset>
<br>
</blockquote></div><div bgcolor="#FFFFFF" text="#000000"><blockquote type="cite"><pre>______________________________<wbr>_________________
Beowulf mailing list, <a class="m_921538342554566001m_-1841834953486676765moz-txt-link-abbreviated" href="mailto:Beowulf@beowulf.org" target="_blank">Beowulf@beowulf.org</a> sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit <a class="m_921538342554566001m_-1841834953486676765moz-txt-link-freetext" href="http://www.beowulf.org/mailman/listinfo/beowulf" target="_blank">http://www.beowulf.org/<wbr>mailman/listinfo/beowulf</a>
</pre>
</blockquote></div><div bgcolor="#FFFFFF" text="#000000">
<br>
<pre class="m_921538342554566001m_-1841834953486676765moz-signature" cols="72">--
regards,
Arif Ali
Mob: <a href="tel:+44%207970%20148122" target="_blank" value="+447970148122">+447970148122</a></pre>
</div>
______________________________<wbr>_________________<br>
Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" target="_blank">Beowulf@beowulf.org</a> sponsored by Penguin Computing<br>
To change your subscription (digest mode or unsubscribe) visit <a href="http://www.beowulf.org/mailman/listinfo/beowulf" target="_blank" rel="noreferrer">http://www.beowulf.org/<wbr>mailman/listinfo/beowulf</a><br>
</blockquote></div><div dir="ltr">-- <br></div><div data-smartmail="gmail_signature"><div dir="ltr"><div>‘[A] talent for following the ways of yesterday, is not sufficient to improve the world of today.’</div><div> - King Wu-Ling, ruler of the Zhao state in northern China, 307 BC</div></div></div>
</div></div><br>______________________________<wbr>_________________<br>
Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org">Beowulf@beowulf.org</a> sponsored by Penguin Computing<br>
To change your subscription (digest mode or unsubscribe) visit <a href="http://www.beowulf.org/mailman/listinfo/beowulf" target="_blank" rel="noreferrer">http://www.beowulf.org/<wbr>mailman/listinfo/beowulf</a><br>
<br></blockquote></div><br></div>