<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>Yeah we've pinged Intel/Whamcloud to find out upgrade paths as we
wanted to know what the recommended procedure is.<br>
</p>
<p>Sure. So we have 3 systems that we want to upgrade 1 that is a PB
and 2 that are 5 PB each. I will just give you a description of
one and assume that everything would scale linearly with size.
They all have the same hardware.</p>
<p>The head nodes are Dell R620's while the shelves are M3420 (mds)
and M3260 (oss). The MDT is 2.2T with 466G used and 268M inodes
used. Each OST is 30T with each OSS hosting 6. The filesystem
itself is 93% full.</p>
<p>-Paul Edmon-<br>
</p>
<br>
<div class="moz-cite-prefix">On 07/23/2018 01:58 PM, Jeff Johnson
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAFCYAse34C8FJukBXhTn__+MBk3R7_9LJTQs2N+H+WDs5=xWHg@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=utf-8">
<div dir="ltr">Paul,
<div><br>
</div>
<div>How big are your ldiskfs volumes? What type of underlying
hardware are they? Running e2fsck (ldiskfs aware) is wise and
can be done in parallel. It could be within a couple of days,
the time all depends on the size and underlying hardware. </div>
<div><br>
</div>
<div>Going from 2.5.34 to 2.10.4 is a significant jump. I would
be sure there isn't a step upgrade advised. I know there has
been step upgrades in the past, not sure about going to/from
these two versions. </div>
<div><br>
</div>
<div>--Jeff</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Jul 23, 2018 at 10:34 AM, Paul
Edmon <span dir="ltr"><<a
href="mailto:pedmon@cfa.harvard.edu" target="_blank"
moz-do-not-send="true">pedmon@cfa.harvard.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<p>Yeah we've found out firsthand that its problematic as
we have been seeing issues :). Hence the urge to
upgrade.</p>
<p>We've begun exploring this but we wanted to reach out
to other people who may have gone through the same thing
to get their thoughts. We also need to figure out how
significant an outage this will be. As if it takes a
day or two of full outage to do the upgrade that is more
acceptable than a week. We also wanted to know if
people had experienced data loss/corruption in the
process and any other kinks.</p>
<p>We were planning on playing around on VM's to test the
upgrade path before committing to upgrading our larger
systems. One of the questions we had though was if we
needed to run e2fsck before/after the upgrade as that
could add significant time to the outage for that to
complete.</p>
<span class="HOEnZb"><font color="#888888">
<p>-Paul Edmon-<br>
</p>
</font></span>
<div>
<div class="h5"> <br>
<div class="m_8894130196693404536moz-cite-prefix">On
07/23/2018 01:18 PM, Jeff Johnson wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">You're running 2.10.4 clients against
2.5.34 servers? I believe there are notable lnet
attrs that don't exist in 2.5.34. Maybe a
Whamcloud wiz might chime in but I think that
version mismatch might be problematic.
<div><br>
</div>
<div>You can do a testbed upgrade to test taking a
ldiskfs volume from 2.5.34 to 2.10.4, just to be
conservative. </div>
<div><br>
</div>
<div>--Jeff</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Jul 23, 2018 at
10:05 AM, Paul Edmon <span dir="ltr"><<a
href="mailto:pedmon@cfa.harvard.edu"
target="_blank" moz-do-not-send="true">pedmon@cfa.harvard.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<p>My apologies I meant 2.5.34 not 2.6.34.
We'd like to get up to 2.10.4 which is
what our clients are running. Recently we
upgraded our cluster to CentOS7 which
necessitated the client upgrade. Our
storage servers though stayed behind on
2.5.34.</p>
<span class="m_8894130196693404536HOEnZb"><font
color="#888888">
<p>-Paul Edmon-<br>
</p>
</font></span>
<div>
<div class="m_8894130196693404536h5"> <br>
<div
class="m_8894130196693404536m_1100149130740236357moz-cite-prefix">On
07/23/2018 01:00 PM, Jeff Johnson
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Paul,
<div><br>
</div>
<div>2.6.34 is a kernel version.
What version of Lustre are you at
now? Some updates are easier than
others.</div>
<div><br>
</div>
<div>--Jeff</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Jul
23, 2018 at 8:59 AM, Paul Edmon <span
dir="ltr"><<a
href="mailto:pedmon@cfa.harvard.edu"
target="_blank"
moz-do-not-send="true">pedmon@cfa.harvard.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">We have
some old large scale Lustre
installs that are running 2.6.34
and we want to get these up to
the latest version of Lustre. I
was curious if people in this
group have any experience with
doing this and if they could
share them. How do you handle
upgrades like this? How much
time does it take? What are the
pitfalls? How do you manage it
with minimal customer
interruption? Should we just
write off upgrading and stand up
new servers that are on the
correct version (in which case
we need to transfer the several
PB's worth of data over to the
new system)?<br>
<br>
Thanks for your wisdom.<br>
<br>
-Paul Edmon-<br>
<br>
______________________________<wbr>_________________<br>
Beowulf mailing list, <a
href="mailto:Beowulf@beowulf.org"
target="_blank"
moz-do-not-send="true">Beowulf@beowulf.org</a>
sponsored by Penguin Computing<br>
To change your subscription
(digest mode or unsubscribe)
visit <a
href="http://www.beowulf.org/mailman/listinfo/beowulf"
rel="noreferrer"
target="_blank"
moz-do-not-send="true">http://www.beowulf.org/mailman<wbr>/listinfo/beowulf</a><br>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div
class="m_8894130196693404536m_1100149130740236357gmail_signature"
data-smartmail="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">------------------------------<br>
Jeff Johnson<br>
Co-Founder<br>
Aeon Computing<br>
<br>
<a
href="mailto:jeff.johnson@aeoncomputing.com"
target="_blank"
moz-do-not-send="true">jeff.johnson@aeoncomputing.com</a><br>
<a
href="http://www.aeoncomputing.com"
target="_blank"
moz-do-not-send="true">www.aeoncomputing.com</a><br>
t: 858-412-3810 x1001 f:
858-412-3845<br>
m: 619-204-9061<br>
<br>
4170 Morena Boulevard, Suite
C - San Diego, CA 92117
<div><br>
</div>
<div>High-Performance
Computing / Lustre
Filesystems / Scale-out
Storage</div>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset
class="m_8894130196693404536m_1100149130740236357mimeAttachmentHeader"></fieldset>
<br>
<pre>______________________________<wbr>_________________
Beowulf mailing list, <a class="m_8894130196693404536m_1100149130740236357moz-txt-link-abbreviated" href="mailto:Beowulf@beowulf.org" target="_blank" moz-do-not-send="true">Beowulf@beowulf.org</a> sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit <a class="m_8894130196693404536m_1100149130740236357moz-txt-link-freetext" href="http://www.beowulf.org/mailman/listinfo/beowulf" target="_blank" moz-do-not-send="true">http://www.beowulf.org/mailman<wbr>/listinfo/beowulf</a>
</pre>
</blockquote>
<br>
</div>
</div>
</div>
<br>
______________________________<wbr>_________________<br>
Beowulf mailing list, <a
href="mailto:Beowulf@beowulf.org"
target="_blank" moz-do-not-send="true">Beowulf@beowulf.org</a>
sponsored by Penguin Computing<br>
To change your subscription (digest mode or
unsubscribe) visit <a
href="http://www.beowulf.org/mailman/listinfo/beowulf"
rel="noreferrer" target="_blank"
moz-do-not-send="true">http://www.beowulf.org/mailman<wbr>/listinfo/beowulf</a><br>
<br>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div class="m_8894130196693404536gmail_signature"
data-smartmail="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">------------------------------<br>
Jeff Johnson<br>
Co-Founder<br>
Aeon Computing<br>
<br>
<a
href="mailto:jeff.johnson@aeoncomputing.com"
target="_blank" moz-do-not-send="true">jeff.johnson@aeoncomputing.com</a><br>
<a href="http://www.aeoncomputing.com"
target="_blank" moz-do-not-send="true">www.aeoncomputing.com</a><br>
t: 858-412-3810 x1001 f: 858-412-3845<br>
m: 619-204-9061<br>
<br>
4170 Morena Boulevard, Suite C - San
Diego, CA 92117
<div><br>
</div>
<div>High-Performance Computing / Lustre
Filesystems / Scale-out Storage</div>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset
class="m_8894130196693404536mimeAttachmentHeader"></fieldset>
<br>
<pre>______________________________<wbr>_________________
Beowulf mailing list, <a class="m_8894130196693404536moz-txt-link-abbreviated" href="mailto:Beowulf@beowulf.org" target="_blank" moz-do-not-send="true">Beowulf@beowulf.org</a> sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit <a class="m_8894130196693404536moz-txt-link-freetext" href="http://www.beowulf.org/mailman/listinfo/beowulf" target="_blank" moz-do-not-send="true">http://www.beowulf.org/<wbr>mailman/listinfo/beowulf</a>
</pre>
</blockquote>
<br>
</div>
</div>
</div>
<br>
______________________________<wbr>_________________<br>
Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org"
moz-do-not-send="true">Beowulf@beowulf.org</a> sponsored
by Penguin Computing<br>
To change your subscription (digest mode or unsubscribe)
visit <a
href="http://www.beowulf.org/mailman/listinfo/beowulf"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://www.beowulf.org/<wbr>mailman/listinfo/beowulf</a><br>
<br>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div class="gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">------------------------------<br>
Jeff Johnson<br>
Co-Founder<br>
Aeon Computing<br>
<br>
<a href="mailto:jeff.johnson@aeoncomputing.com"
target="_blank" moz-do-not-send="true">jeff.johnson@aeoncomputing.com</a><br>
<a href="http://www.aeoncomputing.com" target="_blank"
moz-do-not-send="true">www.aeoncomputing.com</a><br>
t: 858-412-3810 x1001 f: 858-412-3845<br>
m: 619-204-9061<br>
<br>
4170 Morena Boulevard, Suite C - San Diego, CA 92117
<div><br>
</div>
<div>High-Performance Computing / Lustre Filesystems /
Scale-out Storage</div>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Beowulf mailing list, <a class="moz-txt-link-abbreviated" href="mailto:Beowulf@beowulf.org">Beowulf@beowulf.org</a> sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit <a class="moz-txt-link-freetext" href="http://www.beowulf.org/mailman/listinfo/beowulf">http://www.beowulf.org/mailman/listinfo/beowulf</a>
</pre>
</blockquote>
<br>
</body>
</html>