<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <p>Yeah we've pinged Intel/Whamcloud to find out upgrade paths as we
      wanted to know what the recommended procedure is.<br>
    </p>
    <p>Sure. So we have 3 systems that we want to upgrade 1 that is a PB
      and 2 that are 5 PB each.  I will just give you a description of
      one and assume that everything would scale linearly with size. 
      They all have the same hardware.</p>
    <p>The head nodes are Dell R620's while the shelves are M3420 (mds)
      and M3260 (oss).  The MDT is 2.2T with 466G used and 268M inodes
      used.  Each OST is 30T with each OSS hosting 6.  The filesystem
      itself is 93% full.</p>
    <p>-Paul Edmon-<br>
    </p>
    <br>
    <div class="moz-cite-prefix">On 07/23/2018 01:58 PM, Jeff Johnson
      wrote:<br>
    </div>
    <blockquote type="cite"
cite="mid:CAFCYAse34C8FJukBXhTn__+MBk3R7_9LJTQs2N+H+WDs5=xWHg@mail.gmail.com">
      <meta http-equiv="content-type" content="text/html; charset=utf-8">
      <div dir="ltr">Paul,
        <div><br>
        </div>
        <div>How big are your ldiskfs volumes? What type of underlying
          hardware are they? Running e2fsck (ldiskfs aware) is wise and
          can be done in parallel. It could be within a couple of days,
          the time all depends on the size and underlying hardware. </div>
        <div><br>
        </div>
        <div>Going from 2.5.34 to 2.10.4 is a significant jump. I would
          be sure there isn't a step upgrade advised. I know there has
          been step upgrades in the past, not sure about going to/from
          these two versions. </div>
        <div><br>
        </div>
        <div>--Jeff</div>
      </div>
      <div class="gmail_extra"><br>
        <div class="gmail_quote">On Mon, Jul 23, 2018 at 10:34 AM, Paul
          Edmon <span dir="ltr"><<a
              href="mailto:pedmon@cfa.harvard.edu" target="_blank"
              moz-do-not-send="true">pedmon@cfa.harvard.edu</a>></span>
          wrote:<br>
          <blockquote class="gmail_quote" style="margin:0 0 0
            .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div bgcolor="#FFFFFF" text="#000000">
              <p>Yeah we've found out firsthand that its problematic as
                we have been seeing issues :).  Hence the urge to
                upgrade.</p>
              <p>We've begun exploring this but we wanted to reach out
                to other people who may have gone through the same thing
                to get their thoughts.  We also need to figure out how
                significant an outage this will be.  As if it takes a
                day or two of full outage to do the upgrade that is more
                acceptable than a week.  We also wanted to know if
                people had experienced data loss/corruption in the
                process and any other kinks.</p>
              <p>We were planning on playing around on VM's to test the
                upgrade path before committing to upgrading our larger
                systems.  One of the questions we had though was if we
                needed to run e2fsck before/after the upgrade as that
                could add significant time to the outage for that to
                complete.</p>
              <span class="HOEnZb"><font color="#888888">
                  <p>-Paul Edmon-<br>
                  </p>
                </font></span>
              <div>
                <div class="h5"> <br>
                  <div class="m_8894130196693404536moz-cite-prefix">On
                    07/23/2018 01:18 PM, Jeff Johnson wrote:<br>
                  </div>
                  <blockquote type="cite">
                    <div dir="ltr">You're running 2.10.4 clients against
                      2.5.34 servers? I believe there are notable lnet
                      attrs that don't exist in 2.5.34. Maybe a
                      Whamcloud wiz might chime in but I think that
                      version mismatch might be problematic. 
                      <div><br>
                      </div>
                      <div>You can do a testbed upgrade to test taking a
                        ldiskfs volume from 2.5.34 to 2.10.4, just to be
                        conservative. </div>
                      <div><br>
                      </div>
                      <div>--Jeff</div>
                      <div><br>
                      </div>
                    </div>
                    <div class="gmail_extra"><br>
                      <div class="gmail_quote">On Mon, Jul 23, 2018 at
                        10:05 AM, Paul Edmon <span dir="ltr"><<a
                            href="mailto:pedmon@cfa.harvard.edu"
                            target="_blank" moz-do-not-send="true">pedmon@cfa.harvard.edu</a>></span>
                        wrote:<br>
                        <blockquote class="gmail_quote" style="margin:0
                          0 0 .8ex;border-left:1px #ccc
                          solid;padding-left:1ex">
                          <div bgcolor="#FFFFFF" text="#000000">
                            <p>My apologies I meant 2.5.34 not 2.6.34. 
                              We'd like to get up to 2.10.4 which is
                              what our clients are running.  Recently we
                              upgraded our cluster to CentOS7 which
                              necessitated the client upgrade.  Our
                              storage servers though stayed behind on
                              2.5.34.</p>
                            <span class="m_8894130196693404536HOEnZb"><font
                                color="#888888">
                                <p>-Paul Edmon-<br>
                                </p>
                              </font></span>
                            <div>
                              <div class="m_8894130196693404536h5"> <br>
                                <div
                                  class="m_8894130196693404536m_1100149130740236357moz-cite-prefix">On
                                  07/23/2018 01:00 PM, Jeff Johnson
                                  wrote:<br>
                                </div>
                                <blockquote type="cite">
                                  <div dir="ltr">Paul,
                                    <div><br>
                                    </div>
                                    <div>2.6.34 is a kernel version.
                                      What version of Lustre are you at
                                      now? Some updates are easier than
                                      others.</div>
                                    <div><br>
                                    </div>
                                    <div>--Jeff</div>
                                  </div>
                                  <div class="gmail_extra"><br>
                                    <div class="gmail_quote">On Mon, Jul
                                      23, 2018 at 8:59 AM, Paul Edmon <span
                                        dir="ltr"><<a
                                          href="mailto:pedmon@cfa.harvard.edu"
                                          target="_blank"
                                          moz-do-not-send="true">pedmon@cfa.harvard.edu</a>></span>
                                      wrote:<br>
                                      <blockquote class="gmail_quote"
                                        style="margin:0 0 0
                                        .8ex;border-left:1px #ccc
                                        solid;padding-left:1ex">We have
                                        some old large scale Lustre
                                        installs that are running 2.6.34
                                        and we want to get these up to
                                        the latest version of Lustre.  I
                                        was curious if people in this
                                        group have any experience with
                                        doing this and if they could
                                        share them.  How do you handle
                                        upgrades like this?  How much
                                        time does it take?  What are the
                                        pitfalls?  How do you manage it
                                        with minimal customer
                                        interruption? Should we just
                                        write off upgrading and stand up
                                        new servers that are on the
                                        correct version (in which case
                                        we need to transfer the several
                                        PB's worth of data over to the
                                        new system)?<br>
                                        <br>
                                        Thanks for your wisdom.<br>
                                        <br>
                                        -Paul Edmon-<br>
                                        <br>
                                        ______________________________<wbr>_________________<br>
                                        Beowulf mailing list, <a
                                          href="mailto:Beowulf@beowulf.org"
                                          target="_blank"
                                          moz-do-not-send="true">Beowulf@beowulf.org</a>
                                        sponsored by Penguin Computing<br>
                                        To change your subscription
                                        (digest mode or unsubscribe)
                                        visit <a
                                          href="http://www.beowulf.org/mailman/listinfo/beowulf"
                                          rel="noreferrer"
                                          target="_blank"
                                          moz-do-not-send="true">http://www.beowulf.org/mailman<wbr>/listinfo/beowulf</a><br>
                                      </blockquote>
                                    </div>
                                    <br>
                                    <br clear="all">
                                    <div><br>
                                    </div>
                                    -- <br>
                                    <div
                                      class="m_8894130196693404536m_1100149130740236357gmail_signature"
                                      data-smartmail="gmail_signature">
                                      <div dir="ltr">
                                        <div>
                                          <div dir="ltr">------------------------------<br>
                                            Jeff Johnson<br>
                                            Co-Founder<br>
                                            Aeon Computing<br>
                                            <br>
                                            <a
                                              href="mailto:jeff.johnson@aeoncomputing.com"
                                              target="_blank"
                                              moz-do-not-send="true">jeff.johnson@aeoncomputing.com</a><br>
                                            <a
                                              href="http://www.aeoncomputing.com"
                                              target="_blank"
                                              moz-do-not-send="true">www.aeoncomputing.com</a><br>
                                            t: 858-412-3810 x1001   f:
                                            858-412-3845<br>
                                            m: 619-204-9061<br>
                                            <br>
                                            4170 Morena Boulevard, Suite
                                            C - San Diego, CA 92117
                                            <div><br>
                                            </div>
                                            <div>High-Performance
                                              Computing / Lustre
                                              Filesystems / Scale-out
                                              Storage</div>
                                          </div>
                                        </div>
                                      </div>
                                    </div>
                                  </div>
                                  <br>
                                  <fieldset
                                    class="m_8894130196693404536m_1100149130740236357mimeAttachmentHeader"></fieldset>
                                  <br>
                                  <pre>______________________________<wbr>_________________
Beowulf mailing list, <a class="m_8894130196693404536m_1100149130740236357moz-txt-link-abbreviated" href="mailto:Beowulf@beowulf.org" target="_blank" moz-do-not-send="true">Beowulf@beowulf.org</a> sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit <a class="m_8894130196693404536m_1100149130740236357moz-txt-link-freetext" href="http://www.beowulf.org/mailman/listinfo/beowulf" target="_blank" moz-do-not-send="true">http://www.beowulf.org/mailman<wbr>/listinfo/beowulf</a>
</pre>
                                </blockquote>
                                <br>
                              </div>
                            </div>
                          </div>
                          <br>
                          ______________________________<wbr>_________________<br>
                          Beowulf mailing list, <a
                            href="mailto:Beowulf@beowulf.org"
                            target="_blank" moz-do-not-send="true">Beowulf@beowulf.org</a>
                          sponsored by Penguin Computing<br>
                          To change your subscription (digest mode or
                          unsubscribe) visit <a
                            href="http://www.beowulf.org/mailman/listinfo/beowulf"
                            rel="noreferrer" target="_blank"
                            moz-do-not-send="true">http://www.beowulf.org/mailman<wbr>/listinfo/beowulf</a><br>
                          <br>
                        </blockquote>
                      </div>
                      <br>
                      <br clear="all">
                      <div><br>
                      </div>
                      -- <br>
                      <div class="m_8894130196693404536gmail_signature"
                        data-smartmail="gmail_signature">
                        <div dir="ltr">
                          <div>
                            <div dir="ltr">------------------------------<br>
                              Jeff Johnson<br>
                              Co-Founder<br>
                              Aeon Computing<br>
                              <br>
                              <a
                                href="mailto:jeff.johnson@aeoncomputing.com"
                                target="_blank" moz-do-not-send="true">jeff.johnson@aeoncomputing.com</a><br>
                              <a href="http://www.aeoncomputing.com"
                                target="_blank" moz-do-not-send="true">www.aeoncomputing.com</a><br>
                              t: 858-412-3810 x1001   f: 858-412-3845<br>
                              m: 619-204-9061<br>
                              <br>
                              4170 Morena Boulevard, Suite C - San
                              Diego, CA 92117
                              <div><br>
                              </div>
                              <div>High-Performance Computing / Lustre
                                Filesystems / Scale-out Storage</div>
                            </div>
                          </div>
                        </div>
                      </div>
                    </div>
                    <br>
                    <fieldset
                      class="m_8894130196693404536mimeAttachmentHeader"></fieldset>
                    <br>
                    <pre>______________________________<wbr>_________________
Beowulf mailing list, <a class="m_8894130196693404536moz-txt-link-abbreviated" href="mailto:Beowulf@beowulf.org" target="_blank" moz-do-not-send="true">Beowulf@beowulf.org</a> sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit <a class="m_8894130196693404536moz-txt-link-freetext" href="http://www.beowulf.org/mailman/listinfo/beowulf" target="_blank" moz-do-not-send="true">http://www.beowulf.org/<wbr>mailman/listinfo/beowulf</a>
</pre>
                  </blockquote>
                  <br>
                </div>
              </div>
            </div>
            <br>
            ______________________________<wbr>_________________<br>
            Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org"
              moz-do-not-send="true">Beowulf@beowulf.org</a> sponsored
            by Penguin Computing<br>
            To change your subscription (digest mode or unsubscribe)
            visit <a
              href="http://www.beowulf.org/mailman/listinfo/beowulf"
              rel="noreferrer" target="_blank" moz-do-not-send="true">http://www.beowulf.org/<wbr>mailman/listinfo/beowulf</a><br>
            <br>
          </blockquote>
        </div>
        <br>
        <br clear="all">
        <div><br>
        </div>
        -- <br>
        <div class="gmail_signature" data-smartmail="gmail_signature">
          <div dir="ltr">
            <div>
              <div dir="ltr">------------------------------<br>
                Jeff Johnson<br>
                Co-Founder<br>
                Aeon Computing<br>
                <br>
                <a href="mailto:jeff.johnson@aeoncomputing.com"
                  target="_blank" moz-do-not-send="true">jeff.johnson@aeoncomputing.com</a><br>
                <a href="http://www.aeoncomputing.com" target="_blank"
                  moz-do-not-send="true">www.aeoncomputing.com</a><br>
                t: 858-412-3810 x1001   f: 858-412-3845<br>
                m: 619-204-9061<br>
                <br>
                4170 Morena Boulevard, Suite C - San Diego, CA 92117
                <div><br>
                </div>
                <div>High-Performance Computing / Lustre Filesystems /
                  Scale-out Storage</div>
              </div>
            </div>
          </div>
        </div>
      </div>
      <br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <br>
      <pre wrap="">_______________________________________________
Beowulf mailing list, <a class="moz-txt-link-abbreviated" href="mailto:Beowulf@beowulf.org">Beowulf@beowulf.org</a> sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit <a class="moz-txt-link-freetext" href="http://www.beowulf.org/mailman/listinfo/beowulf">http://www.beowulf.org/mailman/listinfo/beowulf</a>
</pre>
    </blockquote>
    <br>
  </body>
</html>