System Issues: 9/17/14

Follow

Comments

7 comments

  • Avatar
    Duncan McCreery

    Everyone,

    Thank you for patience.  We wanted to give a quick update as to our progress.  We have identified the root issue, however, the resolution is more complicated than we previously anticipated.  We have been pursuiing two key options in conjunction with our systems team to restore the front end websites and administrative access.  We will continue to post updates here as we make progress.

  • Avatar
    Mark Sedgley

    Hey Everyone,

    We are still working through the issues that we encountered tonight.  We expect the product to be down over the next several hours.  Our team along with our hosting partner are diligently working through the issues.  This is a tandem effort and one that we are taking very seriously.  We don't currently have an eta in place.  However,  we will update you again no later than 5:00 am EST.

  • Avatar
    Richie Ward

    Hello Everyone,

    We've narrowed in on the issue and expect to have restored service in the next several hours.  We are continuing to investigate root cause while making restoring the product our first and foremost priority.  We expect to have another update by 8 AM EST.

    Thank you.

  • Avatar
    Emily Srygley
    Hello Everyone,

    The product has returned to normal operation at this time and your organization’s website should be loading on both the front end and the back end of the MemberClicks product.  We sincerely apologize for any inconvenience this has caused you.

    The MemberClicks Engineering Team, in conjunction with our data center partner, has worked diligently to identify the root cause of this downtime.  While we’re still working to identify the root cause, we have pinpointed the specific issue and narrowed our focus to our network file system infrastructure.

    It’s important to note that any file that has physically been uploaded into the MemberClicks system since Monday night at midnight will need to be reuploaded.  This would include file uploads to media manager, committee attachments, social community & community forum uploads, etc.  Forms and profile data are not affected and should be current.

    You may also notice older messages prior to January 2014 from Contact Center & E-List may not be immediately accessible.  We will be working throughout the day today to restore these messages so they may be accessed in the product.

    We sincerely apologize for any inconvenience this has caused you.  We have all hands on deck internally to return the product to full operation as soon as possible.

    We will provide an additional update at 1 PM EST.

    Thank your for your patience.
  • Avatar
    Emily Srygley

    Update:  All older messages prior to January 2014 from Contact Center & E-List are now accessible.  We have been working to fully restore these messages and they are now available in the product.

  • Avatar
    Emily Srygley

    Update: We are still experiencing some residual issues as part of yesterday’s outage. Our engineers are working with our data center to resolve this as quickly as possible.  More updates to follow as soon as we have them.

  • Avatar
    Mark Sedgley

     

    Hello Everyone,

     

    We wanted to let you know that as of yesterday early evening we restored the product to normalized performance levels.  We have worked exhaustively over the last two days with our data center provider to establish a root cause for the experienced down time.  Through that process a human error on the part of our data center partner was revealed as the cause for our experienced outage.  On Tuesday evening, our partner was performing routine diagnostic and optimization checks.  Through that process they identified an opportunity to improve the current back up speeds.  Unfortunately, there was a critical mistake made at that time that essentially brought down both our primary and secondary Network File Mounts.  At that point, the only course of action was a full restore of both systems.  These systems are very large and maintain millions of files and are of significant size.  

     

    The restoration and re-starting of the platform finished at 7:32 am on Wednesday morning. Once back up our engineering team went to work to ensure a stable launch and to mitigate any short term production issues as an after affect of the outage and subsequent restoration.  As of yesterday evening, we brought the system totally in line with normal production performance.  

     

    Through this process, our long time data center partner was fully transparent and highly responsive.  We respect that approach.  With that said, your experience is 100% our responsibility and as I have said before I personally take that very seriously.  All of us at MemberClicks apologize for any disruption our outage may have caused your organization.  Rest assured we our reviewing all of our current protocols and that of our providers along with any action items on our side that could help us mitigate this type of event in the future.

Article is closed for comments.