System Slowness: Updated 9/30

Follow

Comments

74 comments

  • Avatar
    Mark Sedgley

    All,

     

    I want to bring you up to speed on a few items.  First, we are planning Maintenance Windows over the next three nights.  We will bring the product down between 12:00 am – 2:00 am EST on Friday, Saturday, and Sunday mornings.  Secondly, please find our available support hours for this weekend.

     Saturday:

     Phone:  9:00 am – 6:00 pm EST

    Email:   9:00 am  - 8:00 pm EST 

    Sunday: 

    Phone:  9:00 am – 6:00 pm EST

    E-mail:  9:00 am – 6:00 pm EST

     

    I personally want to thank each of you that has reached out to us in support of our efforts.  We cherish our relationship with our customers.  Your words of encouragement add to our relentless commitment to bring this to a resolution.  Thank you for your support but we’re not taking it for granted. 

    0
    Comment actions Permalink
  • Avatar
    Duncan McCreery

    Update: Again, thank you for your patience as we continue to work on these issues. Unfortunately, we have to bring the product down from 3:00 p.m. to 3:30 p.m. Eastern time in an effort to expedite resolution.  We will keep you updated as we move through this window.

    0
    Comment actions Permalink
  • Avatar
    Duncan McCreery

    Update:  The product is back up and running at this time.  We're continuing to work with our partners on a complete resolution.

    0
    Comment actions Permalink
  • Avatar
    Duncan McCreery

    Update: Front-end  directory searches will be unavailable from 5:15 to 6:15.  Other areas of the product will not be affected during this period.

    0
    Comment actions Permalink
  • Avatar
    Duncan McCreery

    Update: Searching is available again as of 6:15 p.m. Eastern.

    Quick reminder: Due to the ongoing problems, phone and email support will be available this weekend to help with any issues that arise.  Phone lines will be open  tomorrow from 9 a.m. to 6 p.m. Eastern and email support will be available from 9 a.m. to 8 p.m. Eastern.  On Sunday, both phone and email support will be available from 9 a.m. to 6 p.m. Eastern.

    0
    Comment actions Permalink
  • Avatar
    Duncan McCreery

    Reminder: The application will be unavailable starting in 30 minutes, from 12 a.m. to 2 a.m. Eastern time.

    0
    Comment actions Permalink
  • Avatar
    MemberClicks Engineering

    Hi all,

    We have completed our maintenance and the product is back online.

     

    Thank you for your continued patience,
    Richie Ward
    MemberClicks Engineering

    0
    Comment actions Permalink
  • Avatar
    Mark Sedgley

    Hello Everyone,

    Just a reminder that we are here today for any support issues that you may be working through this weekend.  We continue to work through the issues of this week.  Thank you and have a good day.

     

    -Mark

     

     

    0
    Comment actions Permalink
  • Avatar
    Duncan McCreery

    Hi Everyone,

    We wanted to send a reminder that we are here today for any questions or support issues that come up.  Our teams continue to make progress on the issues from last week.  We will keep you updated as we move forward.  Have a good day!

    0
    Comment actions Permalink
  • Avatar
    Duncan McCreery

    Update and a Reminder: The maintenance window tonight begins in just under an hour.  Apologies for the late notice; we will be extending this window for an extra hour.  The product will be unavailable from 12 a.m. to 3 a.m. Eastern.

    0
    Comment actions Permalink
  • Avatar
    Mark Sedgley

    All,


    As promised, I have put together a summary of events from last week and through this weekend.  You can find this below.  I decided to delay my communication in order to deliver a more thorough picture of exactly where we stand today.  This summary is not a technical document.  For those of you that are interested in such a document, we will be putting together a Post Mortem Report once we’ve collected all of the appropriate data.  


    On Monday 9/23/2013 at around 2:00 pm EST we started to see significant disruption to our Oracle Database.  This disruption presented itself to both our DBAs (Database Administrators) and through our Oracle Enterprise Manager (OEM).  As a quick description, OEM is a tool developed and provided by Oracle to provide current, detailed information about activity within the database.  Our DBAs and OEM captured irregular database spikes that weren’t lining up with the actual load placed on the database.  On Monday night, we made several configuration changes selected by our team to our Oracle database along with changes in the way that our Application interacts with the database.  


    On Tuesday 9/24/2013, the MemberClicks product was performing normally until around 3:00 p.m. EST.  At this time, we started seeing the same indicators from our OEM and the Oracle database despite the suggested configuration changes from our DBAs.   On Tuesday, we opened a Severity 1 (top priority) Support Ticket with Oracle.  From Tuesday on, we had Oracle directly involved in the process in conjunction with our contracted DBA team and an emergency team we employed.  


    Over the course of Wednesday and Thursday, we continued to address the suggestions and implications being delivered to us by our DBA teams and Oracle directly.  Each of these changes only made incremental improvements and were not addressing the underlying issues.  On Friday morning we made the decision to divert to a two pronged strategy.  This was based on two factors:  1. We felt at that time we were reaching an exhaustion point with Oracle as our efforts only resulted in bug submissions for both the database and their OEM product.  2.  We needed to maximize our resources on hand and attempt to deliver real gains as soon as possible.


    Starting Friday, we coded around the clock to substantially alter the way we handle traffic, depend on Oracle, and route traffic through our own server infrastructure.  These changes are not small.  Our testing indicates that these changes will make a marked difference in both performance and stability.  The first set of changes were rolled out over the weekend.  This first set of changes were designed around improving stability.  The next round of changes, aimed at performance, will be delivered over the course of Monday and Tuesday.


    With that said, we are on high alert here at MemberClicks.  Our team has not stopped since these issues surfaced and, as we’ve said all along, we won’t stop until we have a proven, sustained normal state.   We will update you as we go throughout the day today and through the rest of the week.


    Thank you in advance.   


    -MCS

    0
    Comment actions Permalink
  • Avatar
    Mark Sedgley

    Good Evening Everyone!

    I want to touch down and download our next round of changes.  The infrastructure and stability changes that we rolled out over the weekend delivered on it’s intent.  We experienced significant improvement in product stability and overall performance throughout the day today.  We will be rolling out more changes through the evening.  Thankfully, this next round of tuning will not require a maintenance window.  There will be a new element on your admin homepage, however, with this next round of improvements.  When you login you will notice a “manually refresh” button in your admin home statistics box.  We’ve added this component in order to better cache data calls and in simple terms, make the product much faster!  

    So what does this actually mean?  In this new world when any profile is added, deleted, or changed from a group or status found in the Admin Statistics Box, you will need to manually refresh the statistics in order to update the numbers.  These statistics will also update automatically after five minutes without a manual refresh.  

    We look forward to your feedback on these latest changes and let us know if you have any questions.  

    Best,

    -MCS

     

    0
    Comment actions Permalink
  • Avatar
    Duncan McCreery

    Hi Everyone,

    Thank you to everyone who has reached out to us over the last 24 hours.  The changes put in place this weekend have indeed solved the stability issues from last week.  As Mark wrote yesterday, we also had a release dedicated to enhancing product performance.  I wanted to confirm that this release is live and the only functional change you should see is the Manual Refresh button for the admin home statistics.  

    We had a number of bugs fixes ready to go prior to the problems last week which also went live.  I've included the notes for them below:

    • There was a problem in event management that prevented the Save button from properly loading for some administrators.  That issue is now fixed.
    • When creating Blog posts, the Read More link would display whether or not it had been enabled.  Now, the Read More link will obey the settings.
    • We received several reports of users having troubles uploading attachments to forms using Internet Explorer.  The problem is resolved now.
    • Forms without receipts would occasionally throw errors when clicking the link to view receipts.  Now, admins are able to click the link without the error.
    • There was a bug causing attachments posted to an elist to drop when the message was posted through the web interface using Firefox.
    • When members paid invoices, administrators received two payment notification emails when the invoice was paid in full.  You won't receive the duplicates anymore.

    Although the updates will be less frequent, we'll keep you posted throughout the remainder of the week.

    0
    Comment actions Permalink
  • Avatar
    Mark Sedgley

    Good Morning!

     

    I wanted to take a moment to bring the events from last week to a close.  Since Saturday morning we have experienced strong performance and stability in the product.  After the roll-outs through the weekend and yesterday, I hope you have noticed a remarkable difference in both speed and performance of the MemberClicks platform.  Currently, we are seeing response times and page loads that are significantly faster than any we’ve measured in the past twelve months.  This is a direct result of working with our consulting partners, Oracle directly, and our engineering team’s sheer determination.   Each of these entities helped us to deliver a comprehensive solution to the issues we saw through Friday of last week.  As we continue to complete out our Postmortem and deliver our Reason For Outage (RFO) report, it is clear that several factors were at play once we started experiencing irregularities in our Oracle Database.  As I mentioned in my previous posts, we took a multi-faceted approach that focused on many database configuration changes, improving the efficiency and scale of our back end server infrastructure, and architectural changes in our code base to more efficiently leverage the power of our database.

     

    Although we are pleased with the results we are seeing today, we are both extremely disappointed and equally motivated by our failure last week. Therefore, we are going to continue to expedite our existing long-term strategy for fortifying both our platform scale and speed.  The changes we execute over the coming weeks have already been in play over the last year as we have rolled them out gradually.  In other words, there are several parts of our product today that are already utilizing the new architecture.  These are the faster components of our solution.  We are simply expediting our timeline to ensure that we don’t experience a week like this again.  

     

    All of us at MemberClicks were more than disappointed that we let you down last week.  We took it personally.  As a company that prides itself on serving the customer first, last week’s events only strengthened our resolve to exceed your expectations as we move forward.  We’ve served Small Staff Associations for 15 years now.  We are truly blessed to have customers like you.  I have witnessed over the last seven days that adversity only reveals character and I can tell you my confidence in our customers, our team, and our product has never been higher.  

     

    Thank you in advance.  If you would like a copy of the RFO report we will be finishing it up later this week.  Simply e-mail myself (mark@memberclicks.com) or Duncan (duncan@memberclicks.com) and we will be happy to send you a copy.   

    Best,

    -MCS 

    0
    Comment actions Permalink

Please sign in to leave a comment.