System Slowness: Updated 9/30

Follow

Comments

74 comments

  • Avatar
    Mark Sedgley

    All,


    As we continue to try and resolve this matter, we've put into place a stop gap measure to deliver  more consistent performance throughout the remainder of the day.  We are still working, however, toward a full resolution.  With the exception of isolated interruptions and error messages the product should perform at normalized levels.  As we progress I will continue to update you here.  We are making progress but this fix is an interim one.  With that said, it will allow you to operate with minimized interruption and much improved performance through the course of the day. 

    -Mark

    0
    Comment actions Permalink
  • Avatar
    Duncan McCreery

    Update: We're seeing more severe issues right now and the application is effectively down.  We're working through a restart to bring it back up.  This is an unplanned outage and we're working to shorten the downtime as much as possible to minimize the disruption.

    0
    Comment actions Permalink
  • Avatar
    Duncan McCreery

    Update: Our team continues to work through the root issues, but the system is back up and running now.  

    0
    Comment actions Permalink
  • Avatar
    Mary Winder

    Hey Duncan,

    Thanks for all that MC has been doing to fix this.  I know you are frustrated and probably really stressed.  I was able to get in just now and do some work.  You may or may not know that the content management editor is only working in HTML right now.  There are no WYSIWYG buttons or anything for editing articles. 

    0
    Comment actions Permalink
  • Avatar
    Mark Sedgley

    All,

    As a continued effort to ensure proactive communication, we will be brining our product down from 12:00 am - 2:00 am EST to put through several changes.  This is part of our ongoing effort in working with our team and third party experts to resolve this matter.  All of us continue to be fully committed to the issue at hand.

    Best,

    -Mark

    0
    Comment actions Permalink
  • Avatar
    Eric Jacobs

    Not to pile on, but having the MemberClicks status website happily show "All Systems Go!" and all green lights just rubs salt in the wound. This page was set up to keep customers informed of system problems. Showing that everything is fine now, and that yesterday had only "some intermittent slowness", with the 99.83% availability trumpeted in the largest type, makes this seem like MemberClicks views this site more as a marketing piece than a tool designed to help customers struggling when there are problems with the platform. I know there are obviously far bigger issues to resolve right now, but I hope when there's a chance to breathe and conduct a post-mortem on this week's events, senior management will look at making this page more transparently show what's happening. 




    Screen Shot 2013-09-25 at 6.45.22 PM.png
    0
    Comment actions Permalink
  • Avatar
    Mark Sedgley

    All,

    Just a quick update that our planned downtime for tonight is complete.  We are now continuing our work to ensure resolution to the issues that we've seen over the last three days.  We will keep you posted.

    0
    Comment actions Permalink
  • Avatar
    Mark Sedgley

    All,

    We've made progress over the past several hours and we continue to be fully-focused on a complete resolution.  More updates to come as they are available.

    0
    Comment actions Permalink
  • Avatar
    Mark Sedgley

    All,

    After additional analysis, we will be taking a 25 minute downtime beginning at 9:45 a.m. EST.  Front-end sites will continue to load, but log ins, profile access another other core features will be down during this period.  I will post another update as soon as possible.

    0
    Comment actions Permalink
  • Avatar
    Mark Sedgley

    All,

    The system is coming back up as I write this. As per my previous posts we will continue to monitor the performance of the product and keep you posted.  We should be back up shortly.  

     

    0
    Comment actions Permalink
  • Avatar
    Paula Chase-Hyman

    Fingers crossed that the system remain up. It's really quite nice, not to mention efficient, at this new speed.

    0
    Comment actions Permalink
  • Avatar
    Annelise Gillespie

    I'm still experiencing issues when trying to edit/create reports.  All appears only in HTML.  Is this a change or a result of the current issues?  TY

    0
    Comment actions Permalink
  • Avatar
    Duncan McCreery

    Hi Annelise, we're still working through a number of issues, so it's definitely related.  Thank you for bearing with us on this!

     

    0
    Comment actions Permalink
  • Avatar
    Mark Sedgley

    Hi Everyone,

    A quick heads up.  We are going to bring down the E-list processor temporarily.  This will delay any e-list messages currently in queue by 30-45 minutes.  You will not lose any messages.  This action is directly related to our continued efforts in Root Cause Discovery for our system issues.  Thank you in advance and we will continue to keep you in the loop as we progress.  

    Best,

    -Mark

    0
    Comment actions Permalink
  • Avatar
    Mark Sedgley

    All,

    E-lists were restored to their normal operations at 3:23 EST.  As we continue to move through our product, we will be bringing down all search functions from 5:00 pm to 7:00 pm EST.  We will present a custom message to all users attempting to access a search during that time.  Please continue to bare with us.  We are attempting to mitigate the impact to you to our greatest ability while also being as aggressive as we possibly can in solving this matter.

    Regards,

    -Mark 

    0
    Comment actions Permalink
  • Avatar
    Cynthia McCracken

    I find it extremely interesting that the system slows to a crawl when we need speed!!!

    Happened the same time last year during close of entry. 

    This needs to be fixed as others have said After Midnight!! NOT during business hours. 

    Severely frustrated. 

    Cynthia

    0
    Comment actions Permalink
  • Avatar
    Mark Sedgley

    Cynthia, 

    We totally agree with you.  The only reason and I must stress only reason that we are making these changes during normal business operations is to ensure our product doesn't become totally unresponsive to you.  We sincerely understand your frustration.  We feel it, and internally we are responding with extreme urgency.  If we could solve this matter quickly and easily in the after hours windows we definitely would.  We have been working 24/7 since early Tuesday morning.  This time of year is incredibly busy for all of our organizations that's exactly why we are taking the measures you see today.  I know just how unacceptable this is.  I can only continue to assure you that we are doing everything within our power to resolve this matter.  

    Best,

    -Mark

    0
    Comment actions Permalink
  • Avatar
    Lisa Kelsey

    We are also at peak conference registration this week.  Generally we do not experience issues or have problems with the MC maintenance/down time.  I do hope this is resolved quickly as we are getting many many phone calls with log-in issues.  I have confidence that it will.  Duncan - thank you for the personal phone call this morning.  Clearly I was out of the loop!

    Jennifer

    ILOTA

    0
    Comment actions Permalink
  • Avatar
    Christine Meert

    Suggestion: maybe Memberclicks could give us all a substantial discount on our next bill to reward our patience and encourage us to remain patient?????

    Christine

    0
    Comment actions Permalink
  • Avatar
    Linda Krause

    I'm with Christine-I'm 6 days from a conference and it's frustrating. I could be losing a ton of those last minute registrants : (  A discount would be nice as this is the 2nd issue this month with MC. Linda

    0
    Comment actions Permalink
  • Avatar
    Mark Sedgley

    Christine and Linda,

    Thank you for the posts and the suggestion.  Currently we are fully focused and our number one priority is resolving the issues at hand.  Once we have resolved the system performance issues, MemberClicks will formulate an appropriate approach for recognizing our customers' patience.

    Best,

    -MCS

    0
    Comment actions Permalink
  • Avatar
    Mark Sedgley

    All,

    I wanted to give everyone a heads up that we will be bringing the product down from 12:00 am - 3:00 am EST.  This will include several changes to our Oracle Database.   Thank you in advance.

    Best,

    -MCS

    0
    Comment actions Permalink
  • Avatar
    Paul Talbut

    I'd just like to point out that Memberclicks has users outside the US and so any requests for maintenance overnight will mean maintenance during the day for us.

    0
    Comment actions Permalink
  • Avatar
    Christine Meert

    Yes, we have many clients overseas.... will have to email them to let them know, bummer!

    0
    Comment actions Permalink
  • Avatar
    Eric Jacobs

    We're all frustrated by the failures at MemberClicks, but at this point, continuing to complain about the inconvenience and problems, or the times they're shutting down parts of the system, likely doesn't help them or us. Yes, a nighttime shutdown in the US will occur in daytime somewhere else in the world, but what is the alternative? They are clearly picking times designed to inconvenience what they think will be the smallest subset of their customers, but there is no time that will be perfect  for everyone. I think it's clear from the responses and postings, most coming from the president of MemberClicks (who never posts here), that they fully understand how badly and seriously their problems are causing all of us.

    I'm sure it's a nightmare scenario for them, as they have disrupted all their customers and risk losing us if they can't get things back to normal and improve reliability longterm. They're clearly trying their hardest to deal with a serious meltdown problem. So as unhappy as I am, I am also sympathetic. As someone who has complained in the past about lack of clear communication, I want to thank Mark Sedgley and Duncan for stepping up the communications as they try to get system operations corrected.

    I do hope and expect that once the immediate crisis is resolved and the dust settles, MemberClicks will share with us a more detailed explanation of what went wrong, why backups or rollbacks weren't able to be employed, how we will be compensated for the disruption in service, and most importantly, what they can and will do to implement more safeguards and redundancy to insure this can't happen again. For now, we all just have to hold our breath (and our tongues) and hope they are able to get things under control in the near future.

    0
    Comment actions Permalink
  • Avatar
    Perry Mills

    Not being aware of the actual technical issues, if Memberclicks is using Oracle, then that is probably the culprit.  The database on the IBM i server, DB2, is profoundly better and reliable.  I can sell Memberclicks this server if they like, as my company is an IBM Business Partner.  I am only half kidding about this.  We too have experienced some major difficulties, but I trust the team at Memberclicks to get this ironed out shortly.

    0
    Comment actions Permalink
  • Avatar
    Bob Schilmoeller

    Thank you Eric for your words.  I am traveling to California for two weeks for work meetings.   I have watched this all from afar and trying to preach patience to my Board.  Your words matched my sentiments.  I have also looked at the timestamps of the messages coming from Mark and Duncan.  I have been on that end of bad situations like this, and they are not fun.  We do need to give them the room to do their job.  Then we we all deserve all of the information you describe in your last paragraph.   If we don't get it, then I will be on the phone...

    Mark, Duncan and team I wish you the very best in troubleshooting this.  You are feeling our pain.  Thank you for your persistence.

    0
    Comment actions Permalink
  • Avatar
    Mark Sedgley

    All,

    We are bringing the product back up.  At 1:38 a.m. EST the product came back online.  Over the next several hours we will be working through any issues that we are seeing from the maintenance window and continue to monitor the results of our efforts.  

    Thank you,

    -Mark

    0
    Comment actions Permalink
  • Avatar
    Harvey Nation

    9:35 EDT

    For what it's worth, we've had several folks renew their membership this morning. No complaints with response time. And I've had no trouble as ASA.

    I hope it's over...for everyone's sake.

    hn

     

    0
    Comment actions Permalink
  • Avatar
    Mark Sedgley

    All,

    We rolled out several changes last night in an attempt to address our ongoing product issues.  As Harvey mentioned these changes have made incremental improvements.  I'm very disappointed, however, to write that despite these improvements you will experience continued sporadic and intermittent product performance issues.  With that said I want you to know what we are doing at this point.

    1.  We are going to provide ongoing support through the weekend both e-mail and phone.  If we can help we will.  Further details to come.

    2.  We will continue to work relentlessly with our collected team of Oracle experts to resolve the issues we've seen in the Oracle platform. However,  We can no longer afford to execute fixes/changes to "try" and solve the problem.  Therefore, our engineering team will continue to work 24/7 to reduce our dependency on the Oracle database.  We believe these changes will make a significant difference.  All involved agree.  This is no small task, but we are committed to work non-stop to ensure execution.  This strategy positions our team to be optimally effective while we continue to leverage the significant outside expertise we've engage on the Oracle side.

    3.  I will be crafting a communication later today detailing the events of the week and where we stand and post it here.

    4.  Due to significant direct feedback from customers subscribed to this System Performance Digest, we need to turn commenting off of this forum in order to stop inundating all of our customers' inbox that are subscribed to this post.  This does not mean we don't welcome your comments.  As a matter fact we would like to continue to address each of them directly.  My personal e-mail is mark@memberclicks.com and Duncan's is duncan@memberclicks.com.  Please send direct feedback to us and we will continue to post official updates here.

    All of us remain committed to you and we are doing everything within our power to move this forward.  Thank you.

    Best,

    -MCS

    0
    Comment actions Permalink

Please sign in to leave a comment.