posted 11-17-2011 08:05 AM            

After the initial move shakeout, the forum’s performance greatly improved.

Until a couple of days ago the remaining problem, which was not material to our daily operations, looked like it was nearing resolution. Hostopia has kept us informed about their progress to resolve things. Earthlink has been silent.

Then few days ago a new set of issues started to appear. At first, it seemed only material to forum maintenance (i.e, indexing, moving/editing threads, etc) but the issue has continued to worsen to the point that we are now getting user complaints about forum performance.

Our domain www.SMPub.com is on a huge server farm in California.

A server farm or server cluster is a collection of computer servers usually maintained by an enterprise to accomplish server needs far beyond the capability of one machine. Server farms often have backup servers, which can take over the function of primary servers in the event of a primary server failure. Server farms are typically co-located with the network switches and/or routers which enable communication between the different parts of the cluster and the users of the cluster.

The new problem seems to be a good neighbor and policing problem….

Hostopia needs to identify the source and/or sources of the runaway CGI, then to determine why Hostopia's CGI load balancing is not protecting other domains and then to put a fix in place. Hopefully the resolution will be quick. But it could also take some time.

We are expecting to hear more about the steps to resolution on Wednesday. That seems like a long time but apparently trouble shooting these issues is very involved.

Thank you in advance for your continuing patience.

posted 11-18-2011 08:34 AM            

When there is a problem, if you would note the time (and what time zone you are in) and describe the problem, it could help the techs trouble shoot things.

The description can be as simple as:

the main forum page took 50 second to completely finish loading..."

"my new (or edit) completely timed out"

posted 11-18-2011 08:40 AM            

quote:

---

Subject: Things are getting even worse

Over night and this morning are particularly bad. This morning pages take a very long time to be processes by CGI.
Overnight the site monitor detected 5 outages -- a new record! See below.

=======================================
Date: Nov 17, 2011 11:10 PM
The monitor SSF (http://www.smpub.com/cgi-bin/ubb/Ultimate.cgi) is currently down.
Alert Details: 500 - Internal Server Error (The server encountered an unexpected condition that prevented it from fulfilling the request.)

Date: Nov 17, 2011 11:17 PM
The monitor SSF (http://www.smpub.com/cgi-bin/ubb/Ultimate.cgi) is back up (it was down for 07 minute(s) and 22 second(s)).
=============================

Date: Nov 18, 2011 1:13 AM
The monitor SSF (http://www.smpub.com/cgi-bin/ubb/Ultimate.cgi) is currently down.
Alert Details: 500 - Internal Server Error (The server encountered an unexpected condition that prevented it from fulfilling the request.)

Date: Nov 18, 2011 1:21 AM
The monitor SSF (http://www.smpub.com/cgi-bin/ubb/Ultimate.cgi) is back up (it was down for 07 minute(s) and 18 second(s)).
==============================

Date: Nov 18, 2011 2:04 AM
The monitor SSF (http://www.smpub.com/cgi-bin/ubb/Ultimate.cgi) is currently down.
Alert Details: 500 - Internal Server Error (The server encountered an unexpected condition that prevented it from fulfilling the request.)

Date: Nov 18, 2011 2:11 AM
The monitor SSF (http://www.smpub.com/cgi-bin/ubb/Ultimate.cgi) is back up (it was down for 07 minute(s) and 25 second(s)).
===============================

Date: Nov 18, 2011 4:55 AM
The monitor SSF (http://www.smpub.com/cgi-bin/ubb/Ultimate.cgi) is currently down.
Alert Details: 500 - Internal Server Error (The server encountered an unexpected condition that prevented it from fulfilling the request.)

Date: Nov 18, 2011 5:02 AM
The monitor SSF (http://www.smpub.com/cgi-bin/ubb/Ultimate.cgi) is back up (it was down for 07 minute(s) and 38 second(s)).
================================

Date: Nov 18, 2011 6:52 AM
The monitor SSF (http://www.smpub.com/cgi-bin/ubb/Ultimate.cgi) is currently down.
Alert Details: 500 - Internal Server Error (The server encountered an unexpected condition that prevented it from fulfilling the request.)

Date: Nov 18, 2011 6:59 AM
The monitor SSF (http://www.smpub.com/cgi-bin/ubb/Ultimate.cgi) is back up (it was down for 07 minute(s) and 26 second(s)).
====================================

---

posted 11-21-2011 03:12 PM            

I believe our technical contacts are communicating well. Here is the most recent response from Hostopia/Earthlink:

quote:

---

....Right now we are looking into if moving your site onto a different server on the webfarm will correct the issue. If it is being caused by load balancing this should correct the problem for you at least. I have made copy of your site on a test server and it seems to run with out issue, it is at least faster.

---

Thank you everyone for your continued patience.

posted 11-22-2011 09:13 AM            

This is the first overnight period where we have NOT had the our site monitor tell us the SSF went down/up. We have NOT had a site monitor down/up report since Nov 21, 2011 10:27 AM. Also we have NOT gotten contacted by any users.

To me, the SSF forum pages are once again loading well.

The complicated test thread in the new members forum was able to post in about about a minute. This time is similar to what was happening before the CGI resources problem appeared. Re-indexing the SSF went without a problem and seemed to take about as long as in the past

Hopefully, I am not premature in thinking the CGI problems are over. And I am looking forward to finding out Hostopia now understands what caused the CGI issue and have taken steps to see that the problem has been permanently resolved.

Since the problem appeared without warning, what I am fearful of is that problem has only temporally disappeared. Tomorrow we will hear from Hostopia about what they have discovered and the steps they have taken to assure the CGI Issues/problem doesn't return.

posted 11-22-2011 01:44 PM            

After 11 pm EST last night has anyone had any problems?

posted 11-23-2011 11:10 AM            

Since the CGI has been well behaved we did a bunch of forum maintenance that hasn't been done for a while. Today we successfully did:

• We re-built two forum's data that had some corruption.
  
• removed all the editing comments by Scott Martin (i.e.,[This mesage has been edited by Scott Martin (edited *).])
  
• Updated threads in all forums
  
• re-indexed all forums

posted 11-23-2011 03:25 PM            

This afternoon, as promised, we were contacted by Hostopia. They were glad to hear we haven’t had any more CGI issues since Monday 11 pm EST. This coincides with their monitoring of the SSF

They were also forthright about what they discovered and how the CGI problem has been resolved.

The simple answer is they acknowledged our CGI problems and others problems which resulted in adjustments to the server farm configuration and load balancing. They have their suspicions and have taken steps to better isolate possible heavy users and runaway applications.

In short, they don’t know which of the numerous adjusts made to the server farm made the difference for the SFF. They have a special monitor on the SSF and we suspect they will continue monitor the SSF until they are confident we are not experiencing a return of the CGI problems.

If we do see a return of the CGI issue then we may reach out to Hostopia again. But for anything else, we are back to having to deal with Earthlink’s deplorable overseas support.

Thanks to everyone at Hostopia and especially Saul.

Saul is continuing to follow up one additional problem for us. It is a problem that only effects the Administrators of the WEB site, forum and the stats. We believe this will be resolved in the not too distant future.

posted 12-02-2011 01:52 PM            

Some of us (moderators, members), very occasionally, still see a CGI issue. Our site monitor has been quiet for the last 24 hours.

I hope this won't jinks things but.... my personal experience has been good... as the SSF pages are building and loading very fast.

The Hostopia techs are always making adjustments/improvements/fine tuning to their server farm. They have also been clear that nothing was specifically done to resolve the SSF's prior CGI issues but rather it went away after some other "adjustments/improvements/fine tuning" to the server farm. What helpped us they are unsure. I do appreciate the honesty.

Since the big CGI problems, some of us and our site monitor, have seen the occasional CGI stall or failure. A CGI stall is annoying. A CGI failure has resulted in lost threads/posts and creates some extra work to recover most of the lost posts/threads. At this time, the event of a CGI failure is not easily detected and is a rare event. Even so we would prefer things work 99.99% or better.

The Hostopia techs are unable to say much because there is no way to trigger the problem so that they are unable to do further resolution testing.

To their credit, they are keeping their own site monitors running on the SSF. And they have asked for, and we will provide, 2 test forums for them to test with.

If you detect an issue please

1. email us. info@smpub.com subject: SSF technical issue
2. Note the time you are having the issue.
   
3. Describe the issue with as much detail as possible.
   
4. it would be helpful if you could include screen shots

Thanks

posted 12-30-2011 06:56 PM            

The ISP has resolved near all our problems. The remaining issue has been very hard to determine.

Once or twice a day, our site diagnostics indicate there appears to be a few seconds/minute(s) site outage.
The ISP’s own site monitors are NOT showing the same.
Very occasionally, I see a problem but it only last a few seconds. Since it only lasts a few seconds it is difficult to determine if it is a server or an Internet traffic issue.

It would be very helpful if you experience a problem, if you would post here:

1. The time the problem starts (include time zone).
   
2. The duration (length of time)
   
3. Any error messages and/or
   Please describe, in detail, the problem.
   
4. What browser and version.