• Welcome to the Two Wheeled Texans community! Feel free to hang out and lurk as long as you like. However, we would like to encourage you to register so that you can join the community and use the numerous features on the site. After registering, don't forget to post up an introduction!

Server Busy Errors

Joined
Feb 16, 2007
Messages
891
Location
Central Texas
I was thinking that there have been a some announced hardware or software upgrades to the system over the past several months. Sometimes just going back to an earlier ghost or drive image can help isolate problems related to changes in a network environment. I used to dread upgrades because almost invariably it meant incompatibility issues would need to be addressed.

Re-imaging a system means down time, but it's not like you are in a position of having to guarantee uptime - right? I visit other forums and TWT is the fastest as far as response time goes. Unless there are other factors involved in your decision to have your own server, the elimination of a few little glitches we've run into lately hardly seem worth the cost of buying a new server just for this small TWT database.

This is a clean and well run board, so much so that I don't mind going along with whatever it takes for you guys to work through these "server busy" problems. This too shall pass.;-)

just my $.02.
:sun:
 
Joined
Feb 16, 2007
Messages
891
Location
Central Texas
I was thinking that there have been a some announced hardware or software upgrades to the system over the past several months. Sometimes just going back to an earlier ghost or drive image can help isolate problems related to changes in a network environment. I used to dread upgrades because almost invariably it meant incompatibility issues would need to be addressed.

Re-imaging a system means down time, but it's not like you are in a position of having to guarantee uptime - right? I visit other forums and TWT is the fastest as far as response time goes. Unless there are other factors involved in your decision to have your own server, the elimination of a few little glitches we've run into lately hardly seem worth the cost of buying a new server just for this small TWT database.

This is a clean and well run board, so much so that I don't mind going along with whatever it takes for you guys to work through these "server busy" problems. This too shall pass.;-)

Bill, a good point here, "I have noticed that there are errors and dropped packets on the interface and they have been growing, so a new cable or network card would be the next logical step in this process." In the old days when we made our own network cables this was common place, especially when apprentice ISSS's had made some of the cables. I've experienced slow response times when controllers, routers or modems were dieing.

I also noticed this morning I couldn't get in until the third try. My PC timed out and I got "connection failed" errors indicating the server was down.

Could there be problems between our host and there ISP (addressing issues) that could be the cause of our "busy server" or "failed connection" situation?

just my $.02.
:sun:
 

Tourmeister

Keeper of the Asylum
Admin
Joined
Feb 28, 2003
Messages
45,764
Location
Huntsville
:tab QS, there have only be a few upgrades since we moved to this server back around March. The problems started though from day one on this server. The first thing we did was bump the RAM from 1GB to 2GB and that seemed to work up until recently. The other upgrade was when I upgraded the forum software to the latest version a few months back. That did not appear to have any affect on the server performance. What we had was just the occasional user getting a server busy message, similar to what was going on before that upgrade.

:tab Now recently, in the last month or so, the server performance has really gotten MUCH worse even though we have not changed anything. The outages have become more frequent and longer lasting. This is what prompted the kernel upgrade last Saturday. That made things worse and we have rolled back the upgrade to the previous version.

:tab Chris has been working at monitoring the system and looking for problems in config files. We've been trying different MySQL config settings. Those may be helping some, but we are still getting what I would describe the server freezing. Previously, people would get the server busy error, but that means the server was still at least putting out the website content with that message. Also, I was still able to log into the server remotely and restart MySQL to drop the load. What I see happening now is the server is just freezing and not responding at all. So if you type in twtex.com, it simply never loads anything and you get the browser error saying "page not responding". When this happens, I cannot even ping the machine. Also, the server load numbers don't always show a rising load prior to a freeze or after a freeze. This is what has me thinking there may be a hardware issue. It could be anything from a chip getting to hot and shutting itself down until it cools, network cards acting up, network cables going in/out, etc,...

:tab The data center guys are nice, but not real helpful when it comes to getting them to do stuff that they cannot bill for. So I really have to push to get hardware related stuff investigated. I agree that the cost of setting up a new server and moving the site is something to be avoided if at all possible. So we are going to keep :headbang: until we can figure something out. Then if all else fails, I will look at relocating the site.
 
Joined
Feb 16, 2007
Messages
891
Location
Central Texas
I know this is reaching, but I have seen instances where after an thunder storm or an electrical outage a router would need to be brought down due to addressing problems and up again to re-initialize itself and this is all that was needed for the fix.
:sun:
 

TWTim

Forum Supporter
Joined
Feb 19, 2007
Messages
9,907
Location
Midland
So either we have a piece of hardware going bad on us, or there is something in the software that is off... I will probably take the site down again this coming Saturday at noon...
Just so you'll know, I tried to log-in about 20 minutes ago and got a "Server Does Not Exist" error from my browser. It cleared after about 5 minutes.

I think you may have a hardware problem. Check for bad/shorted cabling, especially if you're getting load spikes for no apparent reason.
 

Tourmeister

Keeper of the Asylum
Admin
Joined
Feb 28, 2003
Messages
45,764
Location
Huntsville
Just so you'll know, I tried to log-in about 20 minutes ago and got a "Server Does Not Exist" error from my browser. It cleared after about 5 minutes.

I think you may have a hardware problem. Check for bad/shorted cabling, especially if you're getting load spikes for no apparent reason.
Well, the site may be going down anytime now. I asked them to replace the cable/network card. As soon as the other tech heads get back from lunch, the guy I talked with will be heading down to the cage to make the changes. If you are a praying person... Now would be a good time :pray:
 

TWTim

Forum Supporter
Joined
Feb 19, 2007
Messages
9,907
Location
Midland
Well, the site may be going down anytime now. I asked them to replace the cable/network card. As soon as the other tech heads get back from lunch, the guy I talked with will be heading down to the cage to make the changes. If you are a praying person... Now would be a good time :pray:
[PRAY]Lord, please bless this forum and guide the technicians in returning it to good working order. Thank you for TWT, and for the opportunities and relationships in facilitates. In Christ's holy name, Amen.[/PRAY]

:thumb:
 

Tracker

Forum Supporter
Joined
Dec 25, 2005
Messages
11,105
Location
down in a holler, Catawba County, NC
random thoughts that have helped me in the past.

1. Always start with the basics. Cabling (internal and external). Ports on switch, NIC., clean power.
2. Change management. Always document every change made in some form of log. Document every hiccup reported. Sometimes you don't know you have a trend until you look back 4 weeks later and see change X made on date Y had effect Z, even though X shouldn't have resulted in Z.
3. Prayer. Isn't it amazing to think that God knows where every electron is and he knows exactly what line of code is bad or what component is hiccupping.
4. Systematic elimination. I'm sure you're all over this one.
 
Joined
Dec 18, 2006
Messages
2,016
Location
Houston
With everything said, done and tried thats been listed and judging from the amount of money you appear to be spending per month for this host, I would venture to say that perhaps Sproketdata.com may not be serving your best financial interests. At least the same IP resolves back to Sproketdata.

We hosted through theplanet in dallas, and their facilities are nice, and significantly more affordable than what your paying for these guys. I ve been getting a significant amount of page cannot be displayed errors in the last few days.
 

Tourmeister

Keeper of the Asylum
Admin
Joined
Feb 28, 2003
Messages
45,764
Location
Huntsville
Theplanet has some nice deals for sure. However, I'd be paying a good bit more than I am paying now. Given the cost for dedicated hosting, I'd like to build/purchase my own machine instead of leasing. At their lease rates, I could pay for the machine pretty quick ;-) I may eventually go this route even if we figure out what is going on with the current machine just so that we have room for TWT to continue growing (hopefully without the growing pains :-P).
 
Joined
May 29, 2003
Messages
1,639
Location
Dallas, TX
Theplanet has some nice deals for sure.
The Planet is where we were originally hosted, before I moved equipment down to Houston. They do good work, but like anything you get what you pay for. There's an advantage to leasing equipment from them ... they'll do your break/fix, but it all comes down to what you can afford.
 
Joined
Dec 19, 2004
Messages
3,401
Location
Austin Texas
I know you guys are considering all factors, but we had a server running open solaris that had problems. Rebuilt with CentOS and no hardware changes and the server performs perfectly. All the other apps and services were the same. It was easier than kernel debugging.
 

Tourmeister

Keeper of the Asylum
Admin
Joined
Feb 28, 2003
Messages
45,764
Location
Huntsville
:tab Well, yesterday around 4:30pm or so, we got a different network card installed in the machine. I doubt it was new... Anyway, things seemed to run pretty well initially, but last night we started getting problems again :doh: So, we will keep at it...
 
Joined
Dec 18, 2006
Messages
2,016
Location
Houston
Theplanet has some nice deals for sure. However, I'd be paying a good bit more than I am paying now. Given the cost for dedicated hosting, I'd like to build/purchase my own machine instead of leasing. At their lease rates, I could pay for the machine pretty quick ;-) I may eventually go this route even if we figure out what is going on with the current machine just so that we have room for TWT to continue growing (hopefully without the growing pains :-P).

Noted: We had 1/2 rack from them, and supplied out own equipment. Went from MCI/Worldcom facilities in Richardson over to ThePlanet after we got a much better deal.

No fault can be given for wanting your own equipment in there. If your vision & model is to continue to grow with a compound rate of members and or contributions, then owning your own equipment can get 'spensive pretty quickly. If you are paying this much for service for machines and bandwidth for a site the size of TWTEX, I simply couldnt imagine what Adv Rider costs per month. :eek2:

Good luck guys. I know youll get it worked out.
 

wczimmerman

Forum Supporter
Joined
Apr 1, 2004
Messages
1,204
Location
Renton, WA
In the immortal words of Monty Python, "I'm getting better! No, you're not. You'll be stone dead in a moment!"

That's kinda been the experience on this so far. Where we've tuned things it appears that it has in fact made it run better. This load spike issue seems to be getting less frequent, but it's still there.

TM: does the provider do any denial of service detection? I wonder if the randomness of this is due to an outside flood of traffic in an effort to hack/crash the system?
 

Tourmeister

Keeper of the Asylum
Admin
Joined
Feb 28, 2003
Messages
45,764
Location
Huntsville
Techs say the have been monitoring for DOS attacks and have seen nothing indicating that might be a problem.
 

Tourmeister

Keeper of the Asylum
Admin
Joined
Feb 28, 2003
Messages
45,764
Location
Huntsville
Just got off the phone with techs again. They will be trying to build a new box for us from comparable hardware and then mirroring the site over to the new machine. Ideally, this will confirm/deny hardware issues... Will probably be Monday before they can get the new box up and running to make the switch.
 

wczimmerman

Forum Supporter
Joined
Apr 1, 2004
Messages
1,204
Location
Renton, WA
Ah ha!

TM mentioned to the techs about an excessive number of *.bin files in the mysql directory to which he responded that he thought they were dump/crash files. That got me thinking and researching.

They are binary logs for updates to the db (kinda like transaction logs). According to TM, there are a BUNCH of them (up to 22GB worth) in this same directory. They are also used for DB replication which doesn't apply to us. I happened to catch the system when it spiked this morning at 09:30 with vmstat and I saw lots of I/O for that timeframe. TM also mentioned that the timestamps of the files matched many of the outage windows we've been seeing.

I've sent a PM to him with the following suggestions:
1. Using MySQL commands, trim the binary logs down to just one. This is a normal maintenance thing that needs to take place periodically but doesn't look like it has at all simply because we didn't know to do it. This could fix the situation and preserve the point-in-time recovery of the DB if it is struggling to keep track of a LARGE number of files. Not to mention the filesystem performance can be affected if the number of files gets to be excessive. I don't know the exact number of files that exist, but if it's been running for all this time without a purge, it could be huge.
2. Turn off binary logging altogether. This would remove the ability to do a point in time recovery-meaning if the database died we would only be able to recover back to the data of the last backup (which I think is being done nightly), but this could improve DB performance and stop this pesky issue.

All of the other tuning seems to have really improved the site at all the other times.
 

wczimmerman

Forum Supporter
Joined
Apr 1, 2004
Messages
1,204
Location
Renton, WA
Update:

Well...:headbang: we're getting there...

Think of this like an old analog dial radio. You turn the dial one way, then back the other to find that sweet spot that gives you the best performance. The last round of tweaks from overnight had the system absolutely hammered. So...turned things back today and things are closer to normal. TM and I are working on this pretty solid, so if you get a timeout or server busy error here and there-please be patient.
 

TWTim

Forum Supporter
Joined
Feb 19, 2007
Messages
9,907
Location
Midland
Think of this like an old analog dial radio. You turn the dial one way, then back the other to find that sweet spot that gives you the best performance. The last round of tweaks from overnight had the system absolutely hammered. So...turned things back today and things are closer to normal. TM and I are working on this pretty solid, so if you get a timeout or server busy error here and there-please be patient.
FYI, this morning:

twtdatabaseerror.jpg
 
Joined
Apr 9, 2007
Messages
1,147
Location
Katy
Update:

Well...:headbang: we're getting there...

Think of this like an old analog dial radio. You turn the dial one way, then back the other to find that sweet spot that gives you the best performance. The last round of tweaks from overnight had the system absolutely hammered. So...turned things back today and things are closer to normal. TM and I are working on this pretty solid, so if you get a timeout or server busy error here and there-please be patient.
I don't know what you guys did, but since you posted this I haven't had any issues with the forum. It was really crappy earlier in the morning though... time outs and server busy messages.
 
Joined
May 13, 2004
Messages
3,960
Location
Leander, Tx
In the immortal words of Monty Python, "I'm getting better! No, you're not. You'll be stone dead in a moment!"

That's kinda been the experience on this so far. Where we've tuned things it appears that it has in fact made it run better. This load spike issue seems to be getting less frequent, but it's still there.
"You're not fooling anyone, y'know..."

Update:

Well...:headbang: we're getting there...

Think of this like an old analog dial radio. You turn the dial one way, then back the other to find that sweet spot that gives you the best performance. The last round of tweaks from overnight had the system absolutely hammered. So...turned things back today and things are closer to normal. TM and I are working on this pretty solid, so if you get a timeout or server busy error here and there-please be patient.
Or like jetting a carb using the "pant seat dyno"?
 
R

Red Brown

Howdy,

:tab We are aware of the "Server Busy" messages and of the site simply not responding at all. We're working on tracking down the cause. We're sorry for the interruption of your addiction. For the latest info on what is happening, find the end of this thread ;-)
1. Perhaps provide telnet access to one of the forum leaders who knows UNIX well. You can easily track in real time all processes in a little window.

2. Print out a web access log during around the time the site was having that issue. You can PDF it or post for others to view...perhaps in a private section for privacy.

3. You might consider moving the site to a Yahoo or Google free hosted groups. This will never have overload issues and there might be a way of importing at least part of the messages/content into it. This option is pretty good if the continued economics don't make sense. The other ST-Owners web site does this through MSN Groups.

4. You might consider buying a good web server and having a dedicated hosting option. There are many cheap options out there for dedicated hosting..perhaps outside of the US?

RB
 
Joined
Sep 18, 2006
Messages
276
Location
McKinney Texas
Update:

Well...:headbang: we're getting there...

Think of this like an old analog dial radio. You turn the dial one way, then back the other to find that sweet spot that gives you the best performance. The last round of tweaks from overnight had the system absolutely hammered. So...turned things back today and things are closer to normal. TM and I are working on this pretty solid, so if you get a timeout or server busy error here and there-please be patient.
I'm not seeing server busy errors, I'm seeing "there is no such server" errors. (see Tim Kreitz example below - or above, as the case may be :)
 
Last edited:

Squeaky

2
Forum Supporter
Joined
Mar 6, 2004
Messages
13,258
Location
Katy
Do you want data about outages and down time, or are you guys Ok with that and we should just sit back and wait?

(I'm so used every example being documented at work, it's a habit)
 

Tourmeister

Keeper of the Asylum
Admin
Joined
Feb 28, 2003
Messages
45,764
Location
Huntsville
:tab We are already doing much of what Red suggested regarding logging stuff and monitoring processes. Squeak, we pretty much know when every outage occurs. After last night I had over 1000 emails sitting in my inbox letting me know the site was not responding... hehe.
 
Joined
Jun 7, 2006
Messages
5,846
Location
Exit. Stage West.
:tab We are already doing much of what Red suggested regarding logging stuff and monitoring processes. Squeak, we pretty much know when every outage occurs. After last night I had over 1000 emails sitting in my inbox letting me know the site was not responding... hehe.
I don't know if this means anything or is helpful but I had no access at all from 4:30 am till near noon yesterday. One brief time I did (later morning) I noticed other posts from throughout the am. I posted once then no access again. Denied! ;-)

So access was random? It was slow loading all afternoon and last night I didn't even bother. (too slow to load)

No, I'm not addicted but I had two PM's waiting for me.

Just FYI
 

wczimmerman

Forum Supporter
Joined
Apr 1, 2004
Messages
1,204
Location
Renton, WA
I don't know if this means anything or is helpful but I had no access at all from 4:30 am till near noon yesterday. One brief time I did (later morning) I noticed other posts from throughout the am. I posted once then no access again. Denied! ;-)

So access was random? It was slow loading all afternoon and last night I didn't even bother. (too slow to load)

No, I'm not addicted but I had two PM's waiting for me.

Just FYI
Yeah-the system was held together with duct tape and bailing wire yesterday morning. TM and I worked for a couple of hours on this last night. Let's see how it does today.
 
Joined
May 28, 2007
Messages
2,775
Location
Midlothian, TX
I can't help myself...

open software... the rage... stable... installs and tunes itself... Like Apple... you don't need training... it's intuitive...

The alternative... pay money up front... pay the vendor to "help" you... look for help from your friends...

What's the difference? I can't see one. It's software and will drive you batty.
 

Tourmeister

Keeper of the Asylum
Admin
Joined
Feb 28, 2003
Messages
45,764
Location
Huntsville
:tab Okay, so the new plan is to simply move to new hardware, install clean and up to date versions of all software, then move the site over. I don't have a schedule for this just yet. Right now I am pricing servers and colocation costs. Once we decide what we are going to do, it will still take some time to get it done. I'll post up the details once I know something definite.

:tab I know this has been inconvenient and annoying for folks and I apologize, but we are somewhat constrained by financial concerns. This is why we have been trying to exhaust potential solutions before making this move. I want to get a server that has good performance and growing room. The move to the new server is going to be a BIG out of the ordinary expense. Thus it is not something I can do without serious consideration. Ideally though, this will have us set for a long time.

:tab So I would just ask that you please bear with us and be patient. And don't worry, we are not going to be hosting the site on some machine in India :-P
 

wczimmerman

Forum Supporter
Joined
Apr 1, 2004
Messages
1,204
Location
Renton, WA
The last change we put in at 13:00 today seems to be holding things quite well. If this holds out, we should be good to run while we setup the new server.
 

Tourmeister

Keeper of the Asylum
Admin
Joined
Feb 28, 2003
Messages
45,764
Location
Huntsville
:tab I'll be ordering the new server tomorrow (Wed). It will take about 5 days before it is ready. Ideally, WC will be able to head over to Austin and take delivery of it in person so that we can save the shipping time. It will probably take a week or so to get it setup and then we have to get it up to Dallas to the data center for installation in the rack. After we are satisfied everything is good to go, I'll take the site down, move it over, and then bring it back up. If all goes well (:pray:), it will be a brief downtime.

:tab As I mentioned earlier, this is going to be a significant expense above and beyond our normal operating costs. I could use any and all help that our members might be willing to give in covering that expense. To that end, I have decided to setup a separate server fund:

http://twtex.com/forums/showthread.php?t=21467

You can also access it by clicking the "Support TWT" link on the NavBar at the top of every page.
 

Tourmeister

Keeper of the Asylum
Admin
Joined
Feb 28, 2003
Messages
45,764
Location
Huntsville
The new server is in. It got to Wczimmerman's home last Tuesday. He's getting it set up. Then we'll get it up to the data center in Dallas. If all goes well, there will be a very short down time while I move all the files over to the new server. Once we bring the new server on line, we won't be changing IP addresses like last time. So you won't have all the issues with twtex.com pointing to the old site for some people and to the new site for others. It should be an instantaneous switch :pray: I'll post up to let you know when the site will be down for the copying of files.
 
Joined
Feb 16, 2007
Messages
891
Location
Central Texas
The new server is in. It got to Wczimmerman's home last Tuesday. He's getting it set up. Then we'll get it up to the data center in Dallas. If all goes well, there will be a very short down time while I move all the files over to the new server. Once we bring the new server on line, we won't be changing IP addresses like last time. So you won't have all the issues with twtex.com pointing to the old site for some people and to the new site for others. It should be an instantaneous switch :pray: I'll post up to let you know when the site will be down for the copying of files.
Great news! Will your server be dedicated to TWT Forum? Are you planning to lease out any of this new server's capacity?

Just curious as to why you are having to locate your server in a data center and not your home? My suspicion is that any problems we were having (not having any now) could have been with the data center or somebody was playing with software or config files either accidentally or deliberately.

What is the name of the data center you are using?
:sun:
 

Tourmeister

Keeper of the Asylum
Admin
Joined
Feb 28, 2003
Messages
45,764
Location
Huntsville
Will you two be coming up to Dallas to do this? We could have a little get-together & go eat or something... :eat:

I also offer my services for keeping Wizzerman under control... :lol2:
:tab I don't know. Other than dropping off the server at the data center, there really isn't much that we need to do.

Great news! Will your server be dedicated to TWT Forum? Are you planning to lease out any of this new server's capacity?

Just curious as to why you are having to locate your server in a data center and not your home? My suspicion is that any problems we were having (not having any now) could have been with the data center or somebody was playing with software or config files either accidentally or deliberately.

What is the name of the data center you are using?
:sun:
:tab The server we are on now hosts only TWT and the ST-Owners.com site. We will be moving TWT over first to make sure everything runs properly. Once we are satisfied things are good, we will bring the ST-owners site over. We will watch things for a while and if they stay good, then ST-Owners will share the server with us indefinitely. This will help tremendously with keeping our costs down. At this time, there are no plans to offer any other hosting services.

:tab There are several reasons for not having the server in my home. First is simply the data connection speed. Where I am, it would cost a fortune to get a decent speed connection for uploads, which is the bulk of the server's traffic. The data center is right in downtown Dallas at the hub of numerous high speed backbones and thus have redundant paths to the internet. This assures that I get a fast redundant connection to the outside world. Then there is the physical security of the server. I have two toddlers... NOTHING in my house is safe!! :lol2: They have secured and escorted only access. My power is less than predictable and I can't afford a UPS and generator to run the site for long outages. They have battery and generator backup able to cover extended outages with contracts for indefinite resupply of diesel for the generators. There is the protection of a hardened data center versus my home... They have environmental and fire control systems. They are located far inland so hurricanes aren't much of a threat and the area is siesmically stable. A tornado might be able to get them I guess... :shrug:

:tab The cost is also a big factor. It is just way cheaper to use a data center for colocation. We have been using SprocketNetworks.com and will likely stay with them for the foreseeable future as they are offering a good package. With Wczimmerman's admin services, we will no longer be relying on them for tech support beyond the physical needs of the machine (swapping cables, etc...). The new server includes on next day site hardware replacement from Dell, so we won't really need to rely on the tech support for any issues related to the server hardware.
 

Tourmeister

Keeper of the Asylum
Admin
Joined
Feb 28, 2003
Messages
45,764
Location
Huntsville
As of about 4:00pm this afternoon, the new server is up and running at the data center in downtown Dallas. Now we get to work out the details for moving the site over. Once we have all that ironed out, I'll post a down time notice.
 

wczimmerman

Forum Supporter
Joined
Apr 1, 2004
Messages
1,204
Location
Renton, WA
We're now running on the new server!! Now where's that, "Please be patient, I'm in training" button? Seriously, there may be a couple of hiccups here and there, but we were pretty thorough in our testing so hopefully this will take care of all of the issues.
 
Top