• Welcome to the Two Wheeled Texans community! Feel free to hang out and lurk as long as you like. However, we would like to encourage you to register so that you can join the community and use the numerous features on the site. After registering, don't forget to post up an introduction!

Server Busy Errors

Tourmeister

Keeper of the Asylum
Admin
Joined
Feb 28, 2003
Messages
45,765
Location
Huntsville
Howdy,

:tab We are aware of the "Server Busy" messages and of the site simply not responding at all. We're working on tracking down the cause. We're sorry for the interruption of your addiction. For the latest info on what is happening, find the end of this thread ;-)
 
Last edited:
Joined
Apr 21, 2004
Messages
2,685
Location
Bryan, TX!!
Any word on the cause? I have been getting these for a while. More today. I have wondered if TWT was just too big for the setup you have had going. (All that new stuff still isn't sorted out and it's still fickle.) :doh:
 

Tourmeister

Keeper of the Asylum
Admin
Joined
Feb 28, 2003
Messages
45,765
Location
Huntsville
:tab Well, there are several issues. Right now, we are sharing the server with ST-Owners.com. Their site is about the same size as TWT. Even with both sites, the hardware should not be having a problem. So we are thinking it has to be some kind of configuration snag. So I have gone to the VB support site and gotten the recommended settings and changes to the server. However, it is beyond my abilities so I will be paying the hosting service techs to do it for me. Also, ST-Owners is likely to be moving off to their own dedicated server soon. This will double my hosting costs, but it means that TWT and my own personal sites will be the only sites on the machine. That gives us a little more flexibility with regards to making changes and upgrading stuff because I don't have to check with anyone regarding timing, costs, etc,... Hopefully, all of the above will be sorted out before the end of this month.

:tab Is there any particular time of day you are getting these messages? I am on most of the day from noon until about 2:00am and I pretty much never see the message my self. However, I do get the occasional email from the site letting me know the server stopped responding for a moment or two. I can also monitor the server load live and every now and then it will just spike for no apparent reason... :ponder: Very annoying. We are still working on it though.
 
Joined
Apr 21, 2004
Messages
2,685
Location
Bryan, TX!!
I got a few just before I posted here. It took about 20 minutes before I could get a page to load.

I have gotten them at various times ranging from 6-7am, 1-4pm, 6-11pm. So pretty much anytime. My visits can be a bit sporadic.
 

Tourmeister

Keeper of the Asylum
Admin
Joined
Feb 28, 2003
Messages
45,765
Location
Huntsville
Hmmmm, before people go to work, lunch break time, and after they get home. Makes sense to me!;-)
I thought the same thing, but when I monitor the server loads, it will often spike with very few people on the site :shrug: It will spike for a few minutes and then drop back to normal. Like I said, I am on pretty much noon to 2:00am and I cannot recall the last time I saw it :scratch:
 
Joined
Dec 19, 2004
Messages
3,401
Location
Austin Texas
I'm in a java/apache environment. We see periods where apache consumes more cpu than java apps, which should never/rarely happen, so I bounce apache. I've also seen far too many sql calls being made from an app that appears to have little load, so we've done a lot of work looking at sql and driver updates.

Its not always load related. Interesting little buggers these things are. I assume someone is perusing forums, and its likely someone else is seeing the same thing in their environ. Or it will flush with your next update.
 

Tourmeister

Keeper of the Asylum
Admin
Joined
Feb 28, 2003
Messages
45,765
Location
Huntsville
Well, once we get the server sharing ironed out, I will be providing my tech guys a list of software updates for Apache, MySQL and some other stuff, in addition to tweaking settings. Hopefully that will take care of it. The hardware is a dual 3G Pentium, 2GB Ram, couple hundred Gig hard drive, so it really should be more than enough for a site the size of TWT.
 

Tourmeister

Keeper of the Asylum
Admin
Joined
Feb 28, 2003
Messages
45,765
Location
Huntsville
:tab So we got a HUGE load spike just a few minutes ago (just after noon). A normal server load number should be 5 or less. I was monitoring the load and it spiked up to about 275. People start getting the server busy message when the load hits 10... During that spike, the server memory was pegged at 100% usage and the CPU loads were bouncing around from zero to 100%. As the memory usage finally started to come back down, the server load number dropped. This is strange because we have 2GB RAM in the server which all the tech types are telling me should be plenty for the number of people we are getting on the site :scratch: I may try sticking in another GB of RAM just to see if that helps. I'll keep you posted.
 
Joined
Jan 13, 2007
Messages
3,389
Location
Gravel Switch, KY
I thought the same thing, but when I monitor the server loads, it will often spike with very few people on the site :shrug: It will spike for a few minutes and then drop back to normal. Like I said, I am on pretty much noon to 2:00am and I cannot recall the last time I saw it :scratch:
I was wondering, when you're talking about how many people are on, is this how many members are logged in or how many people, total, including guests are viewing the site?
 

Tourmeister

Keeper of the Asylum
Admin
Joined
Feb 28, 2003
Messages
45,765
Location
Huntsville
I was wondering, when you're talking about how many people are on, is this how many members are logged in or how many people, total, including guests are viewing the site?
It is the number of total users that matters. Also, even if we have hundreds of people on at one time, not all of them are actually doing anything at any given moment. Someone that is just reading a post is not using any server resources. Someone doing a bunch of searches really hammers the server resources.

The interesting thing is that the other night I watched the load spike really high and there were maybe 15 people logged in and maybe 15-20 guests. Yet at other times, we might have over 100 people logged in and over 100 guests and the server load is nice and low. :shrug:
 
Joined
May 28, 2007
Messages
2,775
Location
Midlothian, TX
Does the database have any scheduled maintenance tasks? index reorgs... cleanups... I don't know which one is there, but from the "sound" of the symptoms, it makes sense that something large is getting kicked off periodically... you've probably already thought of that.
 

Tourmeister

Keeper of the Asylum
Admin
Joined
Feb 28, 2003
Messages
45,765
Location
Huntsville
Yeah, most of that stuff happens during off hours. The most intensive one starts every night at 4:00am and runs for maybe a few minutes.
 
Joined
Apr 21, 2004
Messages
2,685
Location
Bryan, TX!!
I still see the messages about once or twice a week. I quit posting about it since I know Scott is already on top of it. I look forward to the solution arriving.
 

Tracker

Forum Supporter
Joined
Dec 25, 2005
Messages
11,105
Location
down in a holler, Catawba County, NC
:tab So we got a HUGE load spike just a few minutes ago (just after noon). A normal server load number should be 5 or less. I was monitoring the load and it spiked up to about 275. People start getting the server busy message when the load hits 10... During that spike, the server memory was pegged at 100% usage and the CPU loads were bouncing around from zero to 100%. As the memory usage finally started to come back down, the server load number dropped. This is strange because we have 2GB RAM in the server which all the tech types are telling me should be plenty for the number of people we are getting on the site :scratch: I may try sticking in another GB of RAM just to see if that helps. I'll keep you posted.
Can you capture what process is creating the spikes? Sounds like a bad process if it's spiking memory and CPU to 100%. You're on linux/mySQL, right?
 

Tourmeister

Keeper of the Asylum
Admin
Joined
Feb 28, 2003
Messages
45,765
Location
Huntsville
So I watched from about 4:30pm-5:30pm today and never saw it spike above 5 despite having nearly 200 people on the site during that time.

Gary, I never really see any one process that goes wild. Each user is listed as a process and those will run up to 100% on the CPU sometimes for a brief moment or two, but they seldom go nuts on the memory. I will be continuing to monitor it. The problem is that I am not always sitting at my computer to watch when a spike hits.
 

wczimmerman

Forum Supporter
Joined
Apr 1, 2004
Messages
1,204
Location
Renton, WA
Have you checked your mysql config? At the shop, we've run into problems with mysql where the number of database connections available to the application was too low for the load causing, obviously, failures to connect. The other thing that comes to mind is some sort of table contention with the DB.

From a system standpoint, check the system log (/var/log/messages depending on your Linux distribution) and run the command dmesg to get console messages. These would point out any hardware errors or software crashes that may be happening.
 

Tourmeister

Keeper of the Asylum
Admin
Joined
Feb 28, 2003
Messages
45,765
Location
Huntsville
:nono:

I am NOT a sysadmin!! I am a user struggling in unfamiliar territory :lol2:

What I have done is get some recommendations from the VB tech heads about appropriate mysql config settings which I am forwarding to the hosting tech heads so they can implement. Also, the will be updating Linux, Apache, MySQL, and a few other software thingies to the latest greatest versions. So hopefully that will clear up the issue.
 

wczimmerman

Forum Supporter
Joined
Apr 1, 2004
Messages
1,204
Location
Renton, WA
:nono:

I am NOT a sysadmin!! I am a user struggling in unfamiliar territory :lol2:

What I have done is get some recommendations from the VB tech heads about appropriate mysql config settings which I am forwarding to the hosting tech heads so they can implement. Also, the will be updating Linux, Apache, MySQL, and a few other software thingies to the latest greatest versions. So hopefully that will clear up the issue.
Sorry. I AM a SysAdmin (Linux/AIX) so sometimes my work face comes out. :giveup:
 

Tourmeister

Keeper of the Asylum
Admin
Joined
Feb 28, 2003
Messages
45,765
Location
Huntsville
The deal is, I don't have full root access to the server so I am not able to make all the changes anyway. Nor am I even remotely familiar with the environment having spent all my years in DOS/OS2 before finally coming over to Winbloze. I have never had any need/reason to take up Linux prior to doing this site. For the first few years of the site's existence, I had a fantastic tech head that just made everything happen with no fuss ;-) No I am just another customer among many when I call tech support :doh:
 

Tourmeister

Keeper of the Asylum
Admin
Joined
Feb 28, 2003
Messages
45,765
Location
Huntsville
:tab Okay, so with the help of Wczimmerman, we think it may be a kernel issue with the version of Linux running our server. I am going to check with the techs at the hosting service to see if I can get it upgraded to the latest version. Ideally, this will only require that the site be down for a few minutes while the machine reboots.
 
T

Texfire

Kernel? Does this mean we have too many posts about corn in the ethanol thread? ;)
 

Tourmeister

Keeper of the Asylum
Admin
Joined
Feb 28, 2003
Messages
45,765
Location
Huntsville
:tab:tab:tab:thumb:
:tab I wouldn't even notice.
:tab It has been locking up more frequently for longer periods of time than a reboot. :mrgreen:
:tab You make an interesting observation that I have been musing over lately. Nothing has really changed in terms of the actual usage of the server or in the software and yet the load spikes seem to be getting more frequent and lasting longer each time! :angryfire: I contacted my tech heads at the hosting service yesterday to see what can be done in terms of getting the server OS updated to the most recent kernel version. Supposedly, this will fix the problem. I am waiting on them to respond to my latest inquiries.
 

Tourmeister

Keeper of the Asylum
Admin
Joined
Feb 28, 2003
Messages
45,765
Location
Huntsville
Okay. We now have the latest greatest version of the kernel for the operating system. I will continue to monitor the server loads. Hopefully, this will have resolved the issues. Thanks for your patience!
 
Joined
Apr 21, 2004
Messages
2,685
Location
Bryan, TX!!
I haven't seen the server busy errors, but I have been unable to load the forum on several occasions since the reboot.
 

kurt

Admin
Joined
Nov 17, 2003
Messages
19,307
Location
Lago Vista, Texas
Did you ever get rid of that free-loading other forum (forgot the name - touring or something like that)? ;-)
I'll bet not. They have he same delays at the same times as TWT. I can try to load both pages in separate tabs and they both have the same load or time out times. :giveup:
 

Texas T

Moderator
Forum Supporter
Joined
Nov 7, 2004
Messages
11,913
Location
Sun Lakes & Show Low, Arizona
Timed out today from 13:13 - 13:19. 51 members and 51 guests online when it came back up.

I posted this at approx 13:22 but the time stamp on the thread was 12:57?
 

Tourmeister

Keeper of the Asylum
Admin
Joined
Feb 28, 2003
Messages
45,765
Location
Huntsville
:tab Yeah, last night it went down around 10:00pm :doh: It was so bad that I could not even telnet into the server to attempt a restart of MySQL. Not sure what caused that. Usually, with a spike, I would still be able to telnet into the server to run command line stuff. The next step is to try making some changes to the MySQL config file to see if that helps.
 

Tourmeister

Keeper of the Asylum
Admin
Joined
Feb 28, 2003
Messages
45,765
Location
Huntsville
Well, if that were the case, I would expect the problem to have gotten much worse since we first moved to this server box. My experience with most hardware failures is that they either bring things to a screeching halt pretty quick, or in short order... I have seldom seen one that lingers for months. I'm not saying it is impossible, just unusual. In the end, if we have all the software upgraded and the problem persists, then it may be time to look at moving to a new physical server. The problem with that is it will cost a LOT of money...
 

Tourmeister

Keeper of the Asylum
Admin
Joined
Feb 28, 2003
Messages
45,765
Location
Huntsville
Well, the upgrade seems to have made things even worse :doh: :headbang: So the server has been rolled back to the previous kernel version for now. I'm working on some other stuff in the mean time. Sorry for the annoyance :oops:
 

Tourmeister

Keeper of the Asylum
Admin
Joined
Feb 28, 2003
Messages
45,765
Location
Huntsville
:tab That should only matter if people are doing lots of searches. I have tried doing searches on broad key words to see if that drives up the load and it does not seem to :shrug:

:tab At this point, I am exploring options to move to a new (as in custom built) server strictly for TWT so that there are no sharing issues to worry about. It will cost quite a bit more initially, but for the peace of mind and hopefully stable environment, it would be worth it.
 
Joined
Apr 4, 2006
Messages
13,843
Location
Arlington, TX
I got the time out thingy today at 14:25. But it only lasted for about 3 or so minutes so my withdrawals didn't get out of hand.... this time...:trust:
 

Tourmeister

Keeper of the Asylum
Admin
Joined
Feb 28, 2003
Messages
45,765
Location
Huntsville
Oh, I know the times down to the minute. I get TONS of error emails every time there is a database error ;-)

I just updated the MySQL config file about five minutes ago (11:15pm). Now we play the waiting game to see if that helps... :ponder:
 
Joined
Apr 9, 2007
Messages
1,147
Location
Katy
Well I don't know if this would help with the diagnosis, or puzzle you even more... but I've noticed when I get the "server busy" error message, that if I click on the link to the home page, and from the home page to the 'forums' link, it will let me in immediately, instead of getting another "server busy" message. It has worked every time since I tried that.
 
Joined
Apr 9, 2007
Messages
1,147
Location
Katy
Well scratch my last post... I should have known better than to write "works every time" .... :doh:
 
Joined
Feb 16, 2007
Messages
891
Location
Central Texas
Would more memory help? I was of the opinion that Unix was bullet proof. Oh wait it's the software, but then the software folks always say it's the hardware. This scenario hasn't changed since I entered the computer/data processing field in the late 60's.:shrug:

Are there any rights issues as far as who has access to which files/data bases? I have know of instances where rights were not verified yet a denial msg would never be issued and the program would just go into a loop or drop off into the black hole ultimately consuming vast amounts of system resources.
 

Tourmeister

Keeper of the Asylum
Admin
Joined
Feb 28, 2003
Messages
45,765
Location
Huntsville
:tab Everything I am being told by the VB support people is that our hardware setup should be over kill for our use levels. So either we have a piece of hardware going bad on us, or there is something in the software that is off... I will probably take the site down again this coming Saturday at noon so the techs at the hosting center can run a hardware check to ensure there is not something wrong with any of the components. Wczimmerman is currently going through all the software stuff checking settings, tweaking config stuff, etc,... trying to eliminate anything from that angle. I have gone over all the settings within VB itself looking for anything that might be consuming server resources and so far as I can tell, there is nothing else to be done in forum software :shrug: If all else fails, I will then consider simply setting up a custom server from scratch with all new software (clean install) and then moving the site again (a pain, but this needs to get fixed!)
 
Joined
Feb 13, 2005
Messages
74
Location
Whitehouse, Texas
Can't we just give the server a break? I mean maybe it has to go to the bathroom or needs a little "me" time. You Guys are so needy. :doh:

Stephen
 

wczimmerman

Forum Supporter
Joined
Apr 1, 2004
Messages
1,204
Location
Renton, WA
I've been working with Scott on this issue for a couple of days now. I think we're making progress, but there's been a whole lot of :headbang:

Quick-Silver: you asking if UNIX is rock solid? In short, yes. :trust:

This issue is a strange one, though. When the system was unresponsive before, it was because the system load would spike suddenly and cross the threshold that the apps were configured to respond with a "server busy" error. What was causing the system load spike is still unknown. :giveup:

Currently, we've tuned MySQL for better memory use (smaller, actually) which has reduced the running system load pretty dramatically. I don't think anyone has seen a "server busy" error since yesterday afternoon, but the remaining issue is one of the connection not responding for a period of time. I have noticed that there are errors and dropped packets on the interface and they have been growing, so a new cable or network card would be the next logical step in this process.

TM: can you setup with the hosting company a time for them to replace the cable with a new one AND move it to a different port on the switch? They need to check the logs on the switch, too for errors.

All in all, I think the system performance is improving. The loading seems to run a bit faster unless it's just my optimistic thinking. :lol2: If we can just nail down this connectivity issue, then I think we'll be good for a while.
 
Top