Forums/Z31P/announcements-and-events announcements-site-and-forum-related/*PLEASE READ*
Imported from Z31Performance.com Forums{{#if:Jason84NA2T| — original post by Jason84NA2T}}{{#if:2009-05-18 18:38:44|, posted 2009-05-18 18:38:44}}{{#if:27| · 27 post{{#ifeq:27|1||s}}}}{{#if:/Z31performance_archive/forum/announcements-and-events/announcements-site-and-forum-related/7451-please-read.html|
Local archive: view archived thread on dev.z31library.com}}
Jason84NA2T: (2009-05-18 18:38:44)
This most recent forum downage was brought to us by the hosing provider. Granted I was out of town when it happened, but essentially something was generating too many queries on the database, causing a server overload. You may have noticed the SQL errors and slowness recently, all stemming from this same issue. Because Z31Performance is hosted on a shared server, this impacted other users and forced the hosing company to take action. Their recommendation was for me to purchase a dedicated server on which to host the forum. Not wanting to spend the $170/month to do that, I persuaded them to give me an opportunity to remedy the situation. I am trying things to reduce the server overload until I can figure out what the exact cause of the problem is.
This server overload is caused by any number of things, including:
1. People using the built-in search function in PHPBB to search the forum - serach function is now disabled - please use google to search the forum
2. Spammers and Bots - I have implemented and am working on new anti-spam and upgrades to control the bots crawling the forum.
3. Lots of users on at one time - Less of an issue, but I have limited it to 50 concurrent users. We also seem to get a lot of guests hogging it up, so for now guest access is disabled. Because people can't register right now, this makes the forum inaccessible unless you already have an account.
4. Doing a lot of things at once, like opening new threads/searches in different windows before others have completed. please avoid doing this.
I have also made numerous forum configuration changes which will result in less server load. I am working on a better solution to all the above issues without any drawbacks for the users.
Thanks,
Jason
1artworkz: (2009-05-18 22:39:12)
Yooo Hooooo! I am ....
Kennedy: (2009-05-19 06:18:13)
Between web hosting and dedicated should be Virtual server ( well.. in somecases they say virtual server is dedicated.. ). Thay should be enable to limit bandwidth and memory use.
But hell.. i'll give you money for hosting money.
I have virtual server in use.. $50/month. 512MB (soon 1GB), 20GB, 10MB link, no bandwidth limit (yet)
http://www.askwebhosting.com/whs/22/…ting/michigan/
http://www.tvog.net/services.php - 100GB disc, 400Gb bandwidth, $50/month.
Get 12 Months for the price of 10! Pay for a year of service on any of our packages, and recieve 2 months absolutely FREE!
The VonOstin Group, Novi 48377, Michigan MI
$500/first year..
http://www.maxvps.com/vps/?id=vps-100 - Virtual Private Servers VPS-100, 20GB, 256MB, 500 GB Bandwidth, $29.95.. ( $360/year)
NEXCESS.NET L.L.C., Ann Arbor, MI
edit: ok.. 500GB/month is not enought.
Have you think Google ad's?
Steve88t: (2009-05-20 14:36:57)
I really don't think it is actual users causing the problems. I'd suspect the bots and spam scripts being the culprit. Does the host have a way for you to see what processes you have running and what resources they consume? You may be able to track it down a little more closely if they do.
Jason84NA2T: (2009-05-20 15:20:58)
Jukka, thanks for the suggestions. For now we get to see how things go. I don't think anyone is going to want to put up with importing our database again (let alone me). The tech support team at the current host is considered to be way above average, but even they had trouble with our 200MB sql DB. This is after I almost crashed their server a couple times trying to import it with bigdump.
Yes, but all I am able to see if number of users running each script. So we have 5 people running posting.php and 25 running index.php, etc etc.
For now, I have all guest access disabled. This will prevent any sort of crosslinking or hotlinking from other forums/websites, which might be one of the bigger issues (like when someone posts a thread here on a huge forum with 100K members).
I also implemented two new fields on registration which will force people to answer two VERY simple questions (are you a bot, what site is this) and should stop most of the automated bots that get past captcha. The spammers that can get in now will most likely be people, which will force us to simply be more reactive to their postings.
The google bot and yahoo bot are on the forum very frequently, but there are ways to control how fast and how often they grab new threads.
I should reiterate after all the e-mails I have received that bandwidth is not an issue. It's server utilization (CPU/RAM, etc)
smelly$cat: (2009-05-20 15:38:21)
I noticed the "recent posts" link is missing. Was that a planned change as well?
Thanks!
Kennedy: (2009-05-20 15:39:57)
View new topics after last visit button i miss.. :-(
ak86na300z: (2009-05-20 16:18:11)
are you bringing the search feature back there are a few posts i am trying to find and cant beacuse there is no search
adamvann3: (2009-05-20 16:30:20)
How the hell were there 50 users on here at 4am last night?
Guess mid day is the best time right now. Glad we are back up and at least limping along. Thanks Jason.
Kennedy: (2009-05-20 16:54:26)
http://www.google.com/search?hl=fi&r…mance.com+&lr=
Use Google with site only tag
For English use this link. Type in the word or text string you wish to search
after "site:z31performance.com' in the Search box.
http://www.google.com/search?q=site%…=Google+Search
Edited by 1artworks, May 20, 2009
FlawleZ: (2009-05-20 22:10:18)
It's very unlikely the server overload was caused from actual registered users or even human interaction at all. Bots seem to get worse as time goes on, plaguing internet message boards and server load and bandwidth relentlessly. A few questions, how long do you have the user account login period set for? Many forums run into too many users logged on because the server isn't set to update immediately once a user has logged out, sometimes leaving accounts logged on for up to 24 hours at a time. Also, your log files in the cpanel should give some evidence as to why the MySQL crashed as well as your awstats clearly pointing as to which bots and spiders are sucking up the most bandwidth. I assume you also have your robots.txt file set to exclude malicious/unwanted bots?
FlawleZ: (2009-05-20 22:14:10)
I'm sure it was. Even if it wasn't, it still uses the "search feature" and wouldn't function anyway until Jason enables searching again.
zachary.ard: (2009-05-21 03:20:45)
Any chance of maybe sponsorship-level type deals? I'd be interested in helping support this site.
Guest: (2009-05-21 04:01:24)
viewtopic.php?f=5&t=1028
Kennedy: (2009-05-21 06:31:31)
http://z31performance.com/ - Make a donation via Paypal.
zachary.ard: (2009-05-21 18:02:01)
Thanks guys. I do plan on donating via Paypal, but was just curious about this -
Jason84NA2T: (2009-05-21 23:17:02)
PLEASE NOTE: There may be momentary lapses in forum accessibility as I make some modifications.
Say goodbye to the spammers!
Search is now enabled. You can only search once every 20 seconds. I am monitoring the load statistics and will compare to yesterday (without search).
Hello! Contact me if you have/are a business that is interested in sponsoring the forum. Sponsorship is yearly and is VERY reasonable. For now, forum sponsorship includes a banner on the main page and a special rank. It may eventually (in the future) include a banner ad displayed at the bottom of the page, but that is still undecided.
1artworkz: (2009-05-22 01:10:44)
Yeaaaaaaaaa Ahaaaaaaaaaa! "search_id=newposts" is back.
Chris86NA2T: (2009-05-22 04:28:20)
z31performance is set as my home page. I often leave my web browser open for long periods of time while logged into the forum. Is this something that bogs things down? I have no idea how this shit works. If it would help to stop doing that please let me know.
adamvann3: (2009-05-22 04:44:19)
Your hard core is z31p is your home page. Nice work
sporklift: (2009-05-24 01:57:36)
It's my home page as well. There is a shitload of amazingly great information on this forum.
PulseCode: (2009-05-25 02:33:12)
THANK YOU Jason. This site has become invaluable to me now that I have begun doing all my own work on my Z. If it ever comes down to it- I would be willing to pay fees to use it. Thanks again man for the hard work. Jeff Z. in SC
floridaZ: (2009-05-27 01:29:29)
That is me right there. I didn't realize that it made a difference if you did them in sequence or all at once.
FlawleZ: (2009-05-27 16:12:39)
[quote]floridaZ wrote:
4. Doing a lot of things at once, like opening new threads/searches in different windows before others have completed. please avoid doing this.That is me right there. I didn't realize that it made a difference if you did them in sequence or all at once.
Well, anytime you click on a link, refresh the page, etc. your sending a request to the server. Each request must be processed and executed and then transmitted to the host machine. Some of these requests require more from the cpu such as the search feature. People often don't realize that the "view new posts", "view your posts", etc. use the search engine on the server. Even clicking the members link at the top uses the engine to pull the database of users.
Jason84NA2T: (2009-06-02 15:58:45)
Nope, that's not really an issue. I guess the simplest way to explain is that load is incurred whenever you click and your browser loads something. This is especially true of anything that uses the search feature, as it combs through a section of the DB indexes looking for specific information.
FlawleZ's explaination above is very much correct. Unfortunately part of this issue is that SQL queries appear to run in realtime, which means that multiple queries get stacked into a queue. When someone on the server (me, or other people like with the SQL errors less than an hour ago) run a script that generates a big string of queries from the DB, it clogs up the CPU for a period of time and nothing else can be done on the server until it works its way through the queue. I know there are some number of ways out there to allow prioritization of these individual queries (so one script cannot hog the entire cpu, the processing is shared), but I don't know the rest of the details, don't own the box, and I'm also not about to tell the hosting company what to do when they have been so (relatively) accommodating already.
phxZ31: (2009-06-21 06:13:25)
[quote]Jason84NA2T wrote:
z31performance is set as my home page. I often leave my web browser open for long periods of time while logged into the forum. Is this something that bogs things down? I have no idea how this shit works. If it would help to stop doing that please let me know.
Nope, that's not really an issue. I guess the simplest way to explain is that load is incurred whenever you click and your browser loads something. This is especially true of anything that uses the search feature, as it combs through a section of the DB indexes looking for specific information.
FlawleZ's explaination above is very much correct. Unfortunately part of this issue is that SQL queries appear to run in realtime, which means that multiple queries get stacked into a queue. When someone on the server (me, or other people like with the SQL errors less than an hour ago) run a script that generates a big string of queries from the DB, it clogs up the CPU for a period of time and nothing else can be done on the server until it works its way through the queue. I know there are some number of ways out there to allow prioritization of these individual queries (so one script cannot hog the entire cpu, the processing is shared), but I don't know the rest of the details, don't own the box, and I'm also not about to tell the hosting company what to do when they have been so (relatively) accommodating already.
I find this whole thing to be quite odd…
We have a 10GB SQL database for a massive product catalog, and the main server doing all the work is a ten year old, dual CPU Pentium 3 933mhz machines with 2GB of single channel, single datarate SDRAM. We have over 300 employees' client PCs constantly making queries, searching the DB, modifying entries, etc, all day long. The business depends on the database running. The same server is also an active directory domain controller hosting roaming windows profiles, functions as a DNS server, and has routing and remote access enabled to control IP traffic to a private internal network.
It seems odd to me that I can do so much with a 10 year old server and it doesn't even give hints of complaining about it, yet this forum with a relatively tiny SQL database size and probably relatively moderate traffic should be so crippling. The problem must be with bots or other software on the forums. Do you have any programs running that constantly monitor your SEO rankings, back links, google optimizations, and stuff like that? I've heard of those types of tools causing problems like this..
Are they sure they can definitively link the resource hogging to the thread/s running this board?
Rat1314: (2009-07-02 12:09:00)
my computer picked up 2 Trojan viruses when I was looking in the Z31 turbo modifications FAQ and The truth about Ebay turbochargers. I only had one page up at a time and I had my virus software notify me of a virus each time.