Contrary to some rumors that may be out there I'm still alive, just popping my head up to see what's going on. As many know, ActiveRain had some, uh, pretty serious "technical issues" for a while which I've been helping to battle. Luckily things have stabilized quite a bit in the last week. I'll try to give you the post mortem on this one, warning tech talk ahead.
Traffic to ActiveRain's site has been steadily increasing month over month, I think we're averaging somewhere north of 80k visitors a day and 15M page views a month. Our servers were able to handle the human traffic fine, but then we started getting absolutely pounded by various bots and spiders trying to crawl the hundreds of thousands, if not millions, of pages on the site. For some reason they liked to all choose the exact same time to hit us too. Some of the RSS feed readers were particularly hideous, as unlike the major search engines that would throttle their requests and only try to say load a page a send, they would trying hitting every RSS feed on the site at the same time. This was causing traffic spikes that were large enough to start taking down our web servers periodically or at least make them so slow for periods of time they were nearly unusable.
About two months ago we ordered upgraded hardware doubling our number of web servers. Our managed hosting company built out this environment, but in the process upgraded almost all the software and operating system versions to the latest ones. As most computer users know, while upgrading software theoretically should help reduce problems it often has the opposite effect. The new environment went through our testing no problem, and seemed to function fine when we put it up. However, when we switched over, every couple hours our servers would just blow up (not litererally), almost at random. We spent many, many hours going line by line through log files, and doing everything we could to diagnose the issue. Not being able to find any reason for the random self destruction we began rolling software versions back methodically trying to find if one of those caused it.
None of the usual suspects seemed to help, finally we ended up rolling the version of linux we're running on back a minor version and the problem disappeared. Almost two weeks after the roll out of our new environent we had a fix. That was about a week ago, and things have greatly stabilized since then. We've got one more major hardware upgrade coming in the next day or two, as we're going to much more powerful database servers (4 core/4 GB of RAM to 16 core/32 GB ram) This should further speed up the site and give us a lot of buffer for future growth, and yes we are making sure they are all running the same software versions this time.
So that's what I've been up to and while all of my hair finally fell out :) No I'm not Jeff Turner's long lost twin...
I understand some of that where as I would not have a year ago. You have been missed but more importantly, we all appreciate you AR guys!
bald is beautiful, don't show us any pictures if you pull more hair out! ooch- please take note that I have not complained, I just use the down time to go sell homes----be good cw
That is too funny. Glad to see you are ok. Good luck -Remember. Energy drinks for the late nights.
Wow! That is a lot of work! Thank you for keeping us going:)
Obsession is a core trait of the Geek personality. A Geek absolutely will not and can not stop until the frolicking thing is fixed. :-)
Good work squashing the bug.
Ok, who featured this post, I was getting used to hiding :)
It's all good, and I would have to agree with Chreyl, the bald thing ain't bad either!
You could almost pass for Jeff Turner in the photo! lol~ Nice to see you pop your head out Matt, glad you finally got the chance. I commend you on your hardwork and team effort!! Thanks man! :-)
I can imagine it's been a bear.
I'm having two new servers installed and the migration is ongoing as I write this. My old server was almost 5 years old, wend periodically and there was no longer any support. Hopefully, by morning, it will be completed and I can enter new nameserver addresses for my domains at the registrar.
Right now, I have no mail. Ugh.
Matt,
LOL! You are definitely one of those guys that prefers a cave setting.
MH,
Great to see you back. It`s only growing pains. You`re treading in waters not seen before,which is really amazing from the days when we were hanging out celebrating 5000 strong!
I`m positive you and your staff have a handle on everything!
I miss the old days,....
Somehow am positive you do as well...")
Thanks for the update Matt. The issue of the RSS feeds all deciding to update en masse is really intriguing. Machines are wonderful when they work, but when they don't, well you've been living with that nightmare! Appreciate all your hardwork.
Oh wow. All we need is Bob to shave his head. :P
Nicole, you mean you're not next???
Matt - It cannot be easy maintaining a network this large. But we are glad you guys are up for it as we love AR soooo much.
Ah, so the servers blew up... that's as techical as I care to get. Glad to see everything back up and running smoothly.... now, about that spellcheck.. :)
Hey Matt,
Good to see you again...you guys are doing a bang up job! Hey...will my spell check ever work again?
I appreciate the update. Never had a doubt that you guys were all over it and it was being worked on. I'm w/ Neal though, would love Spell check back :-)
On the spell check issue...
Matt, hahaha! No thank you.
Hey! Who let you out???....Get back in the basement with the servers!
Actually, Matt, I don't think our members fully appreciate all of the hard work you perform so faithfully behind the scenes. Us Community Builder types get all the recognition, but it's people like you who really make the Network such an amazing community. So, thanks!
Matt~ I thought you really didn't exist, and were just a figment of someone's imagination... :) Nice to see you are a real person! Don't worry your loyal fans in the rain are "hanging in there"
Matt, thanks for the update. I have never doubted you were in the basement working hard. I went to Best Buy the other day to take a microwave back that was less than a year old.
All their computers were down.
Major upgrade...
That's what happens.
We have to desenigrate before we rebuild.
Thanks for posting the photo- I've always wondered what AR Headquarters looked like ;) !
Yikes... Matt Did you really lose all your hair? Wow!
I'll be glad when you get back to writing your economic articles:) You look totally different in those pix!! Nice to hear AR will stabilize. Last couple of weeks have been frustrating.
Matt, So I started reading and was so proud of myself for understanding after your "tech disclaimer"... yah, that didn't last through your whole post - but thanks for the explanation - I'm sure some people "get it"..... I just like knowing it's being taken care of...
Nice bald look.
Darn... saw the pic in my blog line up and thought! HOT DAMN! He's going to comment on the guys who MIGHT get fried at Bear over the MBS - oh well - server talks not my thing - but thanks for worrying over it and working on it anyway!
I told Bob to kick your chair and make you do the Meme I signed you up for... I guess you HAD an excuse to not do it...
And I'm not shaving my head.
Thanks, MAtt. We appreciate all that you do, and there are certainly challenges with a network as large as ours. Appreciate the update. What does Lydia think about your hair...or lack of?
JEff
Matt, glad you are still around! I was getting worried for a while.
Matt, thanks for all you do. This is a FANTASTIC network!
Thanks for keeping us updated, and continuiing to make AR even better.
Matt who? Do I know you?
Matt... can you put that in English please.... lol And what was the answer to the spell check question? But yes, I have seen a huge improvement.. thanks for all that you all do.
Thank you much for your explanation! Nice to know what's going on.
Thanks for the update Matt. It must be gard to keep up with this growth.
Hey Matt,
A little glitch here and there is to be expected when you are growing as fast as you are... I ran into a Proxy Server error earlier which went away about 10 minutes later when I checked back...
I'll probably see you at Real Estate Connect in SF...
All the Best,
Glenn Sanford
Eleanor: I'll try to write some more commentary on the credit market stuff soon, unfortunatly it's almost to scary/depressing to write about right now.
Glenn: New DB server just came online, it's possible we may have had a few seconds of "glitchiness" that caused the 502 errors during the transaction. But it's done now and the site is MUCH faster, and load on the database server is about 1/8th of what it was before.
You guys are great and we appreciate all that you do. You do a great job. All the best.
I almost understood that. What I do appreciate is that you're working to make Active Rain better. Thank you.
Thanks for the update...yes, we ranted and complained but in reality we are very grateful for all that you do...that's why were are all still here!
Quit stressin and have a cocktail Matt :) You guys do a great job, thanks so much!
Its good to hear that the issues should have been worked out. I've notice it was moving very slow in the past and seems to be doing well now.
Renee: Yeah, I'm going to be having several this weekend down in vegas to try and destress.
Who's the bald guy? Oh..you? Does Lydia like it? Why don't you just change your profile pic? The picture is a little shaded...did you shave off your moustache and beard too? If not...will you just be all hairless?
Hey...had to make up for all that time you weren't writing...thanks for everything you do Matt.
Now...can you fix my profile page so I can edit :) :) :) :)
Glad things are running smoother in here now!!! It was indeed a crazy month last month with all of the down times. Thanks for the update!!! We all know you guys were working diligently in the background, trying to take care of the overload!!!!