ForumsNewsRackspace failed us again

Rackspace failed us again

Author	Message
Jake Toodledo Founder	Posted: Jul 07, 2009 Score: 0 Reference Jake (Founder) Posted: Jul 07, 2009 Score: 0 Reference This is getting really really frustrating. Rackspace (who runs our datacenter) had another power outage today (their 4th). This affected us for about 45 minutes. We are currently back up and running without any data loss. However, we are 1 server short right now, so everything is going to run slowly until we get that other server back online. I know that 4 outages in a few weeks makes Toodledo look totally amateurish and ridiculous and I assure you that we are working like crazy to migrate everything to a more reliable datacenter. Migrating everything is a complex coordination issue that takes some time to test and complete, but it can't happen fast enough.
karin.rosner	Posted: Jul 07, 2009 Score: 1 Reference karin.rosner Posted: Jul 07, 2009 Score: 1 Reference I still love you, Toodledoo. I hope we get this sorted soon. I saw the outage and went to lunch.
MFA	Posted: Jul 07, 2009 Score: 0 Reference MFA Posted: Jul 07, 2009 Score: 0 Reference biting my fingernails there for a sec
jalex508	Posted: Jul 07, 2009 Score: 0 Reference jalex508 Posted: Jul 07, 2009 Score: 0 Reference My company uses Rackspace and we have no issues. It's that Dallas Datacenter.
Anders	Posted: Jul 07, 2009 Score: 1 Reference Anders Posted: Jul 07, 2009 Score: 1 Reference Yes, it was at least nicely timed for lunchtime on the east coast. I wouldn't be surprised if Rackspace winds up with a lot of free rackspace before long :)
gadgetude	Posted: Jul 07, 2009 Score: 0 Reference gadgetude Posted: Jul 07, 2009 Score: 0 Reference Do you have a proposed time line for the data center move?
jbeckham	Posted: Jul 07, 2009 Score: 0 Reference jbeckham Posted: Jul 07, 2009 Score: 0 Reference Thanks for keeping us posted on what's going on. Me not having my todo list isn't quite as bad as you having to deal with all this crap... good luck.
Jake Toodledo Founder	Posted: Jul 07, 2009 Score: 0 Reference Jake (Founder) Posted: Jul 07, 2009 Score: 0 Reference Our proposed timeline for the datacenter move is ASAP.
Andrew	Posted: Jul 07, 2009 Score: 0 Reference Andrew Posted: Jul 07, 2009 Score: 0 Reference Dare I ask where you are moving to?
Linden	Posted: Jul 07, 2009 Score: 0 Reference Linden Posted: Jul 07, 2009 Score: 0 Reference Yes, I too lucked out and only experienced the outage over lunch. Sorry to hear that this power issue is STILL ongoing. Best of luck in shopping and migrating hosts. I'm not especially conversant in hosting issues, so forgive me if this is an impossible/expensive/whatever idea, but: Might it be possible to have a server with one host and another with a different host? Rackspace, from what I've been hearing, is generally one of the top hosting solutions available. So I'm gathering that others might have similar problems. If it's possible/affordable, it could be nice to have servers with two companies so that an outage with one still leaves the site operating with the other -- regardless of who it is. In any case, I'm definitely sticking around while you sort it out. The system has been priceless for me identifying and acting on my goals!
bsglaser	Posted: Jul 07, 2009 Score: 0 Reference bsglaser Posted: Jul 07, 2009 Score: 0 Reference I'm suddenly missing my Gmail Gadget...is this related to the outage?
Jake Toodledo Founder	Posted: Jul 07, 2009 Score: 0 Reference Jake (Founder) Posted: Jul 07, 2009 Score: 0 Reference The other server is back online, so we are 100% back up now. To answer Linden's question, while it is possible to have servers at different datacenters, the challenged would be keeping the data 100% in sync between the two datacenters. This would require a high-bandwidth private connection between the two, which might not be possible or might be cost prohibitive. The other issue is that switching traffic from the failed datacenter to the working datacenter would take time due to the way DNS works. So it would really only be protection from a major catastrophe, like a fire that took our server offline for multiple days. But we are exploring this option to see if it is affordable.
Proximo	Posted: Jul 07, 2009 Score: 0 Reference Proximo Posted: Jul 07, 2009 Score: 0 Reference I understand this is frustrating, but I believe many of us can appreciate the complexity of migrating a to a new data center, even if we never experienced doing so at such a large scale. I just switched hosting services for my personal business and website on a much smaller scale. I did not plan as well and lost some data. So please, take your time and do it right. We trust the backup system is working because it's proven itself recently. I am not sure where Toodledo is located, but some suggest you are here in Dallas. If that's the case, I would love to visit one day. :-)
frossie	Posted: Jul 07, 2009 Score: 0 Reference frossie Posted: Jul 07, 2009 Score: 0 Reference Toodledo must have been unlucky - this time some of their Dallas cluster managed to stay up on UPS. Aren't you sure you don't want to stay at Dallas and keep getting the service credits? :-) Seriously, do you really want to spend the effort on moving datacenters? Rackspace are bound to fix this, especially since they are now becoming a bit of a joke over this center.
frossie	Posted: Jul 07, 2009 Score: 0 Reference frossie Posted: Jul 07, 2009 Score: 0 Reference And if it makes you feel any better, check out these poor guys: http://smugmug.wordpress.com/2009/07/07/7709-site-offline/ (they are a great service too, as you can tell from the honesty of their announcement)
Linden	Posted: Jul 07, 2009 Score: 0 Reference Linden Posted: Jul 07, 2009 Score: 0 Reference Posted by Toodledo: To answer Linden's question, while it is possible to have servers at different datacenters, the challenged would be keeping the data 100% in sync between the two datacenters. This would require a high-bandwidth private connection between the two, which might not be possible or might be cost prohibitive. The other issue is that switching traffic from the failed datacenter to the working datacenter would take time due to the way DNS works. So it would really only be protection from a major catastrophe, like a fire that took our server offline for multiple days. But we are exploring this option to see if it is affordable. Thanks for the answer. Sounds to me like the cons/difficulties outweigh the potential pros -- especially since the pros appear to be negligible.
Anders	Posted: Jul 07, 2009 Score: 0 Reference Anders Posted: Jul 07, 2009 Score: 0 Reference @Frossie Wow, that's terrible, and unlike Toodledo, sadly self-induced. They are one of the biggest photo sharing sites out there, aren't they?
frossie	Posted: Jul 07, 2009 Score: 0 Reference frossie Posted: Jul 07, 2009 Score: 0 Reference Anders - yes, they are a photo hosting site and actually they are really great. Not only is the interface really well designed, but they have lots of different levels of privacy control, which is really nice for those of us who are not quite as exhibitionist as the youngsters these days :-) On the overall issue, I think customers are willing to tolerate some level of service interruption if it is infrequent and quickly and honestly dealt with. The thing people really can't tolerate is data loss. I don't think a business could recover from that.
Anders	Posted: Jul 07, 2009 Score: 0 Reference Anders Posted: Jul 07, 2009 Score: 0 Reference I agree. If you can't live with occasional interuptions, then you basically can't use the internet, but people do like to be kept in the loop. Both websites have handled that part quite well.
kaipaniolo	Posted: Jul 07, 2009 Score: 0 Reference kaipaniolo Posted: Jul 07, 2009 Score: 0 Reference Our radio station corporate email is out of the same Rackspace server farm. When Toodledo is down, our email usually is too. Though, we are in Hawaii, so when this happens, we can just take the Cell phone to the beach and wait it out. :-)