Explosion at THE PLANET

[themoose]

The Planet Datacentre Exploded

Well.. kinda :lol:

http://blog.servercomplete.com/tech/the-planet-goes-boom-hostgator-affected/

This evening at 4:55pm CDT in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room. Thankfully, no one was injured. In addition, no customer servers were damaged or lost.

We have just been allowed into the building to physically inspect the damage. Early indications are that the short was in a high-volume wire conduit. We were not allowed to activate our backup generator plan based on instructions from the fire department.

This is a significant outage, impacting approximately 9,000 servers and 7,500 customers. All members of our support team are in, and all vendors who supply us with data center equipment are on site. Our initial assessment, although early, points to being able to have some service restored by mid-afternoon on Sunday. Rest assured we are working around the clock.

 

[J J]

From the thread at WHT, I've read that they have turned the backup generators on and a lot of the servers are up. Though look at the 'The Planet' thread on WHT, it's already had 71,000 + hits in 3 days!

 

[Peo]

statcounter.com was one of the victims.

Due to an explosion in The Planet Datacenter at the weekend, service on this partition was interrupted. Depsite our best efforts 24 to 30 hours of stats spread across Sunday GMT and some of Monday morning have been lost. We sincerely apologise for this. More information is available in the Service Status section of the forum

FWS is also hosted at ThePlanet, but we had no lost data or downtime.

 

[Skylar]

Time to pack up and move? Heh... We got lucky then, I'd have been bored out of my mind of FWS was down!

What I don't really get is if there was an "explosion" that was clearly bad enough to get the fire dept. involved and them not allowed to have their backup generators on, then it must have done more than just "knocked over 3 walls". From the issue at statscounter, it sounds like a FEW servers had been damaged, but not enough of them for ThePlanet to want to make an announcement about it, which is probobly a better idea. If they would have said SOME data was lost, then there would have been a lot more panic.

 

[SC-Daniel]

The Planet currently owns and operates 5 or 6 datacenters in the Houston area, so no doubt that FWS is not in DC1

 

[SC-Jon]

Time to pack up and move? Heh... We got lucky then, I'd have been bored out of my mind of FWS was down!

What I don't really get is if there was an "explosion" that was clearly bad enough to get the fire dept. involved and them not allowed to have their backup generators on, then it must have done more than just "knocked over 3 walls". From the issue at statscounter, it sounds like a FEW servers had been damaged, but not enough of them for ThePlanet to want to make an announcement about it, which is probobly a better idea. If they would have said SOME data was lost, then there would have been a lot more panic.

The most that would happen to servers would be from the power surge, quick charge, end, overload, explode.

That'd probably be the most damage done, if any at all to the actual servers, I don't think they'd stick the electrical room near the server rooms.

 

[HostV]

Just found out about this myself, hope things get resolved soon

 

[James]

One of our servers were effected in phase 2.

They did a great job to get it back running.

 

[Jan]

Yes, top marks to them for getting so many back online in a relatively short time

 

[XFH-Jay]

Look at the positive side. No servers damaged, no people hurt. That's good.

 

[iCorp]

I don't get the reason that everyone is mad on The planet forums. I know that this is bad but the planet have an excellent uptime stats excellent support and realiblity number one for dedicated servers. Mistakes happen and they doing their best to fix it. Thanks god no server lose or damage happened

 

[Dynash]

Yes, top marks to them for getting so many back online in a relatively short time

I agree, they said they would and they followed through with it.
Personally, they need to be praised for it, not bashed.

 

[XFH-Jay]

I'm sure most people are happy about it on the inside, just following the crowd. They sure did a good job.

 

[Skylar]

I'm sure most people are happy about it on the inside, just following the crowd. They sure did a good job.

Well everybody that is in that DC is bound to be mad because of the downtime. They should be happy nothing was lost.

 

[DavidsAwesome]

Well everybody that is in that DC is bound to be mad because of the downtime. They should be happy nothing was lost.

Exactly. Downtime is horrible, but losing your hard drive would be worse. Be thankful they apparently had some form of protection, an explosion like that probably could have taken a few servers with it if they weren't careful about the placement of the electrical gear, and stuff like that.

 

[bryonhost1]

Hi!
I am not offically a engineer (sckooling) but I can tell you from first-hand experience..transformers do indeed expire and there's no way you can predict it. If you are lucky..it just stops working. If you not...well..ummm..yeah.

As a kid..I distinctly remember when the transformer on our pole suddenly died..and exploded. I also remember in Atlanta one of the big boys (power substation) exploding..causing a fire and even traffic havoc (hot oil melting car tires).

I think some times these things happen. Could it have been better planned?
Of course. I've seen the design of many datacenters..and they are not redundant at all. One point of failure..one power entry point..one room.

Sadly..much of the fiber comes in this one point too...and...well...there you go. I think you have the telco industry to spead some blame around on that..they really do design things like that.

Bryon

 

[Dynash]

As a kid..I distinctly remember when the transformer on our pole suddenly died..and exploded. I also remember in Atlanta one of the big boys (power substation) exploding..causing a fire and even traffic havoc (hot oil melting car tires).

One just down the road from me exploded too, about 6ish months back.
but ThePlanet are suitable, they where redundant, it's just they got told they can't turn the generators on. If they could, this wouldn't have happened.

 

[bryonhost1]

Hi!
Suitable? Ahh..well. I'm afraid my opinion of them has not changed..I would not use them. I think they are just too darn big..and the little guy with one server who has a problem really gets the very short end of the stick.

If there were not so many horror stories about..three day hard drive replacement, anyone? Pass. I love hostmysite..and after I've had some time with netdepot..we will see what we will see.

>>>And a big shoutout woot! to Uncle Bill (Microsoft) for making my first two Win Server 2008 servers affordable!<<<<

Bryon

 

[Skylar]

Stuff like this is bound to happen. People just need to realize that and be cool about it.

 

[Dynash]

Bryon, you can't blame ThePlanet directly for following the fire departments orders for safety reasons. They worked hard, and got the network back online within 48 hours, could you do that in those circumstances?