Clean-Solutions.net Status Blog

Server Upgrades Progressing

baptisthost — Sun, 10 Aug 2008 04:08:57 +0000

Hello,

The server upgrade is still progressing. It has taken longer than expected due to the amount of data being moved.

Please be aware that any updates made to your website this weekend will be lost. Once the upgrade is complete you can resume updating your website as before.

I will update you later with status.

Thanks
Andrew
BaptistHost.net

Server Upgrades Tonight

baptisthost — Sat, 09 Aug 2008 02:39:35 +0000

Hello,

First, thanks for being a customer of BaptistHost.net!

As mentioned earlier in the week, we are upgrading all servers this month. This server your website is hosted on is slated to be upgraded starting at 10:00 PM CST. During these upgrades your website and email will be unavailable. We estimate that the process should take between 12-14 hours.

As always, we appreciate your understanding as we complete these exciting upgrades!

Thanks
Andrew Teesdale
Owner
BaptistHost.net

Datacenter Outage Update #11

baptisthost — Wed, 04 Jun 2008 08:18:56 +0000

All services are now up again after app. 12 hours of downtime today. Our datacenter replaced the faulty generator with a backup generator.

I want to thank you for your patience during this trying time.
Thanks

Andrew – Clean-Solutions.net

Datacenter Outage #10

baptisthost — Tue, 03 Jun 2008 15:28:35 +0000

This morning at approximately 2:45 a.m. CST, the temporary generator supplying power to the servers and environmental control systems located in Phase 1 of our H1 facility shut down. This was caused by some faulty current sensors in the output breaker. The sensors detected an out of balance current condition that did not exist.

Technicians from the generator company were onsite within 15 minutes. After working on the breaker for an hour, they believed the issue was remedied, and the generator was restarted. As the servers and environmental control systems were brought back online, the breaker again caused the generator to trip offline.

At this time we have a replacement breaker in route to the site and will get power restored as soon as physically possible.

Datacenter Outage Update #9

baptisthost — Mon, 02 Jun 2008 21:37:20 +0000

The server is currently online at this time. All services are working as intended including SiteStudio. There may be some further outages due to the replacement of electrical gear at the datacenter.

I appreciate your patience and understanding as we work to completely correct this issue.

Thanks
Andrew
Clean-Solutions.net

Datacenter Outage Update #8

baptisthost — Mon, 02 Jun 2008 16:53:01 +0000

Here is the latest from our datacenter:

Onsite technicians are currently working to restore service to the remaining 10% of Phase 2 upstairs servers. The Phase 1 downstairs servers are expected to start coming online late this evening.

Datacenter Outage #7

baptisthost — Mon, 02 Jun 2008 07:06:53 +0000

Dear Customer,

The affected Clean-Solutions.net server is located in Phase 1. It appears that we will not have power restored until the evening of June 2. Thanks for your patience…

Thanks,

Andrew Teesdale

Clean-Solutions.net – Owner

The following is the last communication from our datacenter The Planet.

As previously committed, I would like to provide an update on where we stand following yesterday’s explosion in our H1 data center. First, I would like to extend my sincere thanks for your patience during the past 28 hours. We are acutely aware that uptime is critical to your business, and you have my personal commitment that The Planet team will continue to work around the clock to restore your service.

As you have read, we have begun receiving some of the equipment required to start repairs. While no customer servers have been damaged or lost, we have new information that damage to our H1 data center is worse than initially expected. Three walls of the electrical equipment room on the first floor blew several feet from their original position, and the underground cabling that powers the first floor of H1 was destroyed.

There is some good news, however. We have found a way to get power to Phase 2 (upstairs, second floor) of the data center and to restore network connectivity. We will be powering up the air conditioning system and other necessary equipment within the next few hours. Once these systems are tested, we will begin bringing the 6,000 servers online. It will take four to five hours to get them all running.

We have brought in additional support from Dallas to have more hands and eyes on site to help with any servers that may experience problems. The call center has also brought in double staff to handle the increase in tickets we’re expecting. Hopefully by sunrise tomorrow Phase 2 will be well on its way to full production.

Let me next address Phase 1 (first floor) of the data center and the affected 3,000 servers. The news is not as good, and we were not as lucky. The damage there was far more extensive, and we have a bigger challenge that will require a two-step process. For the first step, we have designed a temporary method that we believe will bring power back to those servers sometime tomorrow evening, but the solution will be temporary. We will use a generator to supply power through next weekend when the necessary gear will be delivered to permanently restore normal utility power and our battery backup system. During the upcoming week, we will be working with those customers to resolve issues.

We know this may not be a satisfactory solution for you and your business but at this time, it is the best we can do.

I recognize that this is not all good news. I can only assure you we will continue to utilize every means possible to fully restore service.

Datacenter Outage #6

baptisthost — Mon, 02 Jun 2008 00:13:32 +0000

We continue to pursue our plans to provide initial power to our H1 data center this evening. It will take several hours to assure power can be safely restored to the facility. Based on how the initial work goes, we will have more information to provide you in the upcoming hours. .

DataCenter Outage #5

baptisthost — Sun, 01 Jun 2008 03:51:35 +0000

This evening at 4:55pm CDT in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room. Thankfully, no one was injured. In addition, no customer servers were damaged or lost.

We have just been allowed into the building to physically inspect the damage. Early indications are that the short was in a high-volume wire conduit. We were not allowed to activate our backup generator plan based on instructions from the fire department.

This is a significant outage, impacting approximately 9,000 servers and 7,500 customers. All members of our support team are in, and all vendors who supply us with data center equipment are on site. Our initial assessment, although early, points to being able to have some service restored by mid-afternoon on Sunday. Rest assured we are working around the clock.

Datacenter Outage #4

baptisthost — Sun, 01 Jun 2008 01:22:23 +0000

In our latest assessment, we have determined that networking gear has not been damaged, but we are without power so assessments continue. All disaster recovery systems are in motion, and we have teams already working in the data center.