Now that we have a better understanding of what happened we would like to give everyone an update.
One of our old generators that have worked for years and was recently load tested had a mechanical failure and caught fire resulting in power being cut to our core routers and fire suppression system controlling the fire. Unfortunately, the fire department opted to cut power to the rest of the building as a precaution even though the power systems were independent. We are currently waiting for an emergency inspector to arrive to give the all-clear so we can bring most of the servers in Ogden back online. Some servers will have an extended outage as they may require rebuilds due to some water damage. Those builds have a high probability that data is intact.
We would like to thank you for your patience and know that we are doing everything we can to get everyone back online.
Update : Monday, April 5, 2021
Some servers should come online today, most servers will hopefully be up in the next day or two, other servers with more complex builds or rare components may take several days.
Current updates project that close to 90-95% of all Servers hardware experienced zero damage. Electricians are working now to restore power to the Ogden Utah data center and are estimating power should return at approximately 3:00 pm MDT tomorrow.
Update : Monday, April 6, 2021
We can confirm that all servers are being manually checked that they're at login screen. Some Servers should be online by now.
If your server isn't online; it most likely still needs power ran to it.
we're hoping all power will be ran by tonight.
Update : Monday, April 7, 2021
- Network Issues - While many sections of the datacenter have had full power and network restored, others are still being worked on.
- Extended Outages - A smaller percentage of services will have extended outages as they may have been affected by water in the area from fire suppression systems. Some services are in the process of being rebuilt to preserve data integrity. If you have not already opened a ticket, please do so and we can help track the status of restoring said services.
- More info to come shortly
Tuesday, April 6, 2021