Cloudflare Dashboard and APIs Down After Data Center Power … – Slashdot

Follow Slashdot blog updates by subscribing to our blog RSS feed




The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
I still have zero trust in CloudFlare.
I trust them completely. To slap their silly little message on my screen, ensuring that I’m running JavaScript and have not installed a popup blocker.
Thank goodness I have a plug-in that auto checks the “I am a human” test.
Nobody sane trusts any modern tech company. They all are far too optimized for profit to deserve any level of trust.
Fukushima did not expect this type of accident, ever, because of the boundless arrogance and stupidity prevalent in the nuclear industry. They could have placed the emergency generators easily on a nearby hill. They could have made sure the mobile generators they had on site could be connected to the cooling pumps. They could have bought the hydrogen release valves offered with that reactor type. They could have done other easy and cheap things. If they just had not completely botched the risk analysis the
Their backup generators were probably not being regularly tested, or whatever caused the outage also blew up their backup, or the power failure wasn’t total (phase loss) and didn’t trigger backup power to start. This isn’t a good excuse, just what I suspect has happened from seeing a local power failure in my area where the backup generator failed to start.
Turns out Cloudflare just rented a shared hosting server from Dreamhost!

Dreamhost had a power outage that took down their Portland datacenter for several hours this morning too (so much for their backup power and 100% uptime guarantee). Are they in the same facility?

Dreamhost had a power outage that took down their Portland datacenter for several hours this morning too (so much for their backup power and 100% uptime guarantee). Are they in the same facility?
Yes, it is Flexential datacenter PDX02 in Portland.
Dreamhost lives there too.
At last update, the datacenter lost utility, switched to generators, and the generators failed (unspecified how) in a way that took out battery backup to the building.
In the case of Cloudflare, they began a sitewide-live-failover when utility was lost, but when the generators took out the UPS systems, the data center had a complete unclean shutdown much sooner than expected.
I can only imagine how that would f-up a full site migration in progress!
Personally I’m a bit surprised Cloudflare is renting datacenter space, instead of having their own.
battery backup just needs to last for the time it takes generators to start and to stabilize the power not run the full site for hours.
From the article:
> …the company revealed that the ongoing issues are due to power outages at multiple data centers.
Odds of that for a 4-9s datacenter is 0.001% multiplied by N datacenter.
Company says:
>Cloudflare is assessing a loss of power impacting data centres while simultaneously failing over services.
This a multiple-hour multiple-service outage.
Odds that they’re “simultaenously” failing over services is a bit lower than the odds of multiple datacenters losing power.
>This is the second large outage that has hit Cloudflare since the start of the week
The odds of such “coincidences” are worse than the odds of winning the grand prize in the Powerball lottery.
What’s more likely is someone pushed a network configuration change outside a maintenance window, with no peer review, and locked themselves out of fixing it, not having used safe-mode or auto-restore procedures.
It’s always fantastic when your redundant capacity knocks you offline when there isn’t a need for the redundancy to be active to begin with.
Reminds me of spanning-tree failures when someone plugs a cable into the wrong ports and causes an unintended loop.
Interesting observation.
Going to become even more interesting if somehow, someway a corporate liability policy actually covers the harm done from a 1-in-Powerball power outage mass event. I’d love to know how much they paid the insurance adjuster.

Was not the cloud supposed to provide cheapo five 9s reliability?

Was not the cloud supposed to provide cheapo five 9s reliability?
Good, Cheap, or Fast. Pick any two.
Alright, alright, you got me. Pick any one.
There may be more comments in this discussion. Without JavaScript enabled, you might want to turn on Classic Discussion System in your preferences instead.
The Final Beatles Song, ‘Now and Then,’ Featuring All Four Members and AI, Released
Arm Acquires Minority Stake in Raspberry Pi
Time to take stock. Go home with some office supplies.

source

Jesse
https://playwithchatgtp.com