Alibaba Cloud disrupted after fire at Digital Realty datacenter in Singapore

A fire at a Digital Realty Singapore datacenter by a lithium-ion battery explosion disrupts Alibaba Cloud services.

Listen to article

A significant portion of Alibaba Cloud services experienced disruptions following a fire at a Digital Realty datacenter in Singapore.

The blaze, which occurred early Tuesday morning on September 10, 2024, is believed to have been triggered by a lithium-ion battery explosion in the facility's battery room.

Local reports confirmed that Singapore’s Civil Defence Force responded to the fire shortly after it broke out.

According to a spokesperson for Digital Realty, the fire alarm at the SIN11 datacenter was triggered at 07:45 local time.

All on-site personnel were safely evacuated by 08:15, and fortunately, no injuries were reported.

Alibaba Cloud, one of the largest cloud service providers in the region, immediately began detecting anomalies in its Singapore-based Availability Zone C around 10:20 SGT.

The Chinese cloud giant released a statement confirming the explosion and subsequent fire, which led to abnormally high temperatures in the affected areas. This, in turn, caused some cloud services to experience disruptions.

Despite the fire, Alibaba Cloud reported that its disaster recovery systems were activated, and failover routines worked as designed.

This meant that its high-availability cloud products continued to operate within promised service levels. However, some users were still required to manually migrate workloads away from the affected zone to ensure continuity.

According to the cloud provider’s status updates, certain services and offerings were temporarily taken offline as temperatures within the datacenter remained dangerously elevated.

By 20:04 SGT on Tuesday, the situation had not yet been fully resolved, with staff unable to re-enter the blaze-hit building.

By early Wednesday morning (01:46 SGT), further complications arose as the datacenter began to experience water leaks, likely caused by firefighting efforts.

This posed a new risk of electrical short circuits, forcing an emergency shutdown of electrical systems in one of the buildings.

Digital Realty confirmed that at 01:45 SGT on Wednesday, portions of the facility had been powered down to prevent further damage. Alibaba Cloud, meanwhile, worked to restore network services to unaffected buildings in the complex.

The incident not only disrupted Alibaba Cloud’s operations but also caused service outages and degradation for other platforms dependent on the same infrastructure. Digital Ocean, IaaS service Coolify, and content delivery network Cloudflare were all affected by the blaze, according to online complaints from users.

This incident adds to the growing list of concerns over lithium-ion battery safety in critical infrastructure like datacenters. French cloud provider OVH lost one of its datacenters in 2021 due to a similar fire in an uninterruptible power supply (UPS) system.