West7Center engineer is trying to locate the root cause, before restoring the power;
The P+R power feed was disconnected due to possible fire hazard when incident occurred.
It was not a power outage; instead it’s a fire prevention lead to power cut.
The P+R power feed was disconnected due to possible fire hazard when incident occurred.
It was not a power outage; instead it’s a fire prevention lead to power cut.
There is
- No sprinkler has been initiated;
- No visible fire hazard;
- No water damage;
- No evacuation;
- A small area has smoke smelled. (Not entire suites)
- only power cut on the cabinet; ambient light remain on.
Based on past information from other site engineers, the power was disconnected just after fire alarm has been triggered.
- No sprinkler has been initiated;
- No visible fire hazard;
- No water damage;
- No evacuation;
- A small area has smoke smelled. (Not entire suites)
- only power cut on the cabinet; ambient light remain on.
Based on past information from other site engineers, the power was disconnected just after fire alarm has been triggered.
W7C response:
They are waiting for UPS system vendor engineer to be onsite.
The electrical engineer has arrived.
The UPS system has encountered some issue which prevent it starting
They are waiting for UPS system vendor engineer to be onsite.
The electrical engineer has arrived.
The UPS system has encountered some issue which prevent it starting
/var/log/DMIT-NOC.log
W7C UPS technician arrives in 30 minutes.
UPS technician is working on restoring power.
All our submarines cables with China are landing at Downtown LA.
This is the main reason DMIT stay in downtown.
DMIT will work with our team and multiple partners to find out a better solution in Los Angeles about this.
A solution will be post later.
This is the main reason DMIT stay in downtown.
DMIT will work with our team and multiple partners to find out a better solution in Los Angeles about this.
A solution will be post later.
Dear customers,
We are sincerely apologize about the outage hours ago.
Below is a preliminary investigation of the LAX data center outage. The full incident investigation report will be shared in this channel once provided by the data center.
DMIT is still waiting report from this location.
At around 4:30 PM CST (7:30AM China Standard Time) on May 7, 2025, the West 7 Center data center located in 1200 W 7TH, Los Angeles, CA, 90017 triggered a fire alarm. To minimize potential damage, the data center's fire suppression system was automatically activated, cutting off power to all cabinets and causing all servers and network equipment in the data center to lose power. Since the UPS battery system in the data center was also disconnected after the fire alarm was triggered, the UPS could not function properly. However, lighting power remained operational.
Subsequently, the following actions were taken on-site, and power was restored at around 9:30PM PST (12:20 PM China Standard Time):
- DMIT dispatched engineers to the site for inspection. The engineers detected an unidentified smoke odor on-site. After inspection, no abnormal were found in our cage and equipment. We waited for the data center to restore power.
- Data center engineers worked to identify the cause of the fire alarm and attempted to restore power while minimizing the impact within a controllable risk range.
- The data center attempted to restore power but found that the UPS was not functioning properly, so UPS engineers were dispatched to the site for inspection.
- UPS engineers arrived on-site, inspected, and repaired the UPS system to make it operational.
After power was restored, DMIT inspected all equipment to confirm that it was unaffected and began gradually restarting all devices.
Currently, all equipment in the DMIT LAX data center has been restored to service. If your service is still inaccessible, please submit a ticket to contact us.
We deeply regret this outage and appreciate your patience. Although the uptime remains above the SLA compensation threshold (the downtime lasted approximately 5 hours), we have decided to provide compensation to all affected customers.
- For all affected LAX services, compensation equivalent to 5 days of service renewal value (if the calculated compensation amount is less than 0.5 USD, a minimum of 0.5 USD will be credited) will be added to your DMIT account.
- For all existing DMIT customers, any purchase of DMIT products with a monthly average cost exceeding 3.33 USD (T1 series products are excluded from this restriction) is eligible for a 10% discount using the coupon code 2025-EXISTING-CUSTOMER-10OFF, valid until June 30, 2025.
The scope and conditions for using the coupon code are as follows:
- Can be applied to special products, if the product is in stock or you are eligible to apply for a special product order. (HKG Pro Mongkok & Nathan not included)
- Your account must have an active service.
Thank you again for your continued support. We will continue to improve the stability and reliability of our services. The detailed investigation results of this incident will be shared once provided by the data center.
We are sincerely apologize about the outage hours ago.
Below is a preliminary investigation of the LAX data center outage. The full incident investigation report will be shared in this channel once provided by the data center.
DMIT is still waiting report from this location.
At around 4:30 PM CST (7:30AM China Standard Time) on May 7, 2025, the West 7 Center data center located in 1200 W 7TH, Los Angeles, CA, 90017 triggered a fire alarm. To minimize potential damage, the data center's fire suppression system was automatically activated, cutting off power to all cabinets and causing all servers and network equipment in the data center to lose power. Since the UPS battery system in the data center was also disconnected after the fire alarm was triggered, the UPS could not function properly. However, lighting power remained operational.
Subsequently, the following actions were taken on-site, and power was restored at around 9:30PM PST (12:20 PM China Standard Time):
- DMIT dispatched engineers to the site for inspection. The engineers detected an unidentified smoke odor on-site. After inspection, no abnormal were found in our cage and equipment. We waited for the data center to restore power.
- Data center engineers worked to identify the cause of the fire alarm and attempted to restore power while minimizing the impact within a controllable risk range.
- The data center attempted to restore power but found that the UPS was not functioning properly, so UPS engineers were dispatched to the site for inspection.
- UPS engineers arrived on-site, inspected, and repaired the UPS system to make it operational.
After power was restored, DMIT inspected all equipment to confirm that it was unaffected and began gradually restarting all devices.
Currently, all equipment in the DMIT LAX data center has been restored to service. If your service is still inaccessible, please submit a ticket to contact us.
We deeply regret this outage and appreciate your patience. Although the uptime remains above the SLA compensation threshold (the downtime lasted approximately 5 hours), we have decided to provide compensation to all affected customers.
- For all affected LAX services, compensation equivalent to 5 days of service renewal value (if the calculated compensation amount is less than 0.5 USD, a minimum of 0.5 USD will be credited) will be added to your DMIT account.
- For all existing DMIT customers, any purchase of DMIT products with a monthly average cost exceeding 3.33 USD (T1 series products are excluded from this restriction) is eligible for a 10% discount using the coupon code 2025-EXISTING-CUSTOMER-10OFF, valid until June 30, 2025.
The scope and conditions for using the coupon code are as follows:
- Can be applied to special products, if the product is in stock or you are eligible to apply for a special product order. (HKG Pro Mongkok & Nathan not included)
- Your account must have an active service.
Thank you again for your continued support. We will continue to improve the stability and reliability of our services. The detailed investigation results of this incident will be shared once provided by the data center.
Incident Report L1-135 (05-16-2025) Update (05-26-2025).pdf
276.6 KB
📢 [Incident Report] L1-135 Power Outage
On May 6, 2025, our upstream data center partner experienced two separate power-related incidents affecting Data Hall L1-135, where your VM(s) be hosted:
1️⃣ Primary Incident – Facility Power Bus Failure
A major arc flash occurred in a 4000A bus duct due to a downstream ground fault, causing a complete loss of utility and generator power. This led to downtime for both critical power and cooling. Power was restored at 5:33 PM, and UPS was fully recovered by 9:00 PM.
2️⃣ Secondary Issue – Faulty Customer PDU
A customer-provided PDU (not part of our equipment) caused intermittent ground faults, which tripped the UPS Static Bypass Module and delayed recovery. The PDU was removed on May 14.
⚠️ DMIT Position:
While we appreciate the transparency of the facility’s investigation, DMIT Cloud Service does not fully agree with the root cause attribution. A single faulty PDU should not be able to take down an entire UPS system, and we believe better electrical isolation should have been in place to localize such failures.
✅ DMIT Cloud Service Response & Improvements:
🔁 Network Resilience: We will relocate all Layer 3 network devices out of W7C, ensuring that IP transit and routing customers across other sites are not impacted by any single-site failure.
🧭 New Availability Zone: We are adding a secondary zone in Los Angeles, expected to come online by the end of 2025.
🔄 Dual-Zone Support: VM customers will soon be able to choose between availability zones or enable dual-zone deployment for improved redundancy.
🤝 Ongoing Coordination: We are working closely with the IDC provider to improve power infrastructure and failure containment.
We apologize for the disruption and thank you for your continued trust. All systems are fully restored and operating normally.
On May 6, 2025, our upstream data center partner experienced two separate power-related incidents affecting Data Hall L1-135, where your VM(s) be hosted:
1️⃣ Primary Incident – Facility Power Bus Failure
A major arc flash occurred in a 4000A bus duct due to a downstream ground fault, causing a complete loss of utility and generator power. This led to downtime for both critical power and cooling. Power was restored at 5:33 PM, and UPS was fully recovered by 9:00 PM.
2️⃣ Secondary Issue – Faulty Customer PDU
A customer-provided PDU (not part of our equipment) caused intermittent ground faults, which tripped the UPS Static Bypass Module and delayed recovery. The PDU was removed on May 14.
⚠️ DMIT Position:
While we appreciate the transparency of the facility’s investigation, DMIT Cloud Service does not fully agree with the root cause attribution. A single faulty PDU should not be able to take down an entire UPS system, and we believe better electrical isolation should have been in place to localize such failures.
✅ DMIT Cloud Service Response & Improvements:
🔁 Network Resilience: We will relocate all Layer 3 network devices out of W7C, ensuring that IP transit and routing customers across other sites are not impacted by any single-site failure.
🧭 New Availability Zone: We are adding a secondary zone in Los Angeles, expected to come online by the end of 2025.
🔄 Dual-Zone Support: VM customers will soon be able to choose between availability zones or enable dual-zone deployment for improved redundancy.
🤝 Ongoing Coordination: We are working closely with the IDC provider to improve power infrastructure and failure containment.
We apologize for the disruption and thank you for your continued trust. All systems are fully restored and operating normally.
Los Angeles Maintenance Emergency Notice
- Time: June 2, 2025, 7 p.m. Pacific Time
- Details: Deploy CoreSite Edge Router, move some IP Transit on it.
- Impact: Some IP Transit customer services will be down, AS58807 and AS9929 will be temporarily interrupted
- Duration: 3 hours
- Time: June 2, 2025, 7 p.m. Pacific Time
- Details: Deploy CoreSite Edge Router, move some IP Transit on it.
- Impact: Some IP Transit customer services will be down, AS58807 and AS9929 will be temporarily interrupted
- Duration: 3 hours
/var/log/DMIT-NOC.log pinned «Los Angeles Maintenance Emergency Notice - Time: June 2, 2025, 7 p.m. Pacific Time - Details: Deploy CoreSite Edge Router, move some IP Transit on it. - Impact: Some IP Transit customer services will be down, AS58807 and AS9929 will be temporarily interrupted…»
/var/log/DMIT-NOC.log
Los Angeles Maintenance Emergency Notice - Time: June 2, 2025, 7 p.m. Pacific Time - Details: Deploy CoreSite Edge Router, move some IP Transit on it. - Impact: Some IP Transit customer services will be down, AS58807 and AS9929 will be temporarily interrupted…
Partial complete;
No service impact right now.
June 3rd night finish the pre-patch work;
June 4th 7pm hot cut rest of fiber, and bring it into network.
No service impact right now.
June 3rd night finish the pre-patch work;
June 4th 7pm hot cut rest of fiber, and bring it into network.
/var/log/DMIT-NOC.log
Partial complete; No service impact right now. June 3rd night finish the pre-patch work; June 4th 7pm hot cut rest of fiber, and bring it into network.
CoreSite LA1 migration and upgrade work has been completed.
DMIT has completed the upgrade of a new border router at CoreSite LA1; however, there are still few 100G customers/transit that have not been migrated to DMIT CoreSite LA1's new location.
Next, DMIT will complete the upgrades at W7C and Digital Realty:
- Move the W7C border to DRT; since W7C does not have critical carrier facilities, establishing a border here is constrained by insufficient carrier transportation and hardware capacity, such as frequent packet loss from AS1299 and AS174.
- Deploy access routers at W7C; provide access for VM customers and IP Transit customers after border router has been moved.
- Upgrade transport routers at W7C and DRT, transitioning from fixed routers to chassis routers.
DMIT currently has a protected Layer 2 transmission network with a capacity of 600G (expandable to 2.4Tbps protected) between three major data centers in downtown Los Angeles.
Please feel free to connect with us if you need:
- IP Transit of Internet
- China Optimized IP Transit (Branding Natural supported; DMIT won't be in trace route, and BGP)
- Ethernet Transportation within Los Angeles.
Next, DMIT will complete the upgrades at W7C and Digital Realty:
- Move the W7C border to DRT; since W7C does not have critical carrier facilities, establishing a border here is constrained by insufficient carrier transportation and hardware capacity, such as frequent packet loss from AS1299 and AS174.
- Deploy access routers at W7C; provide access for VM customers and IP Transit customers after border router has been moved.
- Upgrade transport routers at W7C and DRT, transitioning from fixed routers to chassis routers.
DMIT currently has a protected Layer 2 transmission network with a capacity of 600G (expandable to 2.4Tbps protected) between three major data centers in downtown Los Angeles.
Please feel free to connect with us if you need:
- IP Transit of Internet
- China Optimized IP Transit (Branding Natural supported; DMIT won't be in trace route, and BGP)
- Ethernet Transportation within Los Angeles.
DMIT has completed another upgrade of optimized network to China in collaboration with our partners.
DMIT currently holds
- 40Gbps CN2 GIA (distributed across DRT and CSLA dual cores and submarine system);
- 40Gbps CMIN2;
- 20Gbps CUG VIP.
DMIT currently installed 100G of high-quality optimized bandwidth to China.
DMIT currently holds
- 40Gbps CN2 GIA (distributed across DRT and CSLA dual cores and submarine system);
- 40Gbps CMIN2;
- 20Gbps CUG VIP.
DMIT currently installed 100G of high-quality optimized bandwidth to China.