/var/log/DMIT-NOC.log
SJC Maintenance Time: Apr 10 ~ Apr 30, 2025 Service: All VMs. Impact: Lower IO & Network performance; couple reboot. Duration: Less than 5 hrs per VM. Description: - Merge with LAX.T1. HKG Maintenance Time: Apr 16 ~ Apr 24, 2025 (Changed to Apr 13 ~ Apr…
HKG Maintenance completed.
The timestamp issue happened after migration has been fixed as well.
The timestamp issue happened after migration has been fixed as well.
April 16
HKG one node experience major hardware error;
Working on the repairing.
Working on the repairing.
April 30
April 30
All the services were restored at 10PM HKT.
The compensation has been issued.
The control panel has been restored.
The new built node rather has CPU or motherboard failure which prevent it go thru the POST.
DMIT has ordered couple new nodes.
====
The WHMCS and corresponding modules currently used by DMIT are burdened with too many legacy issues. Even though DMIT has recoded most of it.
The configuration file loss is currently the main reason why VMs cannot recover quickly after node failure
We already figured out a new method for VM management which allows us have more secure and flexible way to manage the VM's configuration files.
This approach will be written in the new architecture and compatible with the current architecture to help smooth out the transition.
The compensation has been issued.
The control panel has been restored.
The new built node rather has CPU or motherboard failure which prevent it go thru the POST.
DMIT has ordered couple new nodes.
====
The WHMCS and corresponding modules currently used by DMIT are burdened with too many legacy issues. Even though DMIT has recoded most of it.
The configuration file loss is currently the main reason why VMs cannot recover quickly after node failure
We already figured out a new method for VM management which allows us have more secure and flexible way to manage the VM's configuration files.
This approach will be written in the new architecture and compatible with the current architecture to help smooth out the transition.
April 30
Our main site at Los Angeles was experiencing power outage.
There is a multiple incident happening in the downtown LA.
West7Center report:
There is a power cut to the suite; the power is restoring;
The issue happened in CoreSite LA1 has been resolved already:
A third-party vendor performing work on the street inadvertently damaged a water pipe. At this time, all building cooling systems are operating normally, and there is no indication of reduced water pressure entering the building. The Los Angeles Department of Water and Power (LADWP) has been notified, and we are currently awaiting an update on their estimated time of arrival.
You may direct any questions or concerns to the below department.
There is a multiple incident happening in the downtown LA.
West7Center report:
There is a power cut to the suite; the power is restoring;
The issue happened in CoreSite LA1 has been resolved already:
A third-party vendor performing work on the street inadvertently damaged a water pipe. At this time, all building cooling systems are operating normally, and there is no indication of reduced water pressure entering the building. The Los Angeles Department of Water and Power (LADWP) has been notified, and we are currently awaiting an update on their estimated time of arrival.
You may direct any questions or concerns to the below department.
May 7
May 7
The website and management systems are in Los Angeles core site. It won’t affected the service in other regions but the control panel will remain offline.
May 7
On-site engineers smell smoke onsite; no visible damage in the suite and in our cage.
May 7
May 7
West7Center engineer is trying to locate the root cause, before restoring the power;
The P+R power feed was disconnected due to possible fire hazard when incident occurred.
It was not a power outage; instead it’s a fire prevention lead to power cut.
The P+R power feed was disconnected due to possible fire hazard when incident occurred.
It was not a power outage; instead it’s a fire prevention lead to power cut.
May 7
There is
- No sprinkler has been initiated;
- No visible fire hazard;
- No water damage;
- No evacuation;
- A small area has smoke smelled. (Not entire suites)
- only power cut on the cabinet; ambient light remain on.
Based on past information from other site engineers, the power was disconnected just after fire alarm has been triggered.
- No sprinkler has been initiated;
- No visible fire hazard;
- No water damage;
- No evacuation;
- A small area has smoke smelled. (Not entire suites)
- only power cut on the cabinet; ambient light remain on.
Based on past information from other site engineers, the power was disconnected just after fire alarm has been triggered.
May 7
W7C response:
They are waiting for UPS system vendor engineer to be onsite.
The electrical engineer has arrived.
The UPS system has encountered some issue which prevent it starting
They are waiting for UPS system vendor engineer to be onsite.
The electrical engineer has arrived.
The UPS system has encountered some issue which prevent it starting
May 7
May 7
/var/log/DMIT-NOC.log
W7C UPS technician arrives in 30 minutes.
UPS technician is working on restoring power.
May 7
May 7
All our submarines cables with China are landing at Downtown LA.
This is the main reason DMIT stay in downtown.
DMIT will work with our team and multiple partners to find out a better solution in Los Angeles about this.
A solution will be post later.
This is the main reason DMIT stay in downtown.
DMIT will work with our team and multiple partners to find out a better solution in Los Angeles about this.
A solution will be post later.
May 7
May 7
Dear customers,
We are sincerely apologize about the outage hours ago.
Below is a preliminary investigation of the LAX data center outage. The full incident investigation report will be shared in this channel once provided by the data center.
DMIT is still waiting report from this location.
At around 4:30 PM CST (7:30AM China Standard Time) on May 7, 2025, the West 7 Center data center located in 1200 W 7TH, Los Angeles, CA, 90017 triggered a fire alarm. To minimize potential damage, the data center's fire suppression system was automatically activated, cutting off power to all cabinets and causing all servers and network equipment in the data center to lose power. Since the UPS battery system in the data center was also disconnected after the fire alarm was triggered, the UPS could not function properly. However, lighting power remained operational.
Subsequently, the following actions were taken on-site, and power was restored at around 9:30PM PST (12:20 PM China Standard Time):
- DMIT dispatched engineers to the site for inspection. The engineers detected an unidentified smoke odor on-site. After inspection, no abnormal were found in our cage and equipment. We waited for the data center to restore power.
- Data center engineers worked to identify the cause of the fire alarm and attempted to restore power while minimizing the impact within a controllable risk range.
- The data center attempted to restore power but found that the UPS was not functioning properly, so UPS engineers were dispatched to the site for inspection.
- UPS engineers arrived on-site, inspected, and repaired the UPS system to make it operational.
After power was restored, DMIT inspected all equipment to confirm that it was unaffected and began gradually restarting all devices.
Currently, all equipment in the DMIT LAX data center has been restored to service. If your service is still inaccessible, please submit a ticket to contact us.
We deeply regret this outage and appreciate your patience. Although the uptime remains above the SLA compensation threshold (the downtime lasted approximately 5 hours), we have decided to provide compensation to all affected customers.
- For all affected LAX services, compensation equivalent to 5 days of service renewal value (if the calculated compensation amount is less than 0.5 USD, a minimum of 0.5 USD will be credited) will be added to your DMIT account.
- For all existing DMIT customers, any purchase of DMIT products with a monthly average cost exceeding 3.33 USD (T1 series products are excluded from this restriction) is eligible for a 10% discount using the coupon code 2025-EXISTING-CUSTOMER-10OFF, valid until June 30, 2025.
The scope and conditions for using the coupon code are as follows:
- Can be applied to special products, if the product is in stock or you are eligible to apply for a special product order. (HKG Pro Mongkok & Nathan not included)
- Your account must have an active service.
Thank you again for your continued support. We will continue to improve the stability and reliability of our services. The detailed investigation results of this incident will be shared once provided by the data center.
We are sincerely apologize about the outage hours ago.
Below is a preliminary investigation of the LAX data center outage. The full incident investigation report will be shared in this channel once provided by the data center.
DMIT is still waiting report from this location.
At around 4:30 PM CST (7:30AM China Standard Time) on May 7, 2025, the West 7 Center data center located in 1200 W 7TH, Los Angeles, CA, 90017 triggered a fire alarm. To minimize potential damage, the data center's fire suppression system was automatically activated, cutting off power to all cabinets and causing all servers and network equipment in the data center to lose power. Since the UPS battery system in the data center was also disconnected after the fire alarm was triggered, the UPS could not function properly. However, lighting power remained operational.
Subsequently, the following actions were taken on-site, and power was restored at around 9:30PM PST (12:20 PM China Standard Time):
- DMIT dispatched engineers to the site for inspection. The engineers detected an unidentified smoke odor on-site. After inspection, no abnormal were found in our cage and equipment. We waited for the data center to restore power.
- Data center engineers worked to identify the cause of the fire alarm and attempted to restore power while minimizing the impact within a controllable risk range.
- The data center attempted to restore power but found that the UPS was not functioning properly, so UPS engineers were dispatched to the site for inspection.
- UPS engineers arrived on-site, inspected, and repaired the UPS system to make it operational.
After power was restored, DMIT inspected all equipment to confirm that it was unaffected and began gradually restarting all devices.
Currently, all equipment in the DMIT LAX data center has been restored to service. If your service is still inaccessible, please submit a ticket to contact us.
We deeply regret this outage and appreciate your patience. Although the uptime remains above the SLA compensation threshold (the downtime lasted approximately 5 hours), we have decided to provide compensation to all affected customers.
- For all affected LAX services, compensation equivalent to 5 days of service renewal value (if the calculated compensation amount is less than 0.5 USD, a minimum of 0.5 USD will be credited) will be added to your DMIT account.
- For all existing DMIT customers, any purchase of DMIT products with a monthly average cost exceeding 3.33 USD (T1 series products are excluded from this restriction) is eligible for a 10% discount using the coupon code 2025-EXISTING-CUSTOMER-10OFF, valid until June 30, 2025.
The scope and conditions for using the coupon code are as follows:
- Can be applied to special products, if the product is in stock or you are eligible to apply for a special product order. (HKG Pro Mongkok & Nathan not included)
- Your account must have an active service.
Thank you again for your continued support. We will continue to improve the stability and reliability of our services. The detailed investigation results of this incident will be shared once provided by the data center.
May 7