DMIT already aware about LAX EB packet loss on China direction.
We observe there is no packet loss on PtP segment.
The CMIN2 NOC response there is an issue on NCP submarine. Their capacity has been impacted due to the submarine issue.
DMIT will keep you updated.
We observe there is no packet loss on PtP segment.
The CMIN2 NOC response there is an issue on NCP submarine. Their capacity has been impacted due to the submarine issue.
DMIT will keep you updated.
/var/log/DMIT-NOC.log
DMIT already aware about LAX EB packet loss on China direction. We observe there is no packet loss on PtP segment. The CMIN2 NOC response there is an issue on NCP submarine. Their capacity has been impacted due to the submarine issue. DMIT will keep…
CTGnoc report, partial Tokyo CN2 CTGnet also been detour because NCP failure.
/var/log/DMIT-NOC.log
DMIT already aware about LAX EB packet loss on China direction. We observe there is no packet loss on PtP segment. The CMIN2 NOC response there is an issue on NCP submarine. Their capacity has been impacted due to the submarine issue. DMIT will keep…
We worked with CMIN2 to have the LAX.EB's current situation to be relieved.
Please check again.
Please check again.
/var/log/DMIT-NOC.log
Unexpected issue happend on TYO.Pro; investigating
Issue resolved;
Encounter JunOS software design defact.
VRF table refresh (deactivate & activate, table name remain unchange) make the VRF obtained a new FIB Table ID;
The PBR (which use "name" of table) does not update the Table ID after that.
Which bring the VM traffic been directed to a blackhole.
Resolved by disable PBR in Firewall, then reactivate.
Encounter JunOS software design defact.
VRF table refresh (deactivate & activate, table name remain unchange) make the VRF obtained a new FIB Table ID;
The PBR (which use "name" of table) does not update the Table ID after that.
Which bring the VM traffic been directed to a blackhole.
Resolved by disable PBR in Firewall, then reactivate.
/var/log/DMIT-NOC.log
DMIT already aware about LAX EB packet loss on China direction. We observe there is no packet loss on PtP segment. The CMIN2 NOC response there is an issue on NCP submarine. Their capacity has been impacted due to the submarine issue. DMIT will keep…
CMI NOC indicated the issue has been resolved.
DMIT has confirmed this resolution after 3 days of observation.
DMIT has confirmed this resolution after 3 days of observation.
From China Telecom,
For Los Angeles CN2 GIA.
====
Dear customer,
Please kindly be informed that there will be an planned maintenance and more information as below.
Time window (Date/Time):
2025-01-10 16:00:00 - 2025-01-22 22:00:00 UTC
2025-01-11 00:00:00 - 2025-01-23 06:00:00 UTC+8
Maintenance Description:
Facilities relocation
Maintenance Location:
American company, International
Service Impact:
The circuit will experience outage up to 360 minutes during the maintenance window.
For Los Angeles CN2 GIA.
====
Dear customer,
Please kindly be informed that there will be an planned maintenance and more information as below.
Time window (Date/Time):
2025-01-10 16:00:00 - 2025-01-22 22:00:00 UTC
2025-01-11 00:00:00 - 2025-01-23 06:00:00 UTC+8
Maintenance Description:
Facilities relocation
Maintenance Location:
American company, International
Service Impact:
The circuit will experience outage up to 360 minutes during the maintenance window.
/var/log/DMIT-NOC.log pinned «From China Telecom, For Los Angeles CN2 GIA. ==== Dear customer, Please kindly be informed that there will be an planned maintenance and more information as below. Time window (Date/Time): 2025-01-10 16:00:00 - 2025-01-22 22:00:00 UTC 2025-01-11 00:00:00…»
Observed CT CU routes has been withdrawn by CMI on Tokyo EB.
Waiting updates from CMI.
Waiting updates from CMI.
DMIT HKG Maintenance:
In 3 hours, upto 30 mins.
Replace / Upgrade Edge Router routing engine.
In 3 hours, upto 30 mins.
Replace / Upgrade Edge Router routing engine.
Network downgrade has been detected
Impact: CMIN2, Los Angeles.
The latency to China Mobile of LAX.EB, and LAX.Pro has increase.
No increaed packet loss rate has been observed.
LAX.Pro used CMIN2 for China Mobile inbound, the latency will also be impacted.
Notes: The only low latency inbound optimization for China Mobile is CMIN2.
CM-on-Net (CMI AS58453) has at least 150-180ms during normal time.
Issue has been reported to China Mobile International NOC already.
===
Jan 3, 2024 1AM EST
No update from CMI, but the latency return to normal hours ago.
Impact: CMIN2, Los Angeles.
The latency to China Mobile of LAX.EB, and LAX.Pro has increase.
No increaed packet loss rate has been observed.
LAX.Pro used CMIN2 for China Mobile inbound, the latency will also be impacted.
Notes: The only low latency inbound optimization for China Mobile is CMIN2.
CM-on-Net (CMI AS58453) has at least 150-180ms during normal time.
Issue has been reported to China Mobile International NOC already.
===
Jan 3, 2024 1AM EST
No update from CMI, but the latency return to normal hours ago.
Out of Stock Notice of DMIT Los Angeles Data Center
DMIT was unable to complete the expansion at the end of December 2024, due to HPE delivery overruns.
To ensure stable operation for existing customers, DMIT suspended sales of most instances in Los Angeles.
We expect to resume sales of all regular instances within 2 weeks.
Please contact us if you have any urgent need on specific one.
DMIT was unable to complete the expansion at the end of December 2024, due to HPE delivery overruns.
To ensure stable operation for existing customers, DMIT suspended sales of most instances in Los Angeles.
We expect to resume sales of all regular instances within 2 weeks.
Please contact us if you have any urgent need on specific one.
Due to the Pacific Palisades wildfire in Los Angeles; there are many place has experience power outage include DTLA.
The following place reported power quality event and power outage in recent hour.
CoreSite LA1 (One Wilshire Building)
Digital Realty LOS1 (600 W 7TH)
Both site has report the they sucessed switch power source to generator or backup.
Please be prepared any possiable service downgrade.
DMIT will keep you posted.
The following place reported power quality event and power outage in recent hour.
CoreSite LA1 (One Wilshire Building)
Digital Realty LOS1 (600 W 7TH)
Both site has report the they sucessed switch power source to generator or backup.
Please be prepared any possiable service downgrade.
DMIT will keep you posted.
CoreSite
Digital Realty
DMIT Notes:
- The fire is still far from the facility and the main risk remains power outages.
- The hill fire in Pasadena is closer to the facility than in Palisades.
- CoreSite, W7C, Digital Realty are well prepared.
- However, the potential risk is that when DTLA is classified as a risk area, there is a possibility that on-site personnel could be ordered to evacuate.
https://www.fire.ca.gov/incidents/2025/1/7/palisades-fire
Description: In response to the active wildfires in Pacific Palisades and Altadena, along with the strong winds we experienced overnight, we are closely monitoring the facility conditions to ensure the site remains stable.
Site conditions remain as follows:
The site is supported by the normal source utility power with the emergency source generators available if required. All mechanical systems related to cooling remain in a normal operational state.
We will be placing the fire system into test mode to avoid any activations due to outside smoke from the wildfires.
Should site conditions change, we will send an updated notification to ensure all clients are made aware.In response to the active wildfires in Pacific Palisades and Altadena, along with the strong winds we experienced overnight, we are closely monitoring the facility conditions to ensure the site remains stable.
Site conditions remain as follows:
The site is supported by the normal source utility power with the emergency source generators available if required. All mechanical systems related to cooling remain in a normal operational state.
We will be placing the fire system into test mode to avoid any activations due to outside smoke from the wildfires.
Should site conditions change, we will send an updated notification to ensure all clients are made aware.
You may direct any questions or concerns to the below department:
Thank you
Digital Realty
Date: January 8, 2025
To: All Digital Realty Operations Customers in Los Angeles Metro Area
The National Weather Service has issued a Red Flag Warning which is in effect for the Los Angeles Metro Area until January 9, 2025, 6:00 PM local time.
In preparation, Digital Realty Operations has implemented its standard severe weather response protocols to ensure appropriate engineering, vendor and security coverage during the event as well as ensuring the availability of critical spares, generator fuel and emergency supplies. Any incidents impacting site operations will be communicated to the affected customers through our standard communication procedures.
A Red Flag Warning means that critical fire weather conditions are either occurring now, or will shortly. Use extreme caution with anything that can spark a wildfire. Residents near wildland
interfaces should be prepared to evacuate if a wildfire breaks out. See www.readyforwildfire.org and www.wildfirerisk.org for information.
If you have any questions regarding our preparation procedures or current site status, please contact your local site operations team.
Thank you,
DIGITAL REALTY GLOBAL OPERATIONS
Digital Realty 2
As a final update to the Utility Interruption at 600 W. 7th, the engineering team reports that utility service has been restored. The site has returned to normal operating configuration. Any further notices relative to this issue will be communicated via the local site team and the normal Change Management process as required.
Summary
CoreSite Update
Event: Wildfires in Pacific Palisades and Altadena, with strong winds.
Status:
- Normal utility power;
- emergency generators on standby;
- cooling systems operational.
- Fire system placed in test mode to prevent false alarms from outside smoke.
Digital Realty
Red Flag Warning – LA Metro (Jan 8-9, 2025)
- Severe weather protocols activated: engineering, vendor, security coverage in place; critical supplies ready.
Advisory:
- Residents near wildland areas should prepare for possible evacuation.
Utility Restored – 600 W. 7th
Power restored; site back to normal.
DMIT Notes:
- The fire is still far from the facility and the main risk remains power outages.
- The hill fire in Pasadena is closer to the facility than in Palisades.
- CoreSite, W7C, Digital Realty are well prepared.
- However, the potential risk is that when DTLA is classified as a risk area, there is a possibility that on-site personnel could be ordered to evacuate.
https://www.fire.ca.gov/incidents/2025/1/7/palisades-fire
There is no further updates. Everything is work as expected.
DMIT’s service is unaffected for now.
Cogent’s capacity is impact for 3 hrs on last 24 hrs. We observed the congestion has been relieved for now.
DMIT’s service is unaffected for now.
Cogent’s capacity is impact for 3 hrs on last 24 hrs. We observed the congestion has been relieved for now.