Los Angeles Maintenance Emergency Notice
- Time: June 2, 2025, 7 p.m. Pacific Time
- Details: Deploy CoreSite Edge Router, move some IP Transit on it.
- Impact: Some IP Transit customer services will be down, AS58807 and AS9929 will be temporarily interrupted
- Duration: 3 hours
- Time: June 2, 2025, 7 p.m. Pacific Time
- Details: Deploy CoreSite Edge Router, move some IP Transit on it.
- Impact: Some IP Transit customer services will be down, AS58807 and AS9929 will be temporarily interrupted
- Duration: 3 hours
/var/log/DMIT-NOC.log pinned «Los Angeles Maintenance Emergency Notice - Time: June 2, 2025, 7 p.m. Pacific Time - Details: Deploy CoreSite Edge Router, move some IP Transit on it. - Impact: Some IP Transit customer services will be down, AS58807 and AS9929 will be temporarily interrupted…»
/var/log/DMIT-NOC.log
Los Angeles Maintenance Emergency Notice - Time: June 2, 2025, 7 p.m. Pacific Time - Details: Deploy CoreSite Edge Router, move some IP Transit on it. - Impact: Some IP Transit customer services will be down, AS58807 and AS9929 will be temporarily interrupted…
Partial complete;
No service impact right now.
June 3rd night finish the pre-patch work;
June 4th 7pm hot cut rest of fiber, and bring it into network.
No service impact right now.
June 3rd night finish the pre-patch work;
June 4th 7pm hot cut rest of fiber, and bring it into network.
/var/log/DMIT-NOC.log
Partial complete; No service impact right now. June 3rd night finish the pre-patch work; June 4th 7pm hot cut rest of fiber, and bring it into network.
CoreSite LA1 migration and upgrade work has been completed.
DMIT has completed the upgrade of a new border router at CoreSite LA1; however, there are still few 100G customers/transit that have not been migrated to DMIT CoreSite LA1's new location.
Next, DMIT will complete the upgrades at W7C and Digital Realty:
- Move the W7C border to DRT; since W7C does not have critical carrier facilities, establishing a border here is constrained by insufficient carrier transportation and hardware capacity, such as frequent packet loss from AS1299 and AS174.
- Deploy access routers at W7C; provide access for VM customers and IP Transit customers after border router has been moved.
- Upgrade transport routers at W7C and DRT, transitioning from fixed routers to chassis routers.
DMIT currently has a protected Layer 2 transmission network with a capacity of 600G (expandable to 2.4Tbps protected) between three major data centers in downtown Los Angeles.
Please feel free to connect with us if you need:
- IP Transit of Internet
- China Optimized IP Transit (Branding Natural supported; DMIT won't be in trace route, and BGP)
- Ethernet Transportation within Los Angeles.
Next, DMIT will complete the upgrades at W7C and Digital Realty:
- Move the W7C border to DRT; since W7C does not have critical carrier facilities, establishing a border here is constrained by insufficient carrier transportation and hardware capacity, such as frequent packet loss from AS1299 and AS174.
- Deploy access routers at W7C; provide access for VM customers and IP Transit customers after border router has been moved.
- Upgrade transport routers at W7C and DRT, transitioning from fixed routers to chassis routers.
DMIT currently has a protected Layer 2 transmission network with a capacity of 600G (expandable to 2.4Tbps protected) between three major data centers in downtown Los Angeles.
Please feel free to connect with us if you need:
- IP Transit of Internet
- China Optimized IP Transit (Branding Natural supported; DMIT won't be in trace route, and BGP)
- Ethernet Transportation within Los Angeles.
DMIT has completed another upgrade of optimized network to China in collaboration with our partners.
DMIT currently holds
- 40Gbps CN2 GIA (distributed across DRT and CSLA dual cores and submarine system);
- 40Gbps CMIN2;
- 20Gbps CUG VIP.
DMIT currently installed 100G of high-quality optimized bandwidth to China.
DMIT currently holds
- 40Gbps CN2 GIA (distributed across DRT and CSLA dual cores and submarine system);
- 40Gbps CMIN2;
- 20Gbps CUG VIP.
DMIT currently installed 100G of high-quality optimized bandwidth to China.
[Network Update] DMIT Now Receiving IP Transit from GSL (AS137409)
DMIT is pleased to confirm IP Transit from GSL (AS137409) is now available at all three of our core locations.
✅ Los Angeles is fully operational.
🔜 Tokyo and Hong Kong will be brought online soon.
Routing Overview:
Los Angeles:
- GSL serves as the primary upstream for the inbound traffic.
- AS3257, AS1299, and AS174 will be used for inbound backup, their customer routes, and outbound. (Selected by BGP)
- This maximize the Internet direction Anti-DDoS capacity for Pro and EB customer.
- T1's inbound will only go thru DMIT's AS3257, AS1299, and AS174 only at this time, but T1's outbound will have access to GSL
Tokyo & Hong Kong (APAC):
- GSL and RETN will operate in a dual-primary configuration with mutual failover.
- RETN remains the preferred path for Europe-bound traffic optimization.
- DMIT will try to make AS58453 primarily go thru GSL(via AS3356, LUMEN) for better connectivity; but DMIT does ***NOT*** give any guarantee for connectivity between T1 and China. Whatever the quality, performance or anything else.
====
DMIT also has on site mitigation capacity for any traffic that comes in from other ISPs. (400Gbps in LAX, 200G in HKG, 200G in Tokyo).
DMIT is pleased to confirm IP Transit from GSL (AS137409) is now available at all three of our core locations.
✅ Los Angeles is fully operational.
🔜 Tokyo and Hong Kong will be brought online soon.
Routing Overview:
Los Angeles:
- GSL serves as the primary upstream for the inbound traffic.
- AS3257, AS1299, and AS174 will be used for inbound backup, their customer routes, and outbound. (Selected by BGP)
- This maximize the Internet direction Anti-DDoS capacity for Pro and EB customer.
- T1's inbound will only go thru DMIT's AS3257, AS1299, and AS174 only at this time, but T1's outbound will have access to GSL
Tokyo & Hong Kong (APAC):
- GSL and RETN will operate in a dual-primary configuration with mutual failover.
- RETN remains the preferred path for Europe-bound traffic optimization.
- DMIT will try to make AS58453 primarily go thru GSL(via AS3356, LUMEN) for better connectivity; but DMIT does ***NOT*** give any guarantee for connectivity between T1 and China. Whatever the quality, performance or anything else.
====
DMIT also has on site mitigation capacity for any traffic that comes in from other ISPs. (400Gbps in LAX, 200G in HKG, 200G in Tokyo).
[APAC Network Notice]
DMIT has connect with GSL in HKG; due to GSL has more peers than RETN, most network are favor GSL rather than RETN.
We cannot force make any networks favor RETN in EU, because
- AS3257 is RETN's IP Transit; they only connect it at EU. This make any AS3356 customer where ever they at, go to EU. (Like APAC/US>EU>RENT>HK); any they only connect it via Frankfurt.
- RETN also only peer with LUMEN(AS3356) in EU, the same reason, and RETN won't pass our communities to AS3356, if in EU, AS3356 is favored thru RETN; The LUMEN Hong Kong will also go to EU.
- same for some peers.
- GSL does not offer a regional prepend community.
It is likely that only some RETN customers are currently connected via RETN, most European targets will now be connected via GSL.
Due to this reason, DMIT NOC needs extra time to build anohter approches. Keep you posted.
DMIT has connect with GSL in HKG; due to GSL has more peers than RETN, most network are favor GSL rather than RETN.
We cannot force make any networks favor RETN in EU, because
- AS3257 is RETN's IP Transit; they only connect it at EU. This make any AS3356 customer where ever they at, go to EU. (Like APAC/US>EU>RENT>HK); any they only connect it via Frankfurt.
- RETN also only peer with LUMEN(AS3356) in EU, the same reason, and RETN won't pass our communities to AS3356, if in EU, AS3356 is favored thru RETN; The LUMEN Hong Kong will also go to EU.
- same for some peers.
- GSL does not offer a regional prepend community.
It is likely that only some RETN customers are currently connected via RETN, most European targets will now be connected via GSL.
Due to this reason, DMIT NOC needs extra time to build anohter approches. Keep you posted.
[HK Maintenance]
Date: June 27 - July 1, 2025
Content:
- Replacement of edge router outdated 10Gbps line cards(2x);
---- CMI expected upto 1hr downgrade;
---- CN2 GIA has redundnacy circuits expected graceful switch over;
- Replacement of main switch;
---- To Juniper QFX5200-32C (from Arista 7280CR)
- Replacement of TOR switch;
---- To Juniper QFX5200-32C (from QFX5100-24Q)
- Replacement of on-site DDoS mitigation facility;
- Repair error virtualization node;
- Adding virtualization nodes;
- Adding 480G edge router line card.
- Adding 2x100G peering PNI;
- Adding ASE connectivity to Tokyo.
Date: June 27 - July 1, 2025
Content:
---- CMI expected upto 1hr downgrade;
---- CN2 GIA has redundnacy circuits expected graceful switch over;
- Replacement of main switch;
---- To Juniper QFX5200-32C (from Arista 7280CR)
- Replacement of TOR switch;
---- To Juniper QFX5200-32C (from QFX5100-24Q)
- Replacement of on-site DDoS mitigation facility;
- Repair error virtualization node;
- Adding virtualization nodes;
- Adding 480G edge router line card.
/var/log/DMIT-NOC.log pinned «[HK Maintenance] Date: June 27 - July 1, 2025 Content: - Replacement of edge router outdated 10Gbps line cards(2x); ---- CMI expected upto 1hr downgrade; ---- CN2 GIA has redundnacy circuits expected graceful switch over; - Replacement of main switch;…»
📢 [Network Update – China Unicom Packet Loss in LAX]
We have observed packet loss on EB and Pro routes via China Unicom (AS9929), primarily caused by a DDoS attack targeting AS9929 customers.
China Unicom does not drop attack traffic within China when AS10099 customer announces RTBH (Blackhole). Instead, traffic is blocked only at overseas Provider Edge (PE) routers. If the attack volume exceeds their backbone capacity, packet loss and jitter are unavoidable.
🔒 DMIT has already implemented second-level DDoS detection and automatic RTBH to minimize impact, with victim IPs blackholed in under few seconds. This protects our uplink with AS10099 from congestion.
However, please note:
This cannot prevent packet loss if the attack targets other AS10099 customers.
It is also ineffective if the attack exceeds China Unicom's international backbone capacity (AS9929/AS10099).
China Unicom does not support RTBH natively on AS4837 and instead sells it as a premium service. Even with this feature, we cannot mitigate attacks aimed at unrelated IPs. (like if there is attack to other AS10099 customers.)
Unfortunately, resolution is not possible unless China Unicom changes its policies—which appears unlikely, as the current structure incentivizes selling RTBH as a product.
We appreciate your understanding and we will continue to monitor and mitigate as much as possible on our end.
We have observed packet loss on EB and Pro routes via China Unicom (AS9929), primarily caused by a DDoS attack targeting AS9929 customers.
China Unicom does not drop attack traffic within China when AS10099 customer announces RTBH (Blackhole). Instead, traffic is blocked only at overseas Provider Edge (PE) routers. If the attack volume exceeds their backbone capacity, packet loss and jitter are unavoidable.
🔒 DMIT has already implemented second-level DDoS detection and automatic RTBH to minimize impact, with victim IPs blackholed in under few seconds. This protects our uplink with AS10099 from congestion.
However, please note:
This cannot prevent packet loss if the attack targets other AS10099 customers.
It is also ineffective if the attack exceeds China Unicom's international backbone capacity (AS9929/AS10099).
China Unicom does not support RTBH natively on AS4837 and instead sells it as a premium service. Even with this feature, we cannot mitigate attacks aimed at unrelated IPs. (like if there is attack to other AS10099 customers.)
Unfortunately, resolution is not possible unless China Unicom changes its policies—which appears unlikely, as the current structure incentivizes selling RTBH as a product.
We appreciate your understanding and we will continue to monitor and mitigate as much as possible on our end.
/var/log/DMIT-NOC.log
[HK Maintenance] Date: June 27 - July 1, 2025 Content: - Replacement of edge router outdated 10Gbps line cards(2x); ---- CMI expected upto 1hr downgrade; ---- CN2 GIA has redundnacy circuits expected graceful switch over; - Replacement of main switch;…
APAC:
There are couple fiber cut inside APAC region;
GSL is now losing both short path connection between HK and TY.
RETN is also experience congestion.
NTT won't be good, as our experience.
DMIT is now configuring router to adapter the MPLS Ethernet connectivity.
After that, we can delivery the PNI and ASE connection.
* The TPE-TYO of GSL shows available but 0% IN/OUT; might need wait for GSL furtuer action.
There are couple fiber cut inside APAC region;
GSL is now losing both short path connection between HK and TY.
RETN is also experience congestion.
NTT won't be good, as our experience.
DMIT is now configuring router to adapter the MPLS Ethernet connectivity.
After that, we can delivery the PNI and ASE connection.
* The TPE-TYO of GSL shows available but 0% IN/OUT; might need wait for GSL furtuer action.
CoreSite Edge Router encounter an unexpected issue bring the rpd process crash; rollback and it was resovled.
There is an uncommon, and undocumented bug on JunOS that make unpredictable error in the routing and rdp.
We’ve ran many test past 2 months to ident the root cause.
The issue on Los Angeles RIB/FIB desync has been resolved.
Like some China Unicom prefer to use CMIN2 even the best route is 9929 in the RIB, and backup in CMIN2.
Also some GTT route will not go to GTT IP Transit even it’s the best and no other comparable selection. (Failover to default, use GSL..)
There is a pending change to fix this problem entirely that might be bring internal BGP reset. DMIT will perform this action at super off-peak time. The estimate impact time is less than 10 minus.
We’ve ran many test past 2 months to ident the root cause.
The issue on Los Angeles RIB/FIB desync has been resolved.
Like some China Unicom prefer to use CMIN2 even the best route is 9929 in the RIB, and backup in CMIN2.
Also some GTT route will not go to GTT IP Transit even it’s the best and no other comparable selection. (Failover to default, use GSL..)
There is a pending change to fix this problem entirely that might be bring internal BGP reset. DMIT will perform this action at super off-peak time. The estimate impact time is less than 10 minus.
/var/log/DMIT-NOC.log
There is an uncommon, and undocumented bug on JunOS that make unpredictable error in the routing and rdp. We’ve ran many test past 2 months to ident the root cause. The issue on Los Angeles RIB/FIB desync has been resolved. Like some China Unicom prefer…
Configuration has been loaded.
This issue has been permanently fixed.
This issue has been permanently fixed.