After inspection, the earthquake in Japan only slightly affected DMIT's data center in Equinix TY: a management node got circuit failed.
All computing and network nodes are running as expected.
Due to the particularity of the Tokyo data center, we still recommend that you schedule regular backups in this data center.
Stay safe.
All computing and network nodes are running as expected.
Due to the particularity of the Tokyo data center, we still recommend that you schedule regular backups in this data center.
Stay safe.
The Issue on the Los Angles is noticed, please wait for our NOC's investigation.
/var/log/DMIT-NOC.log
The Issue on the Los Angles is noticed, please wait for our NOC's investigation.
Solved. DDoS Mitigation Cluster has routing loop issues. Huge DDoS used up the entire bandwidth capacity of the core router. The router tried to reset all interfaces on the PFE to "solve" this issue.
To save FIB, the DDoS Cluster has no full FIB, it shares with other tables.
We'll keep looking into this to avoid any potential problems
The DDoS is not that huge on Internet, due to the looping, it was enlarged by "TTL-times".
If the source is Windows which has 128 TTL then it will be 128 times.
To save FIB, the DDoS Cluster has no full FIB, it shares with other tables.
We'll keep looking into this to avoid any potential problems
The DDoS is not that huge on Internet, due to the looping, it was enlarged by "TTL-times".
If the source is Windows which has 128 TTL then it will be 128 times.
DMIT has been following up on this event since Federal Communications Commission (FCC) issued the news announcement about China Telecom America (CTA).
To ensure the accuracy of the announcement and avoid misunderstanding, DMIT has not published any assumptions about this events until we got confirmation from CTA.
The FCC 214 license is needed to provide mobile, home networks, etc. in the U.S.
DMIT is also an IP Transit provider, and providing IP Transit services in the United States does not require a license. The revocation of CTA's 214 licenses will only affect its CTExcel(MVNO service).
Attached is the official statement of CTA on this event.
https://www.ctexcel.us/commonproblem?activate=28
To ensure the accuracy of the announcement and avoid misunderstanding, DMIT has not published any assumptions about this events until we got confirmation from CTA.
The FCC 214 license is needed to provide mobile, home networks, etc. in the U.S.
DMIT is also an IP Transit provider, and providing IP Transit services in the United States does not require a license. The revocation of CTA's 214 licenses will only affect its CTExcel(MVNO service).
Attached is the official statement of CTA on this event.
https://www.ctexcel.us/commonproblem?activate=28
/var/log/DMIT-NOC.log
Maintenance notice Location: LAX Status: Planing Description: 1. Upgrade Core Router system to avoid knew secure issues. 2. Backbone Connection with HKG and TYO 3. Perform network configuration standardized for future functions. Impact: 1. Unstable network…
Step 2 is completed. Please let us know if you have any routing issues.
Detect unreachable issues between HKG Pro with CFMT.
Under Investigating.
Under Investigating.
/var/log/DMIT-NOC.log
Detect unreachable issues between HKG Pro with CFMT. Under Investigating.
Cloudflare cannot solve this issue quickly. CFMT routing announcement has been withdrawal temporarily.
Initial diagnosis:
A BGP routing collection and analysis project sent unexpected BGP I/O flow, which led to Juniper routing core dump.
Nov 9 21:30:54 re.lax.DMIT.com rpd[5751]: BGP_IO_ERROR_CLOSE_SESSION: BGP peer x.x.x.x (External AS xxxx): Error event Operation timed out(60) for I/O session - closing it (instance master)
Nov 9 21:35:20 re.lax.DMIT.com jlaunchd: routing (PID 5751) terminated by signal number 6. Core dumped!
A BGP routing collection and analysis project sent unexpected BGP I/O flow, which led to Juniper routing core dump.
Nov 9 21:30:54 re.lax.DMIT.com rpd[5751]: BGP_IO_ERROR_CLOSE_SESSION: BGP peer x.x.x.x (External AS xxxx): Error event Operation timed out(60) for I/O session - closing it (instance master)
Nov 9 21:35:20 re.lax.DMIT.com jlaunchd: routing (PID 5751) terminated by signal number 6. Core dumped!
/var/log/DMIT-NOC.log
Solved. DDoS Mitigation Cluster has routing loop issues. Huge DDoS used up the entire bandwidth capacity of the core router. The router tried to reset all interfaces on the PFE to "solve" this issue. To save FIB, the DDoS Cluster has no full FIB, it shares…
This is not the major reason that caused the CR core dump at last month. After reviewing both dump logs. This is caused by unsupported BGP operations sent from the RIS project.
There are no problems at JunOS 15.x; but it happened after we upgrade to Jtac recommended JunOS 20.x. We’ll follow with Jtac once our new MPC line card has arrived.
There are no problems at JunOS 15.x; but it happened after we upgrade to Jtac recommended JunOS 20.x. We’ll follow with Jtac once our new MPC line card has arrived.
LAX.sPro(CFMT)'s IPv4 address space is running out, and it will take time for us to replenish the IPv4 address. Please order ahead if you may need the services recently.
TYO maintain:
- Reboot to apply DMIT kernel patch for PVE.
++ Improve network stability.
++ Drop unrelated Layer 2/3/4 packet with current client VM.
++ Remove PVE heavy network bridge architecture.
++ Remove PVE heavy firewall architecture.
- Reboot to apply DMIT kernel patch for PVE.
++ Improve network stability.
++ Drop unrelated Layer 2/3/4 packet with current client VM.
++ Remove PVE heavy network bridge architecture.
++ Remove PVE heavy firewall architecture.
TYO kernel patch complete. We only rebooted all VMs.
The load increasing caused by Ceph upgrade, data is backfilling.
The load increasing caused by Ceph upgrade, data is backfilling.