Possiable failure on AS174 Los Angeles; We saw Internet routes that annouced by Cogent but unable to reach via Cogent.

DMIT rejected Cogent routes to prevent possible null-route on Internet.

We will recover it once Cogent solved the problem.

Planned maintenance
Date: Jun 15-18 2023 Hong Kong Time

Content:
Fix management node issues; install DDoS mitigation facilities, complete installation of new nodes, rebuild old cabinets.

Possible scenarios: Network downgration, VM reboot, Unable to access control panel.

We are currently migrating your VM to the new nodes;
There will be less than 10min downtime (avg) for each VM.

Since the old infra has huge difference than the new one; reboot is unpreventable.

We are diagnosing an error on one of the new nodes.

We are working on a glitch on this device; there may be multiple reboots.

10% VMs are affected.

problem solved by remove vhost_net experimental_zcopytx=1
It leads to no log system freeze. No kernel panic, no crash log.

Migration has been completed. It might take up to 10 mins to boot all VMs.
The manual boot is also an option.

Emergency maintenance for one of Tokyo nodes.

Discover partial Internet disconnection of LAX; resolved.

The routing issue from HK > CN has been resolved.

Disabled CFMT advance TCP protection for LAX.sPro

It leads to unexplained problems.

The Layer 3 / 4 DDoS filtering on Internet direction has been activated for HKG.T1 HKG.EB HKG.Pro. The RTBH threshold has been increased as well.

Detected CN2 GIA connection issue in Tokyo.

China Telecom interface down.

Already contact China Telecom support.

Confirmed Megaport emergency maintenance.

ETA: 4hrs.

Service has been restored.