Scheduled maintenance
Location: Hong Kong
Duration: Nov 30 - Dec 3, 2023, Hong Kong Time
Duration:
- Less than 30 minutes per hypervisor;
- Max 1hr for VM. (Due to booting sequence.)
Content: Deploy DMIT Full Layer 3 network enhancements. Block ARP, multicast, broadcast, etc. in the customer's network; and provide additional IP address delivery methods: RADVD (IPv6), DHCP (IPv4).
CloudInit will still be the primary method. The SSH Key injection, password update should be updated by this methold.
At Hong Kong location, it saves about ~5GB per month for you.
Location: Hong Kong
Duration: Nov 30 - Dec 3, 2023, Hong Kong Time
Duration:
- Less than 30 minutes per hypervisor;
- Max 1hr for VM. (Due to booting sequence.)
Content: Deploy DMIT Full Layer 3 network enhancements. Block ARP, multicast, broadcast, etc. in the customer's network; and provide additional IP address delivery methods: RADVD (IPv6), DHCP (IPv4).
CloudInit will still be the primary method. The SSH Key injection, password update should be updated by this methold.
At Hong Kong location, it saves about ~5GB per month for you.
The core reason we have to do this is because of ARP is not suit for public cloud.
at Juniper, all ARP response packets are handled by the RE.
a) The CPU usage growth with the network size. It slow down the BGP/OSPF/MPLS/etc. route calculation, also might leads to crash during high load.
at Juniper, the PFE will send ARP if the target in LAN but not in ARP table on each packet received.
b) During the network scanning to all IPv4 we have, it generate huge number of ARP requests when the target does not existed in our network;
at Juniper, the RE has a internal rule to protect it will not be flooded by ARP; it just a simple rate limit.
c) It means the IP on your VM could lost ARP entity because of aging, and ARP rate-limit. It leads to sudden disconnection. (rarely happened, but could be. during full subnet scanning.)
at Juniper, all ARP response packets are handled by the RE.
a) The CPU usage growth with the network size. It slow down the BGP/OSPF/MPLS/etc. route calculation, also might leads to crash during high load.
at Juniper, the PFE will send ARP if the target in LAN but not in ARP table on each packet received.
b) During the network scanning to all IPv4 we have, it generate huge number of ARP requests when the target does not existed in our network;
at Juniper, the RE has a internal rule to protect it will not be flooded by ARP; it just a simple rate limit.
c) It means the IP on your VM could lost ARP entity because of aging, and ARP rate-limit. It leads to sudden disconnection. (rarely happened, but could be. during full subnet scanning.)
/var/log/DMIT-NOC.log
Extend 3 days
Everything is done;
If you have any issues:
- speed slow
- no network
- no IPv6
Stop the server and start it again. ≠ reboot
For IPv6, please login to the server and check the SLAAC assignment.
If you have any issues:
- speed slow
- no network
- no IPv6
Stop the server and start it again. ≠ reboot
For IPv6, please login to the server and check the SLAAC assignment.
/var/log/DMIT-NOC.log
The core reason we have to do this is because of ARP is not suit for public cloud. at Juniper, all ARP response packets are handled by the RE. a) The CPU usage growth with the network size. It slow down the BGP/OSPF/MPLS/etc. route calculation, also might…
After removing ARP from HKG, the CPU load of RE is significantly reduced.
/var/log/DMIT-NOC.log
The core reason we have to do this is because of ARP is not suit for public cloud. at Juniper, all ARP response packets are handled by the RE. a) The CPU usage growth with the network size. It slow down the BGP/OSPF/MPLS/etc. route calculation, also might…
Additionally, after we posted this message, DMIT received an increasing number of network scans of all the IPs we own.
Thanks for take caring of us.
Thanks for take caring of us.
Scheduled maintenance
Location: Tokyo
Duration: Dec 13 - Dec 19, 2023, Japan Standard Time
Duration:
- Less than 30 minutes per hypervisor;
- Max 30 minutes for VM. (Due to booting sequence.)
Content: Deploy DMIT Full Layer 3 network enhancements.
Location: Tokyo
Duration: Dec 13 - Dec 19, 2023, Japan Standard Time
Duration:
- Less than 30 minutes per hypervisor;
- Max 30 minutes for VM. (Due to booting sequence.)
Content: Deploy DMIT Full Layer 3 network enhancements.
/var/log/DMIT-NOC.log pinned «Scheduled maintenance Location: Tokyo Duration: Dec 13 - Dec 19, 2023, Japan Standard Time Duration: - Less than 30 minutes per hypervisor; - Max 30 minutes for VM. (Due to booting sequence.) Content: Deploy DMIT Full Layer 3 network enhancements.»
/var/log/DMIT-NOC.log
CTGnet initial report: The network failure occurred in their security protection system; part of their system does not received the route we announced, which lead to the traffic cannot be transmitted to DMIT. CTGnet already esclated this issue to HQ, and…
SLA Credit has been added as date; The billing date of all VMs in TYO added 1 month (Include EB and Pro.)
Scheduled maintenance
Location: Los Angeles
Duration: Dec 18 - Dec 22, 2023, Pacific Standard Time
Duration:
- Less than 30 minutes per hypervisor;
- Max 30 minutes for VM. (Due to booting sequence.)
Content: Deploy DMIT Full Layer 3 network enhancements.
Location: Los Angeles
Duration: Dec 18 - Dec 22, 2023, Pacific Standard Time
Duration:
- Less than 30 minutes per hypervisor;
- Max 30 minutes for VM. (Due to booting sequence.)
Content: Deploy DMIT Full Layer 3 network enhancements.
/var/log/DMIT-NOC.log pinned «Scheduled maintenance Location: Los Angeles Duration: Dec 18 - Dec 22, 2023, Pacific Standard Time Duration: - Less than 30 minutes per hypervisor; - Max 30 minutes for VM. (Due to booting sequence.) Content: Deploy DMIT Full Layer 3 network enhancements.»
/var/log/DMIT-NOC.log
Scheduled maintenance Location: Tokyo Duration: Dec 13 - Dec 19, 2023, Japan Standard Time Duration: - Less than 30 minutes per hypervisor; - Max 30 minutes for VM. (Due to booting sequence.) Content: Deploy DMIT Full Layer 3 network enhancements.
Done.
If you are using CentOS 7 or Ubuntu 20-;
Please wait 30 mins and STOP/BOOT your VM only if your VM has no network.
If you are using CentOS 7 or Ubuntu 20-;
Please wait 30 mins and STOP/BOOT your VM only if your VM has no network.
/var/log/DMIT-NOC.log
Scheduled maintenance Location: Los Angeles Duration: Dec 18 - Dec 22, 2023, Pacific Standard Time Duration: - Less than 30 minutes per hypervisor; - Max 30 minutes for VM. (Due to booting sequence.) Content: Deploy DMIT Full Layer 3 network enhancements.
Done.
At this site, we performed the most compatible configuration. This is because the site has a large number of legacy configurations. It should be 100% adaptive except for IPv6, which you configure manually.
Few VM are still in booting sequance. Please wait for few minutes.
At this site, we performed the most compatible configuration. This is because the site has a large number of legacy configurations. It should be 100% adaptive except for IPv6, which you configure manually.
Few VM are still in booting sequance. Please wait for few minutes.
For all customers who use their own system copy, and/or want to setup the IPv6 manually without CloudInit.
Please use the following configuration for IPv6.
Then you can assign any IPv6 address to your VM in the /64 we allocated to you in your configuration file.
Please use the following configuration for IPv6.
Then you can assign any IPv6 address to your VM in the /64 we allocated to you in your configuration file.
iface [IF_NAME] inet6 auto
autoconf 1
accept_ra 2
From CTG NOC:
Dear Valued Customer,
According to the trace from TYO.Pro to Mainland China,the latency issue caused by the circuit switched to back path,the switching reason due to sea cable cut on NCP S3,no ETR for repairing. Apologies for any inconvenience.
There is an ongoing attack to our Tokyo PoP;
It aims to all our customers include IP Transit customers.
The attack is sending about 100~400 Mbps UDP too over 400IPs at the same time.
This volume apparently over than our uplink capacity in Tokyo.
DMIT currently moving all the Internet Inbound traffic of Tokyo to San Jose for temporary mitigation. It will be moved back once the DDoS stopped.
It aims to all our customers include IP Transit customers.
The attack is sending about 100~400 Mbps UDP too over 400IPs at the same time.
This volume apparently over than our uplink capacity in Tokyo.
DMIT currently moving all the Internet Inbound traffic of Tokyo to San Jose for temporary mitigation. It will be moved back once the DDoS stopped.
It's been over 12hrs, we received total 110Gbps DNS Reply Reflection on all IPv4 of our Tokyo services. The peak is over 200Gbps.
85G+ from San Jose Mitigation facility
25G+ from Hong Kong Mitigation facility.
DMIT is planing to get 100G access from Cogent, NTT, BBIX, EQIX at Tokyo location to let our Tokyo PoP could have capacity to against with DDoS.
The Tokyo PoP is a minor PoP which has only 5 hypervisor in running.
That's the reason DMIT currently does not offer new order other than Premium. Also it's the reason Tokyo has no much capacity to against DDoS.
We are not able to RTBH single IP to solve this problem; since the attack to the IP is about 400Mbps+ per IP (and up to 800Mbps.). For now you'll experience latency increase temporary but this will not be permanently .
85G+ from San Jose Mitigation facility
25G+ from Hong Kong Mitigation facility.
DMIT is planing to get 100G access from Cogent, NTT, BBIX, EQIX at Tokyo location to let our Tokyo PoP could have capacity to against with DDoS.
The Tokyo PoP is a minor PoP which has only 5 hypervisor in running.
That's the reason DMIT currently does not offer new order other than Premium. Also it's the reason Tokyo has no much capacity to against DDoS.
We are not able to RTBH single IP to solve this problem; since the attack to the IP is about 400Mbps+ per IP (and up to 800Mbps.). For now you'll experience latency increase temporary but this will not be permanently .