It's been over 12hrs, we received total 110Gbps DNS Reply Reflection on all IPv4 of our Tokyo services. The peak is over 200Gbps.
85G+ from San Jose Mitigation facility
25G+ from Hong Kong Mitigation facility.
DMIT is planing to get 100G access from Cogent, NTT, BBIX, EQIX at Tokyo location to let our Tokyo PoP could have capacity to against with DDoS.
The Tokyo PoP is a minor PoP which has only 5 hypervisor in running.
That's the reason DMIT currently does not offer new order other than Premium. Also it's the reason Tokyo has no much capacity to against DDoS.
We are not able to RTBH single IP to solve this problem; since the attack to the IP is about 400Mbps+ per IP (and up to 800Mbps.). For now you'll experience latency increase temporary but this will not be permanently .
85G+ from San Jose Mitigation facility
25G+ from Hong Kong Mitigation facility.
DMIT is planing to get 100G access from Cogent, NTT, BBIX, EQIX at Tokyo location to let our Tokyo PoP could have capacity to against with DDoS.
The Tokyo PoP is a minor PoP which has only 5 hypervisor in running.
That's the reason DMIT currently does not offer new order other than Premium. Also it's the reason Tokyo has no much capacity to against DDoS.
We are not able to RTBH single IP to solve this problem; since the attack to the IP is about 400Mbps+ per IP (and up to 800Mbps.). For now you'll experience latency increase temporary but this will not be permanently .
/var/log/DMIT-NOC.log
From CTG NOC: Dear Valued Customer, According to the trace from TYO.Pro to Mainland China,the latency issue caused by the circuit switched to back path,the switching reason due to sea cable cut on NCP S3,no ETR for repairing. Apologies for any inconvenience.
Kindly be noted, submarine cable NCP S3 is still under repair, ETR is unavailable, the circuit resumed by rerouted, thus the latency will higher than the original path.
/var/log/DMIT-NOC.log
Please be kindly noted that NCP S3 ETR ：2/28.
Kindly be noted due to submarine cable faults, the circuit was rerouted which caused the high delay.
The circuit will revert back to the original route by planned maintenance, however ETR is unavailable.
DMIT noticed the speed issue to China Unicom Guangzhou from LAX Pro CN2 GIA since week ago.
We’ve working on it; it takes time to work with ISP NOC.
The test can only be done during the peak hour.
With the time difference of all parties. The estimate repairs time will be in this week.
We’ve working on it; it takes time to work with ISP NOC.
The test can only be done during the peak hour.
With the time difference of all parties. The estimate repairs time will be in this week.
/var/log/DMIT-NOC.log
DMIT noticed the speed issue to China Unicom Guangzhou from LAX Pro CN2 GIA since week ago. We’ve working on it; it takes time to work with ISP NOC. The test can only be done during the peak hour. With the time difference of all parties. The estimate…
Please have a test and leave us a ticket if there is any issue with this case.
https://www.dmit.io/pages/lax-eyeball
CT > DMIT: - AS4809
CU > DMIT: - AS4134 > AS4809
CM > DMIT: - AS58807
DMIT > CT CU CM: AS58807
CT > DMIT: - AS4809
CU > DMIT: - AS4134 > AS4809
CM > DMIT: - AS58807
DMIT > CT CU CM: AS58807
OpenBeta discount code:
LAX-EB-LAUNCH-NON-MONTHLY-RECURRING-20OFF
www.dmit.io
Now let's set our eyes on Los Angeles. - DMIT
High Performance VM in DMIT.IO Cloud Infrastructure Services
OUTPUT.txt
26.3 KB
LAX.EB:
- Enabled DDoS Always-On for all IPs.
- Traffic counter has been reset.
- Strict rule applied. Inbound network performance might experience impact.
- Local Latency slightly increased.
Attack Model:
A) UDP
Sending 3~10Mbps DNS to each IPv4 in the current /22.
B) TCP
Tiny TCP ACK from actual running Internet server for 1Mbps per IPv4.
- Enabled DDoS Always-On for all IPs.
- Traffic counter has been reset.
- Strict rule applied. Inbound network performance might experience impact.
- Local Latency slightly increased.
Attack Model:
A) UDP
Sending 3~10Mbps DNS to each IPv4 in the current /22.
B) TCP
Tiny TCP ACK from actual running Internet server for 1Mbps per IPv4.
/var/log/DMIT-NOC.log
OUTPUT.txt
Local latency increased because the Internet inbound routing has been redirected to San Jose. There are about 6ms between them.
This is a temporary solution. DMIT has already purchase hardware to upgrade Los Angeles as well.
This is a temporary solution. DMIT has already purchase hardware to upgrade Los Angeles as well.
SJC Maintenance
Start time: April 13, 14:30 EST
Estimated completion time: 30 minutes
Purpose: Switch from KRBD to LibRBD.
If the VM does not come online automatically after completion, please wait patiently or start it by yourself in the control panel. If the current VM is read-only, please reboot it in the panel to recover it.
Start time: April 13, 14:30 EST
Estimated completion time: 30 minutes
Purpose: Switch from KRBD to LibRBD.
If the VM does not come online automatically after completion, please wait patiently or start it by yourself in the control panel. If the current VM is read-only, please reboot it in the panel to recover it.
/var/log/DMIT-NOC.log
Local latency increased because the Internet inbound routing has been redirected to San Jose. There are about 6ms between them. This is a temporary solution. DMIT has already purchase hardware to upgrade Los Angeles as well.
The inbound from China to DMIT is remained the same.
The hardware and circuit scale up will be ready between late April ~ May.
The hardware and circuit scale up will be ready between late April ~ May.
Observed IO error from Ceph in Hong Kong location.
It has been resolved.
It has been resolved.
HKG Maintenance
Start time: UTC+8 2024/04/19 02:00
Scope of Impact: All HKG hosts
Purpose: Upgrade NICs to CX5 Dual Port 100G, resolve high load issues on specific nodes, and upgrade server firmware.
Estimated Completion Time: Approx. 30 minutes per host
Start time: UTC+8 2024/04/19 02:00
Scope of Impact: All HKG hosts
Purpose: Upgrade NICs to CX5 Dual Port 100G, resolve high load issues on specific nodes, and upgrade server firmware.
Estimated Completion Time: Approx. 30 minutes per host
/var/log/DMIT-NOC.log pinned «HKG Maintenance Start time: UTC+8 2024/04/19 02:00 Scope of Impact: All HKG hosts Purpose: Upgrade NICs to CX5 Dual Port 100G, resolve high load issues on specific nodes, and upgrade server firmware. Estimated Completion Time: Approx. 30 minutes per host»
/var/log/DMIT-NOC.log
HKG Maintenance Start time: UTC+8 2024/04/19 02:00 Scope of Impact: All HKG hosts Purpose: Upgrade NICs to CX5 Dual Port 100G, resolve high load issues on specific nodes, and upgrade server firmware. Estimated Completion Time: Approx. 30 minutes per host
All completed. The VMs are booting in sequence.
DIRECT MESSAGE FROM RETN.NET [HKG]
Failure Notification
Description:
Our regional capacity is significantly reduced due to submarine fiber cuts in the Asia region.
During peak hours, multiple destinations may experience overutilization, resulting in packet loss and increased RTT.
Currently, we have registered the following emergency cases:
- Subsea fiber cut on Hong Kong - Taipei section. Started: 15.02 | ETR: 30.04
- Subsea fiber cut on Milan - Singapore section. Started: 24.02 | ETR: 19.04
- Subsea fiber cut on Hong Kong - Singapore section. Started: 10.03 | ETR: 13.05.
- Subsea fiber cut on Hong Kong - Taipei section. Started: 13.03 | approximate ETR: April END.
– Subsea fiber cut on Hong Kong - Singapore section. Started: 08.03 | ETR: 18.05.
- Subsea fiber cut on Hong Kong - Seoul section. Started: 03.04 | ETR: Unknown.
- Emergency situation on the Marseille - Singapore section. Start: 19.04 | Expected recovery: Unknown
- Emergency situation on the Sofia - Singapore section. Start: 19.04 | Expected recovery: Unknown
Failure Notification
Description:
Our regional capacity is significantly reduced due to submarine fiber cuts in the Asia region.
During peak hours, multiple destinations may experience overutilization, resulting in packet loss and increased RTT.
Currently, we have registered the following emergency cases:
- Subsea fiber cut on Hong Kong - Taipei section. Started: 15.02 | ETR: 30.04
- Subsea fiber cut on Milan - Singapore section. Started: 24.02 | ETR: 19.04
- Subsea fiber cut on Hong Kong - Singapore section. Started: 10.03 | ETR: 13.05.
- Subsea fiber cut on Hong Kong - Taipei section. Started: 13.03 | approximate ETR: April END.
– Subsea fiber cut on Hong Kong - Singapore section. Started: 08.03 | ETR: 18.05.
- Subsea fiber cut on Hong Kong - Seoul section. Started: 03.04 | ETR: Unknown.
- Emergency situation on the Marseille - Singapore section. Start: 19.04 | Expected recovery: Unknown
- Emergency situation on the Sofia - Singapore section. Start: 19.04 | Expected recovery: Unknown
China Unicom (the Claimant) was unsuccessful in substantiating their claims in the arbitration case against DMIT (the Respondent) for the provision of services AS4837 and AS10099, valued at half a million US Dollars. Ultimately, it was determined that CUA should reimburse DMIT $x2,880.xx(5 digits) USD in the practical terms.
Initially, DMIT had entered into a significant contract with CUA for close to half a million dollars with the aim of joint development and exploring new market opportunities.
However, our plans changed due to unforeseen issues. CUA failed to fulfill the agreed-upon services and unilaterally altered the order, leading to the failed of DMIT's operations in San Jose and degrading our service quality in Tokyo.
CUA denied all liability and sought an arbitration ruling to compel DMIT to reimburse CUA for the full contract value.
As a leading provider of optimized routing solutions from the US to China, DMIT is disheartened by these events and the unreasonable business practices adopted by CUA. We regret to inform that we probably will no longer be working with CUA moving forward.
DMIT wants to share this story to all our partners; stay alert.
Initially, DMIT had entered into a significant contract with CUA for close to half a million dollars with the aim of joint development and exploring new market opportunities.
However, our plans changed due to unforeseen issues. CUA failed to fulfill the agreed-upon services and unilaterally altered the order, leading to the failed of DMIT's operations in San Jose and degrading our service quality in Tokyo.
CUA denied all liability and sought an arbitration ruling to compel DMIT to reimburse CUA for the full contract value.
As a leading provider of optimized routing solutions from the US to China, DMIT is disheartened by these events and the unreasonable business practices adopted by CUA. We regret to inform that we probably will no longer be working with CUA moving forward.
DMIT wants to share this story to all our partners; stay alert.
DMIT observed CN2 end user routes is now bypassing AS23764 since 3 days ago.
AS4809 is customer of AS1299 in US and EU; but AS23764 is not. This lead to the latency increase recently. DMIT will try our best to fix this ASAP.
AS4809 is customer of AS1299 in US and EU; but AS23764 is not. This lead to the latency increase recently. DMIT will try our best to fix this ASAP.
Forwarded from Austin C.
Standard.inet.0: 946707 destinations, 2465729 routes (946093 active, 0 holddown, 467099 hidden)
+ = Active Route, - = Last Active, * = Both
58.32.0.0/17 *[BGP/170] 11:39:07, MED 154110, localpref 100
AS path: 174 23764 4809 4812 I, validation-state: unknown
> to _ via _
[BGP/170] 3d 22:49:21, MED 0, localpref 100
AS path: 1299 3356 23764 4809 4812 I, validation-state: unknown
> to _ via _
+ = Active Route, - = Last Active, * = Both
58.32.0.0/17 *[BGP/170] 11:39:07, MED 154110, localpref 100
AS path: 174 23764 4809 4812 I, validation-state: unknown
> to _ via _
[BGP/170] 3d 22:49:21, MED 0, localpref 100
AS path: 1299 3356 23764 4809 4812 I, validation-state: unknown
> to _ via _