/var/log/DMIT-NOC.log
4.58K subscribers
187 photos
6 files
117 links
Download Telegram
/var/log/DMIT-NOC.log
Los Angeles Planned Maintenance Date: Oct 16, 2025 ~ Oct 24, 2025 Time: 11AM - 6PM Duration: Less than one hour per instance Expected Behavior: - Server fingerprint changes - Server restarts Content: Surprise upgrades
🇺🇸 Los Angeles Computing Node Expansion Completed

The Los Angeles (LAX) computing node expansion and upgrade have been successfully completed!
As part of our ongoing rolling hardware update strategy, all products are now organized into three hardware tiers:

🧩 Los Angeles (LAX) Platform Structure
🧱 Next Platform (AN5) — latest AMD EPYC 9005 series, cutting-edge performance
🧱 Stable Platform (AN4) — AMD EPYC 9004 series, proven reliability & value
🧱 LTS Platform — ultra-stable, long-term supported hardware (coming later)

🔥 Upgraded to AN5 (Next Platform)
The following LAX packages have been migrated to AN5 for enhanced CPU & network performance:

LAX.AN5.Pro.TINY
LAX.AN5.Pro.Pocket
LAX.AN5.Pro.STARTER
LAX.AN5.Pro.MINI
LAX.AN5.Pro.MICRO
LAX.AN5.Pro.MEDIUM
LAX.AN5.Pro.LARGE
LAX.AN5.Pro.GIANT
LAX.AN5.Pro.SHARE
LAX.AN5.Pro.SHAREv2
LAX.AN5.Pro.BUILDER
LAX.AN5.Pro.BUILDERv2
LAX.AN5.EB.TINY
LAX.AN5.EB.Pocket
LAX.AN5.EB.STARTER
LAX.AN5.EB.MINI
LAX.AN5.EB.MICRO
LAX.AN5.EB.MEDIUM
LAX.AN5.EB.LARGE
LAX.AN5.EB.GIANT

🧠 Other LAX packages will remain on AN4 (Stable Platform) for now.

🌏 Asia-Pacific Plans:
Upgrades for HKG and TYO nodes are in the works.

💡 All renewals, pricing, and discounts remain unchanged for existing users.
We have updated our AUP (Acceptable Use Policy)

1. USE OF SERVICES section

By way of example, and not as a limitation, you agree that when you (or your end users are) using a Service, you (and your end users) will not:
...
(Added)
- Engaging in or facilitating any form of fraud, phishing, impersonation, or deceptive activity, including the creation or distribution of fake websites, emails, or communications intended to mislead others, obtain credentials, or gain unauthorized access, as well as hosting or linking to phishing sites, operating scams, or other fraudulent schemes.


2. TERMINATION/ACCESS RESTRICTION section

Deleted:

At the first time, we will suspend VM for a week. At the second time, we will terminate VM directly

Added:
Depending on the severity of the violation, DMIT may implement a service suspension or null routing action for a period of 7 to 30 days, or proceed with immediate service termination.
/var/log/DMIT-NOC.log pinned «We have updated our AUP (Acceptable Use Policy) 1. USE OF SERVICES section By way of example, and not as a limitation, you agree that when you (or your end users are) using a Service, you (and your end users) will not: ... (Added) - Engaging in or facilitating…»
We have updated our AUP (Acceptable Use Policy) and TOS (Term Of Service)

1. AUP (Acceptable Use Policy) | USE OF SERVICES

By way of example, and not as a limitation, you agree that when you (or your end users are) using a Service, you (and your end users) will not:
...
In the event any instance uses excessive CPU from activities such as cryptocurrency CPU mining, DMIT.IO reserves the right to limit the CPU available to your instances in order to maintain a consistent level of performance on all our nodes. DMIT.IO do not accept long-term occupation of the entire VM allocation rate if the service type is traffic-limited. DMIT.IO reserves the right to put the restrict on the VM Internet Ports. (Deleted)

2. AUP (Acceptable Use Policy) | RESOURCE LIMITATION (Added)
To ensure the stable operation of services and uphold the principle of fair resource usage, DMIT.IO establishes the following policies governing the use of computing and network resources.

- In the event any instance uses excessive CPU from activities such as cryptocurrency CPU mining, DMIT.IO reserves the right to limit the CPU available to your instances in order to maintain a consistent level of performance on all our nodes.
- All products are guaranteed 50% CPU usage by default unless otherwise stated.
- When DMIT.IO has enough and stable virtualization computing resources in the region, and the client instance’s operating system or software is not crashed, DMIT.IO accepts long-term and reasonable high-load usage of computing resources.
However, if the workload or software behavior causes abnormal or excessive CPU consumption due to inefficient or inappropriate program design, DMIT.IO may take actions to limit CPU max performance in order to protect overall node performance and service stability.
- DMIT.IO will actively balance VM/Node loads to maximize resource allocation, and will only implement CPU limitation measures when short-term resource expansion is not feasible.
- DMIT.IO reserves the right to restrict VM Internet Ports in cases where such activities affect the overall stability of the network.


3. TOS (Term Of Service) | Action of Abuse Take Down Request (Deleted)
* This section is detailed in AUP (Acceptable Use Policy).
LAX.Pro:
Since CMIN2 failed to improve backbone packet loss, DMIT temporarily rerouted the inbound traffic from China Mobile for LAX.Pro to the internet. The return routing remains the same.
Latency increased, but packet loss dropped.
The IPv6 remain unchanged.

LAX.EB remain on CMIN2 for inbound until the packet loss is unbearable over than acceptable time.

Reference Link for LAX.Pro
Latency: https://ping.nxtrace.org/goto/iRgzvmmDR?orgId=1
Packet Loss: https://ping.nxtrace.org/goto/zKaiDimDR?orgId=1

Reference Link for LAX.Pro.IPv6
Latency: https://ping.nxtrace.org/goto/c4lWDimvg?orgId=1
Packet Loss: https://ping.nxtrace.org/goto/Iw3GDmmDR?orgId=1
DMIT has donated US$5,000 to the Hong Kong SAR to support relief efforts following the recent severe fires in Hong Kong. We hope this contribution could help affected residents navigate this difficult time.
GSL AS137409 severe packet loss; detour in temporary.
DMIT LAX Network Failure Analysis

At approximately 19:35:00 Pacific Time, DMIT deployed a change within the LAX metro to introduce IPv6 over MPLS and IS-IS for the access switches.

1. DMIT uses loopback addresses for iGP routing on all devices.
2. However, in the IPv6 RR configuration, we did not standardize the next-hop for IPv6 routes received from access switches, meaning Next-Hop was not changed to Peer-Address, it remain the final interface address.
3. Due to iBGP behavior, next-hop addresses will not not automatically converted to Peer-Address.
4. To prevent certain customers from using reserved IPv4/IPv6 addresses as Point-to-Point (PtP) interface address, DMIT's internal network does not propagate specific port addresses. (This made we have to change next-hop for all iBGP routes).
5. The above things make the edge router cannot resolve the acual next-hop to final interface.
5. When DMIT's border router fails to find a specific next-hop, it falls back to a Transit table.
6. On the Transit table, the route in FIB was programmed to the customer table.

These factors collectively caused IPv6 traffic originating from customers to continuously loop through multiple VRFs on a single router until the 128 TTL expired.

This ultimately exhausted backplane bandwidth, resulting in RR disconnections. When RR disconnected, custoemr routing was interrupted, loop traffic dropped, and the network recovered briefly before the looping failure recurred.

This configuration fault caused 3 minutes of downtime and a cumulative 13 minutes of degraded service. DMIT sincerely apologizes for this incident.
We have updated our TOS (Term Of Service)

1. TOS (Term Of Service) | Refund Policy

19.1.1 Full refunds are available in compliance with the rules below as well as 19.3.
1.The service is purchased for no more than 3 days;
1.The service is purchased for no more than 3 days (New order only);

19.2.1 Partial refunds are available in compliance with the rules below as well as 19.3.
1. The service is purchased for no more than 30 days;
1. The service is purchased for no more than 30 days (New order only);


19.3.1 We will not refund to the payment gateway under the following circumstances

1. The order is a renewal order/invoice, and the payment has been successfully processed; (Added)
2. The invoice was paid using existing account credit (refunds can only be returned to the account credit);
(Added)
3. The order/invoice is to add funds to the account credit;
(Added)

2. TOS (Term Of Service) | IP Replacement Policy

20.3 For Tier 1(Standard) network profile:
1. Without IP Guarantee+ Addon: DMIT do not give guarantee the IP is globally accessible for the new order, especially for China, Russia, and all country has national network censorship.
1. Monthly Billing Cycle: DMIT do not give guarantee the IP is globally accessible for the new order, especially for China, Russia, and all country has national network censorship.

2. With IP Guarantee+ Addon: Guaranteed first connection in sensitive areas.
2. Non-Monthly Billing Cycle: Guaranteed first connection in sensitive areas.
LAX Pro:
CN2 AS4809 DRT single point failure. Waiting CT to response.
CN2 misconfigured the interface policer in CSLA failover path lead to the congestion.
HKG:
Observed some host node experienced packet loss.
NOC is working on it
/var/log/DMIT-NOC.log
LAX Pro: CN2 AS4809 DRT single point failure. Waiting CT to response.
DMIT already esclated this issue to highest level of support.
As current information. CTG NOC didn't delivery the service as contract incl. specific BGP configuration and interface rates.
CTG HKG BGP session misconfiguration leads to session reset.
DMIT is esclating this issue with CT Group.

Please wait for the futher response.
Both CN2 LAX and HKG:

China Telecom didn't configure the BGP sessions and interfaces correctly.
The service auto-failover triggered a cascade of failures.

Plus, CTG NOC isn't helpful at all. The esclated NOC team isn't responding.
The HKG CTG CN2 session is recovered.
The routing will be restored soon.

We observed DDoS come in and targeted to 3 of our HKG.Pro subnet which might leads some customer experience packet loss.

The longer than expected downtime are caused by none reponse from CTGnet NOC.
DMIT work with CTG NOC since 9AM EST today. But the issue still not been fully resolved.

The service failure notice will be published once we secured all serivces.
Update:

LAX:
We've connected with CTG NOC and CT Group.
There will be an conference soon for this emergency.

HKG:
The DDoS is relieved. We are diagnosing for the issue on the mitigation facility. The DDoS were mostly filtered. But we observed there are some performance issue on the mitigation device. Some packet loss were caused by the mitigation devices instead of DDoS itselves.
We've used remote DDoS mitigation facilities and applied rule on the IP Transit. So far, it's stable, but we are still working on fix anything that not work as expected.