/var/log/DMIT-NOC.log
4.73K subscribers
189 photos
6 files
117 links
Download Telegram
/var/log/DMIT-NOC.log
Both solved. DMIT NOC will continuously focus on this. Reason: ISP broken the fiber when they removed one of our non-used local-loop service at July 28. NOC is now rebooting the Core Router.
All existing problems have been completely solved.
————
DMIT is preparing various todo of LAX data center recently, including expanding the data center, bandwidth, and upgrading our architecture to be compatible with customers transferred from other IDC. Our apology about the inconvenience.
————
Multiple reasons we gave before are not the critical one lead to packet loss.

The most critical reason for packet loss:
DMIT HKG has 4(four) routing tables, one for standard routing, one for economical routing, one for premium routing, and the last one for peering.

To save FIB, DMIT has applied a static IPv6 default route in the premium and economical table that points to the standard(default) table. DMIT did not remove the default route when copy the client route from premium/economical to the default table. It made a hidden loop route. There will be a problem if any client in the network is trying to transmit the packet to a null route IPv6 destination. Before the TTL is expired, the package will be transmitted between each table infinity. Since the packet was transmitted in the fabric, it's complicated to find where the problem is because the bandwidth of fabric was not shown on our monitoring system. These packets can be transmitted 254 times before dropped. In theory, this traffic will be amplified by 254 times.

These are the issues lead to congestion on the fabric and LUCHIP and caused packet loss.
Giant && Tiny~~

PVM.LAX.Pro.SHAREv2
10OFF for Annually payment
Code: I-HAVE-READ-TOS-SHAREv2

PVM.LAX.Pro.GIANT
8c 16G 100G-SSD 44T@300M/1G
10OFF for non-monthly payment
Code: Welcome-GIANT

PVM
.LAX.Pro.TINY
1c 1G 10G-SSD 550G@200M/1G
48.88 Semi-Annually
88.88 Annually

https://www.dmit.io/cart.php?gid=9
Once per client. ~
/var/log/DMIT-NOC.log pinned «Giant && Tiny~~ PVM.LAX.Pro.SHAREv2 10OFF for Annually payment Code: I-HAVE-READ-TOS-SHAREv2 PVM.LAX.Pro.GIANT 8c 16G 100G-SSD 44T@300M/1G 10OFF for non-monthly payment Code: Welcome-GIANT PVM.LAX.Pro.TINY 1c 1G 10G-SSD 550G@200M/1G 48.88 Semi-Annually…»
Notes:

DMIT is now trying to migrate all PVM.LAX.Lite VMs to the AMD EPYC platform. There is a heavy task load on the background. Also, there is a massive amount of new server delivery tasks for clients that transferred from another IDC.

It takes time if you try to reinstall your VM right now. We suggest you do reinstall after today.

BR,
NOC.
We have sent maintenance notice to all HKG customers by email days ago. The maintenance this week may affect your services, please read the email for detail and be prepared accordingly. We apologize for inconvenience.

BR,
NOC
/var/log/DMIT-NOC.log pinned «We have sent maintenance notice to all HKG customers by email days ago. The maintenance this week may affect your services, please read the email for detail and be prepared accordingly. We apologize for inconvenience. BR, NOC»
90% of HKG servers are migrated finished.
The HK-CN connection already recovered.

There is an issue on some VMs; Our engineers will fix it ASAP.
/var/log/DMIT-NOC.log
90% of HKG servers are migrated finished. The HK-CN connection already recovered. There is an issue on some VMs; Our engineers will fix it ASAP.
100%

Please remove the ISO attachment if you cannot boot your server.
Our tech support will migrate your server to another host later.
The reinstallation is not available. It will be available soon.
There is an emergency maintenance in the HKG datacenter. The network will be interrupted several times.
Due to Executive Order 13873 from the White House, our payment gateway vendor “Stripe” suspended the access from the US customers on WeChat Pay. As a US-based company, DMIT Inc will not cooperate with alternative payment gateway vendors to accept WeChat Pay recently.

DMIT Inc suggested the WeChat Pay client migrate to Alipay to avoid overdue invoices.

Thanks.
/var/log/DMIT-NOC.log pinned «Due to Executive Order 13873 from the White House, our payment gateway vendor “Stripe” suspended the access from the US customers on WeChat Pay. As a US-based company, DMIT Inc will not cooperate with alternative payment gateway vendors to accept WeChat Pay…»
NOC Logs & new infos:

A> DMIT NOC completes the CWDM system upgrade in our LAX location.
DMIT now have 180Gbps capacity between Digital Realty and CoreSite LA1/LA2.

B> Based on this, DMIT now connects AS2914 (NTT) as one of our new IP Transit ISP in the LAX location. The connection is already available for all LAX networks.

C> DMIT will apply one XGE backbone between our HKG and LAX location. It can help our customers get high-quality network between the HKG and LAX location.

D> As you can see, DMIT removed CeRa (AS40065) and Voxility (AS3223) recently; and the premium Anti-DDoS becomes unavailable to choose. This is because DMIT is now choosing Cloudflare (AS13335) as our Anti-DDoS partner. It has a batter network, capacity, and firewall than our previous partner (AS3223). The new product will be released soon. All the current Premium Anti-DDoS IP will be automatically migrated to the new network profile soon. Please contact us if you don't like this, our sales team will help you to cancel the extra Premium Anti-DDoS IP.

*The Anti-DDoS will only available for our premium network due to the transit price.
There will be an upgrade on the host firewall today, which improves PPS performance, fix the error on customized firewall rule, and a long-term defect on Proxmox: unable to use some transparency tunnel protocols when the firewall is enabled ( but it has to be enabled in the public cloud )

There should be no impact on the network, but some flaps. Our system will reboot your VM if there are some changes on your VM that need to reboot to proceed.

After this, we will re-enable the cloud firewall function soon.

Time: 2PM-6PM Sep 29, 2020 EDT
The on-site engineer has emergency maintenance, it should be done without any downtime. We are looking forward to the detail. There are 6 nodes face a downtime. Please wait for a while.
LAX Location emergency maintenance:
Core Router reconfiguration.
This email serves as official notification that Zayo and/or one of its providers will be performing maintenance on its network as described below. This maintenance may affect services you have with us.

1st Activity Date
27-Oct-2020 00:01 to 27-Oct-2020 05:00 ( Pacific )
27-Oct-2020 07:00 to 27-Oct-2020 12:00 ( GMT )

Reason for Maintenance: Zayo will implement maintenance to perform software upgrade to router mpr1.lax12.us to SCBE2s

Expected Impact: Service Affecting Activity: Any Maintenance Activity directly impacting the service(s) of customers. Service(s) are expected to go down as a result of these activities.

DMIT Comment: It might impacts DMIT's Internet connectivity for a while (up to 2Gbps)
But DMIT has NTT, Cogent to keep reachability of our network,