/var/log/DMIT-NOC.log
4.73K subscribers
189 photos
6 files
117 links
Download Telegram
Due to Executive Order 13873 from the White House, our payment gateway vendor “Stripe” suspended the access from the US customers on WeChat Pay. As a US-based company, DMIT Inc will not cooperate with alternative payment gateway vendors to accept WeChat Pay recently.

DMIT Inc suggested the WeChat Pay client migrate to Alipay to avoid overdue invoices.

Thanks.
/var/log/DMIT-NOC.log pinned «Due to Executive Order 13873 from the White House, our payment gateway vendor “Stripe” suspended the access from the US customers on WeChat Pay. As a US-based company, DMIT Inc will not cooperate with alternative payment gateway vendors to accept WeChat Pay…»
NOC Logs & new infos:

A> DMIT NOC completes the CWDM system upgrade in our LAX location.
DMIT now have 180Gbps capacity between Digital Realty and CoreSite LA1/LA2.

B> Based on this, DMIT now connects AS2914 (NTT) as one of our new IP Transit ISP in the LAX location. The connection is already available for all LAX networks.

C> DMIT will apply one XGE backbone between our HKG and LAX location. It can help our customers get high-quality network between the HKG and LAX location.

D> As you can see, DMIT removed CeRa (AS40065) and Voxility (AS3223) recently; and the premium Anti-DDoS becomes unavailable to choose. This is because DMIT is now choosing Cloudflare (AS13335) as our Anti-DDoS partner. It has a batter network, capacity, and firewall than our previous partner (AS3223). The new product will be released soon. All the current Premium Anti-DDoS IP will be automatically migrated to the new network profile soon. Please contact us if you don't like this, our sales team will help you to cancel the extra Premium Anti-DDoS IP.

*The Anti-DDoS will only available for our premium network due to the transit price.
There will be an upgrade on the host firewall today, which improves PPS performance, fix the error on customized firewall rule, and a long-term defect on Proxmox: unable to use some transparency tunnel protocols when the firewall is enabled ( but it has to be enabled in the public cloud )

There should be no impact on the network, but some flaps. Our system will reboot your VM if there are some changes on your VM that need to reboot to proceed.

After this, we will re-enable the cloud firewall function soon.

Time: 2PM-6PM Sep 29, 2020 EDT
The on-site engineer has emergency maintenance, it should be done without any downtime. We are looking forward to the detail. There are 6 nodes face a downtime. Please wait for a while.
LAX Location emergency maintenance:
Core Router reconfiguration.
This email serves as official notification that Zayo and/or one of its providers will be performing maintenance on its network as described below. This maintenance may affect services you have with us.

1st Activity Date
27-Oct-2020 00:01 to 27-Oct-2020 05:00 ( Pacific )
27-Oct-2020 07:00 to 27-Oct-2020 12:00 ( GMT )

Reason for Maintenance: Zayo will implement maintenance to perform software upgrade to router mpr1.lax12.us to SCBE2s

Expected Impact: Service Affecting Activity: Any Maintenance Activity directly impacting the service(s) of customers. Service(s) are expected to go down as a result of these activities.

DMIT Comment: It might impacts DMIT's Internet connectivity for a while (up to 2Gbps)
But DMIT has NTT, Cogent to keep reachability of our network,
We found something wrong in HKG, on investigation
/var/log/DMIT-NOC.log
We found something wrong in HKG, on investigation
Since we are installing the brand new DL325Gen10 with the AMD EPYC server, our on-site engineer needs to move our backup switch to an appropriate location.

Because the backup switch has no enough space but we have to install it at the last time we migrate our servers from TGT to Equinix. So, we did it: Our on-site install it at an inappropriate location.

Also, the datacenter offered us a Cisco unsupported transceiver and install it at the uplink of the main switch.

Here we go,~ We lost the backup switch due to move location. Then, we already lost the uplink of the master switch. Boom~

uhmm, Yes we do have an infrastructure monitoring platform, but due to migration at the last time, we haven't restored the connection with the new network and infrastructure monitoring platform.
BAAM~~ These servers are ready to be installed today. All HKG VMs will be migrated to the new nodes later.
There will be a high availability transformation and upgrading for the LAX datacenter. The switch will be reboot times.
10~20min unstable network.
/var/log/DMIT-NOC.log
BAAM~~ These servers are ready to be installed today. All HKG VMs will be migrated to the new nodes later.
There are already over 400+ VM running well in our HKG EPYC with the Ceph cluster. Your data will be live migrated to the Ceph HA cluster and AMD node. Please try to backup your data to prevent any unexpected event.

As we expected, it is smooth progress. But, please try to backup your data to prevent any unexpected event.

But we cannot live to migrate your VM status from the previous-generation node to the new one, we have to reboot your VM at the last stage. So, your HKG VM might face to 1~10min downtime when we transfer you to the EPYC node.
We will start to do this weekend; end before next Tuesday. (EDT)
/var/log/DMIT-NOC.log
There will be a high availability transformation and upgrading for the LAX datacenter. The switch will be reboot times. 10~20min unstable network.
We will gradually transfer your VM from the stand-alone storage file system to our Ceph HA storage. DMIT LAX cluster used the zfs file system which allows us to live to migrate your disk fille to Ceph.

There should have only an IO delay during migration on your VM. Because our system needs to take a snapshot and transfer it to the Ceph cluster, during the transfer, there will have an IO hold to ensure data consistency.

As we expected, it is smooth progress. But, please try to backup your data to prevent any unexpected event.

This progress starts this weekend and will be ended at end of this month.
There is 2(Two) lite-impact maintenance notice for HKG and LAX.
/var/log/DMIT-NOC.log pinned «There is 2(Two) lite-impact maintenance notice for HKG and LAX.»