Fault report of HKG-FBBJ762
How to resolve: STOP and BOOT your VM.
In the HKG datacenter, there were some legacy guests VM without IOPS limitation. These VMs used too much IOPS abnormally, which leads to the ZFS filesystem has no respond. This fault makes QEMU get into pause status.
We already move these VM out and update the disk configuration.
There is an architecture upgrade for HKG at late Q3, 2020;
After investigation, there is nothing wrong with the LAX datacenter. If your VM has no response, please check the global accessibility first. It probably is blocked by a local firewall.
How to resolve: STOP and BOOT your VM.
In the HKG datacenter, there were some legacy guests VM without IOPS limitation. These VMs used too much IOPS abnormally, which leads to the ZFS filesystem has no respond. This fault makes QEMU get into pause status.
We already move these VM out and update the disk configuration.
There is an architecture upgrade for HKG at late Q3, 2020;
After investigation, there is nothing wrong with the LAX datacenter. If your VM has no response, please check the global accessibility first. It probably is blocked by a local firewall.
/var/log/DMIT-NOC.log
If your PVM.LAX.Pro VM is in the Last Decade Generation Intel Xeon Platform (Xeon E5-4657L v2 with 1333Mhz DDR3); please go ahead to contact us for migrating your VM to the Next Generation Platform (AMD 2nd Gen EPYC 7402P with 2400Mhz DDR4); It's all free.…
DMIT has enough AMD EPYC nodes to host all PVM.LAX.Pro; DMIT NOC & Tech Support Team will start to migrate PVM.LAX.Pro to EPYC in this month.
We have received complaints from customers, from us to Digital Ocean (Hereinafter referred to as DO) SFO, there are packet losses or unreachable situations.
After NOC investigation, this problem should be caused by the DO IX Port & Upstream congestion.
DMIT might restore it to IX connection if DO accept peer with us. DO advise the routes that next-hop is unreachable in IX. To prevent this, DMIT rejects all routes in IX from DO.
After NOC investigation, this problem should be caused by the DO IX Port & Upstream congestion.
DMIT might restore it to IX connection if DO accept peer with us. DO advise the routes that next-hop is unreachable in IX. To prevent this, DMIT rejects all routes in IX from DO.
👀Affiliate upgrade:
🎊🎊🎊
10% Recurring for all PVM.LAX.Pro orders until forever.
👍🏻🙋🏻♂️💪🏻
👩🏻💻🧑🏻💻👨🏻💻: BTW please don’t ref yourself🤙🏻 DMIT has affiliate anti-fraud rule and detection program.
Just in case, the fraud affiliate will lead your original order to become non-refundable.
🎊🎊🎊
10% Recurring for all PVM.LAX.Pro orders until forever.
👍🏻🙋🏻♂️💪🏻
👩🏻💻🧑🏻💻👨🏻💻: BTW please don’t ref yourself🤙🏻 DMIT has affiliate anti-fraud rule and detection program.
Just in case, the fraud affiliate will lead your original order to become non-refundable.
/var/log/DMIT-NOC.log pinned «👀Affiliate upgrade: 🎊🎊🎊 10% Recurring for all PVM.LAX.Pro orders until forever. 👍🏻🙋🏻♂️💪🏻 👩🏻💻🧑🏻💻👨🏻💻: BTW please don’t ref yourself🤙🏻 DMIT has affiliate anti-fraud rule and detection program. Just in case, the fraud affiliate will lead your original…»
🕦 & Quantity Limited Promotion Code:
EPYC-CELEBRATE
25%OFF Discount
* Monthly Available;
* SHAREv2 is not included;
* New order only;
* One time each client.
Everyone should already get it via Email 📧. Just for a reminder.
EPYC-CELEBRATE
25%OFF Discount
* Monthly Available;
* SHAREv2 is not included;
* New order only;
* One time each client.
Everyone should already get it via Email 📧. Just for a reminder.
50%off && 40%off discount for PVM.LAX.Lite
50%DISCOUNTLITE
* Time Limited;
* Life Time Promotion;
* Quarterly, Semi-Annually, Annually payment is required.
** Require you have at least one DMIT LAX Product in your account.
40%DISCOUNTLITE
* Accepting All Payment Cycle;
* Available for any customer;
* Life Time.
50%DISCOUNTLITE
* Time Limited;
* Life Time Promotion;
* Quarterly, Semi-Annually, Annually payment is required.
** Require you have at least one DMIT LAX Product in your account.
40%DISCOUNTLITE
* Accepting All Payment Cycle;
* Available for any customer;
* Life Time.
Our sales team agree to remove the exists product requirement for code 40%DISCOUNTLITE;
Since this is a flash promotion; Sorry for bothering you times. This is the finial version; Please feel free to contact us for more information.
Since this is a flash promotion; Sorry for bothering you times. This is the finial version; Please feel free to contact us for more information.
DMIT now released a new product to help customers to have large China download bandwidth at a low price.
It calls PVM.LAX.rPro.Customize;
You could get China download bandwidth down to 0.75USD/Mbps in LAX, USA; up to 2Gbps.
NEW!rPRO!40% : No limit;
NEW!rPRO!50% : Non-monthly, exists DMIT LAX customer
With promotion code and annually payment; the bandwidth only costs 0.375USD/Mbps.
General Configuration:
1vCore, 1G RAM, 10G SSD, 100Mbps, 1 IPv4 Address:
With 50% code
Monthly: $59.99 USD; ~0.6USD/Mbps
Annually: $538.99 USD; ~0.45/Mbps
Large Configuration:
2vCore, 2G RAM, 20G SSD, 2000Mbps, 1 IPv4 Address:
With 50% code
Monthly: $889.24 USD; ~0.45USD/Mbps
Annually: $8012.99 USD; ~0.33/Mbps
https://www.dmit.io/cart.php
Simple Notice: This product is NOT for you to send traffic to China; even the routing from China and to China is going through
It calls PVM.LAX.rPro.Customize;
You could get China download bandwidth down to 0.75USD/Mbps in LAX, USA; up to 2Gbps.
NEW!rPRO!40% : No limit;
NEW!rPRO!50% : Non-monthly, exists DMIT LAX customer
With promotion code and annually payment; the bandwidth only costs 0.375USD/Mbps.
General Configuration:
1vCore, 1G RAM, 10G SSD, 100Mbps, 1 IPv4 Address:
With 50% code
Monthly: $59.99 USD; ~0.6USD/Mbps
Annually: $538.99 USD; ~0.45/Mbps
Large Configuration:
2vCore, 2G RAM, 20G SSD, 2000Mbps, 1 IPv4 Address:
With 50% code
Monthly: $889.24 USD; ~0.45USD/Mbps
Annually: $8012.99 USD; ~0.33/Mbps
https://www.dmit.io/cart.php
Simple Notice: This product is NOT for you to send traffic to China; even the routing from China and to China is going through
AS4809 GIA network; in alternative, it helps you to get content from China in LAX. Also, it has unmetered bandwidth.Our team recently found several abnormal price renewal invoices; This is caused by the users use the bug in the DMIT billing system.
At present, DMIT has deactivated the renewal function. Please use PayPal auto payment or bind the credit card in the DMIT system if you worried about forgot to pay.
DMIT used Stripe as our Credit Card payment gateway. DMIT server, credit card processing is meeting the PCI requirement. DMIT will not save your CC information. It is directly passed to Stripe.
We have received many complaints about invoices and fraud caused by DMIT Product transfer. This is a major reason that we deactivate it.
Please note: If you find a very cheap DMIT server; In most cases, it is from PayPal fraud, credit card fraud, or DMIT system bug. According to the DMIT Service Agreement (ToS), DMIT has the right to stop loss in time. Including stopping/terminated service immediately.
TOS 18.7 You can transfer your products to any DMIT customer; However, DMIT does not take any responsibility for disputes and misoperation before and after the transfer process.
At present, DMIT has deactivated the renewal function. Please use PayPal auto payment or bind the credit card in the DMIT system if you worried about forgot to pay.
DMIT used Stripe as our Credit Card payment gateway. DMIT server, credit card processing is meeting the PCI requirement. DMIT will not save your CC information. It is directly passed to Stripe.
We have received many complaints about invoices and fraud caused by DMIT Product transfer. This is a major reason that we deactivate it.
Please note: If you find a very cheap DMIT server; In most cases, it is from PayPal fraud, credit card fraud, or DMIT system bug. According to the DMIT Service Agreement (ToS), DMIT has the right to stop loss in time. Including stopping/terminated service immediately.
TOS 18.7 You can transfer your products to any DMIT customer; However, DMIT does not take any responsibility for disputes and misoperation before and after the transfer process.
Fault Report:
After check with Digital Realty (DRT).
The condenser of DRT's air-conditions system damaged hours ago. This issue leads to the extremely high temperature of some of DMIT's virtualized nodes. The extreme temperature exceeds the safety threshold. Our nodes shut themselves down for safety reasons automatically.
DMIT NOC already reboots these nodes. Our NOC will continuously update more detail later.
Some VMs need to be started late. Thank you for your patience.
The downtime of some nodes: 1 hour
After check with Digital Realty (DRT).
The condenser of DRT's air-conditions system damaged hours ago. This issue leads to the extremely high temperature of some of DMIT's virtualized nodes. The extreme temperature exceeds the safety threshold. Our nodes shut themselves down for safety reasons automatically.
DMIT NOC already reboots these nodes. Our NOC will continuously update more detail later.
Some VMs need to be started late. Thank you for your patience.
The downtime of some nodes: 1 hour
Falut Report:
Location: HKG
Detail: Some customers reported serious packet loss.
Status: Solved.
Reason: Some clients route local multicast packets into DMIT VM Network. Due to some HKG node didn't apply the lastest DMIT Firewall but PVE-Firewall, the multicast packet was transmitted to the VMs in these these.
Location: HKG
Detail: Some customers reported serious packet loss.
Status: Solved.
Reason: Some clients route local multicast packets into DMIT VM Network. Due to some HKG node didn't apply the lastest DMIT Firewall but PVE-Firewall, the multicast packet was transmitted to the VMs in these these.
/var/log/DMIT-NOC.log
Falut Report: Location: HKG Detail: Some customers reported serious packet loss. Status: Solved. Reason: Some clients route local multicast packets into DMIT VM Network. Due to some HKG node didn't apply the lastest DMIT Firewall but PVE-Firewall, the multicast…
NOC found multiple problems in the HKG Datacenter after in-deep investigation.
A. Both of upLinks between Core Router and Core Switch face to the serious light attenuation (optical power loss).
**[DMIT already send NOC staff to Datacenter to do in-person check]**
B. Huge DDoS Attack was not mitigated yesterday. it made transit port congested due to inappropriate rule. (Fixed)
A. Both of upLinks between Core Router and Core Switch face to the serious light attenuation (optical power loss).
**[DMIT already send NOC staff to Datacenter to do in-person check]**
B. Huge DDoS Attack was not mitigated yesterday. it made transit port congested due to inappropriate rule. (Fixed)
/var/log/DMIT-NOC.log
NOC found multiple problems in the HKG Datacenter after in-deep investigation. A. Both of upLinks between Core Router and Core Switch face to the serious light attenuation (optical power loss). **[DMIT already send NOC staff to Datacenter to do in-person…
Both solved.
DMIT NOC will continuously focus on this.
Reason: ISP broken the fiber when they removed one of our non-used local-loop service at July 28.
NOC is now rebooting the Core Router.
DMIT NOC will continuously focus on this.
Reason: ISP broken the fiber when they removed one of our non-used local-loop service at July 28.
NOC is now rebooting the Core Router.
/var/log/DMIT-NOC.log
Both solved. DMIT NOC will continuously focus on this. Reason: ISP broken the fiber when they removed one of our non-used local-loop service at July 28. NOC is now rebooting the Core Router.
All existing problems have been completely solved.
————
DMIT is preparing various todo of LAX data center recently, including expanding the data center, bandwidth, and upgrading our architecture to be compatible with customers transferred from other IDC. Our apology about the inconvenience.
————
Multiple reasons we gave before are not the critical one lead to packet loss.
The most critical reason for packet loss:
DMIT HKG has 4(four) routing tables, one for standard routing, one for economical routing, one for premium routing, and the last one for peering.
To save FIB, DMIT has applied a static IPv6 default route in the premium and economical table that points to the standard(default) table. DMIT did not remove the default route when copy the client route from premium/economical to the default table. It made a hidden loop route. There will be a problem if any client in the network is trying to transmit the packet to a null route IPv6 destination. Before the TTL is expired, the package will be transmitted between each table infinity. Since the packet was transmitted in the fabric, it's complicated to find where the problem is because the bandwidth of fabric was not shown on our monitoring system. These packets can be transmitted 254 times before dropped. In theory, this traffic will be amplified by 254 times.
These are the issues lead to congestion on the fabric and LUCHIP and caused packet loss.
————
DMIT is preparing various todo of LAX data center recently, including expanding the data center, bandwidth, and upgrading our architecture to be compatible with customers transferred from other IDC. Our apology about the inconvenience.
————
Multiple reasons we gave before are not the critical one lead to packet loss.
The most critical reason for packet loss:
DMIT HKG has 4(four) routing tables, one for standard routing, one for economical routing, one for premium routing, and the last one for peering.
To save FIB, DMIT has applied a static IPv6 default route in the premium and economical table that points to the standard(default) table. DMIT did not remove the default route when copy the client route from premium/economical to the default table. It made a hidden loop route. There will be a problem if any client in the network is trying to transmit the packet to a null route IPv6 destination. Before the TTL is expired, the package will be transmitted between each table infinity. Since the packet was transmitted in the fabric, it's complicated to find where the problem is because the bandwidth of fabric was not shown on our monitoring system. These packets can be transmitted 254 times before dropped. In theory, this traffic will be amplified by 254 times.
These are the issues lead to congestion on the fabric and LUCHIP and caused packet loss.
Giant && Tiny~~
PVM.LAX.Pro.SHAREv2
10OFF for Annually payment
Code: I-HAVE-READ-TOS-SHAREv2
PVM.LAX.Pro.GIANT
8c 16G 100G-SSD 44T@300M/1G
10OFF for non-monthly payment
Code: Welcome-GIANT
PVM.LAX.Pro.TINY
1c 1G 10G-SSD 550G@200M/1G
48.88 Semi-Annually
88.88 Annually
https://www.dmit.io/cart.php?gid=9
Once per client. ~
PVM.LAX.Pro.SHAREv2
10OFF for Annually payment
Code: I-HAVE-READ-TOS-SHAREv2
PVM.LAX.Pro.GIANT
8c 16G 100G-SSD 44T@300M/1G
10OFF for non-monthly payment
Code: Welcome-GIANT
PVM.LAX.Pro.TINY
1c 1G 10G-SSD 550G@200M/1G
48.88 Semi-Annually
88.88 Annually
https://www.dmit.io/cart.php?gid=9
Once per client. ~
/var/log/DMIT-NOC.log pinned «Giant && Tiny~~ PVM.LAX.Pro.SHAREv2 10OFF for Annually payment Code: I-HAVE-READ-TOS-SHAREv2 PVM.LAX.Pro.GIANT 8c 16G 100G-SSD 44T@300M/1G 10OFF for non-monthly payment Code: Welcome-GIANT PVM.LAX.Pro.TINY 1c 1G 10G-SSD 550G@200M/1G 48.88 Semi-Annually…»
Notes:
DMIT is now trying to migrate all PVM.LAX.Lite VMs to the AMD EPYC platform. There is a heavy task load on the background. Also, there is a massive amount of new server delivery tasks for clients that transferred from another IDC.
It takes time if you try to reinstall your VM right now. We suggest you do reinstall after today.
BR,
NOC.
DMIT is now trying to migrate all PVM.LAX.Lite VMs to the AMD EPYC platform. There is a heavy task load on the background. Also, there is a massive amount of new server delivery tasks for clients that transferred from another IDC.
It takes time if you try to reinstall your VM right now. We suggest you do reinstall after today.
BR,
NOC.