Bandwidth capacity 📈 to the moon

LAX-0290CKN is in maintenance for Ceph. ETA: 30min;
Few VM down.

We found there are lots of orders at yesterday. This full-filled our LAX cluster, even we already added 2 more nodes at last week.

But don’t worry 😧; we already put one more node online to relief the loads. There is another node will be available soon.

Please let us know if your new ordered server cannot boot.

FYI: DMIT plans to install 3 more nodes before we get AMD 3rd GEN EPYC servers.

The below message is for LAX.sPro and HKG.Pro only.

Due to multiple limitations which listed below by our architecture, we need time to do more research on PVE code to make sPro stable and fast.

- Cloudflare required TCP MSS lower than 1450
- Juniper does not support TCP MSS clamping on IRB(VLAN L3) interface
- PVE has no native MTU configuration API.

Our solution is doing modifications on PVE. Before we deploy the code, please use the below scripts to keep your VM’s MTU at 1350.

ip link set mtu 1350 dev eth0
[ ! -e "/var/spool/cron/root" -o -z "`grep 'mtu' /var/spool/cron/root`" ] && { echo -e "@reboot ip link set mtu 1350 dev eth0" >> /var/spool/cron/root;chmod 600 /var/spool/cron/root; }
service crond reload
service crond restart


Above message is for LAX.sPro and HKG.Pro only.

We saw some reports from the client that said noVNC is unusable on the LAX site.
This is due to Proxmox and WHMCS module's limitations;
the DMIT dev team is engaging with this problem. It will be fixed soon.

Fixed; Patch has been loaded.

Clipboard and xTerm.js feature will be available soon.

We will push patch to HKG site ASAP.

This patch will help to reduce the response time of API and VNC.

After a month of cable patching work by CTA, this circuit is ready now. Traffic is loaded and ready to use.

Scheduled maintenance:

Datecenter: LAX

Time: April 10, 4: 00 pm Eastern Time;

Content: DMIT will restart BGP session with AS4809; This will result in the loss of CN2(AS4809) route for 1 hour.

Impact: within the time, China traffic transit by Premium Routing Profile does not pass through AS4809

After maintenance: LAX.sPro no longer needs to modify MTU to avoid packet loss.

Known issues:

Data center: HKG

Issues: high packet loss to China telecom (AS4134).

Service: PVM.HKG (without premium or lite tag)

ETR: April 11th to April 15th.

Fixed at Apr 10;

Fixed but still in observation.

Found new problem; update later.

Due to high deamon on Layer 7 DDoS Protection on sPro serial in LAX;
DMIT team decied to build a in datacenter Layer 7 DDoS firewall soon;
Our management team and NOC team are evaluating firewall appliances from several vendors.

hopefully, the firewall will be ready in June.
At that time, DMIT will offer basic protection for the entire LAX data center(Lite & Pro) and the best protection for sPro.

The diagram shows 3 situations with CFMT, just FYI.

Cloudflare Magic Transit only offer Layer 3/4 protection; so in the basic, all Layer 7 attack will be ignored at present.

The IP will lose routing optimization if the client chose to use Cloudflare CDN/WAF even without CFMT IP.

Telstra in DMIT HKG site is ready to use;

We already ordered the appliances🥳

HKG Down for 30mins+:

Due to power failure in a switch at HKG datacenter and the configuration we made last time was not save;
the default spanning-tree configs start to work after power back; the STP block storage and management VLAN/Ports which leads to Ceph system failure and management lost;

The system should now back to normal;

50% VM was impacted;

For all Lite clients,

Due to insufficient IPv4 subnet resources for the Lite network.

DMIT will not offer free IPv4 replacement for the Lite network start with your next billing date.

We think you accepted this change once you paid for your next invoice.

This will be updated to TOS soon.

There is no impact on Pro network.

Recently, some of our customers have deployed services in DMIT Los Angeles datacenter that violate US laws;

Here, we want to let all our customer know:

DMIT will actively cooperate with the US government in order to ensure the stability of the service for other customers.

DMIT will provide all data we have of the suspect, including the disk of VM if PD, DHS, FBI, or any dept required for it.

Please ensure that your business complies with U.S. law in LAX Datacenter.

FYI: Cogent Co, is the IPv4 vendor of DMIT Inc in LAX Datacenter.