Chine Telecom is still working on this.

This network fault is caused by China Telecom. There is a network issue on the CTGnet device.

Fault 1: The return routing should be CN2 instead of AS4134.

Fault 2: TCP packet loss.

===
DMIT is currently working with China Telecom for restore this service.

Once the service has been restored, we will offer SLA compensation.

Please test the TYO Pro.

CT responds circuit backs to normal. We confirmed this situation based our general diagnosis.

Submit / reply to the ticket ASAP if you still have problems.

We are still following up with CTGnet for incident investigation and report;
SLA compensation detail will be posted here within 2 weeks.

TYO Pro RFO (Reason of Outage) From CTGnet:

Quote:
> hardware fault, and the circuit was restored after manufacturer urgent fixed it.


DMIT will issue SLA compensation for all active orders placed before Feb 8, 2023, for TYO Pro. Please wait for more detail.

We will issue a 7-day equivalent credit to your account.

Please contact us via ticket. In order to process faster, you could add [TYO SLA] to the front of the title. This is optional.

DMIT.io maintenance; ETA 2hr

Extend for 2hr;

The maintenance has complete 2hrs ago;
We've improved the performance of control panel.
Please let us know if you found any error or hard-to-use case in this new control panel.

Thanks for assistance.

DMIT is now support PTR record update;

For IPv6, we also support /64 PTR wildcard record;
But the more specific record for IPv6 will not be supported.

PTR update can be accessed by click the change hostname icon on the right side of your hostname.

The AUP will be also appied to PTR.
DMIT does not support Dest. TCP 25 outbound access.

The snapshot feature has become available in LAX.
Every LAX Pro instance has 1 quota; sPro has 2 quotas.
The extra snapshot upgrade and selection on order will be available soon.

This feature will become available for SJC next, then HKG, last is TYO.

SJC is out of hardware resources.

The new SSD, RAM, and servers are on the way.

1. Two nodes will be rebooted next weekend ( March 17 or 19) for a RAM upgrade.

2. The block storage will be doubled, and the snapshot will be available this weekend (March 10 or 12).

3. Your VM might be cold-migrated (one reboot) without notice to a new node since tight hardware resources.

4. The new instance purchase will be available the next day after the action ( March 11 or 13. ) if the no.2 action is completed on time.

We have to performe emergency reboot for some SJC node. It will be done very soon.

We've noticed IO Error in SJC; Investigating. Keep you posted.

The network components we used implemented hash Layer3+4 for Bond interface, which is not supported by Infiniband.

It caused the disonnection-dead-loop for the entire SJC Ceph cluster.

We've removed the config implemented by component and locked it.

We experience extramly high load in SJC, the new hardware is on the way;
The new NVMe block storage hardware will be installed tomorrow.

We are working on restore Ceph-OSD; The problem is found; It still takes more time to recovery.

We are still working on it; we suggest do not reboot if you still able to run your system, since the I/O is currently suspended.

The remote hand is on the way to the SJC location to implement hardware requirements for repair.

The SLA solution will be posted after repair done.

OSD recovery, backfill in progress.

Step 1 still need ~4hrs; 70% VM will return to normal;

Step 2 will take another ~4hrs; 99% VM will return to normal;

Step 3 needs whole day, it only leads to IO performance impact but not uptime impact.

The SLA is lower than TOS offered. The reimbursement will issue case by case; please submit ticket after the event end.

We are deeply sorry for the recent SLA drop that may have caused inconvenience to your business operations. We understand the importance of our services to your business and we take full responsibility for this interruption.

The fault report will be posted after the event.