/var/log/DMIT-NOC.log
4.71K subscribers
189 photos
6 files
117 links
Download Telegram
The inbound traffic of DMIT LAX sPro is 100% offered by Cloudflare Magic Transit

DMIT has no control over anything about this.
Like sPro packet loss for Chinese direction.

Yes, there will definitely, absolutely, no doubt has packet loss for sPro.

There might be congestion between the peering of GTT/ATT <> CU (AS4837)
Cloudflare is using GTT/ATT as IP Transit to connect with CU(AS4837)

Also, it could happen between AS4134 <> Cloudflare
20.3 For Lite network profile:

Monthly Billing Cycle (New Order): We do not give guarantee the IP is globally accessible for the monthly order, especially for China, Russia, and all country has national network censorship. But you could ask us for the guarantee with a 1USD addon.

Non-Monthly Billing Cycle (New Order): The IP global accessible guarantee costs have been included in the price.

Any existing service: There is no free IP replacement for existing services.

Standard IP replacement fee: 5 USD

We've updated the TOS for the Lite network IP replacement policy.

https://www.dmit.io/pages/tos
HKG Lite IPv6 unreachable issue which started 1hr ago has been resolved.
Time-limited special offer for TYO.Pro

- 3 cycle 50off with the condition.
- Annually only.
- Applied to TYO.Pro.STARTER, MINI, MICRO, MEDIUM.

Restriction:
- The TOS of IP replacement is not available for the order with this code.
- The IP replacement will be restricted up to once per 30 days and charged with 5USD each time.
- The annual transfer booster is not available for this promotion.

The existed user could ask us to change your order by renewing a year, but restrictions will be applied immediately.

Code: Special-Annual-50off-Replace-IP-Restricted-To-Once-Per-30-Days
/var/log/DMIT-NOC.log pinned «Time-limited special offer for TYO.Pro - 3 cycle 50off with the condition. - Annually only. - Applied to TYO.Pro.STARTER, MINI, MICRO, MEDIUM. Restriction: - The TOS of IP replacement is not available for the order with this code. - The IP replacement…»
Our new IP Transit Arelion(Twelve99) in Los Angles is ready.

Due to the AS2914 does not meet our requirement in the NA region, we decided not to use it as our cross continents IP Transit.

AS1299 will be our major NA T1 IP Transit.
Pre notification:
(This will be emailed to all LAX LITE user later)

We plan to smoothly migrate all lite VM in Los Angeles to San Jose after San Jose is launched and stable.

Few key points:
- You will receive an email before we migrate your VM.
- The system IO latency will increase during migration.
- The network latency might be increased during migration.
- Possible downtime.
- Backup is strongly recommended, and required if you have high value data. Please check TOS for detail.

We will email you before the entire progress start.
- Your IP will NOT change.
- Better performance: EPYC 7443p, + NVMe Ceph cluster, + 3200Mhz RECC RAM
- You will have better latency in most U.S. cities:
---- Close to Seattle(one of the major exchange points)
---- Better U.S. eastern connectivity.
---- Better local connectivity (San Jose is close to San Francisco)
---- Major IXP/IPT connectivity: 100G FCIX, 100G Equinix San Jose, and 100G Telia, 100G Cogent...
---- Better APAC connectivity.
- You will have better latency to China. There are 7ms between DMIT SJC and DMIT LAX.
---- LAX Lite now use below link: DMIT LAX <> DMIT SJC <> China Unicom (AS4837) <> China.
---- DMIT has 2x100G CU AS4837 in San Jose.

LAX will have the Pro profile only after this migration.

===
The above message is for LAX LITE
===
/var/log/DMIT-NOC.log pinned «Pre notification: (This will be emailed to all LAX LITE user later) We plan to smoothly migrate all lite VM in Los Angeles to San Jose after San Jose is launched and stable. Few key points: - You will receive an email before we migrate your VM. - The system…»
Welcome to the U.S.
We've noticed the routing issue of LAX Pro.
NOC already reported it to CTG. Please wait for further notice.
Our team recently found the Intel 85299 NIC, and its' driver is unreliable for the long term running virtualization environment. Our new architecture has already switched to Mellanox CX3/4/5.
This caused many problems, including the incident happening at the LAX site today.

One NIC driver dead will leads to a heavy load on other nodes for Ceph rebalance.
Then due to heavy load, the drivers of 85299 on other nodes probably dump again which links to a chain reaction.

We gave up Hyper-converged infrastructure design, the new architecture will have a dedicated Ceph cluster network with dual 100G link over ConnectX-5.
One LAX Node failure, rebooting.
The partial network problems in LAX should already be resolved now.

The NOC of Twelve-99(a.k.a Arelion, Telia) put a wrong interface configuration on their end at 1 AM PDT today without noticing us immediately.

We have withdrawn the BGP announcement temporarily and the network profile will be rolled back once they fix this issue.
/var/log/DMIT-NOC.log
Our team recently found the Intel 85299 NIC, and its' driver is unreliable for the long term running virtualization environment. Our new architecture has already switched to Mellanox CX3/4/5. This caused many problems, including the incident happening at…
DMIT LAX Pro, sPro will be migrated to 7443p / ConnectX node very soon.

There are 3 of the 7402p / 82599 nodes that are under very unstable situations due to memory leaking of ixgbe drivers ( it can only be resolved by reboot; rmmod unable to fix ). We have already isolated these 3 nodes to reduce the impact of the cluster.

You don’t need to worry about your data. It has already been migrated to the NVMe Ceph node days ago.

We’ll try our best to live-migrate your VM to the new node. If unable, your VM will reboot once.
Dear customer,

Please kindly be informed that there will be an urgent maintenance and more information as below.

Time window (Date/Time):
2022-06-08 16:00:00 - 2022-06-08 22:00:00 UTC
2022-06-09 00:00:00 - 2022-06-09 06:00:00 UTC+8

Maintenance Description:
Hidden-faulty troubleshooting

Service Impact:
The circuit will experience outage up to 120minutes during the maintenance window.

Affected Circuit(s):
TYO-GIA

From CTG NOC
DMIT is now replacing our ASN from AS54574 to AS906 step by step.

It might slightly affect the network but we’ll try to lower the impact during this process.

Please let us know if you see there are any routing changes that not as expected.