Our new IP Transit Arelion(Twelve99) in Los Angles is ready.
Due to the AS2914 does not meet our requirement in the NA region, we decided not to use it as our cross continents IP Transit.
AS1299 will be our major NA T1 IP Transit.
Due to the AS2914 does not meet our requirement in the NA region, we decided not to use it as our cross continents IP Transit.
AS1299 will be our major NA T1 IP Transit.
Pre notification:
(This will be emailed to all LAX LITE user later)
We plan to smoothly migrate all lite VM in Los Angeles to San Jose after San Jose is launched and stable.
Few key points:
- You will receive an email before we migrate your VM.
- The system IO latency will increase during migration.
- The network latency might be increased during migration.
- Possible downtime.
- Backup is strongly recommended, and required if you have high value data. Please check TOS for detail.
We will email you before the entire progress start.
- Your IP will NOT change.
- Better performance: EPYC 7443p, + NVMe Ceph cluster, + 3200Mhz RECC RAM
- You will have better latency in most U.S. cities:
---- Close to Seattle(one of the major exchange points)
---- Better U.S. eastern connectivity.
---- Better local connectivity (San Jose is close to San Francisco)
---- Major IXP/IPT connectivity: 100G FCIX, 100G Equinix San Jose, and 100G Telia, 100G Cogent...
---- Better APAC connectivity.
- You will have better latency to China. There are 7ms between DMIT SJC and DMIT LAX.
---- LAX Lite now use below link: DMIT LAX <> DMIT SJC <> China Unicom (AS4837) <> China.
---- DMIT has 2x100G CU AS4837 in San Jose.
LAX will have the Pro profile only after this migration.
===
The above message is for LAX LITE
===
(This will be emailed to all LAX LITE user later)
We plan to smoothly migrate all lite VM in Los Angeles to San Jose after San Jose is launched and stable.
Few key points:
- You will receive an email before we migrate your VM.
- The system IO latency will increase during migration.
- The network latency might be increased during migration.
- Possible downtime.
- Backup is strongly recommended, and required if you have high value data. Please check TOS for detail.
We will email you before the entire progress start.
- Your IP will NOT change.
- Better performance: EPYC 7443p, + NVMe Ceph cluster, + 3200Mhz RECC RAM
- You will have better latency in most U.S. cities:
---- Close to Seattle(one of the major exchange points)
---- Better U.S. eastern connectivity.
---- Better local connectivity (San Jose is close to San Francisco)
---- Major IXP/IPT connectivity: 100G FCIX, 100G Equinix San Jose, and 100G Telia, 100G Cogent...
---- Better APAC connectivity.
- You will have better latency to China. There are 7ms between DMIT SJC and DMIT LAX.
---- LAX Lite now use below link: DMIT LAX <> DMIT SJC <> China Unicom (AS4837) <> China.
---- DMIT has 2x100G CU AS4837 in San Jose.
LAX will have the Pro profile only after this migration.
===
The above message is for LAX LITE
===
/var/log/DMIT-NOC.log pinned «Pre notification: (This will be emailed to all LAX LITE user later) We plan to smoothly migrate all lite VM in Los Angeles to San Jose after San Jose is launched and stable. Few key points: - You will receive an email before we migrate your VM. - The system…»
We've noticed the routing issue of LAX Pro.
NOC already reported it to CTG. Please wait for further notice.
NOC already reported it to CTG. Please wait for further notice.
Our team recently found the Intel 85299 NIC, and its' driver is unreliable for the long term running virtualization environment. Our new architecture has already switched to Mellanox CX3/4/5.
This caused many problems, including the incident happening at the LAX site today.
One NIC driver dead will leads to a heavy load on other nodes for Ceph rebalance.
Then due to heavy load, the drivers of 85299 on other nodes probably dump again which links to a chain reaction.
We gave up Hyper-converged infrastructure design, the new architecture will have a dedicated Ceph cluster network with dual 100G link over ConnectX-5.
This caused many problems, including the incident happening at the LAX site today.
One NIC driver dead will leads to a heavy load on other nodes for Ceph rebalance.
Then due to heavy load, the drivers of 85299 on other nodes probably dump again which links to a chain reaction.
We gave up Hyper-converged infrastructure design, the new architecture will have a dedicated Ceph cluster network with dual 100G link over ConnectX-5.
The partial network problems in LAX should already be resolved now.
The NOC of Twelve-99(a.k.a Arelion, Telia) put a wrong interface configuration on their end at 1 AM PDT today without noticing us immediately.
We have withdrawn the BGP announcement temporarily and the network profile will be rolled back once they fix this issue.
The NOC of Twelve-99(a.k.a Arelion, Telia) put a wrong interface configuration on their end at 1 AM PDT today without noticing us immediately.
We have withdrawn the BGP announcement temporarily and the network profile will be rolled back once they fix this issue.
/var/log/DMIT-NOC.log
Our team recently found the Intel 85299 NIC, and its' driver is unreliable for the long term running virtualization environment. Our new architecture has already switched to Mellanox CX3/4/5. This caused many problems, including the incident happening at…
DMIT LAX Pro, sPro will be migrated to 7443p / ConnectX node very soon.
There are 3 of the 7402p / 82599 nodes that are under very unstable situations due to memory leaking of ixgbe drivers ( it can only be resolved by reboot; rmmod unable to fix ). We have already isolated these 3 nodes to reduce the impact of the cluster.
You don’t need to worry about your data. It has already been migrated to the NVMe Ceph node days ago.
We’ll try our best to live-migrate your VM to the new node. If unable, your VM will reboot once.
There are 3 of the 7402p / 82599 nodes that are under very unstable situations due to memory leaking of ixgbe drivers ( it can only be resolved by reboot; rmmod unable to fix ). We have already isolated these 3 nodes to reduce the impact of the cluster.
You don’t need to worry about your data. It has already been migrated to the NVMe Ceph node days ago.
We’ll try our best to live-migrate your VM to the new node. If unable, your VM will reboot once.
Dear customer,
Please kindly be informed that there will be an urgent maintenance and more information as below.
Time window (Date/Time):
2022-06-08 16:00:00 - 2022-06-08 22:00:00 UTC
2022-06-09 00:00:00 - 2022-06-09 06:00:00 UTC+8
Maintenance Description:
Hidden-faulty troubleshooting
Service Impact:
The circuit will experience outage up to 120minutes during the maintenance window.
Affected Circuit(s):
TYO-GIA
From CTG NOC
Please kindly be informed that there will be an urgent maintenance and more information as below.
Time window (Date/Time):
2022-06-08 16:00:00 - 2022-06-08 22:00:00 UTC
2022-06-09 00:00:00 - 2022-06-09 06:00:00 UTC+8
Maintenance Description:
Hidden-faulty troubleshooting
Service Impact:
The circuit will experience outage up to 120minutes during the maintenance window.
Affected Circuit(s):
TYO-GIA
From CTG NOC
/var/log/DMIT-NOC.log
Our team recently found the Intel 85299 NIC, and its' driver is unreliable for the long term running virtualization environment. Our new architecture has already switched to Mellanox CX3/4/5. This caused many problems, including the incident happening at…
LAX.Pro/sPro should all running on the 7443p + 3200 RAM system right now.
DMIT is now replacing our ASN from AS54574 to AS906 step by step.
It might slightly affect the network but we’ll try to lower the impact during this process.
Please let us know if you see there are any routing changes that not as expected.
It might slightly affect the network but we’ll try to lower the impact during this process.
Please let us know if you see there are any routing changes that not as expected.
/var/log/DMIT-NOC.log
DMIT is now replacing our ASN from AS54574 to AS906 step by step. It might slightly affect the network but we’ll try to lower the impact during this process. Please let us know if you see there are any routing changes that not as expected.
IP Transit BGP flapped in TYO and LAX due to ASN changes.
The Internet route should been recovered 2hr ago for LAX, and 10 mins ago for TYO.
The Internet route should been recovered 2hr ago for LAX, and 10 mins ago for TYO.
DMIT is now offering 10G/100G waves between Equinix SV10 ~ Hurricane Electric FMT2.
10G $500MRC $600NRC
100G $1500MRC $5000NRC
Starting with a 12mo contract
24mo 20% NRC discount
36mo 40% NRC discount
- Delivery starts in late July.
- Both end delivery with 1310nm SMF 10km.
- Prepaid only.
- XC on customers.
- DMIT could help to order Equinix IX San Jose.
Contact with sales@dmit.com or ticket
10G $500MRC $600NRC
100G $1500MRC $5000NRC
Starting with a 12mo contract
24mo 20% NRC discount
36mo 40% NRC discount
- Delivery starts in late July.
- Both end delivery with 1310nm SMF 10km.
- Prepaid only.
- XC on customers.
- DMIT could help to order Equinix IX San Jose.
Contact with sales@dmit.com or ticket
unexpected reboot during kernel patch for 2 nodes in HKG.
You could start the server on your own or wait for an automatic boot-up.
You could start the server on your own or wait for an automatic boot-up.
Maintenance Notice:
Hong Kong and Los Angles
Date & Time: 17:00-21:00, July 8, 2022. Eastern Daylight Time.
Duration: upto 30 mins per location.
What: Major Network Configuration
Impact: Internet fully disconnection is possible.
Hong Kong and Los Angles
Date & Time: 17:00-21:00, July 8, 2022. Eastern Daylight Time.
Duration: upto 30 mins per location.
What: Major Network Configuration
Impact: Internet fully disconnection is possible.
/var/log/DMIT-NOC.log pinned «Maintenance Notice: Hong Kong and Los Angles Date & Time: 17:00-21:00, July 8, 2022. Eastern Daylight Time. Duration: upto 30 mins per location. What: Major Network Configuration Impact: Internet fully disconnection is possible.»