Message from CTG:
Please kindly be informed that there will be an urgent maintenance and more information as below.
Time window (Date/Time):
2023-04-21 16:00:00 - 2023-04-21 22:00:00 UTC
2023-04-22 00:00:00 - 2023-04-22 06:00:00 UTC+8
Maintenance Description:
Hidden-faulty troubleshooting
Maintenance Location:
International/Overseas
[DMIT Location: TYO Pro]
Service Impact:
The circuit will experience outage up to 120minutes during the maintenance window.
Affected Circuit(s):
TYO-GIA-*****CTG
Please kindly be informed that there will be an urgent maintenance and more information as below.
Time window (Date/Time):
2023-04-21 16:00:00 - 2023-04-21 22:00:00 UTC
2023-04-22 00:00:00 - 2023-04-22 06:00:00 UTC+8
Maintenance Description:
Hidden-faulty troubleshooting
Maintenance Location:
International/Overseas
[DMIT Location: TYO Pro]
Service Impact:
The circuit will experience outage up to 120minutes during the maintenance window.
Affected Circuit(s):
TYO-GIA-*****CTG
/var/log/DMIT-NOC.log pinned «Message from CTG: Please kindly be informed that there will be an urgent maintenance and more information as below. Time window (Date/Time): 2023-04-21 16:00:00 - 2023-04-21 22:00:00 UTC 2023-04-22 00:00:00 - 2023-04-22 06:00:00 UTC+8 Maintenance Description:…»
/var/log/DMIT-NOC.log
Maintenance Notice: Region: Hong Kong Time: April 17 ~ April 21, 2023 (Local Time) Length: The total length of interruption will not exceed 7 hours. Service degradation is no later than the termination time. Content: DMIT will perform a complete upgrade to…
We successfully completed the DMIT core network rack migration yesterday in the local time.
At present,
- We are in the process of transferring your data from the Hyper-converged Ceph system to the Standalone NVMe Ceph infrastructure.
- The new set of EPYC servers have been installed.
Pending tasks:
- Ensure the new server set is fully operational.
- NTT (AS2914) has used up their 100G interfaces at Equinix HK; DMIT is awaiting the completion of their DWDM deployment.
- Cogent (AS174) has not met the agreed-upon service delivery timeline. DMIT has pre-patched the cross-connect, and we are now waiting for their LOA to finish the connection.
- CUG (AS10099) is still waiting for the prefix filter to be updated.
At present,
- We are in the process of transferring your data from the Hyper-converged Ceph system to the Standalone NVMe Ceph infrastructure.
- The new set of EPYC servers have been installed.
Pending tasks:
- Ensure the new server set is fully operational.
- NTT (AS2914) has used up their 100G interfaces at Equinix HK; DMIT is awaiting the completion of their DWDM deployment.
- Cogent (AS174) has not met the agreed-upon service delivery timeline. DMIT has pre-patched the cross-connect, and we are now waiting for their LOA to finish the connection.
- CUG (AS10099) is still waiting for the prefix filter to be updated.
Completed emergency maintenance:
Emergency maintenance has been successfully completed.
Our on-site engineer identified a critical hardware failure on one of the nodes and resolved the issue before posting any notice.
We will soon migrate all HKG VMs to a new cluster, and the old cluster will be rebuilt.
This is necessary due to architectural issues that have prevented any possiable updates for the past two years.
Emergency maintenance has been successfully completed.
Our on-site engineer identified a critical hardware failure on one of the nodes and resolved the issue before posting any notice.
We will soon migrate all HKG VMs to a new cluster, and the old cluster will be rebuilt.
This is necessary due to architectural issues that have prevented any possiable updates for the past two years.
/var/log/DMIT-NOC.log
We successfully completed the DMIT core network rack migration yesterday in the local time. At present, - We are in the process of transferring your data from the Hyper-converged Ceph system to the Standalone NVMe Ceph infrastructure. - The new set of EPYC…
AS2914 will be installed at end of next week.
AS174 is still in progress of upgrade, waiting for LOA.
AS174 is still in progress of upgrade, waiting for LOA.
TYO partial routing failure report
Hours earlier, Telstra had prematurely terminated IP Transit services. (scheduled to be terminated on May 1.)
This caused us not to turn the new IP Transit service up on time, which resulted in some Internet null routes. However, this has been solved now;
The refreshed IP Transit vendor of DMIT:
Tokyo: +AS2914, +AS17676, [-AS4637, -AS3491]
Hong Kong: +AS2914, +AS9002, [-AS4637, -AS3491]
Hours earlier, Telstra had prematurely terminated IP Transit services. (scheduled to be terminated on May 1.)
This caused us not to turn the new IP Transit service up on time, which resulted in some Internet null routes. However, this has been solved now;
The refreshed IP Transit vendor of DMIT:
Tokyo: +AS2914, +AS17676, [-AS4637, -AS3491]
Hong Kong: +AS2914, +AS9002, [-AS4637, -AS3491]
/var/log/DMIT-NOC.log
We successfully completed the DMIT core network rack migration yesterday in the local time. At present, - We are in the process of transferring your data from the Hyper-converged Ceph system to the Standalone NVMe Ceph infrastructure. - The new set of EPYC…
The new host servers will be available in late May; All VM's VirtIO drive has been migrated to the new NVMe standalone Ceph cluster now.
HKG.Pro
DMIT completed the new vendor connection for HKG Pro and DMIT would like to make the following adjustments for our customers.
TINY:
- 200GB > 400GB (Transfer)
- 0.75 GB > 1.0GB (RAM)
- 10GB > 20GB (SSD)
- 40Mbps > 100Mbps
STARTER:
- 500GB > 800GB (Transfer)
- 20GB > 40GB (SSD)
- 1.5GB > 2.0GB (RAM)
- $69.9 > $79.9 (Keep the price for existing order)
MINI:
- 800GB > 1200GB (Transfer)
- 40GB > 60GB (SSD)
- 100Mbps > 200Mbps
- $109.9 > $119.9 (Keep the price for existing order)
MICRO:
- 1000GB > 1600GB (Transfer)
- 40GB > 80GB (SSD)
- 100Mbps > 200Mbps
- $139.9 > $159.9 (Keep the price for existing order)
MEDIUM:
- 1500GB > 1800GB (Transfer)
- 80GB > 160GB (SSD)
LARGE:
- 2000GB > 2400GB (Transfer)
- 160GB > 240GB (SSD)
GIANT:
- 4000GB > 4800GB (Transfer)
DMIT completed the new vendor connection for HKG Pro and DMIT would like to make the following adjustments for our customers.
TINY:
- 200GB > 400GB (Transfer)
- 0.75 GB > 1.0GB (RAM)
- 10GB > 20GB (SSD)
- 40Mbps > 100Mbps
STARTER:
- 500GB > 800GB (Transfer)
- 20GB > 40GB (SSD)
- 1.5GB > 2.0GB (RAM)
- $69.9 > $79.9 (Keep the price for existing order)
MINI:
- 800GB > 1200GB (Transfer)
- 40GB > 60GB (SSD)
- 100Mbps > 200Mbps
- $109.9 > $119.9 (Keep the price for existing order)
MICRO:
- 1000GB > 1600GB (Transfer)
- 40GB > 80GB (SSD)
- 100Mbps > 200Mbps
- $139.9 > $159.9 (Keep the price for existing order)
MEDIUM:
- 1500GB > 1800GB (Transfer)
- 80GB > 160GB (SSD)
LARGE:
- 2000GB > 2400GB (Transfer)
- 160GB > 240GB (SSD)
GIANT:
- 4000GB > 4800GB (Transfer)
/var/log/DMIT-NOC.log pinned «HKG.Pro DMIT completed the new vendor connection for HKG Pro and DMIT would like to make the following adjustments for our customers. TINY: - 200GB > 400GB (Transfer) - 0.75 GB > 1.0GB (RAM) - 10GB > 20GB (SSD) - 40Mbps > 100Mbps STARTER: - 500GB > 800GB…»
HKG.Pro
Received notification from the vendor that the AS-Path needs to be corrected; after completing the configuration, the BGP session restarted twice which causing the CN2 routing convergence limit to be triggered.
CN2 has route convergence restrictions, CTGnet accepts the route but CN2 does not, resulting in an null route.
It takes 30 minutes to be back to normal.
Received notification from the vendor that the AS-Path needs to be corrected; after completing the configuration, the BGP session restarted twice which causing the CN2 routing convergence limit to be triggered.
CN2 has route convergence restrictions, CTGnet accepts the route but CN2 does not, resulting in an null route.
It takes 30 minutes to be back to normal.
HKG Update:
- AS10099 finished AS-Path filter update;
- AS2914 finished patching;
- AS2914 - AS4809 direct connection has been recoverd.
TYO Update:
- AS2914 - AS4809 direct connection has been recoverd.
- AS10099 finished AS-Path filter update;
- AS2914 finished patching;
- AS2914 - AS4809 direct connection has been recoverd.
TYO Update:
- AS2914 - AS4809 direct connection has been recoverd.
/var/log/DMIT-NOC.log
Investigation result of the packet loss for LAX.Pro. Keep you posted.
Dear Valued Customer,
Please kindly note that our engineer feedback the services were migrated to an alternate option thus restoring service to a stable state.
Please kindly note that our engineer feedback the services were migrated to an alternate option thus restoring service to a stable state.
Unplanned maintenance
Location: LAX
Restart two hosts to resolve kernel errors.
Location: LAX
Restart two hosts to resolve kernel errors.
/var/log/DMIT-NOC.log
The Ceph of LAX has been upgraded to Quincy; The next release should contain the bug fix.
https://docs.ceph.com/en/latest/releases/quincy/#v17-2-6-quincy
os/bluesore: cumulative backport for Onode stuff and more (pr#50048, Igor Fedotov, Adam Kupczyk)
v17.2.6 QUINCY resolved this problem;
After testing, we've update ceph in all locations to V17.2.6.
os/bluesore: cumulative backport for Onode stuff and more (pr#50048, Igor Fedotov, Adam Kupczyk)
v17.2.6 QUINCY resolved this problem;
After testing, we've update ceph in all locations to V17.2.6.
Possiable failure on AS174 Los Angeles; We saw Internet routes that annouced by Cogent but unable to reach via Cogent.
DMIT rejected Cogent routes to prevent possible null-route on Internet.
We will recover it once Cogent solved the problem.
DMIT rejected Cogent routes to prevent possible null-route on Internet.
We will recover it once Cogent solved the problem.
Planned maintenance
Date: Jun 15-18 2023 Hong Kong Time
Content:
Fix management node issues; install DDoS mitigation facilities, complete installation of new nodes, rebuild old cabinets.
Possible scenarios: Network downgration, VM reboot, Unable to access control panel.
Date: Jun 15-18 2023 Hong Kong Time
Content:
Fix management node issues; install DDoS mitigation facilities, complete installation of new nodes, rebuild old cabinets.
Possible scenarios: Network downgration, VM reboot, Unable to access control panel.
/var/log/DMIT-NOC.log pinned «Planned maintenance Date: Jun 15-18 2023 Hong Kong Time Content: Fix management node issues; install DDoS mitigation facilities, complete installation of new nodes, rebuild old cabinets. Possible scenarios: Network downgration, VM reboot, Unable to access…»