⚠️ ❗️ALL USERS ❗️⚠️
RCA (Root Cause Analysis) of the downtime:
Today we’ve had the worst ever downtime seen in the past 3 years of being operational.
Root cause:
A domino of network switches failure, internal network failure and failure to re route network traffic via backups at the primary data centre of the Service Provider and finally storage being inaccessible due to fail safes that come on by default in the latest generation server architecture that we’re using.
Since the issue has been going on since before 8 AM today, our primary systems had not started and inturn not started backup systems as well. Issue was faced by all primary systems at the same time.
Plan Of Action to avoid a scenario like this:
1. In the near term/medium term, have the backup system deployed and running with a completely different service provider in a different geography.
2. Have more than 2 instances that can act as master systems to trigger different systems. While a version of this was already operational, it was a complete failure at different levels for an extended period of time which din’t allow us to attempt an early morning restoration.
3. Work with multiple generations of server architecture.
Compensation to our users:
1. Intraday Algos: We will be crediting users with 2x credits for today based on the number of Algos subscribed for today.
2. Positional algos: 2x credits will be credited for all positional algos for each day it was running till today.
These credits will automatically be added to your Quantiply accounts tomorrow or day after.
It is an extremely frustrating incident for all users, which were are extremely apologetic about. We want to thank you for your patience, understanding and positive reinforcement. We have identified ways to avoid disruption of services specific to this issue.
Note: Issue is resolved but we're implementing certain things as per the recommendation of the Cloud Service Provider. It will take time till tomorrow morning for the system to be fully operational and come online.
RCA (Root Cause Analysis) of the downtime:
Today we’ve had the worst ever downtime seen in the past 3 years of being operational.
Root cause:
A domino of network switches failure, internal network failure and failure to re route network traffic via backups at the primary data centre of the Service Provider and finally storage being inaccessible due to fail safes that come on by default in the latest generation server architecture that we’re using.
Since the issue has been going on since before 8 AM today, our primary systems had not started and inturn not started backup systems as well. Issue was faced by all primary systems at the same time.
Plan Of Action to avoid a scenario like this:
1. In the near term/medium term, have the backup system deployed and running with a completely different service provider in a different geography.
2. Have more than 2 instances that can act as master systems to trigger different systems. While a version of this was already operational, it was a complete failure at different levels for an extended period of time which din’t allow us to attempt an early morning restoration.
3. Work with multiple generations of server architecture.
Compensation to our users:
1. Intraday Algos: We will be crediting users with 2x credits for today based on the number of Algos subscribed for today.
2. Positional algos: 2x credits will be credited for all positional algos for each day it was running till today.
These credits will automatically be added to your Quantiply accounts tomorrow or day after.
It is an extremely frustrating incident for all users, which were are extremely apologetic about. We want to thank you for your patience, understanding and positive reinforcement. We have identified ways to avoid disruption of services specific to this issue.
Note: Issue is resolved but we're implementing certain things as per the recommendation of the Cloud Service Provider. It will take time till tomorrow morning for the system to be fully operational and come online.
⚠️ ❗️ALL USERS ❗️⚠️
The Subscription module will be unavailable till evening till the compensation credits are added to all the users accounts. This means that you will not be able to upgrade/downgrade plans or increase/decrease algo count temporarily till evening. Once the subscriptions module is activated again, we will announce.
The Subscription module will be unavailable till evening till the compensation credits are added to all the users accounts. This means that you will not be able to upgrade/downgrade plans or increase/decrease algo count temporarily till evening. Once the subscriptions module is activated again, we will announce.
⚠️ ❗️ALL USERS ❗️⚠️
Credits are added. Subscriptions module is active
Credits are added. Subscriptions module is active
⚠️ ❗️ALL USERS ❗️⚠️
Positional algos running from before Friday or Friday will not be launched tomorrow. Those have been terminated to avoid it from running without you knowing it’s running. Fresh Positional algos can be run, but the older ones will not run.
Positional algos running from before Friday or Friday will not be launched tomorrow. Those have been terminated to avoid it from running without you knowing it’s running. Fresh Positional algos can be run, but the older ones will not run.
⚠️ ❗️ALL USERS ❗️⚠️
Live Feed is running a bit slow, we're checking and resolving. It is showing Not Running but it is running slow.
Live Feed is running a bit slow, we're checking and resolving. It is showing Not Running but it is running slow.
⚠️ ❗️ALL USERS ❗️⚠️
Live feed is running slow. Execution will be delayed because of this. Please avoid trading for sometime. We're checking this and resolving it asap.
Live feed is running slow. Execution will be delayed because of this. Please avoid trading for sometime. We're checking this and resolving it asap.
⚠️ ❗️ALL USERS ❗️⚠️
We're terminating all the algos, which are running or are yet to run. Please manage positions manually or square off from the terminal
We're terminating all the algos, which are running or are yet to run. Please manage positions manually or square off from the terminal
⚠️ ❗️ALL USERS ❗️⚠️
All algos are terminated. It will show Trade Active/Ready for sometime, status will change in a few minutes. Algos will not run in the backend
All algos are terminated. It will show Trade Active/Ready for sometime, status will change in a few minutes. Algos will not run in the backend
⚠️ ❗️ALL USERS ❗️⚠️
Our team is currently working with the data centre service provider to address today's issue. Our team is trying our best to have the system working normally and smoothly for tomorrow. To avoid a scenario like today where the algos had to be terminated in the morning, we will take a final call on whether it is safe to run the algos tomorrow and inform our users accordingly. We will be sending out updates as soon as we get them from the team on the ongoing issue.
Our team is currently working with the data centre service provider to address today's issue. Our team is trying our best to have the system working normally and smoothly for tomorrow. To avoid a scenario like today where the algos had to be terminated in the morning, we will take a final call on whether it is safe to run the algos tomorrow and inform our users accordingly. We will be sending out updates as soon as we get them from the team on the ongoing issue.
⚠️ ❗️ALL USERS ❗️⚠️
Update on the ongoing issue:
The ongoing issue is still in the process of being resolved. As the issue has originated at the 3rd party end. ie. the Cloud Service Provider, it is taking a lot more time than expected to get everything in order. Due to this we think it is safer that we pause and keep the platform offline for another day or two, till we can get it running normally and smoothly as usual. So on 26th September, algos will stay paused and not run. Some of our users may not appreciate this but this is a precaution in the right direction.
A snapshot of the issue:
Due to a network issue, the latest 3rd generation servers had switched to backups. While switching to the backups there was a malfunction with storage which inturn became inaccessible. This issue was faced by all customers using 3rd gen systems at the Data Center. Post this incident, we have migrated our critical systems from 3rd gen to 2nd gen, which has been in the process of getting configured and optimised over the weekend. Alongside, we have tried various methods to recover our 3rd gen systems. While the system ran today, there were issues with the live feed and delays in execution due to which we had to terminate the algos and disallow further trading.
We believe this was an unexpected, one-off incident we’ve faced for the first time in 3 years, and our team will implement solutions to avoid such an incident. We deeply regret this downtime.
We will be temporarily disabling messaging on the Telegram Discussions group as we are seeing a lot of people who are not using Quantiply create a lot of noise, spam, and also gas-light other users. We will be available on our support handles for queries at @QuantiplySupport and on Whatsapp.
Update on the ongoing issue:
The ongoing issue is still in the process of being resolved. As the issue has originated at the 3rd party end. ie. the Cloud Service Provider, it is taking a lot more time than expected to get everything in order. Due to this we think it is safer that we pause and keep the platform offline for another day or two, till we can get it running normally and smoothly as usual. So on 26th September, algos will stay paused and not run. Some of our users may not appreciate this but this is a precaution in the right direction.
A snapshot of the issue:
Due to a network issue, the latest 3rd generation servers had switched to backups. While switching to the backups there was a malfunction with storage which inturn became inaccessible. This issue was faced by all customers using 3rd gen systems at the Data Center. Post this incident, we have migrated our critical systems from 3rd gen to 2nd gen, which has been in the process of getting configured and optimised over the weekend. Alongside, we have tried various methods to recover our 3rd gen systems. While the system ran today, there were issues with the live feed and delays in execution due to which we had to terminate the algos and disallow further trading.
We believe this was an unexpected, one-off incident we’ve faced for the first time in 3 years, and our team will implement solutions to avoid such an incident. We deeply regret this downtime.
We will be temporarily disabling messaging on the Telegram Discussions group as we are seeing a lot of people who are not using Quantiply create a lot of noise, spam, and also gas-light other users. We will be available on our support handles for queries at @QuantiplySupport and on Whatsapp.
⚠️ ❗️ALL USERS ❗️⚠️
Quantiply will be online tomorrow. You will be able to run your algos. The issue is resolved.
While the issue is resolved and the system is poised to run normally as usual, we will be running benchmarks on performance, loads and latency during live markets and making necessary tweaks after market hours, if required.
We will be reimbursing credits for today’s algos sometime tomorrow evening.
Quantiply will be online tomorrow. You will be able to run your algos. The issue is resolved.
While the issue is resolved and the system is poised to run normally as usual, we will be running benchmarks on performance, loads and latency during live markets and making necessary tweaks after market hours, if required.
We will be reimbursing credits for today’s algos sometime tomorrow evening.
⚠️ ❗️AC AGARWAL USERS ❗️⚠️
There are timeouts from the broker's side. We've raised the issue with them. They're checking and resolving
All other brokers are running fine
There are timeouts from the broker's side. We've raised the issue with them. They're checking and resolving
All other brokers are running fine
⚠️ ❗️AC AGARWAL USERS ❗️⚠️
Please pause your algos, there are timeouts and there are delays in execution. ACA is checking with Symphony and resolving the issue. Once it is resolved we will announce
Please pause your algos, there are timeouts and there are delays in execution. ACA is checking with Symphony and resolving the issue. Once it is resolved we will announce
⚠️ ❗️AC AGARWAL USERS ❗️⚠️
AC Agarwal has resolved the issue, orders will go through without timeouts and delays. You can resume trading.
AC Agarwal has resolved the issue, orders will go through without timeouts and delays. You can resume trading.
📣 NEW FEATURE 📣
BSE SENSEX & NSE MIDCPNIFTY F&O now available on Quantiply
Release notes & Important pointers: https://quantiply.tech/documentation/important-topics/bse-sensex-and-nse-midcpnifty
Topics covered in the Release notes:
1. Broker compatibility.
2. MIS/NRML order & Order type compatibility.
3. Sensex 'freak trade' warning & SL order placement delay functionality.
BSE SENSEX & NSE MIDCPNIFTY F&O now available on Quantiply
Release notes & Important pointers: https://quantiply.tech/documentation/important-topics/bse-sensex-and-nse-midcpnifty
Topics covered in the Release notes:
1. Broker compatibility.
2. MIS/NRML order & Order type compatibility.
3. Sensex 'freak trade' warning & SL order placement delay functionality.
⚠️ ❗️FLATTRADE USERS ❗️⚠️
Some users would have got the following 2 errors:
16795 Order cancelled due to voluntary close out
16419 This error code will be returned for invalid data in the order packet
Pending SL orders may have been cancelled. Check your algos and use the retry individual leg option to place those orders again
Some users would have got the following 2 errors:
16795 Order cancelled due to voluntary close out
16419 This error code will be returned for invalid data in the order packet
Pending SL orders may have been cancelled. Check your algos and use the retry individual leg option to place those orders again
⚠️ ❗️ANGELONE USERS ❗️⚠️
Lot of users are getting AB2001: Internal Error for order placement
Check if the order is executed in the terminal. if the order is not executed use the Retry individual leg option. If the order is executed, then it will have to be managed manually.
Issue has been raised with AngelOne
Lot of users are getting AB2001: Internal Error for order placement
Check if the order is executed in the terminal. if the order is not executed use the Retry individual leg option. If the order is executed, then it will have to be managed manually.
Issue has been raised with AngelOne
⚠️ ❗️FLATTRADE USERS ❗️⚠️
Users would be getting the error: YEL: Yel not ready for transaction
This error is coming for BSE SENSEX only
Use the retry individual leg option to retry placing the orders again, after checking if the order is placed already or not in the terminal
The issue has been raised with Flattrade. We will announce once the issue is resolved from their side.
Users would be getting the error: YEL: Yel not ready for transaction
This error is coming for BSE SENSEX only
Use the retry individual leg option to retry placing the orders again, after checking if the order is placed already or not in the terminal
The issue has been raised with Flattrade. We will announce once the issue is resolved from their side.
⚠️ ❗️FLATTRADE USERS ❗️⚠️
Getting 2 errors for order placement:
Error: 500:Not_Ok:No response from server with error code: ECONNRESET
Error: 500:Not_Ok:No response from server with error code: ETIMEDOUT
Orders may not be getting placed. Please check your positions and manage manually where required
Getting 2 errors for order placement:
Error: 500:Not_Ok:No response from server with error code: ECONNRESET
Error: 500:Not_Ok:No response from server with error code: ETIMEDOUT
Orders may not be getting placed. Please check your positions and manage manually where required
⚠️ ❗️ZERODHA USERS ❗️⚠️
The following error is being observed for order placement:
-:NetworkException:No response from server with error code: ETIMEDOUT
Check your algos and positions, and use the retry individual leg options after checking your open positions
The following error is being observed for order placement:
-:NetworkException:No response from server with error code: ETIMEDOUT
Check your algos and positions, and use the retry individual leg options after checking your open positions