#eu: downtime started, error: Failed to perform https://flespi.io/gw/xxx GET request. Usually this indicates either flespi telematics hub REST API overload or when the hub is in the maintenance mode.
#eu: downtime started, error: Failed to perform https://flespi.io/gw/xxx GET request. Usually this indicates either flespi telematics hub REST API overload or when the hub is in the maintenance mode.
Downtime Explanation - May 27, 2025
During today's scheduled datacenter switch replacement maintenance, we experienced a network issue that caused service interruption.
An engineer on site inadvertently disconnected a one of fiber links from the secondary gateway. This caused the secondary gateway to lose LAN connectivity while still retaining WAN connection and attempting to assume the primary routing role. This triggered a "split brain" situation where the secondary gateway tried to handle all traffic despite having no functional network connection due to the loose fiber connector.
Service has been restored by operating on the primary gateway only. The maintenance work is continuing as planned.
During today's scheduled datacenter switch replacement maintenance, we experienced a network issue that caused service interruption.
An engineer on site inadvertently disconnected a one of fiber links from the secondary gateway. This caused the secondary gateway to lose LAN connectivity while still retaining WAN connection and attempting to assume the primary routing role. This triggered a "split brain" situation where the secondary gateway tried to handle all traffic despite having no functional network connection due to the loose fiber connector.
Service has been restored by operating on the primary gateway only. The maintenance work is continuing as planned.
#eu: downtime started, error: Failed to perform https://flespi.io/gw/xxx GET request. Usually this indicates either flespi telematics hub REST API overload or when the hub is in the maintenance mode.
Dear flespi users,
We're investigating the datacenters network connectivity issue with our uplink provider.
We're investigating the datacenters network connectivity issue with our uplink provider.
There was a power outage in the critical network segment. This affected both datacenters for us. Engineer on site restored the equipment operation.
Now flespi is operating however our NOC system didn't stopped the downtime counting so we are investigating.
Also we expect a huge but temporary load from devices uploading buffered data which also may affect us for a while.
Now flespi is operating however our NOC system didn't stopped the downtime counting so we are investigating.
Also we expect a huge but temporary load from devices uploading buffered data which also may affect us for a while.
Within a hour since network connectivity was restored (01:27:21 - 02:47:55 UTC is exact time of the downtime) we saw a high load on the system. However the amount of free pre-reserved resources allowed us to quickly mitigate it. At the moment all the systems operating in normal mode. And we will perform an extra maintenance checks later today.
If you have any active problem please let us know through the flespi chat. Keep in mind that your channels or streams that operated close to plan limits may be temporarily blocked due to messages frequency. To mitigate such situations (and they can happen not only due to flespi failure, but for a variety of reasons) we suggest to operate your flespi ecosystem with maximum 50% load of allowed by the plan in normal mode - either picking better plan or splitting the data ingestion over multiple channels.
If you have any active problem please let us know through the flespi chat. Keep in mind that your channels or streams that operated close to plan limits may be temporarily blocked due to messages frequency. To mitigate such situations (and they can happen not only due to flespi failure, but for a variety of reasons) we suggest to operate your flespi ecosystem with maximum 50% load of allowed by the plan in normal mode - either picking better plan or splitting the data ingestion over multiple channels.
👍2
#eu: downtime started, error: Failed to perform https://flespi.io/gw/xxx GET request. Usually this indicates either flespi telematics hub REST API overload or when the hub is in the maintenance mode.
Dear flespi users!
The reason of recent downtime was due to the processing of specifically formatted REST API requests which allowed them to bypass our protection layer on the API gateway system and overloaded few internal components by CPU and net bandwidth. After investigating such a race condition and identifying the problem we quickly patched the responsible component few hours ago effectively preventing similar subsequent cases.
Have a great weekend!
The reason of recent downtime was due to the processing of specifically formatted REST API requests which allowed them to bypass our protection layer on the API gateway system and overloaded few internal components by CPU and net bandwidth. After investigating such a race condition and identifying the problem we quickly patched the responsible component few hours ago effectively preventing similar subsequent cases.
Have a great weekend!
👍5
#eu: downtime started, error: Failed to perform https://flespi.io GET request. Usually this indicates either flespi eu datacenter network uplink connection problem or when the platform is in the maintenance mode.
Dear flespi users,
The last downtime was caused by our datacenters network provider. The issue is already mitigated it and now everything is working fine.
Sorry for any inconvenience.
The last downtime was caused by our datacenters network provider. The issue is already mitigated it and now everything is working fine.
Sorry for any inconvenience.
#eu: downtime started, error: Failed to connect to flespi MQTT Broker. Usually this indicates either the problem with MQTT Broker itself or when the Broker is shutdowned for the maintenance.
Dear flespi users,
The recent downtime was caused by human mistake during router testing. The error was promptly noticed and fixed. However, it still resulted in a short downtime.
We apologize for this issue and any inconvenience it may have caused.
The recent downtime was caused by human mistake during router testing. The error was promptly noticed and fixed. However, it still resulted in a short downtime.
We apologize for this issue and any inconvenience it may have caused.
Dear flespi users,
Today Microsoft blocked access to the tenant that was used for login on flespi.io.
Therefore, currently users using Microsoft accounts for authorization will not be able to log in.
We continue to try to restore access, but we have already decided that we will disable the ability to log in through Microsoft accounts.
If you have saved a token, please log in as soon as possible and add your email in your account settings for recovery.
If you do not have access to your account, please contact us by email info@flespi.com and we will try to help you restore access to your account.
Sorry for any inconvenience.
Today Microsoft blocked access to the tenant that was used for login on flespi.io.
Therefore, currently users using Microsoft accounts for authorization will not be able to log in.
We continue to try to restore access, but we have already decided that we will disable the ability to log in through Microsoft accounts.
If you have saved a token, please log in as soon as possible and add your email in your account settings for recovery.
If you do not have access to your account, please contact us by email info@flespi.com and we will try to help you restore access to your account.
Sorry for any inconvenience.
Flespi
Account — your space inside flespi
How to register and manage the flespi account, upgrade, downgrade, and not be blocked.
👍4