#eu: downtime ended, period: 69 second(s)

Dear flespi users!

We successfully finished the transition of our routing layer from one system to another and later this year will be ready to provide you with datacenter-level redundancy. Sorry for all network interruptions that occurred today. It was really a difficult day with our teams worked hard both in Vilnius (LT) and Groningen (NL) and coordinating all changes on software and hardware layer.

Thanks for being with us and have a great day!
Your flespi team.

Dear flespi users!

We are preparing to launch our services from the second datacenter which may require you to adjust IP address whitelisting (mostly this affects streams pushing data towards your integration). For those who do not use any kind of IP address whitelisting there will be no noticeable changes.

Here are the details of the upcoming update: https://forum.flespi.com/d/25-changelog-flespi-telematics-hub/114

#eu: downtime started, error: Failed to perform https://flespi.io GET request. Usually this indicates either flespi eu datacenter network uplink connection problem or when the platform is in the maintenance mode.

#eu: downtime ended, period: 853 second(s)

Dear flespi users,

The recent downtime was caused by a DoS attack, which has now been mitigated.
Only the REST API endpoint was affected during the downtime; all other parts of the platform, including data receiving from devices, were functioning correctly.
We will analyze the situation and take necessary steps to prevent similar incidents from occurring in the future.

We apologize for any inconvenience caused.

Dear flespi users!

Today we enabled production services in the secondary datacenter and now it works in the parallel to the primary. As indicated earlier this may affect your operations if you are whitelisting IP addresses from which flespi may access you via streams, plugins and webhooks (or any other IP-based filter on your side).
Please double check that everything is operating correctly in your integration now. And do not hesitate to contact us using CHAT button in top-right corner of your flespi.io account if any notice any kind of problem.

More details you can find here: https://forum.flespi.com/d/25-changelog-flespi-telematics-hub/114

#eu: downtime started, error: Failed to receive messages posted by the simulated device with GET /gw/channels/XXX/messages REST API call within 5 seconds. It usually means that flespi storage system is either shutdowned for maintenance or currently operating under high load and some database operations may be delayed.

#eu: downtime ended, period: 123 second(s)

We are performing maintenance operations with telematics messages database and with a rather big item being migrated part of the database system was overloaded enough that our status checking bots indicated this as downtime. This is controlled process we are running today. Except for sometimes slower telematics messages access (and only for channels, devices messages storage is not affected) no more impact should be made during this maintenance.

If interested here is an article on how we are constantly checking our availability from multiple locations 24/7 and never turn them off even during any maintenance: https://flespi.com/blog/flespi-service-level-agreement-how-to-guarantee-999-of-system-uptime

Sorry for any inconvenience and have a great weekend!

Currently we see some periodic and short(packet drops are less then 1 minute) connectivity problems between flespi network and AWS zones, especially: sa-east-1(South Amercia), aws-us-east-2(US East), aws-us-west-1(US West). AWS health console reports no problems with their services so it looks like our uplink and moreover its magistrale provider are currently experiencing some difficulties in the link between America and Europe. The problem is not constant, but appears regularly(3 times) for last 30 minutes. At the same time Google Cloud services nodes located in America works perfectly, so this can be very specific problem in a single overseas cable.

We can do nothing with this, just indicating that for those who is running their services in AWS zones. If the problem will persist further we will report it today later. You can monitor live status of flespi network here: https://status.flespi.io/

#eu: downtime started, error: Failed to perform https://flespi.io/gw/xxx GET request. Usually this indicates either flespi telematics hub REST API overload or when the hub is in the maintenance mode.

#eu: downtime ended, period: 27 second(s)

#eu: downtime started, error: Failed to receive messages posted by the simulated device with GET /gw/devices/XXX/messages REST API call within 5 seconds. It usually means that flespi storage system is either shutdowned for maintenance or currently operating under high load and some database operations may be delayed.

#eu: downtime ended, period: 19 second(s)

#eu: downtime started, error: Failed to add command to device commands queue. Usually this indicates either flespi telematics hub REST API overload or when the hub is in the maintenance mode.

#eu: downtime ended, period: 26 second(s)

#eu: downtime started, error: Failed to receive messages posted by the simulated device with GET /gw/devices/XXX/messages REST API call within 5 seconds. It usually means that flespi storage system is either shutdowned for maintenance or currently operating under high load and some database operations may be delayed.

#eu: downtime ended, period: 23 second(s)

Recent short problems with flespi telematics hub performance (REST API requests to https://flespi.io/gw/...) were the reason of our team performance tests of new functionality. We apologize for any inconveniences and will improve the protection layer of this new functionality.
Have a great day!