flespi noc
68 members
41 links
flespi platform NOC
Download Telegram
to view and join the conversation
Dear flespi users!

We're done working with the metadata database automatic failover subsystem. There was no impact on the user's data, except for the last 22 seconds of unavailability to change metadata of the flespi platform objects like name of the channel or device.

Sorry for any inconvenience!
downtime started, error: Failed to receive messages posted by the simulated device with GET /gw/channels/XXX/messages REST API call within 5 seconds. It usually means that flespi storage system is either shutdowned for maintenance or currently operating under high load and some database operations may be delayed.
Dear flespi users!

We are currently migrating the database mirrors into new format and this operation generate a high load on the telematics database which our bots can notice and even generate downtime status, cause delays in messages processing sometimes can be more then allowed 5 seconds.

Except telematics database access all other systems are almost unaffected.

This is controlled process, please be patient.
downtime ended, period: 365 seconds
downtime started, error: Failed to receive messages posted by the simulated device with GET /gw/channels/XXX/messages REST API call within 5 seconds. It usually means that flespi storage system is either shutdowned for maintenance or currently operating under high load and some database operations may be delayed.
---
downtime ended, period: 1 seconds
downtime started, error: Failed to receive messages posted by the simulated device with GET /gw/channels/XXX/messages REST API call within 5 seconds. It usually means that flespi storage system is either shutdowned for maintenance or currently operating under high load and some database operations may be delayed.
downtime ended, period: 48 seconds
downtime started, error: Failed to perform flespi.io HTTP GET method. Usually this indicates either flespi datacenter network uplink connection problem or when the platform is in the maintenance mode.
downtime ended, period: 117 seconds
Dear flespi users!

We're investigating the last downtime cause. So far, everything indicates a network issue at our uplink provider side.
Sorry for any inconveniences.
downtime started, error: Failed to perform flespi.io HTTP GET method. Usually this indicates either flespi datacenter network uplink connection problem or when the platform is in the maintenance mode.
downtime ended, period: 116 seconds
Dear flespi users!

The last downtime was caused by the same network issue as the previous. We will investigate it with our uplink provider too.

Sorry for any inconveniences.
Dear flespi users!

Last two midnight (EU time) network issues were related to the faulted hardware in AMS-IX our up-link provider PoP. Finally they detected the origins of the problem and rerouted traffic via other links replacing the faulted equipment. We hope that these short two minutes interrupts did not influenced a lot your experience with flespi and of course up-link provider hardware is out of our scope of control.

Wish you nice and sunny Sunday!
downtime started, error: Failed to perform flespi.io HTTP GET method. Usually this indicates either flespi datacenter network uplink connection problem or when the platform is in the maintenance mode.
downtime ended, period: 106 seconds
Dear flespi users! We have some technical issues with main infrastructure right now. The team is working to resolve all the problems. It will be fixed as soon as possible.
All issues have been fixed. Common downtime duration was 801 seconds. The problem was caused by update of our services configuration mechanism. That's why our NOC bots did not reported it here. Sorry for any inconveniences.
Dear flespi users!

This message is important only in case you are using flespi analytics in automated calculation mode, for all other flespi users please ignore it.

At the moment we are performing maintenance operations with services that performs real-time calculations for analytic services described here: https://flespi.com/blog/flespi-analytics-architecture

Due to this maintenance we restart these (calc) services with clean cache and so far it takes 4 hours for each service to synchronize 100% of calculated devices. During synchronization process, devices that are already synchronized operate as usual - it means that in 1 hour 25% of affected devices operating in normal way. You may control the active state of calculation for the specified device with special 'synced' property.

We have a plan to improve in the nearest future the synchronization process, first make it much faster, second make it redundant, so that each device can be served by two separate calculation processes on different servers and never goes into unsynced state for more then few seconds.

We are doing it slowly, restarting no more than 20% of services each 4 hours. The maintenance windows are today and tomorrow. No more notifications about it will be posted.
Dear flespi users!

Tomorrow (13 February 2020) we will update ACL REST API. Details are explained on the forum: https://forum.flespi.com/d/143-changelog-flespi-platform-api
This update will not break backward compatibility but you may start receiving error messages along with data response from our services if you are working with ACL tokens. These errors indicate that your ACL usage is deprecated and will not work as expected after 4 weeks.

If these changes are critical for you please contact us ASAP via helpbox from flespi panel.

Sorry for any inconveniences.