🥁 IOG SPO Announcements
1.15K subscribers
6 photos
2 videos
1 file
256 links
A read-only announcement channel for Cardano SPOs to receive key updates.

Sign-up for the SPO Digest newsletter here: https://services.iohk.io/register-for-the-spo-digest
Download Telegram
SPO call tomorrow at 16:00 UTC+1 https://mailchi.mp/iohk/spo-office-hour-673430. Currently we have a limit of 100 seats for the calls so it's first come, first served until I've upgraded the account (or found an alternative). Thank you.
The following services have been upgraded successfully on mainnet:

🖧 cardano-db-sync 9.0.0
🖧 cardano-graphql 4.0.0
🖧 cardano-explorer-app 1.5.0

Explorer is back online after a short maintenance.
SPO call is in 2 hours.
Daedalus 4.0.5 is now available on Testnet, Flight, and Mainnet

Release notes: https://iohk.zendesk.com/hc/en-us/articles/900006675083
Earlier today, you will likely have noticed a pause in block production of c.25m mins after the recent epoch boundary.

Some spike in CPU usage and network pause is to be expected, and this delay was well within modeled bounds. However, this pause was significantly longer than usual.

The network is functioning normally. However, over the next 24 hours, our engineers will profile the performance of the node to determine the cause, along with additional testing so we can mitigate and optimize as needed.

We’ll be sure to share our findings with you once we have determined next steps.
QUICK UPDATE: Last night we reported on the unexpectedly long pause at the epoch boundary.

After investigation today, the node team has now identified the likely root cause. This was due to the node repeating a computation that should have been done just once at the epoch boundary, due to the way the leaders slots are computed in the consensus rules.

We’ve seen positive results from initial tests on the node, and the team has now determined an update that should alleviate the issue at the next epoch boundary.

We’re continuing to test and will next move the update into QA for final testing. We’ll bring you a further update as soon as we have determined a timeframe for release. Thanks for your cooperation.
UPDATE: On Thursday, we reported on the unexpectedly long pause at the epoch boundary.

After further investigation, we concluded this was due to the node repeating a computation that should have been done just once at the epoch boundary. 

Yesterday, our engineers identified a fix to alleviate the issue. We’ve been doing extensive testing and QA today, and subject to final checks, we will be finalizing a point release (1.26.2) which we aim to release later today.

We’ll alert you as soon as this is available.

This will ensure that block producing nodes do not unnecessarily re-evaluate the stake distribution at the epoch boundary. We’ll also continue to optimize this through future releases.

We are asking that once the release is available, all stake pool operators install this update prior to the next epoch boundary, which take place on Tuesday 20th at 21:44:51 UTC. 

Thanks very much for your support.
NEW RELEASE COMING SOON: We have now fully QAd and tagged release 1.26.2 and will soon push it to our own pools before releasing it publicly. Please expect to see this new release drop within the next few hours.
PLEASE UPDATE YOUR NODES: Last night, after completing QA testing, we successfully released a new node version (1.26.2).

This will address the block production issues we encountered earlier this week, by ensuring that block producing nodes do not unnecessarily re-evaluate the stake distribution at the epoch boundary. We’ll also continue to optimize this through future releases.

We are asking all stake pool operators install this update prior to the next epoch boundary, which takes place on Tuesday 20th at 21:44:51 UTC. 

Full release notes can be accessed here: https://github.com/input-output-hk/cardano-node/releases/tag/1.26.2

Please note: This point release is a recommended upgrade for all stake pool operators. It is not required for relays or other passive nodes, but feel free to update these too if you wish.

Thanks very much for your support and patience during this process, and see you all in the next epoch.
A polite request to fill out our SPO Pulse survey for April https://input-output.typeform.com/to/ifq5D45Y. It helps us to gather feedback into one place and for this reason it's very important to the team.
Well, there we go. A good smooth ride over epoch boundary 261. Thanks to all SPOs who updated to 1.26.2 in good time. If you have yet to do so, please ensure you do over the next few days. We’ll still continue to improve and optimize but thanks to our engineers for a job well done. Onward! 🙏
SPO call tomorrow. Good news we upgraded the Zoom so we have more seats! Email invite has been sent https://mailchi.mp/iohk/spo-community-call
🚢 SPO Digest shipped https://mailchi.mp/iohk/spo-digest-673546

(Twitter Space is today... so click that button for calendar link!)