Update on #RecklessApotheosis: our migration from lnd 0.14.2 to 0.15.1 is ongoing. Software upgrades of lnd, bitcoind, tor, bos, and lndg were all successful. Node is fully synced. LND database pruning has been running for 23 hours and is 46.7% complete: 29/104 channels upgraded. At this rate it still has another day of processing, but it is steadily working. Apologies for the downtime!

#RecklessApotheosis lnd 0.15.1 upgrade update. At 50 hours into the process we are now 64.94% done with the database prune. The limiting factor feels to be the massive number of transactions that had to be processed (just shy of 1 billion txs!) and the slow USB SSD bottleneck. There are under 3 million transactions to go, 50/104 channels have been upgraded, no Force Closures🤞yet! The original eta was around 5 hours ago, but processing transactions has slowed toward the end. Thank you all for your patience! And if you upgrade a routing node to 0.15+ be sure to notify your peers and plan for some significant downtime if you're on a Pi! I'll definitely update again when it's complete or with another progress report.

#RecklessApotheosis Pruning update. We are now 74 hours into this crazy painstaking process! The node is 77.85% done pruning, 64/104 channels complete. We unfortunately did have 2 FCs since yesterday. To prevent it from pruning closed channels I'm considering rebooting. Less than 2 million out of 850 million transactions to go, but these last tx's are definitely taking longer, but disk IO and CPU usage have remained constant, so I must be missing something.

#RecklessApotheosis Pruning update. We are now a stupid number of hours into this crazy painstaking process! The node is 89.07% done pruning, 68/104 channels complete. We unfortunately did have 2 FCs since starting. It only has 928k out of 8.5 million transactions to go, but these last tx's are definitely taking longer, but disk IO and CPU usage have remained constant, so I must be missing something. Hopefully we'll be back in business soon!

#RecklessApotheosis is 99.105% done pruning! It should be up soon! Unfortunately had another FC this morning though. I'm working on making a Telegram channel for node announcements as a result of this for our channel partners.

Okay, so for anyone who wants a dual funded channel with RecklessApotheosis or experienced a Force Close, send a DM @RecklessApotheosis and when the node is up, we'll prioritize getting as many opened as we can!

Bitcoind is catching up still, then I'm going to reboot it to get the database caught up!

Bitcoind has updated and we've rebooted the node to finish the compaction step!

#RecklessApotheosis node pruning update, after 6 days and 4½ hours, we've finally completed node pruning on LND 0.15.1-beta! Our database shrunk from 23GB to 1.6GB and reboots went from 2 hours to under 10 minutes! Unfortunately despite having a long htcl delta and pausing forwards before starting, 6 days was too long and we suffered 5 force closures. A disruption we hope will be far more seldom going forward, particularly among our tor-only channel partners! Now to begin with reopens!

And if you aren't already using GordianWatch for uptime monitoring for your nodes, we highly recommend it, it's a great bot run by an excellent noderunner!

✨ channel with RecklessApotheosis was reactivated after 6D 6H 47m

Day #1 recovery for #RecklessApotheosis went well. We reopened 2 channels that had FCd during the downtime as balanced opens, won a new node partnering with inbound capacity, and cooperatively closed & grew our River Financial channel. We have 4more channels we'd like to reopen and 4 more new outbounds. This week will hopefully be one of heavy daily growth! Remember if you want to, opening inbound to RecklessApotheosis gets you 30 days of 0/0 fees and up-to 300 sats of your opening fees refunded in a keysend! We aim to be one of the go-to routing nodes on the network with great exchange, wallet, and PoS connections across the community! Join us Ina #reckless reinvention of money!

OK, RecklessApotheosis now has a new power supply, one that supports USB-C PD (we're still on a Pi) and we have a new 2TB SSD that just arrived. Power supply is installed and I'm trying to decide between upgrading its SSD or taking the time to migrate it to x86.

Power supply change-up did not improve the node's health unfortunately. I now have a VM syncing on x86 hardware, once it's done I'm going to migrate my latest scb and hope for the best! The node runs now, but the ssd throws errors when it starts writing (example in comments). We apologize for the downtime yet again folks!

So RecklessApotheosis remains unhealthy after today's work. It is now conducting a -reindex-chainstate but I fear the issue is the SSD and not a bad reboot, we'll see. Simultaneously I'm spinning up a new node on a NUC. It's substantially faster than the Pi obviously, but it's only at 21GB synced and much more work to configure still with really limited availability on my part (it's a terrible week for this to go bad). So I apologize to everyone and hope to have RA up again soon, but it's not looking promising for a speedy recovery.