#RecklessApotheosis is 99.105% done pruning! It should be up soon! Unfortunately had another FC this morning though. I'm working on making a Telegram channel for node announcements as a result of this for our channel partners.

Okay, so for anyone who wants a dual funded channel with RecklessApotheosis or experienced a Force Close, send a DM @RecklessApotheosis and when the node is up, we'll prioritize getting as many opened as we can!

Bitcoind is catching up still, then I'm going to reboot it to get the database caught up!

Bitcoind has updated and we've rebooted the node to finish the compaction step!

#RecklessApotheosis node pruning update, after 6 days and 4½ hours, we've finally completed node pruning on LND 0.15.1-beta! Our database shrunk from 23GB to 1.6GB and reboots went from 2 hours to under 10 minutes! Unfortunately despite having a long htcl delta and pausing forwards before starting, 6 days was too long and we suffered 5 force closures. A disruption we hope will be far more seldom going forward, particularly among our tor-only channel partners! Now to begin with reopens!

And if you aren't already using GordianWatch for uptime monitoring for your nodes, we highly recommend it, it's a great bot run by an excellent noderunner!

✨ channel with RecklessApotheosis was reactivated after 6D 6H 47m

Day #1 recovery for #RecklessApotheosis went well. We reopened 2 channels that had FCd during the downtime as balanced opens, won a new node partnering with inbound capacity, and cooperatively closed & grew our River Financial channel. We have 4more channels we'd like to reopen and 4 more new outbounds. This week will hopefully be one of heavy daily growth! Remember if you want to, opening inbound to RecklessApotheosis gets you 30 days of 0/0 fees and up-to 300 sats of your opening fees refunded in a keysend! We aim to be one of the go-to routing nodes on the network with great exchange, wallet, and PoS connections across the community! Join us Ina #reckless reinvention of money!

OK, RecklessApotheosis now has a new power supply, one that supports USB-C PD (we're still on a Pi) and we have a new 2TB SSD that just arrived. Power supply is installed and I'm trying to decide between upgrading its SSD or taking the time to migrate it to x86.

Power supply change-up did not improve the node's health unfortunately. I now have a VM syncing on x86 hardware, once it's done I'm going to migrate my latest scb and hope for the best! The node runs now, but the ssd throws errors when it starts writing (example in comments). We apologize for the downtime yet again folks!

So RecklessApotheosis remains unhealthy after today's work. It is now conducting a -reindex-chainstate but I fear the issue is the SSD and not a bad reboot, we'll see. Simultaneously I'm spinning up a new node on a NUC. It's substantially faster than the Pi obviously, but it's only at 21GB synced and much more work to configure still with really limited availability on my part (it's a terrible week for this to go bad). So I apologize to everyone and hope to have RA up again soon, but it's not looking promising for a speedy recovery.

OK, after 1,600 miles of driving in 2 days and a day of flying, I am back home and working on RecklessApotheosis. I am checking the SSD I was using for errors and moving the blocks and chainstate from my desktop over to the ssd and put it back in node, hopefully this will be the fastest way to get the node up temporarily and clear pending htcls. Then I'll focus on getting my x86 node functioning and migrate RecklessApotheosis to it next! I'll update on progress in the comments on this post.

Alright folks, I had not run bitcoind with the txindex and blockfilterindex options when creating my seed on my desktop. So I added those last night, they finished building this morning and I rsynced them to the node, the rsync just finished and I rebooted it, I'm watching its startup with bated breath!

So new blocks, chainstate, and indexes copied to RecklessApotheosis Pi Umbrel setup, but it still doesn't boot, jam_default was also attempting a resync from 94% and perpetually failing. Turns out that's for joinmarket, and was hanging the whole startup process. So I attempted to stop and then uninstall joinmarket, but it threw errors:

umbrel@umbrel:~ $ sudo ~/umbrel/scripts/app uninstall jam
Removing images for app jam...
Network umbrel_main_network is external, skipping
Removing image ghcr.io/joinmarket-webui/joinmarket-webui-standalone:v0.0.5-clientserver-v0.9.5@sha256:5fbbc766b25449e87f5fffb689d236b0a0c9b6332c1cb75478bcbc7225d04ad9
WARNING: Image ghcr.io/joinmarket-webui/joinmarket-webui-standalone:v0.0.5-clientserver-v0.9.5@sha256:5fbbc766b25449e87f5fffb689d236b0a0c9b6332c1cb75478bcbc7225d04ad9 not found.
Deleting app data for app jam...
Removing app jam from DB...
Successfully uninstalled app jam

Working on the node again today after a few hours yesterday. Because of earlier mistakes, and the giant transaction bug, I rolled back my VM to a clean Ubuntu and restarted yesterday. I also got a new 1TB SSD, and installed it in the computer, so no more USB disks! I now have the latest raspiblitz with lnd 0.15.2-beta, I copied over my synced chain, and my old lnd directory, and am working on getting it stood up, hopefully soon.

Well, we have hit a brick wall I think. I've been working with the folks on the Noderunners channel and using chantools against my channel.db on the node, and the bolt command itself against a copy of the file. No dice, I can get some of the buckets from the database, but bolt and LND crash when attempting to read from it, compact it, or do much of anything against it. I'm worried at the moment I'm going to have to restart from channel.backup and seedphrase, and just restart RecklessApotheosis from scratch.

OK, well, not the worst event possible, but not great. I have exhausted all options restoring our channel.db file or repairing it with bolt or chantool. I have the latest channel backup and my seed phrase. We will be force closing all channels when the mempool is light. I sincerely appreciate the community's help debugging the failure and my fellow noderunners' patience. RecklessApotheosis will be reborn on much better hardware and hopefully some redundancy next. If anyone has guides for using lnd with postgres or using etcd to replicate lnd live across 2 servers, let me know in the comments! I'll begin rebuilding as soon as I have the onchain funds established, continuing as prior FCs settle.

OK, used the raspiblitz interface to perform the LND recovery of my Umbrel seed phrase & latest SCB. Now, we wait. You should all be getting FC notifications soon.

So reprocessing the scb recovery on the Umbrel Pi because no matter what I do, raspiblitz' lncli won't perform the recovery from scb (can't connect to lnd). So restarted the sequence on Umbrel on the Pi from a preexisting chainstate and it is processing each chunk of 2k blocks every 35 minutes or so. It won't let me run the scb recovery until that's done, which should be in about 7 hours, then hopefully the bulk FCs will be successful! Boy this recovery has been a pain!