Power supply change-up did not improve the node's health unfortunately. I now have a VM syncing on x86 hardware, once it's done I'm going to migrate my latest scb and hope for the best! The node runs now, but the ssd throws errors when it starts writing (example in comments). We apologize for the downtime yet again folks!

So RecklessApotheosis remains unhealthy after today's work. It is now conducting a -reindex-chainstate but I fear the issue is the SSD and not a bad reboot, we'll see. Simultaneously I'm spinning up a new node on a NUC. It's substantially faster than the Pi obviously, but it's only at 21GB synced and much more work to configure still with really limited availability on my part (it's a terrible week for this to go bad). So I apologize to everyone and hope to have RA up again soon, but it's not looking promising for a speedy recovery.

OK, after 1,600 miles of driving in 2 days and a day of flying, I am back home and working on RecklessApotheosis. I am checking the SSD I was using for errors and moving the blocks and chainstate from my desktop over to the ssd and put it back in node, hopefully this will be the fastest way to get the node up temporarily and clear pending htcls. Then I'll focus on getting my x86 node functioning and migrate RecklessApotheosis to it next! I'll update on progress in the comments on this post.

Alright folks, I had not run bitcoind with the txindex and blockfilterindex options when creating my seed on my desktop. So I added those last night, they finished building this morning and I rsynced them to the node, the rsync just finished and I rebooted it, I'm watching its startup with bated breath!

So new blocks, chainstate, and indexes copied to RecklessApotheosis Pi Umbrel setup, but it still doesn't boot, jam_default was also attempting a resync from 94% and perpetually failing. Turns out that's for joinmarket, and was hanging the whole startup process. So I attempted to stop and then uninstall joinmarket, but it threw errors:

umbrel@umbrel:~ $ sudo ~/umbrel/scripts/app uninstall jam
Removing images for app jam...
Network umbrel_main_network is external, skipping
Removing image ghcr.io/joinmarket-webui/joinmarket-webui-standalone:v0.0.5-clientserver-v0.9.5@sha256:5fbbc766b25449e87f5fffb689d236b0a0c9b6332c1cb75478bcbc7225d04ad9
WARNING: Image ghcr.io/joinmarket-webui/joinmarket-webui-standalone:v0.0.5-clientserver-v0.9.5@sha256:5fbbc766b25449e87f5fffb689d236b0a0c9b6332c1cb75478bcbc7225d04ad9 not found.
Deleting app data for app jam...
Removing app jam from DB...
Successfully uninstalled app jam

Working on the node again today after a few hours yesterday. Because of earlier mistakes, and the giant transaction bug, I rolled back my VM to a clean Ubuntu and restarted yesterday. I also got a new 1TB SSD, and installed it in the computer, so no more USB disks! I now have the latest raspiblitz with lnd 0.15.2-beta, I copied over my synced chain, and my old lnd directory, and am working on getting it stood up, hopefully soon.

Well, we have hit a brick wall I think. I've been working with the folks on the Noderunners channel and using chantools against my channel.db on the node, and the bolt command itself against a copy of the file. No dice, I can get some of the buckets from the database, but bolt and LND crash when attempting to read from it, compact it, or do much of anything against it. I'm worried at the moment I'm going to have to restart from channel.backup and seedphrase, and just restart RecklessApotheosis from scratch.

OK, well, not the worst event possible, but not great. I have exhausted all options restoring our channel.db file or repairing it with bolt or chantool. I have the latest channel backup and my seed phrase. We will be force closing all channels when the mempool is light. I sincerely appreciate the community's help debugging the failure and my fellow noderunners' patience. RecklessApotheosis will be reborn on much better hardware and hopefully some redundancy next. If anyone has guides for using lnd with postgres or using etcd to replicate lnd live across 2 servers, let me know in the comments! I'll begin rebuilding as soon as I have the onchain funds established, continuing as prior FCs settle.

OK, used the raspiblitz interface to perform the LND recovery of my Umbrel seed phrase & latest SCB. Now, we wait. You should all be getting FC notifications soon.

So reprocessing the scb recovery on the Umbrel Pi because no matter what I do, raspiblitz' lncli won't perform the recovery from scb (can't connect to lnd). So restarted the sequence on Umbrel on the Pi from a preexisting chainstate and it is processing each chunk of 2k blocks every 35 minutes or so. It won't let me run the scb recovery until that's done, which should be in about 7 hours, then hopefully the bulk FCs will be successful! Boy this recovery has been a pain!

Node all caught up and force closures running from the scb! If you get a FC from RecklessApotheosis let me know in the comments!

OK, hardware has all arrived and our plan is set. All but 9 FCs have settled, tomorrow we install the OS and hopefully before the end of Saturday the node will be up! There'll definitely be sats for balanced opens, so watch here when we announce channel reopens, it'll be a bos balanced-open extravaganza! If you do the Twitter thing, we made a thing you can follow, but mostly we'll be on Telegram still. https://twitter.com/RecklessNode

RecklessApotheosis Rebirth Update: Friday I set up the node hardware, RAID, and host OS. Then I set up the VM, and the general Linux stuff. Saturday installed bitcoind, made sure it was fully synced, installed postgres 15, LND, and got through much of the configuration. Got stuck on a 4 hour call with a friend debugging an nftables port forwarding issue, and then debugging why LND can't talk to postgres. Today I'm working on tor, LNDg, bos, and continuing to figure out the postgres issues. Sincerest hopes are to have the node working today or tomorrow!

OK, thanks to the tremendous help of two friends, and the NodeRunner community (@Osito_ln, @Hakezuba, @allyourbankarebelongtous, @NguTechnologies, and others!) we now have LND working on clearnet (still had issues with Tor) and caught up with rescanning my old wallet (way faster on x86 than on the Pi). I also now have LNDg & bos installed and running (LNDg is having nginx isues but that's just cosmetic, but it is running from postgres!), and bos-telegram running! Last steps will be to get RTL & TH installed and running on boot, but those don't gate getting the node up. So now we're updating our list of channels to open and their capacities. Once those are open, we'll make a broad announcement of RecklessApotheosis being successfully reborn and ready for new inbound connections! We are also trying to come up with a good strategy for balanced reopens! This has definitely been a journey, but quite the learning experience, and I'm overall really satisfied with how the node is put together now! We can't wait to welcome everyone back into the #Reckless Family!

First new channel being opened after a long arduous journey! I'm going to bed, and tomorrow I'm doing an all-day site visit to a Bitcoin Mine. But assuming this works, I'll post another update saying that RecklessApotheosis is back up and open for business!

⏳ Opening new 0.01000000 channel to Amboss.Space 03006fcf3312dae8d068ea297f58e2bd00ec1ffe214b793eda46966b6294a53ce6.
RecklessApotheosis 03c72f89

At long last, in block 760425, #RecklessApotheosis is reborn! 960 sats to open 21 channels! That's incredible! At this point, with the bulk creation of core channels underway, we are pleased to announce that RecklessApotheosis is welcoming all plebs wishing to open inbound channels to the node, and reward them with 30 days of 0/0 fees! At the moment, we haven't migrated over our scripts that refund opening fees but the intent remains to refund up to the first 300 sats of all inbound opens! We'll be announcing balanced opens capacity & policy in around 18 hours once everything settles. Happy routing, and stay #Reckless!

OK, Status Update! We transferred our scripts & LNDg settings over to the new node, and rewrote some of the scripting to be more automated. We have retro-actively refunded all inbound channels that have already been opened to us, and as we're currently trying to build our node back up, and since ZFR is spinning down, all inbound opened to #RecklessApotheosis will benefit from 0/0 fees until January 1st, 2023! This does not apply to dual-funded channels, which, by-the-way, we are now also accepting with the following criteria: dual-funded channels must be a minimum of 5m sats, fees should be 1s/vB, and dual-funded channels are immediately subject to auto-fees & do not get their opening fees refunded (but they're split anyway, so it doesn't really matter). Once we run out of sats for dual-funded opens, we run out! Act today! Thank you all again for your patience, and the community support of the awesome Plebnet folks!

24 hours in and we've opened 11 channels and grown by over 118m sats! We still have on chain sats for balanced opens and 0 fees until 2023 for inbound opens!

A happy 14th Birthday to the Whitepaper and the world changing machination it set into motion! Here's to the next 14 years!

Good morning. LND is broken. Wake up!

Details:
https://github.com/lightningnetwork/lnd/issues/7096

brqgoo commented 5 minutes ago
Bitcoin has a consensus rule that limits the number of stack items in a row to 1000. However, a P2TR spend containing OP_SUCCESSx precedes this rule regardless.

I made a P2TR spend containing an OP_SUCCESSx opcode with 500,001 empty pushes, which as a result, caused a consensus conflict between btcd and core:
https://blockstream.info/tx/73be398c4bdc43709db7398106609eea2a7841aaf3a4fa2000dc18184faa2a7e

Changing the maxWitnessItemsPerInput parameter from 500,000 to 4,000,000 solves the issue:
https://github.com/btcsuite/btcd/blob/master/wire/msgtx.go#L103


To limit your forwards, you can run:

bos limit-forwarding --disable-forwards

and leave it running in the background or inside a tmux session.