Optimistic block production due to RPC error

[Dapper (Miriam Srapyan)]

Hi everyone!

I need help to get my mainnet validators running again. I do not know for sure the root cause, but I can start with what happened:
After a normal restart of my DAppNode Extreme, my UI told me I needed to re-open the ports / enable UPnP again. I do not know why they closed in the first place, but I got UPnP enabled again and a consecutive “no NAT loopback” message appeared which persists until today. Since I get access via my own WIFI, this is not a real issue for me.

However, the following is troubling me:
Since then, my Geth client does not sync, my prysm client does.
My web3signer logs report this:

But looking at my prysm logs, the following messages appear:
My validators say my node is producing optimistic blocks due to a rpc error and therefore cannot request validation requests:

My beacon chain node shows the following:

My geth client seems to be waiting for the right prysm inputs, because now the beacon chain backfill keeps failing:

I hope I provided enough info, I am greatful for any help!

Before filing a new topic, please provide the following information.

Core DAppNode Packages versions

• bind.dnp.dappnode.eth: 0.2.6
• core.dnp.dappnode.eth: 0.2.64
• dappmanager.dnp.dappnode.eth: 0.2.59, commit: 26522964
• https.dnp.dappnode.eth: 0.1.4
• ipfs.dnp.dappnode.eth: 0.2.18
• vpn.dnp.dappnode.eth: 0.2.8, commit: f9a8743e
• wifi.dnp.dappnode.eth: 0.2.8

System info

• dockerComposeVersion: 1.25.5
• dockerServerVersion: 19.03.8
• dockerCliVersion: 19.03.8
• os: debian
• versionCodename: bullseye
• architecture: amd64
• kernel: 5.9.0-1-amd64
• Disk usage: 50%

[Dapper (Miriam Srapyan)]

UPDATE:
For those interested, I solved the mystery myself. As I already assumed, I am a terrible analyser and thought the mistake to lay in the Prysm client, but actually it turned out to be Geth. Web3signer and internet is working fine as well. I completely discarded Geth and its files, then reinstalled from scratch. 1 Day of waiting and now I am validating again.

Once again, thank you DAppNode for making this a reality!

[peatrick (peatrick.eth)]

Thank you for posting your follow up with the solution. I’ve been experiencing some similar “weird” (for lack of a better, or more technical term) network or connectivity issues / errors that I’ve struggled to troubleshoot on my own.

I was afraid of just starting to blow things away, but might attempt your path if i’m still unable to make any progress. Glad you’re back in business. Stay safe.