Constantly losing sync, switched execution and consensus clients, problem persists

[gtluke]

I can’t quite figure this out. For about 3-4 months I’ve had multiple sync issues a day. I had been using Nethermind/Lighthouse and recently switched to Geth/Lodestar. It’s definitely worse now though as I’m missing about half my attestations.
I even went as far as to buy a new home router as my old one was crashing occasionally and seemed hot. No change.
My dashboard looks fine, but I am currently sitting at 14GB of 16GB of memory being used (is that too high?)
It’s a home built NUC that was working for years no problem. I didn’t notice anything that changed hardware/software wise that coincided with my issues. It may have been around the time I added Mev Boost? But I have recently deleted that just too see, and no change.

I’ll post my Geth and Lodestar logs when they lose sync next, should be any minute now.

Any suggestions?

Before filing a new topic, please provide the following information.

Core DAppNode Packages versions

• bind.dnp.dappnode.eth: 0.2.11
• core.dnp.dappnode.eth: 0.2.96
• dappmanager.dnp.dappnode.eth: 0.2.90, commit: df351d59
• https.dnp.dappnode.eth: 0.2.2
• ipfs.dnp.dappnode.eth: 0.2.23
• vpn.dnp.dappnode.eth:
• wifi.dnp.dappnode.eth: 0.2.9
• wireguard.dnp.dappnode.eth: 0.1.3

System info

• dockerComposeVersion: 1.25.5
• dockerServerVersion: 20.10.6
• dockerCliVersion: 20.10.6
• os: debian
• versionCodename: bullseye
• architecture: amd64
• kernel: 5.10.0-11-amd64
• Disk usage: 66%

[gtluke]

Lodestar lost sync

2024-09-13T13:55:05.000831780Z Sep-13 13:55:05.000[]                 info: Synced - slot: 9950973 - head: 0x1506…5ae4 - exec-block: valid(20742124 0x1875…) - finalized: 0x46f4…105a:310965 - peers: 100
2024-09-13T13:55:17.001706322Z Sep-13 13:55:17.001[]                 info: Synced - slot: 9950974 - head: 0xfc14…6826 - exec-block: valid(20742125 0x5654…) - finalized: 0x46f4…105a:310965 - peers: 99
2024-09-13T13:55:29.000323311Z Sep-13 13:55:29.000[]                 info: Synced - slot: 9950975 - head: 0x4653…1cda - exec-block: valid(20742126 0x42db…) - finalized: 0x46f4…105a:310965 - peers: 98
2024-09-13T13:55:41.137419117Z Sep-13 13:55:41.135[]                 info: Lodestar network=mainnet, version=v1.21.0/ae1f9d5, commit=ae1f9d5d8896cf8e5e1dc42cfeea5fbb4b69e99a
2024-09-13T13:55:41.192474283Z Sep-13 13:55:41.192[]                 info: Connected to LevelDB database path=/var/lib/data/chain-db
2024-09-13T13:56:17.573958041Z Sep-13 13:56:17.573[]                 info: Initializing beacon from a valid db state slot=9950880, epoch=310965, stateRoot=0xad0393c161af3ee308b628396807b0f7109443c1a7922bfe9d28b6e2d8bd234a, isWithinWeakSubjectivityPeriod=true
2024-09-13T13:56:19.710015184Z Sep-13 13:56:19.707[chain]            info: Historical state worker started
2024-09-13T13:56:19.732427856Z Sep-13 13:56:19.732[eth1]             info: Eth1 provider urls=http://geth.dappnode:8551
2024-09-13T13:56:19.743468765Z Sep-13 13:56:19.743[execution]        info: Execution client urls=http://geth.dappnode:8551
2024-09-13T13:57:06.842144279Z Sep-13 13:57:06.841[network]          info: running libp2p instance in worker thread
2024-09-13T13:57:07.859404383Z Sep-13 13:57:07.842[network]          info: libp2p worker started peer=16Uiu2HAm7FPosoFYPFPeh3MMV1tfzwt5ziC3NxnMLQiaHsmvDiWd
2024-09-13T13:57:10.622575133Z Sep-13 13:57:10.622[network]          info: PeerId 16Uiu2HAm7FPosoFYPFPeh3MMV1tfzwt5ziC3NxnMLQiaHsmvDiWd, Multiaddrs /ip4/0.0.0.0/tcp/9112
2024-09-13T13:57:10.627028574Z Sep-13 13:57:10.626[metrics]          info: Started metrics HTTP server address=http://0.0.0.0:8008
2024-09-13T13:57:10.830843102Z Sep-13 13:57:10.830[rest]             info: Started REST API server address=http://0.0.0.0:3500
2024-09-13T13:57:10.831013197Z Sep-13 13:57:10.830[rest]             warn: REST API server is exposed, ensure untrusted traffic cannot reach this API
2024-09-13T13:57:10.832349521Z Sep-13 13:57:10.832[]                 info: Searching peers - peers: 0 - slot: 9950983 - head: (slot -103) 0x46f4…105a - exec-block: syncing(20742032 0xc41b…) - finalized: 0x1aac…b2fd:310963
2024-09-13T13:57:10.835330238Z Sep-13 13:57:10.610[network]          info: discv5 worker started peerId=16Uiu2HAm7FPosoFYPFPeh3MMV1tfzwt5ziC3NxnMLQiaHsmvDiWd, initialENR=enr:-IO4QKIcbn_i5qLwV2NETYTNOdEBSvMmqLHwZjMNFcm0EhkKDAjoteShd8d_J8--5lJ3bsEtvkk5vOr6QuZVsMZUVZcBgmlkgnY0iXNlY3AyNTZrMaECr6omgzAB2Uw9oQOBctNx9ycZeDMuvOq5wvCqHENdpqqDdGNwgiOYg3VkcIIjmA, bindAddr4=/ip4/0.0.0.0/udp/9112
2024-09-13T13:57:20.728782835Z Sep-13 13:57:20.728[]                 info: Syncing - ? left - 0.00 slots/s - slot: 9950984 - head: (slot -104) 0x46f4…105a - exec-block: syncing(20742032 0xc41b…) - finalized: 0x1aac…b2fd:310963 - peers: 14
2024-09-13T13:57:28.531626126Z Sep-13 13:57:28.531[execution]        info: Execution client is synced oldState=ONLINE, newState=SYNCED
2024-09-13T13:57:29.072716708Z Sep-13 13:57:29.072[]                 info: Syncing - ? left - 0.00 slots/s - slot: 9950985 - head: (slot -105) 0x46f4…105a - exec-block: syncing(20742032 0xc41b…) - finalized: 0x1aac…b2fd:310963 - peers: 16
2024-09-13T13:57:41.341070111Z Sep-13 13:57:41.340[]                 info: Syncing - ? left - 0.00 slots/s - slot: 9950986 - head: (slot -106) 0x46f4…105a - exec-block: valid(20742032 0xc41b…) - finalized: 0x1aac…b2fd:310963 - peers: 15
2024-09-13T13:57:54.165036099Z Sep-13 13:57:54.164[]                 info: Syncing - 36 seconds left - 1.22 slots/s - slot: 9950987 - head: (slot -44) 0x2463…6acc - exec-block: valid(20742094 0xa4c6…) - finalized: 0x46f4…105a:310965 - peers: 33
2024-09-13T13:58:05.177577123Z Sep-13 13:58:05.177[]                 info: Syncing - 30 seconds left - 1.39 slots/s - slot: 9950988 - head: (slot -41) 0xd17e…a76d - exec-block: valid(20742098 0x61e0…) - finalized: 0x46f4…105a:310965 - peers: 34