Prysm Missed Attestations

famouspickle · 28 January 2021 10:57

Issue

Multiple missed attestations starting at slot 416194 affecting 4/4 validators. I restarted the Prysm package without effect, then rebooted the device which seemed to help.

Logs at time of incident

time="2021-01-28 07:25:11" level=info msg="Attestation schedule" attesterDutiesAtSlot=1 prefix=validator pubKeys=[0x917a1d8d7f90] slot=416227 slotInEpoch=3 totalAttestersInEpoch=4
time="2021-01-28 07:25:11" level=info msg="Attestation schedule" attesterDutiesAtSlot=1 prefix=validator pubKeys=[0x98f594432f67] slot=416230 slotInEpoch=6 totalAttestersInEpoch=4
time="2021-01-28 07:25:11" level=info msg="Attestation schedule" attesterDutiesAtSlot=1 prefix=validator pubKeys=[0xa2d19ba28533] slot=416238 slotInEpoch=14 totalAttestersInEpoch=4
time="2021-01-28 07:25:11" level=info msg="Attestation schedule" attesterDutiesAtSlot=1 prefix=validator pubKeys=[0x82cb16a804ce] slot=416249 slotInEpoch=25 totalAttestersInEpoch=4
time="2021-01-28 07:25:51" level=info msg="Submitted new attestations" AggregatorIndices=[] AttesterIndices=[34263] BeaconBlockRoot=0x63a30508bfc1 CommitteeIndex=16 Slot=416227 SourceEpoch=13005 SourceRoot=0xedae451cee23 TargetEpoch=13007 TargetRoot=0x63a30508bfc1 prefix=validator
time="2021-01-28 07:26:27" level=info msg="Submitted new attestations" AggregatorIndices=[] AttesterIndices=[1503] BeaconBlockRoot=0x63a30508bfc1 CommitteeIndex=12 Slot=416230 SourceEpoch=13005 SourceRoot=0xedae451cee23 TargetEpoch=13007 TargetRoot=0x63a30508bfc1 prefix=validator
time="2021-01-28 07:28:03" level=info msg="Submitted new attestations" AggregatorIndices=[] AttesterIndices=[34264] BeaconBlockRoot=0x63a30508bfc1 CommitteeIndex=16 Slot=416238 SourceEpoch=13005 SourceRoot=0xedae451cee23 TargetEpoch=13007 TargetRoot=0x63a30508bfc1 prefix=validator
time="2021-01-28 07:30:15" level=info msg="Submitted new attestations" AggregatorIndices=[] AttesterIndices=[1504] BeaconBlockRoot=0x63a30508bfc1 CommitteeIndex=11 Slot=416249 SourceEpoch=13005 SourceRoot=0xedae451cee23 TargetEpoch=13007 TargetRoot=0x63a30508bfc1 prefix=validator
time="2021-01-28 07:31:23" level=info msg="Previous epoch voting summary" correctlyVotedHead=false correctlyVotedSource=false correctlyVotedTarget=false epoch=13006 inclusionDistance=18446744073709551615 inclusionSlot=18446744073709551615 newBalance=32.691491478 oldBalance=32.691523284 percentChange="-0.00010%" percentChangeSinceStart="0.09424%" prefix=validator pubKey=0x98f594432f67 startBalance=32.6607109
time="2021-01-28 07:31:23" level=info msg="Previous epoch voting summary" correctlyVotedHead=false correctlyVotedSource=false correctlyVotedTarget=false epoch=13006 inclusionDistance=18446744073709551615 inclusionSlot=18446744073709551615 newBalance=32.693821357 oldBalance=32.693853163 percentChange="-0.00010%" percentChangeSinceStart="0.09419%" prefix=validator pubKey=0x82cb16a804ce startBalance=32.663055507
time="2021-01-28 07:31:23" level=info msg="Previous epoch voting summary" correctlyVotedHead=false correctlyVotedSource=false correctlyVotedTarget=false epoch=13006 inclusionDistance=18446744073709551615 inclusionSlot=18446744073709551615 newBalance=32.466533637 oldBalance=32.466565443 percentChange="-0.00010%" percentChangeSinceStart="0.09474%" prefix=validator pubKey=0x917a1d8d7f90 startBalance=32.435804156
time="2021-01-28 07:31:23" level=info msg="Previous epoch voting summary" correctlyVotedHead=false correctlyVotedSource=false correctlyVotedTarget=false epoch=13006 inclusionDistance=18446744073709551615 inclusionSlot=18446744073709551615 newBalance=32.471899256 oldBalance=32.471931062 percentChange="-0.00010%" percentChangeSinceStart="0.09482%" prefix=validator pubKey=0xa2d19ba28533 startBalance=32.441138462
time="2021-01-28 07:31:35" level=info msg="Attestation schedule" attesterDutiesAtSlot=1 prefix=validator pubKeys=[0x82cb16a804ce] slot=416269 slotInEpoch=13 totalAttestersInEpoch=4
time="2021-01-28 07:31:35" level=info msg="Attestation schedule" attesterDutiesAtSlot=1 prefix=validator pubKeys=[0x98f594432f67] slot=416273 slotInEpoch=17 totalAttestersInEpoch=4
time="2021-01-28 07:31:35" level=info msg="Attestation schedule" attesterDutiesAtSlot=1 prefix=validator pubKeys=[0x917a1d8d7f90] slot=416280 slotInEpoch=24 totalAttestersInEpoch=4
time="2021-01-28 07:31:35" level=info msg="Attestation schedule" attesterDutiesAtSlot=1 prefix=validator pubKeys=[0xa2d19ba28533] slot=416284 slotInEpoch=28 totalAttestersInEpoch=4
time="2021-01-28 07:34:19" level=info msg="Submitted new attestations" AggregatorIndices=[1504] AttesterIndices=[1504] BeaconBlockRoot=0x63a30508bfc1 CommitteeIndex=3 Slot=416269 SourceEpoch=13005 SourceRoot=0xedae451cee23 TargetEpoch=13008 TargetRoot=0x63a30508bfc1 prefix=validator
time="2021-01-28 07:35:03" level=info msg="Submitted new attestations" AggregatorIndices=[] AttesterIndices=[1503] BeaconBlockRoot=0xf96aadc7a2aa CommitteeIndex=6 Slot=416273 SourceEpoch=13005 SourceRoot=0xedae451cee23 TargetEpoch=13008 TargetRoot=0xf96aadc7a2aa prefix=validator
time="2021-01-28 07:36:27" level=info msg="Submitted new attestations" AggregatorIndices=[] AttesterIndices=[34263] BeaconBlockRoot=0xf96aadc7a2aa CommitteeIndex=2 Slot=416280 SourceEpoch=13005 SourceRoot=0xedae451cee23 TargetEpoch=13008 TargetRoot=0xf96aadc7a2aa prefix=validator
time="2021-01-28 07:37:15" level=info msg="Submitted new attestations" AggregatorIndices=[] AttesterIndices=[34264] BeaconBlockRoot=0xf96aadc7a2aa CommitteeIndex=1 Slot=416284 SourceEpoch=13005 SourceRoot=0xedae451cee23 TargetEpoch=13008 TargetRoot=0xf96aadc7a2aa prefix=validator
time="2021-01-28 07:37:47" level=info msg="Previous epoch voting summary" correctlyVotedHead=false correctlyVotedSource=false correctlyVotedTarget=false epoch=13007 inclusionDistance=18446744073709551615 inclusionSlot=18446744073709551615 newBalance=32.691460654 oldBalance=32.69149246 percentChange="-0.00010%" percentChangeSinceStart="0.09415%" prefix=validator pubKey=0x98f594432f67 startBalance=32.6607109
time="2021-01-28 07:37:47" level=info msg="Previous epoch voting summary" correctlyVotedHead=false correctlyVotedSource=false correctlyVotedTarget=false epoch=13007 inclusionDistance=18446744073709551615 inclusionSlot=18446744073709551615 newBalance=32.693790533 oldBalance=32.693822339 percentChange="-0.00010%" percentChangeSinceStart="0.09410%" prefix=validator pubKey=0x82cb16a804ce startBalance=32.663055507
time="2021-01-28 07:37:47" level=info msg="Previous epoch voting summary" correctlyVotedHead=false correctlyVotedSource=false correctlyVotedTarget=false epoch=13007 inclusionDistance=18446744073709551615 inclusionSlot=18446744073709551615 newBalance=32.466502813 oldBalance=32.466534619 percentChange="-0.00010%" percentChangeSinceStart="0.09464%" prefix=validator pubKey=0x917a1d8d7f90 startBalance=32.435804156
time="2021-01-28 07:37:47" level=info msg="Previous epoch voting summary" correctlyVotedHead=false correctlyVotedSource=false correctlyVotedTarget=false epoch=13007 inclusionDistance=18446744073709551615 inclusionSlot=18446744073709551615 newBalance=32.47186745 oldBalance=32.471899256 percentChange="-0.00010%" percentChangeSinceStart="0.09472%" prefix=validator pubKey=0xa2d19ba28533 startBalance=32.441138462

Device

DAppNodeAdvanced

Core DAppNode Packages versions

bind.dnp.dappnode.eth: 0.2.6
core.dnp.dappnode.eth: 0.2.42
dappmanager.dnp.dappnode.eth: 0.2.38, commit: 85e2b4cf
ipfs.dnp.dappnode.eth: 0.2.14
vpn.dnp.dappnode.eth: 0.2.8, commit: f9a8743e
wifi.dnp.dappnode.eth: 0.2.6

System info

docker version: Docker version 18.09.8-ce, build 0dd43dd87fd530113bf44c9bba9ad8b20ce4637f
docker compose version: docker-compose version 1.25.5, build unknown
platform: linux, x64, 5.5.0-2-amd64
Disk usage: 51%

famouspickle · 29 January 2021 00:47

Not sure if it’s related to memory usage?

pablo · 29 January 2021 09:31

Yes you are right, during last weeks seems like a few users reported the same issue: memory usage goes to very high values and start missing attestations.

We have already reported an issue to Prysm and we are working to detect the problem source.

Anyway the problem was solved rebooting DAppNode.

Will let you know with any update

famouspickle · 29 January 2021 14:44

I don’t know if that’s really an effective solution. After the first restart, my validators again started missing attestations without warning and required another restart.

pablo · 29 January 2021 15:14

Absolutely it is not.

I though that after rebooting DAppNode it was solved. Then it seems to be a prysm bug 99%. We are in contact with prysm team to solve it ASAP.

Could you paste here the logs please?

famouspickle · 29 January 2021 16:51

Thanks for the help. I posted the relevant logs at the top of the issue. Any other specific logs to include?

famouspickle · 30 January 2021 19:24

Feel like it’s about to happen again:

gunninginc · 2 February 2021 17:28

I’m having the same issue… saw my cpu usage on my NUC jumping to 98% which may or may not have been correlated to prysm starting to miss attestations. fact is though that i missed 6 hours of attestations while I slept… restarted prysm this morning and i’m back to proper attesting. Any known resolution coming from prysm? Cheers.

Poweriser · 8 February 2021 11:25

A similar problem. Stopping Prysm for no apparent reason. Reloading the package helped and everything is fine again.

mihailo · 8 February 2021 22:10

Same with me. I haven’t checked memory usage, but Prysm just stops synchronizing every few days, and after restarting(package, or dappnode as a whole), it starts again normally.
It’s happening every 3-4 days since last upgrade.

GrayETH · 7 May 2021 22:56

Did you guys ever find a viable solution to fix this? It is happening to me also for the last few months.