W3S Prater (and main) v0.1.26 crashed after add more than 17 keys

Technical Support
1

Before filing a new topic, please provide the following information.

Core DAppNode Packages versions

  • bind.dnp.dappnode.eth: 0.2.10
  • core.dnp.dappnode.eth: 0.2.87
  • dappmanager.dnp.dappnode.eth: 0.2.81, commit: 7bf8e6f5
  • https.dnp.dappnode.eth: 0.2.1
  • ipfs.dnp.dappnode.eth: 0.2.22
  • vpn.dnp.dappnode.eth:
  • wifi.dnp.dappnode.eth: 0.2.9
  • wireguard.dnp.dappnode.eth: 0.1.3

System info

  • dockerComposeVersion: 1.25.0
  • dockerServerVersion: 25.0.0
  • dockerCliVersion: 25.0.0
  • os: debian
  • versionCodename: bullseye
  • architecture: amd64
  • kernel: 5.10.0-27-amd64
  • Disk usage: 36%

grep: /data/keyfiles/.yaml: No such file or directory
2024-01-23 16:23:59.322+00:00 | main | INFO | Web3SignerApp | Web3Signer has started with args --key-store-path=/data/keyfiles,–http-listen-port=9000,–http-listen-host=0.0.0.0,–http-host-allowlist=web3signer.web3signer-prater.dappnode,brain.web3signer-prater.dappnode,validator.prysm-prater.dappnode,–http-cors-origins=,–metrics-enabled=true,–metrics-host,0.0.0.0,–metrics-port,9091,–metrics-host-allowlist=*,–idle-connection-timeout-seconds=900,eth2,–network=prater,–slashing-protection-db-url=jdbc:postgresql://postgres.web3signer-prater.dappnode:5432/web3signer,–slashing-protection-db-username=postgres,–slashing-protection-db-password=password,–slashing-protection-pruning-enabled=true,–slashing-protection-pruning-epochs-to-keep=500,–key-manager-api-enabled=true
2024-01-23 16:23:59.331+00:00 | main | INFO | Web3SignerApp | Version = web3signer/v24.1.0/linux-x86_64/-eclipseadoptium-openjdk64bitservervm-java-17
2024-01-23 16:23:59.879+00:00 | main | INFO | Eth2SubCommand | Network: prater
Spec Name: PHASE0, Fork Epoch: 0, First Slot: 0
Spec Name: ALTAIR, Fork Epoch: 36660, First Slot: 1173120
Spec Name: BELLATRIX, Fork Epoch: 112260, First Slot: 3592320
Spec Name: CAPELLA, Fork Epoch: 162304, First Slot: 5193728
Spec Name: DENEB, Fork Epoch: 231680, First Slot: 7413760

2024-01-23 16:23:59.906+00:00 | main | INFO | HikariDataSource | HikariPool-1 - Starting…
2024-01-23 16:24:00.188+00:00 | main | INFO | HikariPool | HikariPool-1 - Added connection org.postgresql.jdbc.PgConnection@7d97e06c
2024-01-23 16:24:00.189+00:00 | main | INFO | HikariDataSource | HikariPool-1 - Start completed.
2024-01-23 16:24:03.640+00:00 | main | INFO | HikariDataSource | HikariPool-2 - Starting…
2024-01-23 16:24:03.645+00:00 | main | INFO | HikariPool | HikariPool-2 - Added connection org.postgresql.jdbc.PgConnection@73c48264
2024-01-23 16:24:03.646+00:00 | main | INFO | HikariDataSource | HikariPool-2 - Start completed.
2024-01-23 16:24:03.751+00:00 | main | INFO | MetricsHttpService | Starting metrics http service on 0.0.0.0:9091
2024-01-23 16:24:03.801+00:00 | vert.x-eventloop-thread-0 | INFO | MetricsHttpService | Metrics service started and listening on 0.0.0.0:9091
2024-01-23 16:24:03.811+00:00 | pool-2-thread-1 | INFO | SignerLoader | Loading signer configuration metadata files from /data/keyfiles
2024-01-23 16:24:03.813+00:00 | pool-2-thread-1 | INFO | SignerLoader | Signer configuration metadata files read in memory 0 in 00:00:00.001
2024-01-23 16:24:03.814+00:00 | pool-2-thread-1 | INFO | SignerLoader | Total Artifact Signer loaded via configuration files: 0
Error count 0
Time Taken: 00:00:00.000.
2024-01-23 16:24:03.816+00:00 | pool-2-thread-1 | WARN | Eth2Runner | No BLS keys loaded. Check that the key store has BLS key config files
2024-01-23 16:24:03.817+00:00 | pool-2-thread-1 | INFO | DefaultArtifactSignerProvider | Total signers (keys) currently loaded in memory: 0
2024-01-23 16:24:03.828+00:00 | main | INFO | Runner | Web3Signer has started with TLS disabled, and ready to handle signing requests on 0.0.0.0:9000
2024-01-23 16:24:38.836+00:00 | ForkJoinPool.commonPool-worker-1 | WARN | PoolBase | HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@7d97e06c (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
2024-01-23 16:25:38.833+00:00 | ForkJoinPool.commonPool-worker-1 | WARN | PoolBase | HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@2d1e0c3b (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
2024-01-23 16:25:43.839+00:00 | ForkJoinPool.commonPool-worker-1 | WARN | PoolBase | HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@517ec1cd (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
2024-01-23 16:26:08.833+00:00 | ForkJoinPool.commonPool-worker-1 | WARN | PoolBase | HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@37a33203 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
2024-01-23 16:26:38.833+00:00 | ForkJoinPool.commonPool-worker-1 | WARN | PoolBase | HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@5792f9b9 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
2024-01-23 16:28:38.833+00:00 | ForkJoinPool.commonPool-worker-1 | WARN | PoolBase | HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@6763c78c (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
2024-01-23 16:28:43.837+00:00 | ForkJoinPool.commonPool-worker-1 | WARN | PoolBase | HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@3fd41820 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
2024-01-23 16:29:43.415+00:00 | pool-2-thread-1 | INFO | DefaultArtifactSignerProvider | Removed signer with identifier ‘0x8cc1761eb0bec79d06da9a18ebe366e7a60235c727a70dedc954cec88b172d299ba3580078b06279a875b72a170f896d’
2024-01-23 16:30:32.636+00:00 | vert.x-worker-thread-5 | INFO | BLS | BLS: loaded BLST library
2024-01-23 16:30:32.649+00:00 | pool-2-thread-1 | INFO | DefaultArtifactSignerProvider | Loaded new signer for identifier ‘0x8cc1761eb0bec79d06da9a18ebe366e7a60235c727a70dedc954cec88b172d299ba3580078b06279a875b72a170f896d’
2024-01-23 16:30:32.674+00:00 | vert.x-worker-thread-5 | INFO | RegisteredValidators | Validators registered successfully in database:1
2024-01-23 16:31:38.833+00:00 | ForkJoinPool.commonPool-worker-1 | WARN | PoolBase | HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@2151180c (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
2024-01-23 16:31:39.306+00:00 | vert.x-worker-thread-16 | WARN | PoolBase | HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@6340305c (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
2024-01-23 16:31:40.028+00:00 | pool-2-thread-1 | INFO | DefaultArtifactSignerProvider | Loaded new signer for identifier ‘0x9557f0239d34190a0f21a2340eb1ad682a0be78bc60c5d3a5697f07fba71d9d8974954ca11df5bc43ef7f4464263b79b’
2024-01-23 16:31:40.033+00:00 | vert.x-worker-thread-16 | INFO | RegisteredValidators | Validators registered successfully in database:1
2024-01-23 16:31:40.700+00:00 | pool-2-thread-1 | INFO | DefaultArtifactSignerProvider | Loaded new signer for identifier ‘0xb2e0592f65b24998fbac522193a8d4a777150b553b6b5eda6e6cd476702fa6225549e192f97a73995886548c25727e7a’
2024-01-23 16:31:42.380+00:00 | vert.x-worker-thread-16 | INFO | RegisteredValidators | Validators registered successfully in database:1
2024-01-23 16:31:43.023+00:00 | pool-2-thread-1 | INFO | DefaultArtifactSignerProvider | Loaded new signer for identifier ‘0x83048b7dba56b6af870f9aa9b77efcf45485de9e7900c74fab1773b48c3ccb3896dcd4bc81ddfc613f12374a39132d29’
2024-01-23 16:31:43.026+00:00 | vert.x-worker-thread-16 | INFO | RegisteredValidators | Validators registered successfully in database:1
2024-01-23 16:31:43.680+00:00 | pool-2-thread-1 | INFO | DefaultArtifactSignerProvider | Loaded new signer for identifier ‘0xa6481e64d13809b2b4c0ada329e42b3d6d6357aa150cc0084d7f0e79a408394d1bd660963debb78090cd854a43cc9901’
2024-01-23 16:31:44.524+00:00 | vert.x-worker-thread-16 | INFO | RegisteredValidators | Validators registered successfully in database:1
2024-01-23 16:31:45.164+00:00 | pool-2-thread-1 | INFO | DefaultArtifactSignerProvider | Loaded new signer for identifier ‘0x8d0fea58acb07cc712492b77ace81343028ba5f5f1aa8a891f11ffffc6dd08c8224166c66108f1f2942688e39f939113’
2024-01-23 16:32:08.833+00:00 | ForkJoinPool.commonPool-worker-1 | WARN | PoolBase | HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@4dbb94d9 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
2024-01-23 16:32:34.738+00:00 | vertx-blocked-thread-checker | WARN | BlockedThreadChecker | Thread Thread[vert.x-worker-thread-16,5,main] has been blocked for 60440 ms, time limit is 60000 ms
io.vertx.core.VertxException: Thread blocked
at sun.nio.ch.Net.poll(Native Method) ~[?:?]
at sun.nio.ch.NioSocketImpl.park(Unknown Source) ~[?:?]
at sun.nio.ch.NioSocketImpl.timedRead(Unknown Source) ~[?:?]
at sun.nio.ch.NioSocketImpl.implRead(Unknown Source) ~[?:?]
at sun.nio.ch.NioSocketImpl.read(Unknown Source) ~[?:?]
at sun.nio.ch.NioSocketImpl$1.read(Unknown Source) ~[?:?]
at java.net.Socket$SocketInputStream.read(Unknown Source) ~[?:?]
at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:161) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:128) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:113) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.core.PGStream.receiveChar(PGStream.java:465) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2120) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:356) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:496) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:413) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:333) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:319) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:295) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.jdbc.PgConnection.execSQLQuery(PgConnection.java:537) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.jdbc.PgConnection.execSQLQuery(PgConnection.java:530) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.jdbc.PgConnection.getTransactionIsolation(PgConnection.java:986) ~[postgresql-42.5.3.jar:42.5.3]
at com.zaxxer.hikari.pool.HikariProxyConnection.getTransactionIsolation(HikariProxyConnection.java) ~[HikariCP-5.0.1.jar:?]
at org.jdbi.v3.core.Handle.getTransactionIsolationLevel(Handle.java:860) ~[jdbi3-core-3.36.0.jar:3.36.0]
at org.jdbi.v3.core.Handle.inTransaction(Handle.java:756) ~[jdbi3-core-3.36.0.jar:3.36.0]
at org.jdbi.v3.core.Jdbi.lambda$inTransaction$7(Jdbi.java:441) ~[jdbi3-core-3.36.0.jar:3.36.0]
at org.jdbi.v3.core.Jdbi$$Lambda$768/0x0000000800666178.withHandle(Unknown Source) ~[?:?]
at org.jdbi.v3.core.Jdbi.withHandle(Jdbi.java:361) ~[jdbi3-core-3.36.0.jar:3.36.0]
at org.jdbi.v3.core.Jdbi.inTransaction(Jdbi.java:441) ~[jdbi3-core-3.36.0.jar:3.36.0]
at tech.pegasys.web3signer.slashingprotection.RegisteredValidators.registerValidators(RegisteredValidators.java:77) ~[web3signer-slashing-protection-24.1.0.jar:24.1.0]
at tech.pegasys.web3signer.slashingprotection.DbValidatorManager.lambda$addValidator$1(DbValidatorManager.java:59) ~[web3signer-slashing-protection-24.1.0.jar:24.1.0]
at tech.pegasys.web3signer.slashingprotection.DbValidatorManager$$Lambda$749/0x0000000800617a40.useHandle(Unknown Source) ~[?:?]
at org.jdbi.v3.core.HandleConsumer.lambda$asCallback$0(HandleConsumer.java:32) ~[jdbi3-core-3.36.0.jar:3.36.0]
at org.jdbi.v3.core.HandleConsumer$$Lambda$751/0x000000080061c000.withHandle(Unknown Source) ~[?:?]
at org.jdbi.v3.core.transaction.LocalTransactionHandler$BoundLocalTransactionHandler.inTransaction(LocalTransactionHandler.java:208) ~[jdbi3-core-3.36.0.jar:3.36.0]
at org.jdbi.v3.core.transaction.SerializableTransactionRunner.inTransaction(SerializableTransactionRunner.java:55) ~[jdbi3-core-3.36.0.jar:3.36.0]
at org.jdbi.v3.core.Handle.inTransaction(Handle.java:725) ~[jdbi3-core-3.36.0.jar:3.36.0]
at org.jdbi.v3.core.Handle.useTransaction(Handle.java:736) ~[jdbi3-core-3.36.0.jar:3.36.0]
at org.jdbi.v3.core.Jdbi.lambda$useTransaction$6(Jdbi.java:416) ~[jdbi3-core-3.36.0.jar:3.36.0]
at org.jdbi.v3.core.Jdbi$$Lambda$750/0x0000000800617c78.useHandle(Unknown Source) ~[?:?]
at org.jdbi.v3.core.HandleConsumer.lambda$asCallback$0(HandleConsumer.java:32) ~[jdbi3-core-3.36.0.jar:3.36.0]
at org.jdbi.v3.core.HandleConsumer$$Lambda$751/0x000000080061c000.withHandle(Unknown Source) ~[?:?]
at org.jdbi.v3.core.Jdbi.withHandle(Jdbi.java:367) ~[jdbi3-core-3.36.0.jar:3.36.0]
at org.jdbi.v3.core.Jdbi.useHandle(Jdbi.java:383) ~[jdbi3-core-3.36.0.jar:3.36.0]
at org.jdbi.v3.core.Jdbi.useTransaction(Jdbi.java:416) ~[jdbi3-core-3.36.0.jar:3.36.0]
at tech.pegasys.web3signer.slashingprotection.DbValidatorManager.addValidator(DbValidatorManager.java:56) ~[web3signer-slashing-protection-24.1.0.jar:24.1.0]
at tech.pegasys.web3signer.core.service.http.handlers.keymanager.imports.ImportKeystoresHandler.handle(ImportKeystoresHandler.java:161) ~[web3signer-core-24.1.0.jar:24.1.0]
at tech.pegasys.web3signer.core.service.http.handlers.keymanager.imports.ImportKeystoresHandler.handle(ImportKeystoresHandler.java:51) ~[web3signer-core-24.1.0.jar:24.1.0]
at io.vertx.ext.web.impl.BlockingHandlerDecorator.lambda$handle$0(BlockingHandlerDecorator.java:48) ~[vertx-web-4.4.6.jar:4.4.6]
at io.vertx.ext.web.impl.BlockingHandlerDecorator$$Lambda$728/0x00000008005f5068.handle(Unknown Source) ~[?:?]
at io.vertx.core.impl.ContextBase.lambda$executeBlocking$1(ContextBase.java:180) ~[vertx-core-4.4.6.jar:4.4.6]
at io.vertx.core.impl.ContextBase$$Lambda$710/0x00000008005d6b40.handle(Unknown Source) ~[?:?]
at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:277) ~[vertx-core-4.4.6.jar:4.4.6]
at io.vertx.core.impl.ContextBase.lambda$internalExecuteBlocking$2(ContextBase.java:199) ~[vertx-core-4.4.6.jar:4.4.6]
at io.vertx.core.impl.ContextBase$$Lambda$712/0x00000008005d7648.run(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
at java.lang.Thread.run(Unknown Source) ~[?:?]
2024-01-23 16:32:35.736+00:00 | vertx-blocked-thread-checker | WARN | BlockedThreadChecker | Thread Thread[vert.x-worker-thread-16,5,main] has been blocked for 61439 ms, time limit is 60000 ms
io.vertx.core.VertxException: Thread blocked
at sun.nio.ch.Net.poll(Native Method) ~[?:?]
at sun.nio.ch.NioSocketImpl.park(Unknown Source) ~[?:?]
at sun.nio.ch.NioSocketImpl.timedRead(Unknown Source) ~[?:?]
at sun.nio.ch.NioSocketImpl.implRead(Unknown Source) ~[?:?]
at sun.nio.ch.NioSocketImpl.read(Unknown Source) ~[?:?]
at sun.nio.ch.NioSocketImpl$1.read(Unknown Source) ~[?:?]
at java.net.Socket$SocketInputStream.read(Unknown Source) ~[?:?]
at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:161) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:128) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:113) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.core.PGStream.receiveChar(PGStream.java:465) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2120) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:356) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:496) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:413) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:333) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:319) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:295) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.jdbc.PgConnection.execSQLQuery(PgConnection.java:537) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.jdbc.PgConnection.execSQLQuery(PgConnection.java:530) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.jdbc.PgConnection.getTransactionIsolation(PgConnection.java:986) ~[postgresql-42.5.3.jar:42.5.3]
at com.zaxxer.hikari.pool.HikariProxyConnection.getTransactionIsolation(HikariProxyConnection.java) ~[HikariCP-5.0.1.jar:?]
at org.jdbi.v3.core.Handle.getTransactionIsolationLevel(Handle.java:860) ~[jdbi3-core-3.36.0.jar:3.36.0]
at org.jdbi.v3.core.Handle.inTransaction(Handle.java:756) ~[jdbi3-core-3.36.0.jar:3.36.0]
at org.jdbi.v3.core.Jdbi.lambda$inTransaction$7(Jdbi.java:441) ~[jdbi3-core-3.36.0.jar:3.36.0]
at org.jdbi.v3.core.Jdbi$$Lambda$768/0x0000000800666178.withHandle(Unknown Source) ~[?:?]
at org.jdbi.v3.core.Jdbi.withHandle(Jdbi.java:361) ~[jdbi3-core-3.36.0.jar:3.36.0]
at org.jdbi.v3.core.Jdbi.inTransaction(Jdbi.java:441) ~[jdbi3-core-3.36.0.jar:3.36.0]
at tech.pegasys.web3signer.slashingprotection.RegisteredValidators.registerValidators(RegisteredValidators.java:77) ~[web3signer-slashing-protection-24.1.0.jar:24.1.0]
at tech.pegasys.web3signer.slashingprotection.DbValidatorManager.lambda$addValidator$1(DbValidatorManager.java:59) ~[web3signer-slashing-protection-24.1.0.jar:24.1.0]
at tech.pegasys.web3signer.slashingprotection.DbValidatorManager$$Lambda$749/0x0000000800617a40.useHandle(Unknown Source) ~[?:?]
at org.jdbi.v3.core.HandleConsumer.lambda$asCallback$0(HandleConsumer.java:32) ~[jdbi3-core-3.36.0.jar:3.36.0]
at org.jdbi.v3.core.HandleConsumer$$Lambda$751/0x000000080061c000.withHandle(Unknown Source) ~[?:?]
at org.jdbi.v3.core.transaction.LocalTransactionHandler$BoundLocalTransactionHandler.inTransaction(LocalTransactionHandler.java:208) ~[jdbi3-core-3.36.0.jar:3.36.0]
at org.jdbi.v3.core.transaction.SerializableTransactionRunner.inTransaction(SerializableTransactionRunner.java:55) ~[jdbi3-core-3.36.0.jar:3.36.0]
at org.jdbi.v3.core.Handle.inTransaction(Handle.java:725) ~[jdbi3-core-3.36.0.jar:3.36.0]
at org.jdbi.v3.core.Handle.useTransaction(Handle.java:736) ~[jdbi3-core-3.36.0.jar:3.36.0]
at org.jdbi.v3.core.Jdbi.lambda$useTransaction$6(Jdbi.java:416) ~[jdbi3-core-3.36.0.jar:3.36.0]
at org.jdbi.v3.core.Jdbi$$Lambda$750/0x0000000800617c78.useHandle(Unknown Source) ~[?:?]
at org.jdbi.v3.core.HandleConsumer.lambda$asCallback$0(HandleConsumer.java:32) ~[jdbi3-core-3.36.0.jar:3.36.0]
at org.jdbi.v3.core.HandleConsumer$$Lambda$751/0x000000080061c000.withHandle(Unknown Source) ~[?:?]
at org.jdbi.v3.core.Jdbi.withHandle(Jdbi.java:367) ~[jdbi3-core-3.36.0.jar:3.36.0]
at org.jdbi.v3.core.Jdbi.useHandle(Jdbi.java:383) ~[jdbi3-core-3.36.0.jar:3.36.0]
at org.jdbi.v3.core.Jdbi.useTransaction(Jdbi.java:416) ~[jdbi3-core-3.36.0.jar:3.36.0]
at tech.pegasys.web3signer.slashingprotection.DbValidatorManager.addValidator(DbValidatorManager.java:56) ~[web3signer-slashing-protection-24.1.0.jar:24.1.0]
at tech.pegasys.web3signer.core.service.http.handlers.keymanager.imports.ImportKeystoresHandler.handle(ImportKeystoresHandler.java:161) ~[web3signer-core-24.1.0.jar:24.1.0]
at tech.pegasys.web3signer.core.service.http.handlers.keymanager.imports.ImportKeystoresHandler.handle(ImportKeystoresHandler.java:51) ~[web3signer-core-24.1.0.jar:24.1.0]
at io.vertx.ext.web.impl.BlockingHandlerDecorator.lambda$handle$0(BlockingHandlerDecorator.java:48) ~[vertx-web-4.4.6.jar:4.4.6]
at io.vertx.ext.web.impl.BlockingHandlerDecorator$$Lambda$728/0x00000008005f5068.handle(Unknown Source) ~[?:?]
at io.vertx.core.impl.ContextBase.lambda$executeBlocking$1(ContextBase.java:180) ~[vertx-core-4.4.6.jar:4.4.6]
at io.vertx.core.impl.ContextBase$$Lambda$710/0x00000008005d6b40.handle(Unknown Source) ~[?:?]
at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:277) ~[vertx-core-4.4.6.jar:4.4.6]
at io.vertx.core.impl.ContextBase.lambda$internalExecuteBlocking$2(ContextBase.java:199) ~[vertx-core-4.4.6.jar:4.4.6]
at io.vertx.core.impl.ContextBase$$Lambda$712/0x00000008005d7648.run(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
at java.lang.Thread.run(Unknown Source) ~[?:?]
2024-01-23 16:32:36.736+00:00 | vertx-blocked-thread-checker | WARN | BlockedThreadChecker | Thread Thread[vert.x-worker-thread-16,5,main] has been blocked for 62439 ms, time limit is 60000 ms
io.vertx.core.VertxException: Thread blocked
at sun.nio.ch.Net.poll(Native Method) ~[?:?]
at sun.nio.ch.NioSocketImpl.park(Unknown Source) ~[?:?]
at sun.nio.ch.NioSocketImpl.timedRead(Unknown Source) ~[?:?]
at sun.nio.ch.NioSocketImpl.implRead(Unknown Source) ~[?:?]
at sun.nio.ch.NioSocketImpl.read(Unknown Source) ~[?:?]
at sun.nio.ch.NioSocketImpl$1.read(Unknown Source) ~[?:?]
at java.net.Socket$SocketInputStream.read(Unknown Source) ~[?:?]
at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:161) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:128) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:113) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.core.PGStream.receiveChar(PGStream.java:465) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2120) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:356) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:496) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:413) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:333) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:319) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:295) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.jdbc.PgConnection.execSQLQuery(PgConnection.java:537) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.jdbc.PgConnection.execSQLQuery(PgConnection.java:530) ~[postgresql-42.5.3.jar:42.5.3]
at org.postgresql.jdbc.PgConnection.getTransactionIsolation(PgConnection.java:986) ~[postgresql-42.5.3.jar:42.5.3]
at com.zaxxer.hikari.pool.HikariProxyConnection.getTransactionIsolation(HikariProxyConnection.java) ~[HikariCP-5.0.1.jar:?]
at org.jdbi.v3.core.Handle.getTransactionIsolationLevel(Handle.java:860) ~[jdbi3-core-3.36.0.jar:3.36.0]
at org.jdbi.v3.core.Handle.inTransaction(Handle.java:756) ~[jdbi3-core-3.36.0.jar:3.36.0]
at org.jdbi.v3.core.Jdbi.lambda$inTransaction$7(Jdbi.java:441) ~[jdbi3-core-3.36.0.jar:3.36.0]
at org.jdbi.v3.core.Jdbi$$Lambda$768/0x0000000800666178.withHandle(Unknown Source) ~[?:?]
at org.jdbi.v3.core.Jdbi.withHandle(Jdbi.java:361) ~[jdbi3-core-3.36.0.jar:3.36.0]
at org.jdbi.v3.core.Jdbi.inTransaction(Jdbi.java:441) ~[jdbi3-core-3.36.0.jar:3.36.0]
at tech.pegasys.web3signer.slashingprotection.RegisteredValidators.registerValidators(RegisteredValidators.java:77) ~[web3signer-slashing-protection-24.1.0.jar:24.1.0]
at tech.pegasys.web3signer.slashingprotection.DbValidatorManager.lambda$addValidator$1(DbValidatorManager.java:59) ~[web3signer-slashing-protection-24.1.0.jar:24.1.0]
at tech.pegasys.web3signer.slashingprotection.DbValidatorManager$$Lambda$749/0x0000000800617a40.useHandle(Unknown Source) ~[?:?]
at org.jdbi.v3.core.HandleConsumer.lambda$asCallback$0(HandleConsumer.java:32) ~[jdbi3-core-3.36.0.jar:3.36.0]
at org.jdbi.v3.core.HandleConsumer$$Lambda$751/0x000000080061c000.withHandle(Unknown Source) ~[?:?]
at org.jdbi.v3.core.transaction.LocalTransactionHandler$BoundLocalTransactionHandler.inTransaction(LocalTransactionHandler.java:208) ~[jdbi3-core-3.36.0.jar:3.36.0]
at org.jdbi.v3.core.transaction.SerializableTransactionRunner.inTransaction(SerializableTransactionRunner.java:55) ~[jdbi3-core-3.36.0.jar:3.36.0]
at org.jdbi.v3.core.Handle.inTransaction(Handle.java:725) ~[jdbi3-core-3.36.0.jar:3.36.0]
at org.jdbi.v3.core.Handle.useTransaction(Handle.java:736) ~[jdbi3-core-3.36.0.jar:3.36.0]
at org.jdbi.v3.core.Jdbi.lambda$useTransaction$6(Jdbi.java:416) ~[jdbi3-core-3.36.0.jar:3.36.0]
at org.jdbi.v3.core.Jdbi$$Lambda$750/0x0000000800617c78.useHandle(Unknown Source) ~[?:?]
at org.jdbi.v3.core.HandleConsumer.lambda$asCallback$0(HandleConsumer.java:32) ~[jdbi3-core-3.36.0.jar:3.36.0]
at org.jdbi.v3.core.HandleConsumer$$Lambda$751/0x000000080061c000.withHandle(Unknown Source) ~[?:?]
at org.jdbi.v3.core.Jdbi.withHandle(Jdbi.java:367) ~[jdbi3-core-3.36.0.jar:3.36.0]
at org.jdbi.v3.core.Jdbi.useHandle(Jdbi.java:383) ~[jdbi3-core-3.36.0.jar:3.36.0]
at org.jdbi.v3.core.Jdbi.useTransaction(Jdbi.java:416) ~[jdbi3-core-3.36.0.jar:3.36.0]
at tech.pegasys.web3signer.slashingprotection.DbValidatorManager.addValidator(DbValidatorManager.java:56) ~[web3signer-slashing-protection-24.1.0.jar:24.1.0]
at tech.pegasys.web3signer.core.service.http.handlers.keymanager.imports.ImportKeystoresHandler.handle(ImportKeystoresHandler.java:161) ~[web3signer-core-24.1.0.jar:24.1.0]
at tech.pegasys.web3signer.core.service.http.handlers.keymanager.imports.ImportKeystoresHandler.handle(ImportKeystoresHandler.java:51) ~[web3signer-core-24.1.0.jar:24.1.0]
at io.vertx.ext.web.impl.BlockingHandlerDecorator.lambda$handle$0(BlockingHandlerDecorator.java:48) ~[vertx-web-4.4.6.jar:4.4.6]
at io.vertx.ext.web.impl.BlockingHandlerDecorator$$Lambda$728/0x00000008005f5068.handle(Unknown Source) ~[?:?]
at io.vertx.core.impl.ContextBase.lambda$executeBlocking$1(ContextBase.java:180) ~[vertx-core-4.4.6.jar:4.4.6]
at io.vertx.core.impl.ContextBase$$Lambda$710/0x00000008005d6b40.handle(Unknown Source) ~[?:?]
at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:277) ~[vertx-core-4.4.6.jar:4.4.6]
at io.vertx.core.impl.ContextBase.lambda$internalExecuteBlocking$2(ContextBase.java:199) ~[vertx-core-4.4.6.jar:4.4.6]
at io.vertx.core.impl.ContextBase$$Lambda$712/0x00000008005d7648.run(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
at java.lang.Thread.run(Unknown Source) ~[?:?]
2024-01-23 16:32:37.736+00:00 | vertx-blocked-thread-checker | WARN | BlockedThreadChecker | Thread Thread[vert.x-worker-thread-16,5,main] has been blocked for 63440 ms, time limit is 60000 ms

1 Like
2

here is logs from Prysm Prater (W3S not worked properly)

time=“2024-01-23 16:52:48” level=info msg=“Attestation schedule” attesterDutiesAtSlot=1 prefix=validator pubKeys=[0x90e0fa4be4ec] slot=7460064 slotInEpoch=0 totalAttestersInEpoch=13
time=“2024-01-23 16:52:48” level=info msg=“Attestation schedule” attesterDutiesAtSlot=1 prefix=validator pubKeys=[0x8ebd17b1660b] slot=7460065 slotInEpoch=1 timeTillDuty=12s totalAttestersInEpoch=13
time=“2024-01-23 16:52:48” level=info msg=“Attestation schedule” attesterDutiesAtSlot=1 prefix=validator pubKeys=[0xb58994e366ba] slot=7460069 slotInEpoch=5 timeTillDuty=1m0s totalAttestersInEpoch=13
time=“2024-01-23 16:52:48” level=info msg=“Attestation schedule” attesterDutiesAtSlot=1 prefix=validator pubKeys=[0x83048b7dba56] slot=7460070 slotInEpoch=6 timeTillDuty=1m12s totalAttestersInEpoch=13
time=“2024-01-23 16:52:48” level=info msg=“Attestation schedule” attesterDutiesAtSlot=1 prefix=validator pubKeys=[0x8f715f8aa2c9] slot=7460073 slotInEpoch=9 timeTillDuty=1m48s totalAttestersInEpoch=13
time=“2024-01-23 16:52:48” level=info msg=“Attestation schedule” attesterDutiesAtSlot=1 prefix=validator pubKeys=[0xa60a6edb7547] slot=7460075 slotInEpoch=11 timeTillDuty=2m12s totalAttestersInEpoch=13
time=“2024-01-23 16:52:48” level=info msg=“Attestation schedule” attesterDutiesAtSlot=1 prefix=validator pubKeys=[0xa6481e64d138] slot=7460078 slotInEpoch=14 timeTillDuty=2m48s totalAttestersInEpoch=13
time=“2024-01-23 16:52:48” level=info msg=“Attestation schedule” attesterDutiesAtSlot=2 prefix=validator pubKeys=[0x9557f0239d34 0x8e31cce646d0] slot=7460082 slotInEpoch=18 timeTillDuty=3m36s totalAttestersInEpoch=13
time=“2024-01-23 16:52:48” level=info msg=“Attestation schedule” attesterDutiesAtSlot=1 prefix=validator pubKeys=[0xb2e0592f65b2] slot=7460083 slotInEpoch=19 timeTillDuty=3m48s totalAttestersInEpoch=13
time=“2024-01-23 16:52:48” level=info msg=“Attestation schedule” attesterDutiesAtSlot=1 prefix=validator pubKeys=[0xa5d43de78692] slot=7460085 slotInEpoch=21 timeTillDuty=4m12s totalAttestersInEpoch=13
time=“2024-01-23 16:52:48” level=info msg=“Attestation schedule” attesterDutiesAtSlot=1 prefix=validator pubKeys=[0x80d9c455cb06] slot=7460086 slotInEpoch=22 timeTillDuty=4m24s totalAttestersInEpoch=13
time=“2024-01-23 16:52:48” level=info msg=“Attestation schedule” attesterDutiesAtSlot=1 prefix=validator pubKeys=[0x974e7a735917] slot=7460094 slotInEpoch=30 timeTillDuty=6m0s totalAttestersInEpoch=13
time=“2024-01-23 16:52:48” level=info msg=“Submitted builder validator registration settings for custom builders” prefix=validator
time=“2024-01-23 16:53:00” level=error msg=“Could not sign attestation” error=“failed to execute json request: Post “http://web3signer.web3signer-prater.dappnode:9000/api/v1/eth2/sign/0x90e0fa4be4ecd470276735a4aed59e9fdf69ff2fd163d6800b57c691d42d9dbb8f70e522c4f2ba8979d6ee55df2ea86a”: context deadline exceeded” prefix=validator pubKey=0x90e0fa4be4ec slot=7460064
time=“2024-01-23 16:53:07” level=info msg=“Submitted new attestations” AggregatorIndices= AttesterIndices=[608606] BeaconBlockRoot=0x1fc103cb2466 CommitteeIndex=57 Slot=7460065 SourceEpoch=233126 SourceRoot=0x6ec4cfbe1080 TargetEpoch=233127 TargetRoot=0xeec194a69f4f prefix=validator
time=“2024-01-23 16:53:52” level=info msg=“Submitted new attestations” AggregatorIndices= AttesterIndices=[608611] BeaconBlockRoot=0xfd0984526419 CommitteeIndex=31 Slot=7460069 SourceEpoch=233126 SourceRoot=0x6ec4cfbe1080 TargetEpoch=233127 TargetRoot=0xeec194a69f4f prefix=validator
time=“2024-01-23 16:54:12” level=error msg=“Could not request attestation to sign at slot” error=“rpc error: code = DeadlineExceeded desc = context deadline exceeded” prefix=validator pubKey=0x83048b7dba56 slot=7460070
time=“2024-01-23 16:54:27” level=error msg=“Could not receive slots from beacon node, could not connect” error=“rpc error: code = Unavailable desc = error reading from server: read tcp 172.33.0.11:52444->172.33.0.4:4000: read: connection reset by peer” prefix=validator
time=“2024-01-23 16:54:27” level=warning msg=“slots stream interrupted” error=“rpc error: code = Unavailable desc = error reading from server: read tcp 172.33.0.11:52444->172.33.0.4:4000: read: connection reset by peer: could not connect” prefix=validator
time=“2024-01-23 16:54:47” level=info msg=“Submitted new attestations” AggregatorIndices= AttesterIndices=[608608] BeaconBlockRoot=0x8415e202c88e CommitteeIndex=32 Slot=7460073 SourceEpoch=233126 SourceRoot=0x6ec4cfbe1080 TargetEpoch=233127 TargetRoot=0xeec194a69f4f prefix=validator

1 Like
3

More Logs from W3S Prater

2024-01-23 16:42:10.566+00:00 | pool-2-thread-1 | INFO | DefaultArtifactSignerProvider | Removed signer with identifier ‘0x9422c484cfb2763998d5aa944a996adffe850f8a637d37302eab2a0eaeb94c28b794d1143133a79f4c6ef94069b1dbba’
2024-01-23 16:43:08.833+00:00 | ForkJoinPool.commonPool-worker-1 | WARN | PoolBase | HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@4831ba69 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
2024-01-23 16:43:13.839+00:00 | ForkJoinPool.commonPool-worker-1 | WARN | PoolBase | HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@6be3f467 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
2024-01-23 16:43:38.833+00:00 | ForkJoinPool.commonPool-worker-1 | WARN | PoolBase | HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@50609d0e (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
2024-01-23 16:44:08.833+00:00 | ForkJoinPool.commonPool-worker-1 | WARN | PoolBase | HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@54c267aa (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
2024-01-23 16:44:38.833+00:00 | ForkJoinPool.commonPool-worker-1 | WARN | PoolBase | HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@74990317 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
2024-01-23 16:45:38.833+00:00 | ForkJoinPool.commonPool-worker-1 | WARN | PoolBase | HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@49f828ac (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
2024-01-23 16:47:08.834+00:00 | ForkJoinPool.commonPool-worker-1 | WARN | PoolBase | HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@36d5ce56 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
2024-01-23 16:47:13.837+00:00 | ForkJoinPool.commonPool-worker-1 | WARN | PoolBase | HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@4e21f3f6 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
2024-01-23 16:48:21.649+00:00 | vert.x-worker-thread-9 | WARN | PoolBase | HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@42e559f8 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
2024-01-23 16:49:08.833+00:00 | ForkJoinPool.commonPool-worker-1 | WARN | PoolBase | HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@268fca13 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
2024-01-23 16:50:21.005+00:00 | vert.x-worker-thread-12 | WARN | PoolBase | HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@6ae7679a (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
2024-01-23 16:50:38.833+00:00 | ForkJoinPool.commonPool-worker-1 | WARN | PoolBase | HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@3e4252f8 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
2024-01-23 16:51:21.005+00:00 | vert.x-worker-thread-2 | WARN | PoolBase | HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@5858a710 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
2024-01-23 16:51:26.009+00:00 | vert.x-worker-thread-2 | WARN | PoolBase | HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@1965ad2b (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
2024-01-23 16:51:57.005+00:00 | vert.x-worker-thread-7 | WARN | PoolBase | HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@3be99281 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
2024-01-23 16:52:02.009+00:00 | vert.x-worker-thread-7 | WARN | PoolBase | HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@7ad498ba (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
2024-01-23 16:52:57.005+00:00 | vert.x-worker-thread-7 | WARN | PoolBase | HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@6007106b (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
2024-01-23 16:53:02.010+00:00 | vert.x-worker-thread-7 | WARN | PoolBase | HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@6cad78c1 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.

1 Like
4

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.