INCIDENT DIAGNOSTICS & RECOVERY

Node Troubleshooting Center

Field-tested diagnostic runbooks for recovering stuck, crashing, or desynced blockchain nodes. Includes safe state pruning procedures, firewall checks, and port conflict resolution.

DiskPruningStorageGethSystemd

Resolving Node Disk Space Exhaustion & Safe Pruning

Symptoms & Error Logs

Node service stops abruptly; error message 'no space left on device'; state database corruption warning in system logs.

Likely Root Cause

Ancient block headers, debug traces, or state trie accumulation filled available disk volume.

Diagnostic Commands:

bash

df -h /
du -sh /var/lib/* 2>/dev/null | sort -h

Safe Recovery Steps:

1. Stop the node daemon cleanly using systemctl: sudo systemctl stop geth 2. Check if client supports online or offline snapshot pruning. 3. For Geth: execute prune-state: geth snapshot prune-state --datadir /var/lib/ethereum/geth 4. Restart service and monitor free disk: sudo systemctl start geth && df -h

Risk Warning: NEVER run 'rm -rf' on client data directories while the process is running. State pruning can take 3 to 10 hours; do not interrupt it.

Verification Check: Run 'df -h' and confirm at least 20% free space on the data volume. Verify node resumes sync from previous block height.

P2PPeersFirewallNTPNetworking

Zero Peers Connected: P2P Discovery and Port Forwarding Failure

Symptoms & Error Logs

Node boots but peer count stays at 0; block height does not advance; 'no suitable peers found' in logs.

Likely Root Cause

Inbound P2P ports blocked by VPS firewall, missing bootnodes in config, or clock skew on the host machine.

Diagnostic Commands:

bash

sudo ufw status
timedatectl status
curl -s http://localhost:8545 -X POST -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"net_peerCount","params":[],"id":1}'

Safe Recovery Steps:

1. Verify system clock is synchronized via NTP: sudo timedatectl set-ntp true 2. Confirm P2P ports (e.g. 30303 or 26656) allow both TCP and UDP: sudo ufw allow 30303/tcp && sudo ufw allow 30303/udp 3. Append official static bootnodes to client configuration file. 4. Restart node service and tail logs for peer handshakes.

Risk Warning: Ensure you do not accidentally open RPC ports (8545, 8551, 26657) while modifying firewall rules.

Verification Check: Query peer count via RPC or CLI and verify count is > 10 within 3 minutes.

EthereumConsensusJWTEngine APIAuth

Consensus & Execution Engine API JWT Authentication Failure

Symptoms & Error Logs

Lighthouse/Prysm logs: 'HTTP error 401 Unauthorized from execution client' or 'Invalid JWT token'.

Likely Root Cause

Geth and Lighthouse point to different JWT secret files, or file permissions prevent one daemon from reading it.

Diagnostic Commands:

bash

ls -la /var/lib/ethereum/jwt/jwt.hex
sudo diff <(sudo cat /var/lib/ethereum/jwt/jwt.hex) <(sudo cat /var/lib/consensus/jwt.hex)

Safe Recovery Steps:

1. Ensure both clients point to the EXACT same path: /var/lib/ethereum/jwt/jwt.hex. 2. Grant read permissions to both service accounts or shared group: sudo chmod 644 /var/lib/ethereum/jwt/jwt.hex 3. Restart both services: sudo systemctl restart geth && sudo systemctl restart lighthouse-beacon

Risk Warning: Do not leave JWT directory world-writable (777).

Verification Check: Inspect consensus client logs with 'journalctl -u lighthouse-beacon -f' and verify 'Engine API connection established'.

Loading NodeDrops...

Synchronizing state with peers