Validator security: sentry nodes, remote signing and firewalls

Lock down a CometBFT validator so it peers only through sentry nodes, signs through a remote signer, and sits behind ufw rules, key-only SSH and offline key backups. It is for operators who already run a synced validator on a Cosmos SDK chain.

Prerequisites#

• A running, synced validator. If you still need to sync a node, see Run a Cosmos Hub full node with state sync.
• One validator and two sentry hosts, plus a signer host on its own network segment, as the Cosmos-KMS best practices advise. Size them using your chain's hardware guidance.
• A private network between all hosts, such as a VPC, VLAN or WireGuard tunnel.
• Ubuntu Server 24.04 LTS or 26.04 LTS with ufw, OpenSSH, curl, jq, nc and gpg.
• On the signer host, git, make, a C compiler (build-essential) and Go (opens in a new tab) 1.26 or later. The Cosmos-KMS Makefile builds with cgo.
• Your chain's binary, written here as <appd>, and its home directory, written as <chain-home>.
• Your CometBFT version, from <appd> comet version. Current lines are v0.38.x (Cosmos SDK v0.53), v0.39.x (SDK v0.54) and v0.40.x (SDK v0.55). The config keys below exist in v0.38 and v0.40.

Other placeholders: <...-id> values are node IDs, <...-ip> values are host addresses, <admin-ip> is the address you SSH from, <admin-user> is your login user, <node-service> is the systemd unit that runs your node, and <your-chain-id> is the chain ID.

1. Plan the layout and collect node IDs#

The validator talks only to your sentries, over the private network. The sentries connect to the public P2P network and never gossip the validator's address. The signer holds the consensus key and dials out to the validator.

Each sentry costs money and adds a network hop. In return, a traffic flood hits a sentry you can replace instead of the machine that signs.

Run this on every node and write down each ID:

<appd> comet show-node-id --home <chain-home>

2. Configure the sentry nodes#

Edit <chain-home>/config/config.toml on each sentry. Keep your chain's seeds value.

1[p2p]
2laddr = "tcp://0.0.0.0:26656"
3external_address = "<sentry-public-ip>:26656"
4pex = true
5persistent_peers = "<validator-node-id>@<validator-private-ip>:26656"
6private_peer_ids = "<validator-node-id>"
7unconditional_peer_ids = "<validator-node-id>"
8addr_book_strict = false

private_peer_ids stops the sentry from gossiping the validator's address. unconditional_peer_ids keeps the sentry connected to the validator when it hits its peer limits. addr_book_strict = false lets the node accept private addresses. You can add the other sentry to persistent_peers too.

3. Configure the validator node#

Peering and double-sign check#

1[p2p]
2pex = false
3seeds = ""
4persistent_peers = "<sentry-1-id>@<sentry-1-private-ip>:26656,<sentry-2-id>@<sentry-2-private-ip>:26656"
5unconditional_peer_ids = "<sentry-1-id>,<sentry-2-id>"
6private_peer_ids = ""
7addr_book_strict = false
8
9[consensus]
10double_sign_check_height = 10

With pex = false, the validator connects only to persistent_peers and shares no addresses. The value 10 for double_sign_check_height comes from the CometBFT validator docs.

If your [blocksync] section has adaptive_sync (v0.40), keep it false. The config reference warns that it risks equivocation, which neither the state file nor a KMS can stop.

Keep the APIs local#

In config.toml:

[rpc]
laddr = "tcp://127.0.0.1:26657"
unsafe = false

In <chain-home>/config/app.toml:

[api]
enable = false

[grpc]
address = "localhost:9090"

The CometBFT production docs advise against public RPC on validators. Serve public RPC from a separate full node behind a reverse proxy.

Restart the sentries. Then stop the validator, wait more than 10 blocks and start it. A faster restart trips the double-sign check and makes the node panic.

4. Move signing to a remote signer#

Pick a signer#

Plan to move off Horcrux. Cosmos-KMS and TMKMS run one signer per key, so if that signer stops, you miss blocks.

Do not fix that with a second signer or a standby validator on the same key. Downtime gets you jailed, and you can unjail. A double sign gets you tombstoned, and a tombstoned validator cannot rejoin the set.

Build Cosmos-KMS#

On the signer host, build the tagged release. Check the releases page (opens in a new tab) for a newer tag first.

git clone https://github.com/cosmos/kms
cd kms
git checkout v0.1.0
make install
export PATH="$PATH:$(go env GOPATH)/bin"
kms version
kms init --home ~/.kms

kms init writes kms.yaml and identity.json. Pass --home on every kms command, or the signer uses the current directory.

Start ~/.kms/kms.yaml with the chain and validator entries:

chains:
  - id: <your-chain-id>

validators:
  - chain_id: <your-chain-id>
    addr: tcp://<validator-private-ip>:26659
    identity_key: identity.json

The keys entry depends on which key you sign with. Path A moves your current key, and Path B uses a new HSM key.

Path A. Move your existing key#

The Cosmos-KMS README says the file backend is not for production custody. Treat it as a step toward an HSM key. Cosmos-KMS v0.1.0 builds against CometBFT v0.40, so if your chain runs an older line, rehearse on a testnet first.

Stop the validator and read its last signed height and round:

sudo systemctl stop <node-service>
jq -r '.height, .round' <chain-home>/data/priv_validator_state.json

In the validator's config.toml, edit the existing top-level line:

priv_validator_laddr = "tcp://<validator-private-ip>:26659"

The node then signs nothing locally and waits for the signer. The Cosmos-KMS docs use port 26659. With tcp://, the firewall is the only thing that keeps other hosts off this port, and the ufw rules in step 6 allow only the signer.

From your admin workstation, copy the key straight from the validator to the signer:

scp -3 <admin-user>@<validator-private-ip>:<chain-home>/config/priv_validator_key.json <admin-user>@<signer-ip>:.kms/

Add this to kms.yaml:

keys:
  - chain_ids: [<your-chain-id>]
    backend: file
    algorithm: ed25519
    key_file: priv_validator_key.json

On the signer, write the double-sign floor and start the signer. Use the height and round you read earlier, and wait until more than 10 blocks have passed since you stopped the validator.

chmod 600 ~/.kms/priv_validator_key.json
kms state init --chain <your-chain-id> --height <last-height> --round <last-round> --home ~/.kms
kms start --home ~/.kms

Then start the validator within five seconds:

sudo systemctl start <node-service>

Path B. Use an HSM key#

The signer only uses keys that already exist, so create the key with your HSM vendor's tooling first. <pkcs11-module-path> is the vendor's PKCS#11 library. Set KMS_PIN in the signer's service environment, not in shell history.

1keys:
2  - chain_ids: [<your-chain-id>]
3    backend: pkcs11
4    algorithm: ed25519
5    module: <pkcs11-module-path>
6    token_label: <token-label>
7    key_label: <key-label>
8    pin_env: KMS_PIN

An HSM-generated key is a new consensus key. Never point your live validator at it.

On SDK v0.55 with CometBFT v0.40 or later, follow Rotate a consensus key held in Cosmos-KMS (opens in a new tab). That procedure runs a second node with priv_validator_laddr set and its own signer. Use --allow-fresh-state only on that new signer's first start. You can rotate once per unbonding period, and each rotation burns the key_rotation_fee. On older chains, the backend guide (opens in a new tab) suggests registering a new validator instead.

If you stay on TMKMS for now#

TMKMS needs Rust 1.85 or later, pkg-config and libusb-1.0-0-dev. Install the current release with your HSM feature and pin the version:

cargo install tmkms --features=yubihsm --version=0.15.0
tmkms init <kms-home>

Set protocol_version in the [[validator]] block of tmkms.toml to match your chain. Version 0.15.0 added "v0.38".

The deprecation window closes six months after SDK v0.55.0, which was released on July 28, 2026. Follow Migrate from TMKMS (opens in a new tab) before then, and do not skip its state file translation step.

5. Apply double-sign rules#

• Run exactly one signer per key. Never run TMKMS and Cosmos-KMS at the same time against the same key.
• Never restore an old priv_validator_state.json or signer state file. A stale file lowers the double-sign floor.
• When you move a validator or signer, confirm the old process is dead before the new one starts.
• Once signing works and you have an offline backup from step 8, remove priv_validator_key.json from the validator. The node then writes a fresh, unused key file on start, so <appd> comet show-validator shows the wrong key. Check /status instead.

6. Set firewall rules with ufw#

Add the SSH rule before ufw enable. The ufw manual warns that enabling it may drop existing SSH sessions.

On the validator:

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow proto tcp from <admin-ip> to any port 22
sudo ufw allow proto tcp from <sentry-1-private-ip> to any port 26656
sudo ufw allow proto tcp from <sentry-2-private-ip> to any port 26656
sudo ufw allow proto tcp from <signer-ip> to any port 26659
sudo ufw enable

On each sentry:

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow proto tcp from <admin-ip> to any port 22
sudo ufw allow 26656/tcp
sudo ufw enable

On the signer, which dials out and needs no other inbound port:

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow proto tcp from <admin-ip> to any port 22
sudo ufw enable

Allow port 26660 only from your monitoring host. If you cannot pin an admin IP, use sudo ufw limit ssh/tcp. It denies an IP that opens 6 or more connections within 30 seconds.

7. Harden SSH#

Create an SSH key on your workstation and copy it to each host:

ssh-keygen -t ed25519
ssh-copy-id <admin-user>@<host-ip>

In a new terminal, confirm that key login works. Then, on each host, replace <admin-user> and run:

1sudo tee /etc/ssh/sshd_config.d/00-hardening.conf > /dev/null <<'EOF'
2PasswordAuthentication no
3KbdInteractiveAuthentication no
4PermitRootLogin no
5PubkeyAuthentication yes
6MaxAuthTries 3
7AllowUsers <admin-user>
8EOF
9sudo sshd -t
10sudo systemctl restart ssh

For most keywords, sshd keeps the first value it reads. Ubuntu reads the sshd_config.d/*.conf files first, in lexical order. Some cloud images ship a 50-cloud-init.conf that turns password login on, and the 00- prefix puts your file ahead of it. Keep your current session open until a new login succeeds.

8. Back up keys#

• Encrypt the consensus key, priv_validator_key.json, and store two copies offline in separate places. After a rotation, keep the old key until the unbonding period ends.
• Keep the operator mnemonic on a hardware wallet or an offline paper backup. It never belongs on a validator, sentry or signer host.
• node_key.json is low risk. A backup of it keeps your node IDs stable.
• For HSM keys, follow your vendor's backup procedure.

On the host that holds the consensus key, encrypt it before it leaves that host:

gpg --symmetric --cipher-algo AES256 priv_validator_key.json

Copy the .gpg file to offline media and test decryption offline. Then remove the plaintext:

shred -u priv_validator_key.json

The shred man page warns that it is not reliable on journaling, RAID or snapshot filesystems. Treat any server that held the plaintext key as exposed.

Verify it works#

On the validator, list its peers. Only your sentries should appear:

curl -s localhost:26657/net_info | jq -r '.result.peers[] | "\(.node_info.moniker) \(.remote_ip)"'

Check that the validator has voting power and that the block height climbs:

curl -s localhost:26657/status | jq '.result.validator_info'
curl -s localhost:26657/status | jq -r '.result.sync_info.latest_block_height'

On the signer, check that the state file advances:

jq . ~/.kms/state/<your-chain-id>.json

Run the next checks from a machine outside your network. The sentry check should succeed and the validator checks should fail. If the validator has no public IP, skip its checks.

nc -zv -w 3 <sentry-public-ip> 26656
nc -zv -w 3 <validator-public-ip> 26656
nc -zv -w 3 <validator-public-ip> 26657

Confirm the firewall and SSH settings on each host:

sudo ufw status verbose
sudo sshd -T | grep -E '^(passwordauthentication|kbdinteractiveauthentication|permitrootlogin)'

Troubleshooting#

The validator has zero peers#

Check every node ID in persistent_peers and confirm you used private IPs. Set addr_book_strict = false on the validator and on the sentries. Check your cloud security groups as well as ufw.

The node exits with a pubkey timeout#

The error reads can't get pubkey: ... endpoint connection timed out. The signer backed off before the node came up. Restart the signer, then start the node within five seconds. Confirm that the validator's ufw rules allow <signer-ip> on port 26659.

The signer refuses to start at height 0#

The error reads sign-state file ... is missing or empty; refusing to start at height 0. Write the double-sign floor with kms state init, as in Path A. Use --allow-fresh-state only for a key that has never signed on this chain.

The node panics on restart#

With double_sign_check_height set, the node found its own recent votes. Wait more than 10 blocks and start it again. If the node was down for longer than that, look for another process signing with the same key.

Next steps#

• Read the Cosmos-KMS best practices (opens in a new tab). If the connection between the signer and the validator crosses a network you do not fully control, move to the noise:// transport.
• Plan your move off Horcrux or TMKMS with the migration guide (opens in a new tab).
• Set prometheus = true under [instrumentation] and alert on missed blocks and peer count.
• Rehearse a signer restart on a testnet and document the steps.