Monitor a CometBFT node with Prometheus and Grafana

Set up Prometheus to scrape CometBFT and host metrics, Grafana to chart them, and alerts that fire when the node stalls, loses peers or misses blocks. It is for validators and operators running Cosmos SDK nodes on their own Linux servers.

Prerequisites#

• A node on CometBFT v0.38 or later. The metric names in this guide come from the v0.40.0 docs and source, and they match v0.38 and v0.39. If you do not have a node yet, start with Run a Cosmos Hub full node with state sync.
• The node runs on Linux under systemd. You need sudo access and the path to its home directory.
• A second Linux server for monitoring. This is recommended, not required. The Grafana steps use the Debian/Ubuntu APT repository.
• Disk space for Prometheus, which keeps 15 days of data by default. You can change this under storage.tsdb.retention in prometheus.yml, as the configuration docs (opens in a new tab) describe.
• A private network or firewall rules between the two servers.
• curl, wget, tar, sha256sum and jq on both servers.

Check the CometBFT version your node runs. The RPC listens on 127.0.0.1:26657 by default. If you changed laddr under [rpc], use your own port.

curl -s http://localhost:26657/status | jq -r '.result.node_info.version'

Nothing in this setup touches priv_validator_key.json, node_key.json or any mnemonic. Do not copy key files to the monitoring server. Keep validator keys on the signing machine or offline.

Placeholders used in this guide:

1. Enable CometBFT metrics#

CometBFT metrics are off by default. When you turn them on, the node serves them at /metrics on port 26660.

Edit config.toml#

Open <node-home>/config/config.toml and find the [instrumentation] section. Set prometheus to true. The other values below are the defaults.

[instrumentation]
prometheus = true
prometheus_listen_addr = ":26660"
max_open_connections = 3
namespace = "cometbft"

With ":26660", the node listens on every interface. To limit it, set "<node-ip>:26660" for a remote Prometheus, or "127.0.0.1:26660" if Prometheus runs on the same server.

You can also make the change from the shell:

sed -i 's/^prometheus = false/prometheus = true/' <node-home>/config/config.toml
grep -A12 '^\[instrumentation\]' <node-home>/config/config.toml

Restart the node and check the endpoint#

sudo systemctl restart <node-service>
curl -s http://localhost:26660/metrics | grep -E '^cometbft_(consensus_height|p2p_peers)'

If you bound the listener to <node-ip>, use that address instead of localhost. The cometbft_ prefix comes from the namespace value, so consensus_height becomes cometbft_consensus_height. Every CometBFT metric also carries a chain_id label.

The metrics to watch#

The two validator_* metrics appear only on the node that signs for your validator. They carry a validator_address label.

2. Install node_exporter on the node server#

The node_exporter binary reports CPU, memory, disk and network metrics on port 9100. At the time of writing, the latest release is 1.12.1. Check the node_exporter releases page (opens in a new tab) for a newer one.

Download and verify#

NE_VERSION=1.12.1
cd /tmp
wget https://github.com/prometheus/node_exporter/releases/download/v${NE_VERSION}/node_exporter-${NE_VERSION}.linux-amd64.tar.gz
wget https://github.com/prometheus/node_exporter/releases/download/v${NE_VERSION}/sha256sums.txt
sha256sum --check --ignore-missing sha256sums.txt
tar xzf node_exporter-${NE_VERSION}.linux-amd64.tar.gz
sudo install -m 0755 node_exporter-${NE_VERSION}.linux-amd64/node_exporter /usr/local/bin/

The checksum line must end in OK. If it does not, delete the file and stop. On ARM servers, replace linux-amd64 with linux-arm64.

Run it as a service#

Create a system user for the service:

sudo useradd --system --no-create-home --shell /usr/sbin/nologin node_exporter

Create /etc/systemd/system/node_exporter.service:

1[Unit]
2Description=Prometheus node_exporter
3Wants=network-online.target
4After=network-online.target
5
6[Service]
7User=node_exporter
8Group=node_exporter
9ExecStart=/usr/local/bin/node_exporter --web.listen-address=<node-ip>:9100
10Restart=on-failure
11
12[Install]
13WantedBy=multi-user.target

The --web.listen-address flag binds node_exporter to the private interface. Without it, node_exporter listens on all interfaces.

Start it and check the output:

sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter
curl -s http://<node-ip>:9100/metrics | grep -m1 node_cpu_seconds_total

Open the ports to the monitoring server only#

If the server uses ufw, allow both ports from the monitoring server and nowhere else. With another firewall, add the equivalent rules.

sudo ufw allow from <monitoring-ip> to any port 26660 proto tcp
sudo ufw allow from <monitoring-ip> to any port 9100 proto tcp

Do not expose 26660 or 9100 to the internet. They reveal details about your node, its host and its peers.

3. Install Prometheus on the monitoring server#

At the time of writing, the latest release is Prometheus 3.15.1. Check the Prometheus releases page (opens in a new tab) for a newer one.

Download and verify#

PROM_VERSION=3.15.1
cd /tmp
wget https://github.com/prometheus/prometheus/releases/download/v${PROM_VERSION}/prometheus-${PROM_VERSION}.linux-amd64.tar.gz
wget https://github.com/prometheus/prometheus/releases/download/v${PROM_VERSION}/sha256sums.txt
sha256sum --check --ignore-missing sha256sums.txt
tar xzf prometheus-${PROM_VERSION}.linux-amd64.tar.gz
sudo install -m 0755 prometheus-${PROM_VERSION}.linux-amd64/prometheus prometheus-${PROM_VERSION}.linux-amd64/promtool /usr/local/bin/

Create the user and directories#

sudo useradd --system --no-create-home --shell /usr/sbin/nologin prometheus
sudo mkdir -p /etc/prometheus/rules /var/lib/prometheus
sudo chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus

Write the scrape config#

Create /etc/prometheus/prometheus.yml:

1global:
2  scrape_interval: 15s
3  evaluation_interval: 15s
4
5rule_files:
6  - /etc/prometheus/rules/cometbft.yml
7
8scrape_configs:
9  - job_name: cometbft
10    static_configs:
11      - targets: ["<node-ip>:26660"]
12        labels:
13          node: "<your-moniker>"
14
15  - job_name: node
16    static_configs:
17      - targets: ["<node-ip>:9100"]
18        labels:
19          node: "<your-moniker>"

Add one target per node. The node label gives each target a readable name, and the alert summaries in step 4 print it.

Create the service#

Create /etc/systemd/system/prometheus.service:

1[Unit]
2Description=Prometheus
3Wants=network-online.target
4After=network-online.target
5
6[Service]
7User=prometheus
8Group=prometheus
9ExecStart=/usr/local/bin/prometheus \
10  --config.file=/etc/prometheus/prometheus.yml \
11  --storage.tsdb.path=/var/lib/prometheus \
12  --web.listen-address=127.0.0.1:9090
13ExecReload=/bin/kill -HUP $MAINPID
14Restart=on-failure
15
16[Install]
17WantedBy=multi-user.target

By default, Prometheus listens on 0.0.0.0:9090 and has no login. The --web.listen-address flag keeps it on localhost. Grafana runs on the same server, so it can still reach Prometheus there. Do not start the service yet. Step 4 adds the alert rules first.

4. Add alert rules#

Create /etc/prometheus/rules/cometbft.yml:

1groups:
2  - name: cometbft
3    rules:
4      - alert: CometBFTMetricsDown
5        expr: up{job="cometbft"} == 0
6        for: 2m
7        labels:
8          severity: critical
9        annotations:
10          summary: "Metrics endpoint unreachable on {{ $labels.node }}"
11
12      - alert: CometBFTHeightStuck
13        expr: (changes(cometbft_consensus_height[5m]) == 0) unless on (instance) (cometbft_blocksync_syncing == 1)
14        for: 1m
15        labels:
16          severity: critical
17        annotations:
18          summary: "Height has not moved for 5 minutes on {{ $labels.node }}"
19
20      - alert: CometBFTLowPeers
21        expr: cometbft_p2p_peers < 3
22        for: 5m
23        labels:
24          severity: warning
25        annotations:
26          summary: "{{ $labels.node }} has {{ $value }} peers"
27
28      - alert: CometBFTMissedBlocks
29        expr: delta(cometbft_consensus_validator_missed_blocks[10m]) > 5
30        labels:
31          severity: critical
32        annotations:
33          summary: 'Validator on {{ $labels.node }} missed about {{ printf "%.0f" $value }} blocks in 10 minutes'

How the rules work:

• The stuck height rule uses changes() because the height is a gauge. The consensus height does not move during block sync, so the unless clause skips nodes that report cometbft_blocksync_syncing == 1. On chains with slow blocks, widen the 5-minute window.
• The low peers rule uses 3 as a starting point. Set it to what is normal for your chain and your sentry layout.
• The missed blocks rule uses delta(), which the Prometheus docs recommend for gauges. Because delta() extrapolates, the value can be a fraction. The gauge resets when the node restarts, so a restart gives a negative delta and the alert does not fire. Match the threshold to your chain's slashing parameters.

Fix ownership, check both files, then start Prometheus:

sudo chown -R prometheus:prometheus /etc/prometheus
promtool check config /etc/prometheus/prometheus.yml
sudo systemctl daemon-reload
sudo systemctl enable --now prometheus

promtool check config also checks the files listed under rule_files. After later edits, run the check again and reload Prometheus:

sudo systemctl reload prometheus

Prometheus evaluates alerts but does not deliver them. To send them to Telegram, Slack, PagerDuty or email, you need Alertmanager (opens in a new tab).

5. Install Grafana#

Grafana 13.2.4 is the latest release at the time of writing. The official APT repository installs the latest OSS release:

1sudo apt-get install -y apt-transport-https wget gnupg
2sudo mkdir -p /etc/apt/keyrings
3sudo wget -O /etc/apt/keyrings/grafana.asc https://apt.grafana.com/gpg-full.key
4sudo chmod 644 /etc/apt/keyrings/grafana.asc
5echo "deb [signed-by=/etc/apt/keyrings/grafana.asc] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
6sudo apt-get update
7sudo apt-get install grafana
8sudo systemctl daemon-reload
9sudo systemctl enable --now grafana-server

Grafana listens on port 3000. Keep that port closed in the firewall and reach Grafana through an SSH tunnel. Run this on your own computer:

ssh -L 3000:localhost:3000 <user>@<monitoring-ip>

Open http://localhost:3000 and sign in with the username admin and the password admin. Grafana then asks you to set a new password. Choose a strong one.

Connect Prometheus#

1. Click Connections in the left menu, then Add new connection.
2. Enter Prometheus in the search bar and click Prometheus data source.
3. Click Add new data source.
4. Enter http://localhost:9090 in Prometheus server URL.
5. Click Save & test.

Build the first panels#

Create a dashboard and add one panel per query:

cometbft_consensus_height
deriv(cometbft_consensus_height[5m]) * 60
cometbft_p2p_peers
rate(cometbft_consensus_block_interval_seconds_sum[5m]) / rate(cometbft_consensus_block_interval_seconds_count[5m])
cometbft_consensus_height - ignoring (validator_address) cometbft_consensus_validator_last_signed_height
cometbft_mempool_size

In order, the queries show height, blocks per minute, peers, average block time in seconds, blocks since your validator last signed, and mempool size. On a healthy validator, the fifth panel stays near 1. If that value keeps rising, your validator has stopped signing.

For host metrics, import the community Node Exporter Full dashboard. Open Dashboards, click New, then Import dashboard. Enter ID 1860, click Load, select your Prometheus data source and click Import. The dashboard's page suggests starting node_exporter with --collector.systemd --collector.processes so that every panel has data.

Verify it works#

On the monitoring server, check that both targets are up:

curl -s http://localhost:9090/api/v1/targets | jq -r '.data.activeTargets[] | "\(.labels.job) \(.labels.instance) \(.health)"'

Each line should end in up. Next, compare the node's height with the height Prometheus recorded. On the node server:

curl -s http://localhost:26657/status | jq -r '.result.sync_info.latest_block_height, .result.sync_info.catching_up'

On the monitoring server:

curl -s 'http://localhost:9090/api/v1/query?query=cometbft_consensus_height' | jq -r '.data.result[].value[1]'

The two heights should be within a few blocks of each other. The metric reads one above latest_block_height, and scrape delay can make it lag behind. The second value from the node, catching_up, should be false.

To test alerting without touching your validator, add a target that nothing listens on, such as <node-ip>:26661, to the cometbft job. Reload Prometheus, wait about 3 minutes, then list the alerts:

sudo systemctl reload prometheus
curl -s http://localhost:9090/api/v1/alerts | jq -r '.data.alerts[] | "\(.labels.alertname) \(.state)"'

You should see CometBFTMetricsDown firing. Remove the test target and reload again.

Troubleshooting#

Metric names start with tendermint_#

Config files created before the CometBFT rename can still have namespace = "tendermint". Your metrics are then named tendermint_consensus_height and so on. Change the namespace to cometbft and restart the node, or change your queries to use the old prefix. A new namespace starts new series, so older history stays under the old names.

Target is down with connection refused#

Check what is listening on the node:

ss -ltn | grep -E ':(26660|9100)'

If there is no line for 26660, the prometheus = true change did not save or the node has not restarted. If the line shows 127.0.0.1:26660, a remote Prometheus cannot reach it. Also check the firewall rules from step 2.

Missed blocks or last signed height are missing#

These series exist only on the node that signs for an active validator. Sentries and RPC nodes never report them. The missed blocks series also appears only after the first missed block since the process started, so an empty missed blocks result on a healthy validator is normal.

Scrapes time out with several collectors#

max_open_connections defaults to 3. If several Prometheus servers or agents scrape the same node, raise it and restart the node. The comment in the config file warns that higher values may also need higher OS limits.

Next steps#

• Configure Alertmanager (opens in a new tab) routes so alerts reach your on-call channel.
• Add every sentry and RPC node as its own target, each with its own node label. For the sentry layout itself, see Validator security: sentry nodes, remote signing and firewalls.
• Point an external uptime check at the CometBFT /health RPC endpoint. The running in production (opens in a new tab) guide describes it along with /status, /net_info and /validators.
• Collect application metrics from the Cosmos SDK. For SDK v0.54 and later, the telemetry docs (opens in a new tab) cover the OpenTelemetry setup.
• Review the full CometBFT metrics list (opens in a new tab) for consensus round, vote and P2P traffic metrics.