Quick Answer: Solana validator monitoring means tracking your vote-account runway, BLS registration, delinquency, leader-slot production, and host health from the validator server itself. Vigil is a single static Linux binary that runs these checks, scores them from 0 to 100, and exports Prometheus metrics. It is read-only: it never loads keys, signs transactions, or changes host settings.

Most operators end up gluing Solana validator monitoring together from a few shell scripts, a dashboard someone copied, and an alert bot nobody remembers configuring. Each piece covers one slice, and the gaps show up at the worst time. The question isn't whether you can watch your validator. It's whether the checks you rely on would actually tell you when something is quietly going wrong. Here's how to think about that, and what a single read-only tool can cover.

What a complete Solana validator monitoring setup actually checks

Most guides list metrics without explaining what they're for. A useful setup covers five areas, and each one fails in a different way.

  • Vote-account funding, including VAT runway and whether the next payment is covered.
  • Consensus registration, such as BLS registration status.
  • Delinquency and vote lag, meaning whether your vote account is keeping up.
  • Leader duties, meaning slots assigned, blocks produced, and missed slots.
  • Host state, covering CPU, memory, sysctl, time sync, NIC, process limits, and voting mode.

Vigil bundles these into vigil check, vigil score, and vigil audit. The host audit matters more than people expect. Its results are meaningful only when Vigil runs on the validator being monitored, not on an RPC-only node that happens to expose similar data. Run one instance per validator server, pointed at that server's local RPC on http://127.0.0.1:8899 by default.

Why VAT runway catches the quietest failure

Vote-account funding is the check most likely to fail silently. Vigil reads the vote-account balance, subtracts the rent-exempt minimum for the account's actual size, and divides what's left by your configured per-epoch fee:

spendable = max(0, vote_account_balance - rent_exempt_minimum)
runway_epochs = spendable / configured_vat_per_epoch

Here's the trap. Suppose your vote account holds 2.1 SOL, the rent-exempt minimum is 0.3 SOL, and the per-epoch fee is 0.8 SOL. Your balance looks comfortable, but only 1.8 SOL is spendable, which gives you 2.25 epochs of runway. Drop the balance to 1.2 SOL and the runway falls under 1.2 epochs with almost no visible sign in a balance-only dashboard.

The default fee in Vigil is 0.8 SOL per epoch (800000000 lamports). That's configuration, not a promise about future governance, so verify the current fee and slot-time assumptions for your cluster. A runway below 1 is critical because the next payment can't be covered. Below 2, Vigil warns. Set vat_lamports_per_epoch to 0 only if you're intentionally disabling VAT monitoring.

BLS registration: "missing" is not the same as "broken"

Many exporters collapse three different states into one "not registered" flag. Vigil separates them:

  1. Registered: blsPubkeyCompressed is a valid base58 value that decodes to 48 bytes.
  2. Critical: the field is explicitly null or empty, or the value is malformed.
  3. Unknown: the field is absent from the RPC response.

The third state is the one worth caring about. An absent field usually means a schema mismatch between your RPC client and what Vigil expects, not a broken validator. Treating it as healthy hides the problem, and treating it as critical sends you chasing a fault that doesn't exist. Vigil reports it as unknown and tells you to verify the RPC or client schema. That's the honest answer, and it keeps your on-call rotation out of a wild goose chase.

Leader duties and the 32-slot confirmation trap

Leader production sounds simple: you were assigned slots, you produced blocks, and anything else is a miss. In practice, the naive version pages you for forks. A slot can look missed moments after it's assigned because the block hasn't been confirmed yet, or because the chain took a different branch.

Vigil handles this in watch mode. It caches the leader schedule and checks newly assigned slots with getBlocks, but only once they're confirmed, using a default depth of 32 slots. That delay is deliberate. Alerting on unconfirmed misses would turn normal fork churn into noise, and noise is how real incidents get ignored.

This is the counter-intuitive part. A faster alert is often a worse alert. Waiting for confirmation adds a short delay, but it removes the false positives that train people to mute the channel. Per-slot events also never become permanent Prometheus labels, which keeps cardinality bounded.

Alert design: fewer pages beat faster pages

Alerting is where most homegrown setups go wrong. They fire on the first bad sample, flap on every transient error, and never escalate. Vigil's defaults push the other way:

  • Critical conditions notify immediately.
  • Warnings and unknown states need two consecutive observations.
  • Recovery needs two consecutive healthy observations.
  • A severity change notifies immediately.
  • An active incident repeats at most once per hour.
  • Failed deliveries are retried on the next observation, not after the repeat interval.

Secrets stay out of the JSON config. Webhook URLs, Telegram tokens, and PagerDuty routing keys come from environment variables. The supported destinations and their required settings look like this:

TypeRequired settingSecret stored inNotes
slackurl_envEnvironment variableIncoming webhook URL
discordurl_envEnvironment variableIncoming webhook URL
telegramtoken_env, chat_idEnvironment variable (token)Chat ID is plain config
pagerdutyrouting_key_envEnvironment variableRouting key for Events routing

Always verify delivery before you trust it. Run vigil test-alert --config vigil.json. It proves delivery works without changing incident state, so you won't create a fake page in the process.

Running Vigil as a hardened service

For a validator that runs 24/7, skip the terminal session and use systemd. Vigil ships unit files in deploy/, and the flow is straightforward:

  1. Create a locked-down system user and state directory.
  2. Install vigil.json as root:vigil with mode 0640.
  3. Install the vigil-watch and vigil-export unit files.
  4. Run vigil check once as the service user to confirm the config works.
  5. Enable both services with systemctl enable --now.

The exporter binds to loopback on port 9109 by default. Keep it that way. If Prometheus runs on another machine, reach it over a private network, a firewall rule, or a secure tunnel, never the public internet. Once scraped, you can load the ready-made alert rules and import the Grafana dashboard for VAT, BLS, delinquency, leader production, slot lag, and check status. The metrics that matter most include solana_vat_runway_epochs, solana_vat_next_epoch_sufficient, and solana_bls_registered.

Here's the catch with dashboards: a green panel only means what the exporter could observe. That's why vigil score also reports coverage. A run that could only observe half the cells should never read as a clean bill of health, and if your monitoring doesn't show coverage, it's worth adding that to your checklist.

Where a single tool falls short

No monitoring tool replaces the operator. Vigil is read-only by design, so it will tell you that your identity and vote account are misaligned, but it won't fix that. It also depends on the RPC endpoints you give it. If your local RPC lags or returns partial data, you'll see unknown rather than a fabricated healthy value, which is the right behavior, but it still means you need to investigate.

The other limit is scope. Vigil checks what it can observe from the validator's vantage point. A second RPC on mainnet-beta helps you measure slot lag independently, but only if its genesis hash matches. Treat cross-checks like that as evidence, not proof.

Vigil's installer verifies SHA-256 for release binaries and checks the keyless Sigstore signature when cosign is installed. Verify those checks before you run anything with root access on a production host.

Frequently Asked Questions

What is Solana validator monitoring?

Solana validator monitoring is the ongoing tracking of your validator's health, including vote-account funding, consensus registration, delinquency, leader-slot production, and host metrics. Good monitoring alerts you before a failure becomes an outage. Tools like Vigil run these checks read-only and export results to Prometheus and alert channels.

How do I check my Solana validator's health from the command line?

Run vigil check --config vigil.json on the validator server for a one-shot health report. Use vigil score for a single 0–100 number with the fix for each failing check. Exit codes are 0 OK, 1 warning, 2 critical, and 3 unknown, so you can script around them.

Why does my Solana validator show unknown instead of healthy?

Vigil reports unknown when a field or observation is missing, such as an absent BLS field or a failed RPC call. It deliberately avoids guessing healthy. Check your RPC endpoint and client schema first, then rerun the check. An unknown result is a prompt to investigate, not a false alarm.

How do I monitor missed slots without getting false alerts?

Wait for confirmation before alerting. Vigil checks newly assigned slots only after they reach the default confirmation depth of 32 slots, which filters out unconfirmed forks. Pair that with the two-observation threshold for warnings, and a single transient miss won't page you at 3 a.m.

Solana validator monitoring works best when it's read-only, runs on the validator itself, and tells you honestly what it couldn't see. Install Vigil on one validator server this week, run vigil score, and note the coverage line alongside the number. Then pass this setup to a colleague who is still gluing scripts together for their own Solana validator monitoring.