How to Self-Host a Monitoring Stack with Prometheus and Grafana

Why this matters

Uptime Kuma tells you that something is down. It doesn’t tell you why, or when it started going bad. For that you need metrics over time — CPU, memory, disk, response latency, request rates — and that means Prometheus and Grafana.

This guide is the setup I run to monitor a small home server. It’s not a production-grade cluster; it’s the minimum useful thing.

What I tested on

• OS: Debian 13 (trixie)
• Prometheus: 2.53.x
• Grafana: 11.x
• Node Exporter: 1.8.x
• Hardware: 4 GB RAM VPS, monitoring 4 nodes
• Duration: running for 7 months at the time of writing

⚠️ Grafana and Prometheus both ship with authentication disabled by default. Put them behind a reverse proxy with basic auth or a VPN before exposing them.

How I tested

I didn’t benchmark it. I ran it, broke things on purpose (killed a node, filled a disk, dropped network), and checked whether the dashboards and alerts told me what happened.

Step 1: Install Prometheus

# Download and extract (check the latest version first)
wget https://github.com/prometheus/prometheus/releases/download/v2.53.0/prometheus-2.53.0.linux-amd64.tar.gz
tar xvf prometheus-2.53.0.linux-amd64.tar.gz
cd prometheus-2.53.0.linux-amd64

Create a config file:

# prometheus.yml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "node"
    static_configs:
      - targets: ["10.0.0.5:9100", "10.0.0.6:9100"]

Run it:

./prometheus --config.file=prometheus.yml

Step 2: Add Node Exporter on each machine

# On each machine you want to monitor
wget https://github.com/prometheus/node_exporter/releases/download/v1.8.0/node_exporter-1.8.0.linux-amd64.tar.gz
tar xvf node_exporter-1.8.0.linux-amd64.tar.gz
cd node_exporter-1.8.0.linux-amd64
./node_exporter

That’s it. It listens on :9100 and exposes metrics. Prometheus scrapes it.

Step 3: Install Grafana

# Add Grafana's repo
sudo apt install -y software-properties-common
wget -q -O - https://packages.grafana.com/gpg.key | sudo gpg --dearmor -o /etc/apt/keyrings/grafana.gpg
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://packages.grafana.com/oss/deb stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt update
sudo apt install -y grafana
sudo systemctl enable --now grafana-server

Add Prometheus as a data source in the Grafana UI (Connection → Data sources → Add → Prometheus, URL http://localhost:9090), then import dashboard 1860 — the Node Exporter Full dashboard. It just works.

Step 4: Alerting that actually fires

The part everyone skips: alerts. In prometheus.yml:

rule_files:
  - "alerts.yml"

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["localhost:9093"]

# alerts.yml
groups:
  - name: node
    rules:
      - alert: NodeDown
        expr: up{job="node"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Node {{ $labels.instance }} is down"

The for: 1m is important — without it you get paged for every 10-second blip.

How to actually verify it works

Kill a node exporter (pkill node_exporter), wait a minute, and confirm the alert fires in Prometheus’s Alerts tab. If it doesn’t, check prometheus.yml syntax with promtool check config prometheus.yml — a typo is the usual reason.

Alert fatigue is the real failure mode

The first month I ran this stack I had 40 alert rules and ignored all of them. That is worse than having no monitoring, because it trains you to swipe notifications away without reading.

What survived, after deleting most of it:

Page on symptoms, not causes. “Disk almost full” is a cause you can watch on a dashboard. “Backups have not completed in 36 hours” is a symptom that something is actually broken. Only the second earns a notification.

One alert per user-visible failure. If the reverse proxy is down, forty container checks are also red. Alert on the entry point and let the dashboard explain the rest — the Uptime Kuma review covers that layer.

Every alert needs an action in the message. “Load average high” tells me nothing to do. “Load 14 on a 4-core box for 10 minutes, check the media VM” does. If you cannot write the action, the alert should not exist.

Silence anything that fires more than twice a week. Either the threshold is wrong or the system is genuinely broken. Both need fixing, neither needs another notification.

The number of rules I keep is eleven. Eight of them have fired for real in the last year. That ratio is the whole point: a monitor is only useful if you believe it, and belief is a budget you spend carefully.

If the host itself is the thing rebooting, the Proxmox review covers what that looks like from the hypervisor side.

Pros and cons

Pros:

• You see trends, not just current state — “the disk filled over 6 hours” instead of “the disk is full”
• Grafana dashboards are genuinely beautiful and highly shareable
• Prometheus is the industry standard; every guide and every exporter speaks its language

Cons:

• Steep learning curve. PromQL is not intuitive at first.
• Resource-heavy for a small setup — Grafana alone is a lot for a Pi.
• Retention is short by default; long-term storage means extra components.

Best for / Skip if

Best for: anyone who wants to understand why their services misbehave, not just that they did.

Skip if: you just want “is it up?” pings. Uptime Kuma is cheaper and easier. Run both — Uptime Kuma for the simple question, Prometheus for the hard one.

Alternatives

• Netdata — zero-config, gorgeous, but heavier and more opinionated
• Zabbix — older, agent-based, powerful but clunky UI
• Uptime Kuma + a shell script — genuinely fine for very small setups

Verdict

Prometheus + Grafana is overkill for a one-server hobby setup, but it’s the right choice as soon as you have more than a couple of machines or you’ve been woken up by a vague “something’s wrong” and wished you had data. The setup above takes an evening; the insight it gives you lasts.

References

• Prometheus documentation — scrape configs and recording rules
• Grafana documentation — dashboards, provisioning, and alerting