Who This Is For
Intermediate.
Think of it like a hospital's full vital-signs monitor instead of just checking someone's pulse once. You're not just confirming something's alive, you can watch the trend and catch trouble before it becomes an emergency.
What You'll Build
A Grafana dashboard showing metrics for at least one host and one container host.
Prerequisites
- Docker host
- At least one additional Linux host to monitor
Prometheus Architecture
- Prometheus server, scrapes and stores metrics
- Targets, the things being scraped
- Exporters, expose metrics in a format Prometheus understands
- Scrape interval, how often metrics are pulled
Deploy Prometheus and Grafana
$services:
$ prometheus:
$ image: prom/prometheus:latest
$ ports: ["9090:9090"]
$ grafana:
$ image: grafana/grafana:latest
$ ports: ["3000:3000"]
$ prometheus:
$ image: prom/prometheus:latest
$ ports: ["9090:9090"]
$ grafana:
$ image: grafana/grafana:latest
$ ports: ["3000:3000"]
Add Node Exporter to Each Host
$docker run -d --name node-exporter -p 9100:9100 \
$ prom/node-exporter
$ prom/node-exporter
Configure Scrape Targets
$scrape_configs:
$ - job_name: "nodes"
$ static_configs:
$ - targets: ["host1:9100", "host2:9100"]
$ - job_name: "nodes"
$ static_configs:
$ - targets: ["host1:9100", "host2:9100"]
Add Prometheus as a Grafana Data Source
Grafana UI → Connections → Data Sources → Prometheus → point it at the Prometheus URL.
Essential Panels
- CPU utilization
- Load average
- Memory usage
- Disk capacity
- Disk I/O
- Network throughput
- Container state
Set Alert Thresholds
- Disk almost full
- Sustained high CPU
- Host unreachable
- SMART warning fed in from a custom exporter
Security & Backup Notes
- Monitor the monitoring stack itself, a Prometheus/Grafana outage is invisible unless something else is watching it
Troubleshooting
- Prometheus can't scrape an exporter, check the target is actually reachable on that port from the Prometheus host
- Firewall blocks the metrics port, open it internally, never expose it publicly
- Labels cause duplicate targets, check for accidental target overlap in your scrape config
- Grafana dashboard has no data, verify the data source URL and that the query's time range actually overlaps with when data exists
Lab Finish Line
Grafana dashboard showing at least one host and one container host.