Who This Is For
Intermediate.
Think of it like a shared family group chat instead of texting five people separately. One place to search "who said what and when" instead of digging through five different phones.
What You'll Build
Search Docker and Linux logs from one Grafana interface instead of multiple SSH sessions.
Prerequisites
- Docker host
- Grafana already deployed (see the Grafana + Prometheus article)
Logs vs. Metrics
Metrics tell you something is wrong; logs tell you why. Centralized logging closes that gap.
Loki Architecture
- Loki server, stores logs, indexed by label rather than full-text
- Log shipper (Grafana Alloy or Promtail), collects and forwards logs
- Grafana Explore, where you actually search
Deploy Loki
$services:
$ loki:
$ image: grafana/loki:latest
$ ports: ["3100:3100"]
$ loki:
$ image: grafana/loki:latest
$ ports: ["3100:3100"]
Ship Docker Container Logs
Configure the Docker daemon's logging driver to the Loki driver, or run a shipper container that reads container logs directly.
Label Design
hostserviceenvironmentcontainer- Avoid high-cardinality labels (like request IDs), Loki indexes on labels, and too many unique values will blow up performance
Useful Searches
- Authentication failures across all hosts at once
- 502 errors from the reverse proxy
- Container restart loops
- Failed backup jobs
Alert From Logs
Grafana can alert directly on Loki query results, e.g. more than N auth failures in 5 minutes.
Security & Backup Notes
- Sensitive data (tokens, passwords) sometimes ends up in application logs by accident, review what's actually being shipped before treating Loki as a safe long-term store
Troubleshooting
- Shipper can't reach Loki, check network connectivity and the Loki push endpoint URL
- Labels too granular, high-cardinality labels will make Loki slow and memory-hungry; move that data into the log line body instead
- Logs consume disk too fast, set retention limits
- Sensitive data appears in logs, add redaction rules at the shipper level if this happens regularly
Lab Finish Line
Search Docker and Linux logs from one Grafana interface.