Prometheus + Grafana Deploy — Docker + systemd Side-by-Side

Q: What's the minimum Prometheus stack for a production server?

Prometheus server (port 9090), node_exporter on every node (port 9100), and Grafana (port 3000). That's it for v1. Later you add Alertmanager and Pushgateway for jobs.

Q: Docker or systemd for Prometheus?

Docker if you're going to run it with other containerized services (easier to update, cleaner config). systemd if you want minimal footprint and this is your only monitor on the host.

Q: Where do Prometheus metrics get stored?

In Prometheus's local TSDB on disk by default. Without a named Docker volume, a container restart wipes it. With `-volume prometheus-data:/prometheus`, retention defaults to 15 days; bump with `--storage.tsdb.retention.time=30d`.

Q: What's a good PromQL starting point?

`rate(node_network_receive_bytes_total{device!~"lo|veth.*"}[5m])` for network RX traffic. Full templates in [PromQL Recipes](prometheus-query.md).

Q: How do I add alerts once this is running?

Prometheus doesn't send alerts natively — you add Alertmanager. See [Prometheus Alerting & Alertmanager](prometheus-alerting.md).

TL;DR

Prometheus on port 9090 — scrapes targets on schedule, stores metrics in TSDB
Grafana on port 3000 — queries Prometheus via PromQL, renders dashboards
node_exporter on port 9100 on every Linux node — exposes CPU, memory, disk, network
Docker deployment for speed, systemd deployment for minimum footprint

Both paths are covered below with full commands and configs. Pick one and run.

Deployment Option A: Docker Compose (Recommended for Most Cases)

Why Docker

One command to bring up the whole stack
Upgrades: docker compose pull && docker compose up -d
Data persists in named volumes across container restarts
Same topology as production (each component isolated)

`docker-compose.yml`

version: "3.9"

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus-data:/prometheus
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      - "--storage.tsdb.retention.time=30d"
      - "--web.enable-lifecycle"   # Allows /-/reload without restart
    networks:
      - monitoring

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    restart: unless-stopped
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=changeme
      - GF_USERS_ALLOW_SIGN_UP=false
      - GF_ANALYTICS_REPORTING_ENABLED=false
    volumes:
      - grafana-data:/var/lib/grafana
    depends_on:
      - prometheus
    networks:
      - monitoring

  node-exporter:
    image: prom/node-exporter:latest
    container_name: node-exporter
    restart: unless-stopped
    ports:
      - "9100:9100"
    pid: host
    network_mode: host   # node-exporter needs host PID namespace
    command:
      - "--path.procfs=/host/proc"
      - "--path.sysfs=/host/sys"
      - "--collector.filesystem.mount-points-exclude=^/(dev|proc|sys|var/lib/docker/.+)(/$)?"
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/host/root:ro
    networks:
      - monitoring

volumes:
  prometheus-data:
  grafana-data:

networks:
  monitoring:
    driver: bridge

`prometheus.yml`

global:
  scrape_interval: 30s
  evaluation_interval: 30s

scrape_configs:
  # Prometheus scrapes itself
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']

  # Linux node metrics
  - job_name: node
    static_configs:
      - targets:
          - 'localhost:9100'      # The host Prometheus runs on
          # - '10.0.0.5:9100'      # Add more hosts here
          # - '10.0.0.6:9100'

  # Grafana metrics (optional but useful)
  - job_name: grafana
    static_configs:
      - targets: ['grafana:3000']

alerting:
  alertmanagers:
    - static_configs:
        targets: []
      # - static_configs:
      #     targets: ['alertmanager:9093']   # Once you add Alertmanager

Start It Up

# 1. Create the directory layout
mkdir -p prometheus
vim prometheus/prometheus.yml          # Paste config above
vim docker-compose.yml                   # Paste config above

# 2. Bring it up
docker compose pull
docker compose up -d

# 3. Verify
curl -s http://localhost:9090/-/ready | grep SUCCESS
curl -s http://localhost:9100/metrics | head -5
curl -s http://localhost:3000/api/health

# 4. Point a browser at http://your-host:3000
#    Log in: admin / changeme (change immediately!)

Add a Grafana Dashboard

Once Grafana is up:

Add a data source → Prometheus → URL: http://prometheus:9090
Import dashboards:
Node Exporter Full (ID 1860) — the industry standard
Prometheus Stats (ID 3662)
Create a panel with PromQL: rate(node_network_receive_bytes_total[5m])

Deployment Option B: systemd (Minimal Footprint)

Prometheus

# 1. Install binary
sudo -u prometheus wget \
  https://github.com/prometheus/prometheus/releases/download/v2.53.0/prometheus-2.53.0.linux-amd64.tar.gz
sudo -u prometheus tar xzvf prometheus-*.tar.gz
cd prometheus-*/

# 2. Create user
sudo useradd --no-create-home --shell /usr/sbin/nologin prometheus

# 3. Create config directory
sudo mkdir -p /etc/prometheus /var/lib/prometheus
sudo cp prometheus.yml /etc/prometheus/
sudo chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus

# 4. systemd unit
sudo tee /etc/systemd/system/prometheus.service >/dev/null <<'EOF'
[Unit]
Description=Prometheus Time Series Collection and Processing Server
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
ExecStart=/opt/prometheus/prometheus \
    --config.file=/etc/prometheus/prometheus.yml \
    --storage.tsdb.path=/var/lib/prometheus \
    --storage.tsdb.retention.time=30d \
    --web.enable-lifecycle
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

# 5. Move binary
sudo mv prometheus /opt/prometheus/
sudo chmod +x /opt/prometheus/prometheus

# 6. Start
sudo systemctl daemon-reload
sudo systemctl enable --now prometheus
sudo systemctl status prometheus

Grafana via systemd

# 1. Add repo
sudo wget -q -O - https://apt.grafana.com/gpg.key | sudo gpg --dearmor -o /usr/share/keyrings/grafana.gpg
echo "deb [signed-by=/usr/share/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt update && sudo apt install -y grafana

# 2. Start
sudo systemctl enable --now grafana-server

node_exporter via systemd

sudo useradd --system --no-create-home node_exporter
sudo curl -L https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz | tar xz -C /tmp
sudo mv /tmp/node_exporter-1.8.2.linux-amd64/node_exporter /opt/node_exporter/

sudo tee /etc/systemd/system/node_exporter.service >/dev/null <<'EOF'
[Unit]
Description=Node Exporter
After=network-online.target

[Service]
User=node_exporter
ExecStart=/opt/node_exporter/node_exporter \
    --collector.filesystem.mount-points-exclude=^/(dev|proc|sys|var/lib/docker/.+)(/$)?

Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl enable --now node_exporter

Deploy Option Comparison

Aspect	Docker Compose	systemd
Setup time	~5 min	~10 min
Disk footprint	Larger (base images)	Smaller
Upgrade speed	`docker compose pull`	Download new tar, restart
Data isolation	Named volumes	Direct `/var/lib/*`
Multiple Prometheus in one VM	Easy	Complex
Containerized app coexistence	Natural	Requires coordination
Portainer / Rancher friendly	Yes	No

Pick Docker if this host also runs containers (very common). Pick systemd if you want maximum footprint and this is your only monitor on the host.

What to Do Next After Deploy

Once Prometheus and Grafana are both up:

1. Check Target Health

Open http://prometheus:9090/targets — every target should be UP. If any is DOWN, the scrape is failing; check lastError in the same table.

2. Import Dashboards

Node Exporter Full: Grafana dashboards #1860
Prometheus Stats: #3662
Alertmanager (if you add it): #6674

3. Configure Retention

Default is 15 days. To keep 90 days:

# docker-compose.yml command
- "--storage.tsdb.retention.time=90d"

Or in systemd unit:

ExecStart=... --storage.tsdb.retention.time=90d

4. Reload Config Without Restart

curl -X POST http://localhost:9090/-/reload

5 Real Incidents and Fixes

Incident 1: Prometheus Data Disappears After Container Restart

Symptom: docker restart prometheus → all dashboards empty for the last few minutes.

Root cause: Prometheus was running with --storage.tsdb.path=/prometheus but no named volume was mounted. Container restart = container filesystem reset = data gone.

Fix: Add prometheus-data:/prometheus to volumes: in docker-compose.yml.

Incident 2: node_exporter Shows `container_fs_*` Metrics, Which We Don't Want

Symptom: Dashboards show noisy container filesystem metrics.

Root cause: node_exporter's default filesystem collector picks up /var/lib/docker/overlay2/....

Fix: Add --collector.filesystem.mount-points-exclude=^/(dev|proc|sys|var/lib/docker/.+)(/$)? to node_exporter args.

Incident 3: Targets Show DOWN but Service Is Running

Symptom: Prometheus UI shows target DOWN, error: connection refused.

Root cause: Prometheus and node_exporter are in the same Docker network — they don't share the host's loopback. localhost:9100 inside the container doesn't reach the host's 9100.

Fix: Two options: 1. Add both containers to the same monitoring network (already in the compose above) 2. Or use host.docker.internal:9100 with extra_hosts (less clean)

Incident 4: Grafana Says "Datasource is unreachable"

Symptom: Grafana data source test fails with connect: connection refused.

Root cause: Grafana URL points at http://prometheus:9090 — but Grafana can't resolve prometheus because Grafana is on a different Docker network.

Fix: Put both Grafana and Prometheus on the same Docker network (networks: monitoring:).

Incident 5: Prometheus Memory Keeps Growing

Symptom: After a week, Prometheus uses 8 GB and starts swapping.

Root cause: scrape_interval: 30s is too aggressive for a small cluster. Also, --storage.tsdb.retention.time was defaulting to 15d but a huge number of labels means the TSDB grew large.

Fix: 1. Reduce scrape_interval back to 60s unless you truly need 30s 2. Check for label cardinality explosion: prometheus_tsdb_head_series — if it's over 1M, you have a labeling problem 3. Add --storage.tsdb.retention.size=50GB to cap disk growth

PromQL Recipes — 10 copy-paste queries
Prometheus Alerting & Alertmanager — the missing piece for production alerts
Grafana Dashboard Guide — variables, panels, and layout

Quick Reference — Common Prometheus Endpoints

Once Prometheus is running, these endpoints are useful every day:

Endpoint	What it tells you
`/-/ready`	`Prometheus Server is Ready.` — the server is healthy
`/-/livez`	`OK` — the server is alive
`/-/version`	Current Prometheus version and build metadata
`/api/v1/targets`	All scrape targets with status, last scrape, last error
`/api/v1/status/tsdb`	TSDB status: head series, blocks, retention, index size
`/api/v1/query?query=up`	Which targets are currently UP (1) or DOWN (0)
`/api/v1/query?query=up{job="prometheus"}`	Whether Prometheus is scraping itself (health check)
`/-/reload` (POST)	Reload config without restart
`/-/quit` (POST)	Stop Prometheus (graceful shutdown)

Quick Reference — Common node_exporter Collectors

The node_* metrics come from different collectors. If one isn't showing up in Grafana, the collector is probably disabled:

# See which collectors are enabled
curl -s http://localhost:9100/metrics | grep '^node_exporter_collector_enabled'

# Disable a noisy collector via systemd
systemctl edit node_exporter.service
[Service]
Environment="NODE_EXPORTER_ARGS=--no-collector.infiniband --no-collector.hwmon"

Common ones you'll want:

node_cpu_seconds_total — CPU usage by mode (user, system, iowait, steal)
node_memory_MemAvailable_bytes — the metric Grafana uses for "available memory"
node_disk_io_time_seconds_total — disk I/O wait time
node_network_receive_bytes_total — network bytes in
node_filesystem_avail_bytes — free space on each mount
node_processes — process count by state (running, sleeping, zombie)