The default homelab log strategy is SSH and journalctl, one machine at a time, after the fact. By the time you look, the container was recreated and its logs went with it, or the journal on a small root disk already rotated.
The second trap is building a log stack from an old tutorial. Many still use Promtail, which reached end of life on March 2, 2026. Grafana's documentation states that it receives no further updates and that future development happens in Grafana Alloy, so new setups should start with Alloy.
How the pieces fit
Loki stores logs. Unlike a full-text search engine it indexes only a small set of labels (host, unit, container) and stores the log lines compressed in chunks, which keeps it light enough for a homelab. Alloy is the agent: it reads the journal and Docker's container logs on each host, attaches labels, and pushes the lines to Loki. Grafana queries Loki with LogQL.
Install Alloy on each host
Alloy runs as a native service, which makes reading the journal straightforward. The steps follow Grafana's Debian instructions (Alloy 1.20 at the time of writing, October 2026). On the Proxmox host and on each Docker host:
apt install gpg
mkdir -p /etc/apt/keyrings
wget -O /etc/apt/keyrings/grafana.asc https://apt.grafana.com/gpg-full.key
chmod 644 /etc/apt/keyrings/grafana.asc
echo "deb [signed-by=/etc/apt/keyrings/grafana.asc] \
https://apt.grafana.com stable main" > /etc/apt/sources.list.d/grafana.list
apt-get update && apt-get install alloy
usermod -aG adm,systemd-journal alloy
The last line matters: Alloy's documentation warns that without membership in adm and systemd-journal the journal source starts without an error and collects nothing. On Docker hosts, also add the alloy user to the docker group so it can read container logs through the socket (with the usual caveat that the socket means full control of the host).
The configuration
It lives in /etc/alloy/config.alloy and is a set of components wired together: sources forward to a writer.
// /etc/alloy/config.alloy on docker1
loki.write "default" {
endpoint {
url = "http://192.168.10.20:3100/loki/api/v1/push"
}
}
// systemd journal, with unit and level as labels
loki.relabel "journal" {
forward_to = []
rule {
source_labels = ["__journal__systemd_unit"]
target_label = "unit"
}
rule {
source_labels = ["__journal_priority_keyword"]
target_label = "level"
}
}
loki.source.journal "host" {
forward_to = [loki.write.default.receiver]
relabel_rules = loki.relabel.journal.rules
labels = {host = "docker1", job = "journal"}
}
// Docker containers, labelled by container name
discovery.docker "local" {
host = "unix:///var/run/docker.sock"
}
discovery.relabel "containers" {
targets = discovery.docker.local.targets
rule {
source_labels = ["__meta_docker_container_name"]
regex = "/(.*)"
target_label = "container"
}
}
loki.source.docker "containers" {
host = "unix:///var/run/docker.sock"
targets = discovery.relabel.containers.output
forward_to = [loki.write.default.receiver]
labels = {host = "docker1", job = "docker"}
}
On the Proxmox host, keep the loki.write and journal blocks and drop the Docker ones; the journal there carries pvedaemon, pveproxy, pvestatd, backup jobs and the kernel. Restart with systemctl restart alloy and check systemctl status alloy. Alloy also serves a debugging UI on port 12345 when started with a listen address, which shows each component and whether it is healthy.
Labels: few and boring
Loki is fast because its index is small. Every distinct combination of label values creates a new stream, and too many streams make Loki slow and heavy. Keep labels to things with a handful of values that you filter by: host, job, unit, container, level. Never turn request IDs, user names, IP addresses or full paths into labels; search for them in the line instead.
Querying with LogQL
In Grafana, open Explore, pick the Loki data source and start with a label selector, then filter lines:
{host="pve1", unit="pvedaemon.service"}
{job="docker", container="immich_server"} |= "error"
{job="journal", level=~"err|crit|alert|emerg"}
sum by (host) (count_over_time({level="err"}[1h]))
The first shows the Proxmox API daemon's log. The second shows only lines containing "error" from one container. The third shows every high-priority journal line across the lab. The last turns logs into a metric: errors per host per hour, which makes a good panel on an overview dashboard.
Don't forget the local copies
Loki gives you a searchable copy, but Docker still keeps container logs on the host, and with the default json-file driver they grow without limit unless configured. Set limits in /etc/docker/daemon.json on each Docker host and restart Docker (existing containers pick it up when recreated):
{
"log-driver": "json-file",
"log-opts": { "max-size": "10m", "max-file": "3" }
}
On the hosts themselves, cap the journal too: SystemMaxUse=500M in /etc/systemd/journald.conf, then systemctl restart systemd-journald. Loki is now the long memory; local logs only need to cover a few days.
Checklist
- Alloy, not Promtail, on every host.
- The alloy user is in
admandsystemd-journal(anddockeron Docker hosts). - Labels: host, job, unit, container, level. Nothing high-cardinality.
- Docker's json-file logs capped in daemon.json; the journal capped with SystemMaxUse.
- A Logs panel under the graphs on your host dashboard, so a spike and its log lines sit together.
This is adapted from my book Watch Your Homelab (Homelab the Right Way, Book 7). The free sample is here and the full book is here.
Want the week's homelab articles, videos and one tip from the books in your inbox every Sunday? Get the free weekly email.
Drafted with AI help and edited by the author; based on his book.
