The stack from Part 3 is live: multiple domains, isolated namespaces, TLS everywhere. But a production system you cannot observe is a production system you do not control. In this part we install a full observability stack on a modest 2 vCPU, 8 GB VPS, then wire up a GitHub Actions pipeline so that a git push becomes a deployment. Both came with surprises. I will share all of them.
The monitoring stack and its budget
The classic trio:
- Prometheus scrapes and stores metrics from every pod and from the node itself
- Grafana turns those metrics into dashboards
- Loki with Promtail aggregates logs from every container in the cluster
On a small VPS the default Helm values of these charts will eat your memory alive. Here is the budget that works for me, about 1.2 GB total for the whole namespace:
prometheus:
prometheusSpec:
retention: 15d
retentionSize: "8GB"
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi
Retention matters more than people think. Fifteen days of metrics on an 8 GB size cap keeps the disk predictable on a 100 GB volume. For logs, seven days is plenty for a personal infrastructure:
loki:
config:
limits_config:
retention_period: 168h
compactor:
retention_enabled: true
One K3s specific detail: disable the components that do not exist on a lightweight distribution, otherwise Prometheus fills up with permanently failing targets:
kubeEtcd:
enabled: false
kubeControllerManager:
enabled: false
kubeScheduler:
enabled: false
kubeProxy:
enabled: false
Three installation traps on K3s
The Helm install of kube-prometheus-stack failed for me three times, each for a different reason. Documenting them here so you skip the debugging session.
Trap one: the admission webhook job hangs. The chart runs a pre install job that generates TLS certificates for its admission webhooks. On my cluster that job ran for nineteen minutes without finishing and blocked everything behind a "timed out waiting for the condition" error. On a single node personal cluster the admission webhooks are dispensable. Disable them:
helm install monitoring prometheus-community/kube-prometheus-stack \
--namespace ns-monitoring \
--values monitoring-values.yaml \
--set prometheusOperator.admissionWebhooks.enabled=false \
--set prometheusOperator.admissionWebhooks.patch.enabled=false \
--set prometheusOperator.tls.enabled=false
Trap two: half measures crash loop. My first attempt disabled the webhook job but not the TLS mount that depended on it. The operator pod then crash looped on a missing secret called monitoring-kube-prometheus-admission. If you disable the webhooks, disable the TLS block too. And if the release ends up in a broken state, uninstall it cleanly and reinstall rather than patching a corpse.
Trap three: quotas versus third party charts. This is the lesson from Part 3 striking again. My ResourceQuota on the monitoring namespace required every container to declare resources, but the Grafana chart ships init containers without any. Result: zero pods, zero error messages at the Deployment level, and the real reason buried in the ReplicaSet events:
Error creating: pods "monitoring-grafana-..." is forbidden:
failed quota: must specify limits.cpu for: init-chown-data, ...
The pragmatic fix on a personal cluster is to drop the quota on the monitoring namespace. The purist fix is to override the resources of every sub chart container. I chose pragmatism.
The general debugging rule that emerged: when a Deployment shows desired pods but zero current pods, run kubectl describe on the ReplicaSet. Quota rejections live there and nowhere else.
Exposing Grafana safely
Grafana lives on its own subdomain behind Traefik, with two layers of authentication: a basic auth middleware at the proxy level, then the Grafana login itself.
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: grafana-auth
namespace: ns-monitoring
spec:
basicAuth:
secret: grafana-basic-auth
Generate the htpasswd value, base64 encode it, store it in a Secret, reference it from the middleware. Two minutes of work for a serious reduction of exposed attack surface on a service that holds every metric of your infrastructure.
With Loki registered as a Grafana datasource, the payoff is immediate: one interface where you can jump from a CPU spike on a dashboard straight to the logs of the pod that caused it.
The CI/CD pipeline
Manual deployment was fine while building. It stops being fine the third time you type the same six commands. The target workflow:
git push main
detect which apps changed
lint and test only those apps
build Docker images, push to GitHub Container Registry
SSH into the VPS, update the image, wait for the rollout
Path filtering keeps the pipeline fast in a monorepo. Touching only the frontend must not rebuild the blog image:
- uses: dorny/paths-filter@v3
with:
filters: |
web:
- 'apps/web/**'
worker:
- 'apps/crosspost-worker/**'
Images are tagged twice, latest and the short commit SHA. The SHA tag is your rollback button: redeploying any previous version is one kubectl set image away.
Five pipeline failures and their fixes
My pipeline went red five times before going green. Every failure was a lesson.
Failure one: pnpm version conflict. The package.json declared a packageManager field while the workflow pinned a version on the pnpm setup action. The action refuses ambiguity and aborts. Fix: remove the version from the workflow and let packageManager be the single source of truth.
Failure two: module level SDK instantiation. The contact route created its Resend client at import time. Next.js executes route modules while collecting page data during the build, where no API key exists, so the build crashed. Fix: lazy initialization.
let client: Resend | null = null;
function getClient(): Resend {
if (!client) client = new Resend(process.env.RESEND_API_KEY);
return client;
}
The wider rule: never instantiate SDK clients at module scope in a Next.js app. Always create them inside the request handler path.
Failure three: wrong Docker build context. In a monorepo the lockfile lives at the root. Building with the app folder as context means the Dockerfile cannot copy it. The build must run from the repository root with the Dockerfile referenced by path, and the standalone output of Next.js lands in a nested folder, which changes the runtime paths too:
COPY --from=builder /app/apps/web/.next/standalone ./
COPY --from=builder /app/apps/web/.next/static ./apps/web/.next/static
CMD ["node", "apps/web/server.js"]
Failure four: registry permissions. The first image push was done manually with a personal access token, which made me the package owner. The automatic GITHUB_TOKEN of the workflow then received a 403 on push. Fix: in the package settings on GitHub, grant the repository write access under Manage Actions access. One click per package, easy to forget, guaranteed 403 if you do.
Failure five: a corrupted SSH secret. The deploy step failed with "error in libcrypto", which is OpenSSH speak for "this private key file is malformed". The key had been copied by hand into the GitHub secret and lost its line breaks along the way. Fix: pipe the file straight into the clipboard and paste it without touching it. On Windows:
Get-Content "$env:USERPROFILE\.ssh\deploy_key" -Raw | Set-Clipboard
The bonus bug: a page frozen in time
With the pipeline green, one ghost remained. The portfolio kept showing placeholder articles even though the blog API returned the real ones. Environment variables were correct inside the pod. Fetching from inside the pod worked. No errors anywhere.
The cause was subtle. The articles component guards its fetch: if the Ghost credentials are missing, it returns demo content without fetching. During the Docker build no credentials exist, so no fetch ran during prerendering. And Next.js only marks a page as revalidating if it actually observes a fetch with a revalidate option during the build. No fetch observed means fully static page, cached forever, never regenerated. My revalidate option inside the fetch call was correct and completely unused.
The fix is one line at the route level, which forces incremental static regeneration regardless of what happens during the build:
export const revalidate = 300;
After the next deployment, the first visit still serves the stale page but triggers regeneration in the background, and the second visit shows real data. Every future article now appears on the portfolio within five minutes, no redeploy needed.
Where we stand
The infrastructure is now complete:
- Full metrics and log visibility through Grafana on a protected subdomain
- Alerting foundation in place with Alertmanager
- A pipeline where git push main lints, tests, builds, publishes and deploys only what changed
- Rollback to any previous version in one command thanks to SHA tagged images
The whole system, applications and observability included, runs within 8 GB of RAM on a single VPS. Self hosting like a pro is not about big hardware. It is about isolation, budgets, and automating yourself out of the deployment loop.
Next up: backups and disaster recovery, because an infrastructure you cannot restore is an infrastructure you do not own.
