A .tfstate file on your laptop is a single point of failure with no redundancy and no audit trail. The first time a GitHub Actions runner applies against the same infrastructure you just touched locally, you get drift, a lock error, or the failure mode nobody warns you about: a perfectly clean apply that quietly tears down a container. None of that is a state-storage problem in isolation, but remote state is where you fix the storage half and where you have to start thinking about the destruction half.
If you run infrastructure-as-code against a homelab or any self-hosted environment, you already have a perfectly good S3 endpoint sitting there: MinIO. You probably stood it up for backups. It can hold your OpenTofu state too, inside your own perimeter, with no Terraform Cloud account and no real AWS bill. The catch is that "S3-compatible" hides a few sharp edges, and pointing CI at your real compute changes the blast radius of a typo.
Who should care
This is for people running OpenTofu (or Terraform) against on-prem providers like Proxmox, where the resources under management are real boxes you'd rather not lose. If your state already lives in AWS S3 with DynamoDB locking and you're happy, you don't need this. The interesting territory is self-hosted: you want cloud-native state management without renting the cloud, and the provider you're driving can replace resources in ways a managed cloud rarely does.
What I tried first
The obvious starting point is local state. It works, right up until it doesn't. You write your .tf files, run tofu apply, and the state lands in terraform.tfstate next to your code. Then you add a CI pipeline, the runner starts from an empty checkout with no state, and OpenTofu happily plans to create everything you already created. Now you've got two sources of truth and a plan that wants to make duplicates.
The next instinct is to gitignore the state and commit it anyway "just for CI," or to scp it into the runner. Both are bad. State holds resource IDs and, depending on your providers, secrets in plaintext. It does not belong in Git. So remote state it is, and since MinIO is already running, the S3 backend looks like a five-minute job.
It is not a five-minute job the first time. My first MinIO backend block was copied from an AWS example with the endpoint swapped in:
terraform {
backend "s3" {
bucket = "opentofu-state"
key = "proxmox/terraform.tfstate"
region = "us-east-1"
endpoint = "https://minio.example.com"
}
}
tofu init rejected endpoint as deprecated, and once I got past that, the AWS SDK underneath tried to reach opentofu-state.minio.example.com. That hostname does not exist and the wildcard cert doesn't cover it, so I got TLS and DNS errors that looked like a MinIO problem and were actually an addressing-style problem. Then OpenTofu tried to call the AWS instance-metadata endpoint to figure out a region and credentials, which obviously goes nowhere on a bare-metal runner. Every error pointed at a different layer. None of them said "you forgot path-style addressing," which is what every single one of them meant.
The actual solution
Here's the backend block that actually works against MinIO with current OpenTofu. The shape of this changed across versions, so the keys matter:
terraform {
backend "s3" {
bucket = "opentofu-state"
key = "proxmox/terraform.tfstate"
region = "us-east-1" # arbitrary, but required by the SDK
endpoints = {
s3 = "https://minio.example.com"
}
use_path_style = true # bucket in the path, not the hostname
use_lockfile = true # S3-native locking, no DynamoDB
skip_credentials_validation = true
skip_requesting_account_id = true
skip_metadata_api_check = true
skip_region_validation = true
}
}
A few of those lines are the whole game. endpoints = { s3 = ... } replaced the old top-level endpoint attribute. use_path_style = true is the rename of what older guides call force_path_style. The four skip_* flags stop the AWS SDK from trying to validate credentials, look up an account ID, hit the metadata service, or sanity-check the region against AWS's real region list. On MinIO every one of those checks either fails or wastes time.
Credentials do not go in the block. Keep them out of .tf files entirely and feed them through the standard AWS environment variables, which the S3 backend reads natively:
export AWS_ACCESS_KEY_ID="$(op read 'op://Infra/minio-opentofu/access-key')"
export AWS_SECRET_ACCESS_KEY="$(op read 'op://Infra/minio-opentofu/secret-key')"
tofu init
That access key should not be your MinIO root user. Create a dedicated user with a policy scoped to exactly the one bucket OpenTofu touches. Least privilege here is the difference between "a leaked CI secret can read one state file" and "a leaked CI secret owns your entire object store." This is the same mindset I wrote about for agent service accounts: give the automation the smallest grant that lets it do its job.
The MinIO policy looks like this:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["s3:ListBucket", "s3:GetBucketLocation"],
"Resource": ["arn:aws:s3:::opentofu-state"]
},
{
"Effect": "Allow",
"Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
"Resource": ["arn:aws:s3:::opentofu-state/*"]
}
]
}
ListBucket and GetBucketLocation are on the bucket itself. The object actions live on the /* path. DeleteObject is non-optional even though that surprises people: the lockfile is written before an operation and deleted after, so without delete you can acquire a lock and never release it. Apply it with the MinIO client:
mc admin policy create local opentofu-state ./opentofu-state-policy.json
mc admin user add local opentofu-ci "$(op read 'op://Infra/minio-opentofu/secret-key')"
mc admin policy attach local opentofu-state --user opentofu-ci
Wiring it into CI
In GitHub Actions, the MinIO credentials come from repository secrets and get injected as the AWS env vars the backend expects. The job stays small:
- name: OpenTofu Init
env:
AWS_ACCESS_KEY_ID: ${{ secrets.MINIO_ACCESS_KEY }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.MINIO_SECRET_KEY }}
run: tofu init
- name: OpenTofu Plan
env:
AWS_ACCESS_KEY_ID: ${{ secrets.MINIO_ACCESS_KEY }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.MINIO_SECRET_KEY }}
run: tofu plan -detailed-exitcode -out=tfplan
-detailed-exitcode is the piece most pipelines skip. It returns 0 for no changes, 2 for a non-empty plan, and 1 for an error. That lets you gate apply on human approval whenever the plan is non-empty, instead of letting every push to main run straight through to apply -auto-approve. If your runner can't reach MinIO over the public internet, this is also where a Tailscale step earns its keep; I covered that access pattern in Tailscale subnet routers.
The safety net that actually matters
Storing state remotely solves the "where does the truth live" problem. It does nothing for the "a successful apply destroyed something" problem, and on Proxmox that problem is real. The bpg and Telmate providers both have attributes marked ForceNew. Change one of those on an LXC, the network bridge or the rootfs storage, and the provider cannot update in place. It plans a destroy followed by a create. Replacing an LXC means deleting the container and its filesystem, then building a fresh one. The data that lived on it does not come back.
Read that plan by hand and you'd catch it instantly: -/+ destroy and then create replacement. Run it under -auto-approve in CI off a push to main, and the box is gone before anyone looks at the diff. Remote state didn't cause that, but remote state is what made unattended applies easy, so it's the moment to add guards. The first guard is a lifecycle block on anything stateful:
resource "proxmox_virtual_environment_container" "app" {
# ... container config ...
lifecycle {
prevent_destroy = true
}
}
With prevent_destroy = true, any plan that would delete or replace this resource fails at plan time with an error instead of proceeding. That's the behavior you want: the pipeline stops and shouts rather than quietly doing the destructive thing. When you genuinely need to replace the resource, you remove the block deliberately, in a reviewed commit, with your eyes open. The same defensive instinct shows up in provider-driven recreation elsewhere; I hit a version of it with hardware, where a card got replaced out from under me, in the GPU passthrough field guide.
Why it works
Two mechanics are worth understanding instead of cargo-culting, because both will bite you again in other contexts.
The path-style flag comes down to URL construction. AWS S3 prefers virtual-hosted-style addressing: the bucket becomes a subdomain, bucket.s3.amazonaws.com. MinIO serves buckets path-style by default: minio.example.com/bucket. When the SDK defaults to virtual-hosted style against MinIO, it builds opentofu-state.minio.example.com, which has no DNS record and no matching certificate. Setting use_path_style = true tells the SDK to keep the bucket in the path. Every confusing TLS-and-DNS error I got was this and only this, wearing four different costumes.
State locking is the other one. The classic Terraform/OpenTofu S3 backend needed a DynamoDB table to hold a lock item, because plain S3 historically had no way to do an atomic "create this only if it doesn't already exist." Self-hosters without DynamoDB faked it with sidecar services or just turned locking off and prayed. Recent OpenTofu (1.10+) added use_lockfile = true, which writes a .tflock object next to your state and relies on S3 conditional writes (PutObject with If-None-Match) to make lock acquisition atomic. MinIO implements conditional writes, so the lock is honest: a second apply that arrives while the first holds the lock gets refused at the object-store level, not by a polite convention. No extra database, no extra moving part. That's why the same MinIO instance you use for Velero Kubernetes backups can back your IaC state without bolting anything else on. One S3 layer, two jobs.
prevent_destroy works because it intercepts the plan graph before execution. OpenTofu walks the dependency graph, decides each resource needs no-op, update, create, or replace, and a lifecycle guard converts any planned delete on that resource into a hard error. It is a plan-time veto, not a runtime one, which is exactly why it's safe in CI: nothing has touched real infrastructure yet when it fires.
Lessons learned
The version churn on the S3 backend is the thing that wasted the most of my time, and most of it is invisible until init complains. endpoint became endpoints.s3. force_path_style became use_path_style. DynamoDB locking became use_lockfile. If you're following a guide written for an older OpenTofu or for Terraform pre-1.10, half the keys are renamed and you'll burn an afternoon on deprecation warnings that don't clearly explain the replacement. Check the backend docs for your exact version before copying anyone's block, including mine.
prevent_destroy is a seatbelt, not an airbag, and it has a real gap: it only protects a resource that's still in your configuration. Delete the resource block from your .tf files and the lifecycle rule goes with it, so the next plan cheerfully schedules the destroy with nothing left to veto it. Treat removing a stateful resource from config as a destructive operation in its own right, and review those diffs as carefully as you'd review an apply.
The destruction risk reframed how I think about remote state entirely. Moving state into MinIO felt like a storage decision, but the real change is that you've made unattended applies cheap, and cheap unattended applies against real compute need guardrails that local-state workflows never forced you to build. -detailed-exitcode to gate on human review, prevent_destroy on anything holding data, and a habit of actually reading replacement plans matter more than the backend block itself. This is the part of the workflow that complements the GitHub Actions automation I described in automating OpenTofu with GitHub Actions: the automation gets you speed, the guards keep the speed from being a liability.
Credentials were the last lesson, and an easy one to get lazy about. The MinIO key OpenTofu uses should be scoped to one bucket, stored in a secret manager, and pulled into env vars at runtime rather than parked in a .tf file or, worse, a shell history. If you've ever fought your shell over special characters in an automation token, you already know how these leak; I wrote about one such trap in Proxmox API tokens and the ! character. A scoped key plus a lifecycle guard turns the worst-case CI accident from "lost the cluster" into "lost one state file I can restore from a versioned bucket."
That last point is the quiet upside of running this on MinIO instead of a SaaS backend. Turn on bucket versioning and your state file gets the same history your backups do, inside the same perimeter, under the same retention you already control. If you want help designing that kind of self-hosted IaC and storage layer for a production environment, that's the sort of thing I do through GuatuLabs. The state of your infrastructure is some of the most sensitive data you have. Keeping it on hardware you own, behind credentials you scoped, with guards that refuse to delete the wrong box, is a reasonable thing to want, and it's maybe a weekend of work once you know which four flags actually matter.
