Somebody on r/Terraform is watching their Terragrunt CI re-download the internet on every single run — and every obvious fix has a trap in it.
The setup, from the thread: a multi-component Terragrunt project (Terragrunt is a wrapper that keeps Terraform code DRY — don’t repeat yourself — across many components), CI on GitLab running on Kubernetes runners, providers and modules re-downloaded every run. Downloads are slow and sometimes flaky. They tried caching to S3 as compressed archives — the compress and decompress time ate the saving. Pre-baking every provider into the runner image? By their estimate that image lands around 80 GB. A shared EFS volume is now on the table.
Friend, you do not have a download problem. You have a cache-placement problem. And Terraform has a very specific opinion about where provider binaries may safely be shared — ignore it and your parallel runs will corrupt each other’s downloads in fun, non-reproducible ways.
Every init downloading the same provider is not thoroughness. It is your pipeline paying the same toll twenty times a day.
What is actually being downloaded, again and again
A provider is the plugin binary that lets Terraform talk to a cloud — the AWS provider alone is a large binary, and a multi-component project downloads the same providers once per component, per run, because each component initialises in its own working directory. Modules (reusable Terraform building blocks) get fetched per component too, into each unit’s .terragrunt-cache. Multiply by every pipeline, every day. That is the bill — in minutes, and in flaky failures when a registry hiccup fails a job that changed three lines.
Trap 1: the plain plugin cache directory, shared by parallel runs
Terraform’s own shared plugin cache — plugin_cache_dir in the CLI config, or the TF_PLUGIN_CACHE_DIR environment variable — sounds like the answer: point every run at one folder, download once. Read the fine print in Terraform/OpenTofu’s own docs, though: that cache directory is not guaranteed to be concurrency safe, and the provider installer’s behaviour with multiple concurrent init calls is undefined. Two runners populating the same cold cache at the same moment can meet a half-unpacked provider. Terragrunt’s own performance docs add the other half: at large scale, filesystem lock contention on that shared directory becomes the bottleneck. It works in a demo. CI is not a demo.
Trap 2: zip it, ship it, unzip it
The thread’s S3 proof-of-concept failed for a boring reason: compression is a tax you pay twice — once packing, once unpacking — on every run, warm or cold. A cache that costs minutes to restore is just a slower download wearing a costume. Same family: pre-baking providers into the runner image trades download time for an image so large the runner spends its life pulling it instead. The thread’s ~80 GB estimate is their number, not a law of physics — but the direction is right: the image becomes the download.
The fix Terragrunt already built: the Provider Cache Server
Terragrunt ships a provider cache server made for exactly this shape — many units, parallel runs, one shared cache. Turn it on per run:
terragrunt run --all --provider-cache plan
or with environment variables, which is how you will actually do it in CI:
TG_PROVIDER_CACHE=1 \
TG_PROVIDER_CACHE_DIR=/shared/terragrunt-provider-cache \
terragrunt run --all plan
What happens then: Terragrunt serves providers to its own Terraform/OpenTofu runs from the shared cache directory, downloading from the registry only on a genuine miss, and the cache directory uses the same layout as Terraform’s plugin cache — so an existing warmed cache folder can be reused by pointing --provider-cache-dir (or TG_PROVIDER_CACHE_DIR) at it. By default it caches providers from registry.terraform.io and registry.opentofu.org; private registries can be added with --provider-cache-registry-names / TG_PROVIDER_CACHE_REGISTRY_NAMES. One warning from experience-shaped common sense: the cache dir must be a filesystem your runners actually share (your EFS idea works for this), and it must survive between runs — a cache on an ephemeral runner disk that dies with the pod is a diary, not a cache.
On newer OpenTofu (1.10 and later), Terragrunt also has an automatic provider cache dir mode that sets TF_PLUGIN_CACHE_DIR to a central location for you, since that OpenTofu line handles concurrent cache access safely — Terragrunt’s docs position the cache server as the pick for Terraform and older OpenTofu, or when you need the advanced sharing. Know which of the two you turned on; running both by accident is how cache behaviour becomes folklore.
And the modules?
Providers are the heavy half, but do not ignore the other download: Terragrunt fetches each unit’s module source into its per-unit .terragrunt-cache. Persisting that between CI runs helps, but key the cache on your lockfiles and config hashes, and prune it — Terragrunt and env0-style guidance both warn the cache directory grows large in CI if you never clean it. A cache with no eviction plan is a disk-full incident with a waiting period.
The CI caching checklist
Cache server on:
TG_PROVIDER_CACHE=1set for Terragrunt runs in CI.Shared, persistent dir:
TG_PROVIDER_CACHE_DIRpoints at storage that outlives the runner pod (shared volume, not pod disk).Private registries listed: any registry beyond the two defaults added via
TG_PROVIDER_CACHE_REGISTRY_NAMES.No naked shared plugin cache under parallelism: plain
TF_PLUGIN_CACHE_DIRsharing is not concurrency-guaranteed — that is the cache server’s job.Module cache keyed + pruned:
.terragrunt-cachepersisted with a lockfile-based key and a size cap.No zip archives: if restoring the cache means decompressing it, you rebuilt the download with extra steps.
No provider-stuffed runner image: the image stays small; the cache lives outside it.
Staging warning: test cache behaviour in staging first — a poisoned shared cache fails every pipeline at once.
The fix, in one line: stop downloading providers per run and stop sharing the naive cache folder across parallel inits — switch on Terragrunt’s provider cache server, park its directory on storage that survives the runner, and let the registry become the fallback instead of the routine.
Disclaimer
This post describes my personal experience with my own systems and accounts, in my own words. It is a first-person account of what I observed and what I do in my own work — not a statement of any company’s intentions, policies, or internal decisions. Where I do not know something, the post says so.
I am not affiliated with, endorsed by, or speaking for any company, product, or vendor named in this post. All product and company names are trademarks of their respective owners and are mentioned only to describe the experience accurately.
Any error messages, documentation excerpts, or figures quoted here are reproduced from my own records or from the public sources linked in the post, believed accurate at the time of writing. Product behavior changes over time; verify against current official documentation before relying on any detail.
Nothing in this post is professional, financial, legal, or security advice for your specific situation. Examples are illustrative — validate in your own environment before running anything in production.
Corrections are welcome. If any company named here publishes an official statement, or if any detail is wrong, I will update this post and note the correction. The goal is an accurate public record, not a verdict.
© 2026 Honey Kiran. All rights reserved.
Honey Kiran · DevOps & Platform Engineering
Writing on monitoring, SRE, Terraform, Kubernetes and platform engineering — one hard-won lesson at a time.
Tell me in the replies: what does your CI re-download every run that it has no business downloading twice?
Examples are illustrative — validate in your own environment before running in production.
