Building an AWS + Snowflake with Terraform: Lessons from My CI/CD Review

Every analytics platform has an unglamorous foundation: the raw/landing tier where source data first lands before anything clever happens to it. It's easy to click that together by hand once. It's much harder to make it reproducible across dev, UAT, and prod and to prove to yourself that what's in prod is what you reviewed.

So I built the landing tier of a banking-style (FCC/AML) data platform entirely as Terraform: a single root module orchestrating five child modules, with remote state, environment promotion, and a GitLab pipeline gating every change. Then I reviewed my own code as if it were a PR and found a handful of design gaps that are worth more than the happy-path walkthrough. Both halves are below.

What the stack provisions

One root config wires together five modules across AWS and Snowflake:

Why modular, not one big file

The temptation with Terraform is to drop everything into main.tf. Splitting into aws_vpc, aws_ec2, aws_s3, aws_glue, and snowflake_database modules buys three things:

• Explicit dependencies. The EC2 module consumes the VPC module's subnet_id and security_group_id outputs an implicit dependency Terraform resolves for you. The Glue module carries an explicit depends_on = [module.s3_raw] so the bucket exists before the job references it. S3 and Snowflake share nothing, so they apply in parallel.
• Reuse. The same VPC module can back a completely different stack.
• Reviewable blast radius. A change to the Glue module can't silently touch networking.

State that won't corrupt itself

The backend is S3 for the state file plus a DynamoDB table for locking, both encrypted:

terraform {
  backend "s3" {
    bucket         = "your-tf-state-bucket"
    key            = "tf_state/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "your-tf-lock-table"
    encrypt        = true
  }
}

The DynamoDB lock is the part people skip and regret. Without it, two applys running at once — a teammate and a CI job, say can race and corrupt state. The lock makes the second one wait.

Promotion by branch, not by copy-paste

Instead of separate root configs per environment, one config selects its variable file from the branch, driven by workflow:rules:

• main → main.tfvars
• uat → uat.tfvars
• everything else (dev + feature branches) → dev.tfvars

First match wins, and merge-request events are dropped up front. The pipeline stages are validate → plan → apply → destroy:

plan writes a saved plan file and apply consumes exactly that artifact, so you apply the plan you reviewed - not a fresh one computed against drifted state.

Here's the pipeline running for real across the dev, uat, and main branches:

Now the part that actually taught me something

Reviewing this as a PR, three design gaps stood out. These are the interesting bit, because they're the mistakes that look fine until they bite:

1. Merge requests produce no pipeline at all. Because MR events are killed with when: never, there's no validate and no plan at review time. Reviewers approve blind - they never see what the change would actually do to infrastructure. The fix is a dedicated MR job that runs validate and plan and posts the plan output on the merge request, so approval is based on evidence.

2. main auto-applies to prod with no gate - but prod can't be destroyed. The pipeline applies to production automatically on merge to main, yet destroy is manual and blocked on main. That asymmetry is backwards: teardown is protected, but the far riskier unattended apply to prod isn't. Production apply is exactly where you want a manual approval gate.

3. Secrets in *.tfvars, and a .gitignore that misses them. The variable files carried credentials, and .gitignore only excluded *.tfvars.json - so *.tfvars was tracked. Credentials belong in CI variables or a secrets manager, the variables should be marked sensitive = true, *.tfvars should be ignored, and any exposed secret must be rotated and scrubbed from history.

A few smaller ones: the Glue module hardcoded its job name and script path instead of using the variables passed in, so it pointed at a bucket the stack never created; the pipeline had no terraform fmt check or security scanner (tfsec/Checkov); and the manual destroy used --auto-approve.

Takeaways

If you're standing up a landing zone of your own, the patterns that held up: split into modules with clear input/output contracts, lock your state, and promote one config across environments by variable file rather than duplicating configs. And the lesson the gaps taught me: the CI/CD design is the real security boundary, not the Terraform. A tidy .tf file with a pipeline that lets prod apply unreviewed and secrets slip into git isn't safe - it just looks safe.

Terraform makes infrastructure reproducible. Your pipeline is what makes it reviewable. Get that second part right.

The full code

If you want to dig into the actual Terraform - all five modules, the GitLab CI pipeline, and the environment variable files - the complete project is here:

🔗 https://gitlab.com/kiran.gntdm-group/terraform

Questions and feedback are welcome - happy to talk through any of the design choices.