Deploying to a Server You Can't Reach: Building a CI/CD Pipeline with AWS SSM and OIDC

From pushing code manually to building, testing, containerizing, and deploying every change automatically.

In the previous article, we learned how to spin up a functioning network and compute resources on AWS, and deploy our application on them. Let's remember a key architectural decision: our EC2 instance is in a private subnet, shielded from the internet.

Now we have a problem. It gets tedious to log in, pull code, and build our Docker images every time we make changes to our codebase.

Enter CI/CD.

CI/CD stands for Continuous Integration and Continuous Delivery/Deployment. It aims to streamline and accelerate the software development lifecycle.

Continuous Integration (CI) refers to the practice of automatically integrating code changes into a shared source code repository. Continuous Delivery and/or Deployment (CD) is a two-part process that refers to the integration, testing, and delivery of code changes. Continuous delivery stops short of automatic production deployment, while continuous deployment automatically releases the updates into the production environment. This means our deployment should happen every time we push code to our GitHub repo.

(If you need a refresher on our setup, check it out here: Terraform Deployment)

Here is the catch: our server has no public IP address. It sits in a private subnet behind a load balancer. GitHub Actions cannot SSH into it. There is no port 22 open to the internet. There is no bastion host.

So how do you deploy to a server you can't reach from the internet?

Why This is More Complex Than a Typical CI/CD Setup

Typically, we would set up a pipeline that builds a Docker image, pushes it to a registry, then SSHes into a server with a stored SSH key to pull the image and restart the container.

None of that works for this setup.

The First Problem: The server is unreachable. The EC2 instance lives in a private subnet (10.0.10.0/24). Traffic from the internet goes through the Application Load Balancer, not directly to the server. There is no public IP. GitHub Actions cannot SSH in.

The Second Problem: Four containers, not one. This is not a single Docker image deployment. It is a docker-compose.yml stack with four interdependent services:

PostgreSQL → (health check passes) → Backend → (health check passes) → Frontend → Nginx

If the backend starts before PostgreSQL is healthy, the Alembic database migrations crash. If Nginx starts before the backend is healthy, it throws a 502 Bad Gateway. Docker Compose manages this dependency chain with health checks and depends_on conditions. So the deployment tool needs to orchestrate Docker Compose on the server, not just swap one container.

Problem 3: No stored credentials. I did not want AWS access keys sitting in GitHub Secrets. Keys do not expire. If they leak, they are valid until someone notices and manually revokes them. That could be weeks. Or months. In a personal project, probably never.

I needed a solution that was:

• Keyless (no stored AWS credentials)
• Reachable (can talk to a private subnet server)
• Composable (can orchestrate a multi-container stack)

The answer turned out to be three AWS services I had already partially set up: IAM OIDC, Systems Manager (SSM), and Terraform.

The Architecture

Here is how the pipeline works end to end.

Developer pushes to main
         │
         ▼
┌─────────────────────────────┐
│  GitHub Actions CI Workflow  │
│  ┌───────────────────────┐  │
│  │ Docker Compose Build  │  │
│  │ Frontend Build (Vite) │  │
│  │ Backend Lint (Flake8) │  │
│  └───────────────────────┘  │
└──────────────┬──────────────┘
               │ (all checks pass)
               ▼
┌─────────────────────────────┐
│  GitHub Actions CD Workflow  │
│                             │
│  1. Request OIDC token from │
│     GitHub's token service  │
│                             │
│  2. Present token to AWS    │
│     STS: "I am repo X,      │
│     running workflow Y"     │
│                             │
│  3. AWS returns temporary   │
│     credentials (1 hour)    │
│                             │
│  4. Use AWS CLI to send     │
│     SSM command to EC2      │
│                             │
│  5. EC2 executes deploy.sh  │
│     (git pull, compose up)  │
│                             │
│  6. Fetch logs, verify      │
│     success or fail build   │
└─────────────────────────────┘

No SSH. No stored keys. No open ports. The only thing stored in GitHub Secrets is the ARN of an IAM role — a reference, not a credential.

Part 1: The CI Pipeline — Catching Problems Before They Reach the Server

The CI workflow runs on every push to main and every pull request. Its job is simple: make sure the code is not broken before we even think about deploying.

name: CI

on:
  push:
    branches: [ "main" ]
  pull_request:
    branches: [ "main" ]

jobs:
  build-and-test:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout code
      uses: actions/checkout@v4

    - name: Set up Docker Buildx
      uses: docker/setup-buildx-action@v3

    - name: Validate Docker Compose build
      run: docker compose build

    - name: Set up Node.js for Frontend
      uses: actions/setup-node@v4
      with:
        node-version: '20'

    - name: Check Frontend build
      working-directory: ./fe-apartment
      run: |
        npm install
        npm run build

    - name: Set up Python for Backend
      uses: actions/setup-python@v5
      with:
        python-version: '3.12'

    - name: Lint Backend (Flake8)
      working-directory: ./be-apartment
      run: |
        python -m pip install --upgrade pip
        pip install flake8
        flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
        flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics

Three things happen:

1. Docker Compose build validation. This builds all four container images. If a Dockerfile is broken, a requirements.txt has a bad package, or Nginx config has a syntax error — it fails here, not on the server.
2. Frontend build. React with Vite. npm install then npm run build. If someone introduces a TypeScript error or a bad import, the build fails. The CI catches it.
3. Backend lint. Flake8 on the FastAPI codebase. The first pass (--select=E9,F63,F7,F82) catches hard errors — syntax errors, undefined names, things that will crash at runtime. The second pass reports style warnings without failing the build. I am strict on things that break. I am lenient on things that are ugly.

Why run Flake8 in two passes? Because if you block every single style violation from day one on a codebase that was not linted before, you will spend your first week fixing hundreds of trailing whitespace warnings instead of shipping features. The first pass is a wall. The second pass is a nudge.

Part 2: No Stored Keys — The OIDC Trust Relationship

This is the part I am most proud of.

To deploy, the pipeline needs to talk to AWS. The traditional approach: create an IAM user, generate access keys, store them as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in GitHub Secrets.

That works. But those keys never expire. If they leak through a log, a screenshot, or a misconfigured workflow that echoes environment variables, they are valid forever. Or until someone remembers to rotate them. On a personal project? That is never.

OIDC flips the model. Instead of storing a key, you set up a trust relationship. GitHub says: "I am the repository Israel-dot-com/apartment-deployment-main, running a workflow on the main branch." AWS checks its trust policy and says: "I trust that repository. Here are temporary credentials that expire in one hour."

No keys stored anywhere. If the token somehow leaks, it is useless in 60 minutes. And it only works from my specific repository — not from a fork, not from a different org, not from someone who copied my workflow file.

Setting It Up With Terraform

I did not want to click through the AWS Console to set this up. The whole point of this project is Infrastructure as Code. So the OIDC provider and IAM role live in Terraform:

# terraform/github_oidc.tf

resource "aws_iam_openid_connect_provider" "github" {
  url             = "https://token.actions.githubusercontent.com"
  client_id_list  = ["sts.amazonaws.com"]
  # AWS recommends these official GitHub thumbprints
  thumbprint_list = [
    "6938fd4d98bab03faadb97b34396831e3780aea1",
    "1c58a3a8518e8759bf075b76b750d4f2df264fcd",
    "06d927fecd0a84aeba28aad1d808139470fe95c3"
  ]
}

resource "aws_iam_role" "github_actions" {
  name = "${var.project_name}-github-actions-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = {
        Federated = aws_iam_openid_connect_provider.github.arn
      }
      Action    = "sts:AssumeRoleWithWebIdentity"
      Condition = {
        StringEquals = {
          "token.actions.githubusercontent.com:aud" = "sts.amazonaws.com"
        }
        StringLike = {
          # Pro-Tip: Add wildcards around your org/repo names! 
          # GitHub occasionally appends internal numeric IDs to the claim.
          "token.actions.githubusercontent.com:sub" = "repo:Israel-dot-com*/apartment-deployment-main*"
        }
      }
    }]
  })
}

The Condition block is critical. StringLike on the sub claim locks this role to my specific repository. No other GitHub repository in the world can assume it. If someone forks my repo and runs the workflow, AWS rejects the token.

(Pro Tip:* Notice the * after the organization and repository names! GitHub recently started occasionally appending internal numeric IDs to OIDC tokens, so a strict StringEquals might randomly fail. Using StringLike with wildcards saves you hours of debugging!)*

Least Privilege: Only What the Pipeline Needs

The role gets exactly three permissions. Notice how we split the ssm:SendCommand resource array to properly accommodate tag conditions:

resource "aws_iam_role_policy" "github_actions_ssm" {
  role = aws_iam_role.github_actions.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
        Action   = ["ec2:DescribeInstances"]
        Resource = "*"
      },
      {
        # Allow running SSM commands on our specific tagged instance
        Effect   = "Allow"
        Action   = ["ssm:SendCommand"]
        Resource = ["arn:aws:ec2:${var.aws_region}:${data.aws_caller_identity.current.account_id}:instance/*"]
        Condition = {
          StringEquals = {
            "ssm:ResourceTag/Name" = "${var.project_name}-server"
          }
        }
      },
      {
        # Allow access to the AWS managed shell script document
        Effect   = "Allow"
        Action   = ["ssm:SendCommand"]
        Resource = ["arn:aws:ssm:${var.aws_region}::document/AWS-RunShellScript"]
      },
      {
        # Allow retrieving the command logs
        Effect   = "Allow"
        Action   = ["ssm:GetCommandInvocation"]
        Resource = "arn:aws:ssm:${var.aws_region}:${data.aws_caller_identity.current.account_id}:*"
      }
    ]
  })
}

That is it. The role cannot create resources. It cannot delete infrastructure. It cannot read secrets. It cannot modify IAM. It can do exactly one thing: tell one specific server to run a shell script, and read the result.

Part 3: Deploying to a Server Nobody Can Reach

Here is the problem again: the EC2 instance has no public IP. It lives in a private subnet. GitHub Actions cannot SSH into it.

AWS Systems Manager (SSM) solves this. The SSM Agent runs on the EC2 instance and maintains a persistent outbound connection to the AWS SSM service. When you send a command through the AWS API, SSM relays it to the agent on the instance. The server reaches out. Nobody reaches in.

No open inbound ports. No SSH keys. No bastion hosts. And every command is logged in CloudTrail — who ran what, when, and what happened.

The EC2 instance already had the AmazonSSMManagedInstanceCore policy attached from when I built the Terraform infrastructure, and the SSM Agent comes pre-installed on Ubuntu AMIs. So this just worked.

The Deploy Script

The script that actually runs on the server is deliberately simple:

#!/bin/bash
set -euo pipefail

echo "========================================="
echo "Starting deployment at $(date)"
echo "========================================="

PROJECT_DIR="/home/ubuntu/apartment-deployment"
cd "$PROJECT_DIR"

# Pull the latest code
echo "--> Pulling latest code from main branch..."
git fetch origin main
git checkout main
git reset --hard origin/main

# Build and start the containers
echo "--> Rebuilding and starting Docker containers..."
docker compose build
docker compose up -d

# Clean up dangling images to free up space
echo "--> Cleaning up unused Docker images..."
docker image prune -f

echo "========================================="
echo "Deployment completed successfully at $(date)"
echo "========================================="

set -euo pipefail is important. If any command fails — git fetch times out, docker compose build hits a bad Dockerfile, docker compose up crashes — the script exits immediately with a non-zero status. SSM reports it as a failure, the GitHub Action fails, and I get notified.

git reset --hard origin/main instead of git pull is intentional. The server is not a workspace; it is a deployment target. It should always precisely mirror main.

docker image prune -f is the line that saves you at 3am. Every docker compose build creates new images. On a standard EBS volume, it takes about two weeks of deploys before the disk fills up and Docker refuses to build anything. Pruning after every deploy prevents that.

The CD Workflow

name: CD

on:
  push:
    branches: [ "main" ]

permissions:
  id-token: write    # Required for OIDC
  contents: read

jobs:
  deploy:
    name: Deploy to EC2
    runs-on: ubuntu-latest

    steps:
    - name: Checkout code
      uses: actions/checkout@v4

    - name: Configure AWS Credentials
      uses: aws-actions/configure-aws-credentials@v4
      with:
        role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
        aws-region: us-east-1

    - name: Deploy via AWS Systems Manager
      run: |
        INSTANCE_ID=$(aws ec2 describe-instances \
          --filters "Name=tag:Name,Values=apartment-server" \
                    "Name=instance-state-name,Values=running" \
          --query "Reservations[*].Instances[*].InstanceId" \
          --output text)

        if [ -z "$INSTANCE_ID" ]; then
          echo "Error: Could not find apartment-server"
          exit 1
        fi

        # Note: We run the command as the 'ubuntu' user to avoid Git 
        # "dubious ownership" errors caused by SSM running as root.
        COMMAND_ID=$(aws ssm send-command \
          --instance-ids "$INSTANCE_ID" \
          --document-name "AWS-RunShellScript" \
          --parameters 'commands=["sudo -i -u ubuntu bash /home/ubuntu/apartment-deployment/deploy.sh"]' \
          --query "Command.CommandId" \
          --output text)

        echo "Waiting for deployment to complete (timeout: 10 minutes)..."
        # We use a custom polling loop here because the default 'aws ssm wait' 
        # times out after 100 seconds, which isn't long enough for Docker builds!
        for i in {1..60}; do
          STATUS=$(aws ssm get-command-invocation --command-id "$COMMAND_ID" --instance-id "$INSTANCE_ID" --query "Status" --output text)
          if [[ "$STATUS" != "InProgress" && "$STATUS" != "Pending" && "$STATUS" != "Delayed" ]]; then
            break
          fi
          sleep 10
        done

        echo "Deployment Output:"
        aws ssm get-command-invocation \
          --command-id "$COMMAND_ID" \
          --instance-id "$INSTANCE_ID" \
          --query "StandardOutputContent" --output text

        if [ "$STATUS" != "Success" ]; then
          echo "Error log:"
          aws ssm get-command-invocation \
            --command-id "$COMMAND_ID" \
            --instance-id "$INSTANCE_ID" \
            --query "StandardErrorContent" --output text
          echo "Deployment failed with status: $STATUS"
          exit 1
        fi

        echo "Deployment completed successfully!"

Let me walk through what happens:

1. OIDC authentication. aws-actions/configure-aws-credentials@v4 handles the entire OIDC handshake. It requests a token from GitHub, presents it to AWS STS, receives temporary credentials, and configures the AWS CLI. One action. Zero stored keys.
2. Find the server by tag. The pipeline does not hardcode an instance ID. If I destroy and recreate the infrastructure with Terraform, the pipeline finds the new instance dynamically using its name tag.
3. Send the command. ssm:SendCommand tells the EC2 instance to run deploy.sh. (Notice we use sudo -i -u ubuntu to ensure we execute as the correct user).
4. Wait and verify. Because docker compose build can take a few minutes, we use a custom bash loop to poll SSM for up to 10 minutes. Once it completes, we fetch the logs.

The only thing stored in GitHub Secrets is AWS_OIDC_ROLE_ARN — the ARN of the IAM role. This is not a credential. It is a reference. AWS will reject the OIDC token from any other source.

Part 4: What Happens When It Breaks

A green pipeline does not mean your application works. It means your code compiled, your images built, and the deploy script exited with status 0. But Docker Compose reporting healthy containers and your load balancer agreeing are two different things.

The ALB health check hits / every 30 seconds. If the Nginx container is up but the backend behind it is crashing, the ALB marks the target as unhealthy. Users get a 502. The pipeline is green. The app is down.

This is why monitoring exists alongside CI/CD. The pipeline catches code problems. Monitoring catches runtime problems. They are complementary, not interchangeable.

The Rollback

When a bad deploy makes it through, the rollback is fast. Identify the bad commit and revert it:

git revert <bad-commit-sha> --no-edit
git push origin main

That is it. git revert creates a new commit that undoes the bad change. The push triggers the CD pipeline. The pipeline authenticates with OIDC, sends the SSM command, the server pulls the reverted code, rebuilds the containers, and Docker Compose brings everything back up in the correct dependency order.

What I Learned Building This

1. The server should not be special

Before CI/CD, the server accumulates manual changes. Over time, the server drifts from what the code says it should be. git reset --hard origin/main enforces the truth: the server is a mirror of the main branch. Nothing more. If it is not in git, it does not exist.

2. Security and convenience are not a trade-off

I expected the OIDC setup to be painful. It was about 50 lines of Terraform and one GitHub Secret. And now I never think about key rotation, key expiration, or key leakage. The credentials do not exist until the pipeline needs them, and they stop existing an hour later.

SSM was similar. No port 22 open. No SSH keys to distribute. No bastion host to maintain. And I get a full audit trail in CloudTrail for free.

3. Health check chains are the real deployment logic

My docker-compose.yml has a strict dependency chain:

postgres (healthy) → backend (healthy) → frontend (started) → nginx

Each service waits for the previous one to pass its health check before starting. This means docker compose up -d is not just "start everything." It is an orchestrated, ordered, health-verified rollout.

4. Disk space kills you silently

Every docker compose build creates new image layers. docker image prune -f after every deploy is not optional. It is the difference between a pipeline that works for a month and one that works forever.

The Full Picture

Terraform creates:                     GitHub Actions uses:
├── VPC + Private Subnets              ├── OIDC → Temporary AWS credentials
├── ALB (public-facing)                ├── SSM → Deploy to private EC2
├── EC2 (private, no public IP)        ├── CI  → Build + Lint + Validate
├── NAT Gateway (outbound only)        └── CD  → git pull + compose up
├── IAM Role for EC2 (SSM Agent)
├── OIDC Provider (GitHub trust)
└── IAM Role for GitHub (SSM perms)

Clean separation.

Next Steps

This pipeline deploys on every push to main. That is continuous deployment. For a production application with real users, I would add:

• A staging environment. Deploy to a staging EC2 first. Run integration tests against it. Only promote to production after validation.
• Manual approval gates. A github_environment with required reviewers. The pipeline pauses and waits for a human to click "Approve" before deploying to production.
• Container image registry. Push built images to AWS ECR with commit-hash tags. Instead of building on the server, pull pre-built images. Build once, deploy many times.
• Secrets in Parameter Store. Move .env values into AWS Systems Manager Parameter Store. The deploy script pulls them at runtime instead of relying on manually created files on the server.

Each of these is a future article. I'll link them here when they're ready.

The full infrastructure code (Terraform), CI/CD workflows, and Docker Compose configuration are all available on GitHub. If this helped you, drop a like or leave a comment — I'd love to hear how you're solving deployment to private subnets.