From Broken Ingress to Healthy ALB: An EKS and Splunk Hands-On Lab

You’ll take an EKS Ingress with an empty ADDRESS and turn it into a working internet-facing ALB, then prove it with curl, healthy targets, controller reconciliation logs, and Splunk metrics. This is the kind of failure I see in real clusters: the Kubernetes YAML can look right while IAM, Pod Identity, subnet tags, and load balancer reconciliation decide whether traffic ever reaches a pod.

In this lab, the before-state is one specific case: the alb Ingress exists before the AWS Load Balancer Controller exists. The lab still creates the IAM role, Pod Identity association, and public subnet tags that the controller needs after it is installed.

In my run, the proof ended with HTTP 200, 2/2 healthy ALB targets, 1 successful reconciliation, 0 reconciler errors, and 1,968 Splunk cluster time series. Here’s how it works, and how to build it yourself.

TL;DR

• What you get: EKS 1.32, two t3.large worker nodes, nginx, an alb Ingress, an internet-facing ALB, and Splunk Observability Cloud data.
• The trick: the Ingress only becomes an ALB after the AWS Load Balancer Controller is installed with AWS permissions through EKS Pod Identity.
• The proof: curl returned HTTP 200, ALB target health was 2/2, the controller logged 1 successful reconciliation and 0 errors, and Splunk returned cluster/controller/app series.
• Time and cost: the reproduced-and-fixed run took 28.3 minutes. The lab uses one EKS control plane, two t3.large instances, and one ALB for about an hour, estimated as well under 1 USD.

Why a valid Ingress can still have no ALB

A Kubernetes Ingress is a request. It is not the load balancer by itself.

Terraform creates the VPC, two public subnets tagged kubernetes.io/role/elb=1, the EKS cluster, two EC2 worker nodes, the Pod Identity agent, and the IAM role for the controller. Then reproduce.sh deploys the app and Ingress before any controller is installed.

The Deployment is ready and the Service exists, but nothing is watching ingressClassName: alb yet:

== Ingress status (ADDRESS should be empty: nothing reconciles it)
NAME   CLASS   HOSTS   ADDRESS   PORTS   AGE
web    alb     *                 80      34s
Ingress hostname: ''

== Ingress events
Events:       <none>

== IngressClass objects and controller pods
No resources found
aws-load-balancer-controller deployments: 0

That is the useful gap: no controller, no IngressClass, no ALB, no target group, and no Splunk data for this cluster yet.

The trick: give the alb Ingress a controller with AWS permissions

The AWS Load Balancer Controller watches Kubernetes resources and creates AWS load balancer resources from them. In this lab, Terraform binds the controller IAM role to the service account kube-system/aws-load-balancer-controller with EKS Pod Identity.

The Ingress stays small:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/healthcheck-path: /healthz
spec:
  ingressClassName: alb
...

For your service, change the namespace, service name, port, health check path, and whether the ALB should be internet-facing.

Build it yourself in 4 steps

You need Terraform, aws CLI v2, kubectl, helm 3.16 or newer, jq, and curl, plus a Splunk Observability Cloud ingest token and API token. The access token sends data; the API token is used to query for proof.

Step 1: Clone the lab and set the names once

This keeps Terraform, Kubernetes labels, and Splunk queries on the same lab name.

git clone https://github.com/Abhimanyu9988/Abhimanyu9988.github.io.git
cd Abhimanyu9988.github.io/labs/from-broken-ingress-to-healthy-alb-an-eks-and-splunk-hands

export AWS_REGION=eu-central-1
export LAB_NAME=alb-ingress-lab
export TF_VAR_lab_name=$LAB_NAME
export SPLUNK_REALM=us1
export SPLUNK_ACCESS_TOKEN=...
export SPLUNK_API_TOKEN=...

Use your own Splunk realm and tokens from your Splunk Observability Cloud org.

Step 2: Create the EKS foundation (609 seconds in my run)

The repo includes the Terraform. The provider setup supplies the region and permissions boundary, so do not pass those as Terraform variables here.

terraform init && terraform apply

My apply created 23 resources and printed the cluster name, VPC ID, region, and controller role ARN.

Step 3: Reproduce the empty ADDRESS state (48 seconds in my run)

This deploys nginx and the Ingress before the controller is installed.

./reproduce.sh

You should see an empty Ingress hostname, no IngressClass, 0 controller deployments, and 0 ALBs tagged for the cluster.

Step 4: Install the controller and prove the ALB (148 seconds in my run)

The script installs the AWS Load Balancer Controller chart 1.13.0 for app v2.13.0, installs the Splunk OpenTelemetry Collector chart 0.116.0, waits for the ALB, checks targets, curls the ALB, checks logs, and queries Splunk.

./fix.sh

One lab detail: recent Helm versions reject that Splunk chart schema because one deprecated field is a string where the metaschema wants a boolean, so the script uses --skip-schema-validation when Helm supports it.

The proof: HTTP 200, 2 healthy targets, and Splunk series

The target health and curl output prove AWS is sending traffic to pod IPs:

== 4. ALB target health (target-type ip, one target per pod)
10.20.0.106 80  healthy
10.20.0.53  80  healthy
healthy targets: 2/2

== 5. curl through the load balancer: http://k8s-demo-web-70bb130243-28212986.eu-central-1.elb.amazonaws.com/
HTTP status: 200

The controller logged the reconciliation path:

{"level":"info","ts":"2026-10-22T09:51:01Z","logger":"controllers.ingress","msg":"creating targetGroup","stackID":"demo/web","resourceID":"demo/web-web:80"}
{"level":"info","ts":"2026-10-22T09:51:02Z","logger":"controllers.ingress","msg":"created loadBalancer","stackID":"demo/web","resourceID":"LoadBalancer","arn":"arn:aws:elasticloadbalancing:eu-central-1:<account-id>:loadbalancer/ap
{"level":"info","ts":"2026-10-22T09:51:02Z","logger":"controllers.ingress","msg":"successfully deployed model","ingressGroup":"demo/web"}
{"level":"info","ts":"2026-10-22T09:51:03Z","msg":"registering targets","arn":"arn:aws:elasticloadbalancing:eu-central-1:<account-id>:targetgroup/k8s-demo-web-f26d64a1b9/dc37538e55c037db","targets":[{"Id":"10.20.0.106","Availabili
successful reconciliations: 1, reconciler errors: 0

Splunk had data for the same lab name on the first attempt:

attempt 1: cluster=1968 controller-pods=1621 app-pods=34
-- controller Prometheus metrics (best effort):
controller_runtime_reconcile_total series: 12

Proof: curl 200 via k8s-demo-web-70bb130243-28212986.eu-central-1.elb.amazonaws.com, 2/2 targets healthy, controller reconciled, Splunk has cluster/controller/app series.

Use it for real ingress checks

• Check reconciliation before changing the app. If ADDRESS is empty, confirm the IngressClass, controller deployment, Pod Identity association, subnet tags, and controller logs.
• Keep health checks boring. This lab uses /healthz, a 10-second interval, and a healthy threshold of 2.
• Watch Kubernetes and AWS. Pod readiness tells you the app is ready. ALB target health tells you AWS can reach it. Splunk gives you the cluster, controller, and app view together.

Clean up (the ALB costs money while it runs)

Delete Kubernetes-created load balancer resources before destroying Terraform. The cleanup script waits for the controller-managed ALB, target groups, Helm releases, and security groups to go away.

./cleanup.sh
terraform destroy

My cleanup script finished after 37 seconds, and Terraform destroyed 23 resources.

What I didn't test

• HTTPS listeners, ACM certificates, or redirects from HTTP to HTTPS.
• Private/internal ALBs.
• Production sizing, autoscaling, or multi-AZ pod spread beyond this two-node lab.

Every file in this lab

Everything above runs from these files. Open any of them to copy it, or clone the repo as shown:

• README.md
• cleanup.sh
• fix.sh
• k8s/app.yaml
• k8s/ingress.yaml
• main.tf
• providers.tf
• reproduce.sh
• evidence/: the raw output of my run

Sources

• AWS Load Balancer Controller documentation
• Amazon EKS Pod Identities
• AWS Load Balancer Controller v3.6.1 release — background source; this lab pins chart 1.13.0 and app v2.13.0 so Terraform downloads the matching IAM policy.

Disclosure: I work at Splunk, part of Cisco. This is my personal blog; the post isn't sponsored or reviewed by my employer.

I built this in a disposable AWS lab. The lab code and raw output are in the repo. Drafted with AI assistance and reviewed by me.