Focus: A practical record of the VPC networking work completed in AWS.
This document records what was built, tested, observed, troubleshot, and learned during the hands-on sessions.
📚 VPC Theory & Concepts
The concepts behind these hands-on practicals are covered in the following VPC articles:
VPC Part 1 — Networking Fundamentals: VPC, Subnets, Internet Gateway, NAT Gateway, Security Groups vs NACLs
https://dev.to/tejas_shinkar/aws-networking-fundamentals-vpc-subnets-igw-nat-the-sg-vs-nacl-battle-eelVPC Part 2 — Extended Networking Concepts: Route Tables, Blackhole Routes, VPC Flow Logs, VPC Endpoints, Transit Gateway
https://dev.to/tejas_shinkar/aws-vpc-extended-route-tables-blackhole-routes-flow-logs-vpc-endpoints-transit-gateway-og4
These articles cover the underlying concepts in detail, while this document focuses specifically on the hands-on implementation, testing, troubleshooting, and observations from the VPC practicals.
1.Building a VPC from Scratch with Public and Private Subnets
Objective
Build the core VPC networking architecture manually instead of using the AWS VPC and more wizard. The goal was to understand how VPC, subnets, route tables, an Internet Gateway, a NAT Gateway, and EC2 instances fit together to provide public and private connectivity.
Architecture
VPC: 10.200.0.0/16
Internet
|
IGW
|
+-------------+-------------+
| |
Public Subnet Private Subnet
10.200.0.0/24 10.200.1.0/24
| |
Public EC2 Private EC2
| |
+---- NAT Gateway <---------+
|
Public Subnet
What We Built
- VPC:
lab-vpc - CIDR:
10.200.0.0/16 - Public subnet:
10.200.0.0/24 - Private subnet:
10.200.1.0/24 - Internet Gateway:
lab-igw - Public route table
- Private route table
- NAT Gateway:
lab-nat-gw - Elastic IP for NAT Gateway
- Public EC2
- Private EC2
Implementation
VPC
Created VPC lab-vpc with CIDR 10.200.0.0/16, using VPC only so the networking components could be built manually.
Subnets
Created:
-
public-subnet-1—10.200.0.0/24—ap-south-1a -
private-subnet-1—10.200.1.0/24—ap-south-1b
The subnets were placed in different Availability Zones to introduce a basic multi-AZ layout.
Internet Gateway
Created and attached lab-igw. The IGW alone was not enough to make the subnet public; the public subnet also needed a route to the IGW.
Public Route Table
Configured 0.0.0.0/0 → Internet Gateway and associated it with public-subnet-1.
NAT Gateway
Created lab-nat-gw. The NAT Gateway was placed in the public subnet and given an Elastic IP.
The private route table was then configured with 0.0.0.0/0 → NAT Gateway. This produced the outbound path:
Private EC2 → Private Route Table → NAT Gateway → Internet Gateway → Internet
EC2 Testing
Created public-ec2 and private-ec2.
The public EC2 had a public IP. The private EC2 had no public IP.
From the public EC2, we connected to the private EC2 using its private IP.
From the private EC2, we tested outbound Internet access:
curl https://checkip.amazonaws.com
The returned public IP corresponded to the NAT Gateway's Elastic IP rather than the private EC2's private address.
Evidence / Result
The lab successfully demonstrated:
Public EC2 → Internet
Private EC2 → NAT Gateway → Internet
The private EC2 could reach the Internet without having a public IP.
Problems / Troubleshooting
EC2 Instance Connect
Browser-based EC2 Instance Connect caused connectivity issues during the lab. The SSH Security Group rule was temporarily broadened to 0.0.0.0/0 to avoid making SSH access itself the blocker.
This was a temporary lab workaround, not a production configuration.
NAT Gateway placement
The placement initially raised the question of why the NAT Gateway belongs in a public subnet. The final working architecture made the reason clear:
NAT Gateway → Public Subnet → Public Route Table → IGW → Internet
Hands-On Takeaways
- A subnet uses the route table associated with it.
- The public route table pointed to the IGW.
- The private route table pointed to the NAT Gateway.
- The NAT Gateway provided outbound Internet access for the private EC2.
- A private EC2 does not need a public IP to initiate outbound Internet traffic.
- NAT Gateway placement and routing must be considered together.
2.Blocking HTTP While Allowing HTTPS with a Custom NACL
Objective
Create a custom Network ACL for a public subnet and use an Apache web server to prove that NACL rules can selectively allow or reject traffic.
The practical goal was simple: HTTP should fail, while HTTPS should work, allowing the NACL decision to be observed through an actual application.
Architecture
Internet
|
v
Internet Gateway
|
v
Custom NACL
|
v
Public Subnet
|
v
EC2
|
Apache
What We Built
- Custom VPC for the lab
- Public subnet
- Custom Network ACL
- Security Group
- Public EC2
- Apache HTTP server
- HTTPS configuration using a temporary self-signed certificate
NACL Configuration
The important rules were:
| Rule | Port | Action |
|---|---|---|
| HTTP | 80 | DENY |
| HTTPS | 443 | ALLOW |
| SSH | 22 | ALLOW |
| Return traffic | 1024–65535 (TCP) | ALLOW |
The SSH rule was temporarily broadened during connectivity troubleshooting.
Apache Setup
Installed Apache:
sudo dnf install httpd -y
Created a test page:
echo "<h1>NACL Lab - HTTP/HTTPS Test</h1>" | sudo tee /var/www/html/index.html
Enabled HTTPS using mod_ssl and a temporary self-signed certificate. Apache was verified to be listening on port 443.
HTTP Test
Opened http://<EC2-public-IP>.
Result: ERR_CONNECTION_TIMED_OUT — the request was blocked by the NACL.
HTTPS Test
Opened https://<EC2-public-IP>.
The browser displayed the expected self-signed certificate warning. After proceeding through the warning, the Apache page loaded successfully.
Evidence / Result
HTTP :80 → BLOCKED
HTTPS :443 → WORKED
This gave us a direct application-level demonstration of the NACL rules.
Problems / Troubleshooting
During Apache installation, the package manager initially timed out.
The problem turned out to be the NACL's stateless return traffic. The EC2 could send EC2:ephemeral-port → Internet:443, but the response returned to the EC2's ephemeral port and was not initially allowed.
Adding an inbound rule (1024-65535 → ALLOW) resolved the package installation.
This was an important real failure rather than a theoretical example.
Hands-On Takeaways
- NACL rules apply at the subnet level.
- Security Group and NACL rules both affect whether traffic reaches the instance.
- A restrictive NACL must account for the return path.
- Ephemeral ports become important when working with stateless NACLs.
3.Observing Network Traffic with VPC Flow Logs and CloudWatch
Objective
Enable VPC Flow Logs and use the resulting CloudWatch records to inspect real network traffic.
The goal was to move from simply configuring networking to observing what actually happened on the network.
What We Configured
Flow Logs were configured for all traffic, sent to CloudWatch Logs, with 1-minute aggregation, in log group /vpc/flow-logs/nacl-lab.
AWS created/used the required IAM service role for publishing Flow Logs to CloudWatch.
Traffic Generated
From the EC2:
curl http://checkip.amazonaws.com
curl https://checkip.amazonaws.com
SSH traffic had also been generated by connecting to the instance.
CloudWatch Investigation
CloudWatch → Logs → Log groups →
/vpc/flow-logs/nacl-lab→ Log stream
The records exposed fields such as Source IP, Destination IP, Source Port, Destination Port, Protocol, Action, Bytes, and Packets.
One observed HTTPS-related flow contained the important values:
Source IP: 13.213.253.85
Destination IP: 10.50.1.165
Source Port: 443
Destination Port: 32798
Protocol: 6 (TCP)
Action: ACCEPT
The reverse direction showed the corresponding flow back to the EC2's ephemeral port. We also observed REJECT records.
Evidence / Result
The Flow Logs successfully showed both ACCEPT and REJECT traffic, and allowed us to identify:
- who was communicating
- where traffic was going
- which ports were involved
- which protocol was used
- whether the network flow was accepted or rejected
Hands-On Takeaways
- Flow Logs are useful for troubleshooting network behavior.
- They can expose rejected traffic that is otherwise difficult to understand.
- They provide network-level evidence but do not replace application or authentication logs.
- Flow Log records may take some time to appear because of aggregation.
4.Proving the Stateless NACL Gotcha with a Private EC2
Objective
Build a fresh private-network scenario specifically to demonstrate why a restrictive NACL can allow an outbound HTTPS request but still break the connection when the response returns.
The experiment was deliberately designed to fail first and then be fixed.
Architecture
Internet
|
IGW
|
Public Route Table
|
NAT Gateway
|
Private Route Table
|
Private Subnet
|
Private EC2
Lab CIDRs:
- VPC:
10.60.0.0/16 - Public subnet:
10.60.0.0/24 - Private subnet:
10.60.1.0/24
Routing Configuration
- Public route table:
0.0.0.0/0 → Internet Gateway - Private route table:
0.0.0.0/0 → NAT Gateway
The NAT Gateway was placed in the public subnet. The EC2 was placed in the private subnet with Auto-assign Public IP: Disabled.
Initial NACL
Outbound: 100 → HTTPS 443 → ALLOW, * → ALL → DENY
Inbound: * → ALL → DENY
The deliberate design was:
Allow the HTTPS request out, but do not allow the return traffic back in.
First Test — Failure
From the private EC2:
curl https://checkip.amazonaws.com
The command hung. The request path was:
EC2 → NACL outbound → HTTPS 443 ALLOW → NAT Gateway → Internet
The response path returned to an ephemeral port:
Internet:443 → EC2:ephemeral-port → NACL inbound → DENY
Fix
Added the missing inbound rule: 100 → TCP 1024-65535 → ALLOW. The catch-all deny remained below it.
Second Test — Success
Ran again:
curl https://checkip.amazonaws.com
Result: 13.202.19.75 — the HTTPS request now completed successfully.
Evidence / Result
Before: Outbound 443 → ALLOW. Return traffic → BLOCKED.
curl→ timeout.After: Outbound 443 → ALLOW. Inbound ephemeral ports → ALLOW.
curl→ public IP returned.
Problems / Troubleshooting
The main issue was initially easy to misinterpret as a NAT or Internet problem.
Following the traffic path isolated the failure to the inbound NACL rule. The lab therefore became a useful troubleshooting exercise:
Request works? → check return path → NACL is stateless → check ephemeral ports
Hands-On Takeaways
- A private EC2 can reach the Internet through a NAT Gateway without a public IP.
- The private route table must point to the NAT Gateway.
- The public route table used by the NAT Gateway must point to the IGW.
- Restrictive NACLs must explicitly permit both sides of a connection.
- Ephemeral ports are not an abstract networking detail; they directly affected the lab's outcome.
- Following the packet path was more useful than randomly changing AWS resources.
Practical Reasoning Drills
P1.Longest Prefix Match — Choosing the Most Specific Route
Objective
Use a route table with overlapping routes and determine which route AWS would select for different destination IPs.
Given:
| Destination | Target |
|---|---|
10.0.0.0/16 |
Local |
10.0.5.0/24 |
pcx-abc |
0.0.0.0/0 |
igw-xyz |
Results
-
10.0.5.10→10.0.5.0/24→pcx-abc(reason:/24is more specific than/16and/0) -
10.0.9.10→10.0.0.0/16→ Local (reason: it does not belong to10.0.5.0/24, but it does belong to10.0.0.0/16) -
52.1.1.1→0.0.0.0/0→igw-xyz(reason: it does not match either10.x.x.xroute, so the default route is used)
Takeaway
More specific prefix wins:
/24 > /16 > /0
p2.NAT Gateway Deletion — Diagnosing a Blackhole Route
Objective
Work through a realistic failure where a NAT Gateway is deleted but private EC2 instances still depend on it for outbound Internet access.
Scenario
Private route table: 0.0.0.0/0 → NAT Gateway. NAT Gateway is deleted. The route can become 0.0.0.0/0 → NAT Gateway (BLACKHOLE).
Troubleshooting Path
Follow the traffic path:
EC2 → Security Group → NACL → Private Route Table → NAT Gateway → Public Route Table → IGW → Internet
In this exact scenario, the private route table is the high-value first check.
Takeaway
Don't immediately recreate the NAT Gateway. First identify where the traffic path breaks and check whether the route points to a valid target.
P3.Choosing Gateway vs Interface VPC Endpoints
Objective
Decide which endpoint type should be used when private workloads need AWS service access without relying on a public Internet path.
| Service | Endpoint Type |
|---|---|
| S3 | Gateway Endpoint |
| DynamoDB | Gateway Endpoint |
| Secrets Manager | Interface Endpoint |
Takeaway
Gateway endpoints integrate with route tables. Interface endpoints use private network interfaces/PrivateLink.
P4.Transit Gateway vs VPC Peering — Scaling the Connectivity Model
Objective
Choose an architecture for an environment that has 3 VPCs today, expects 12 additional VPCs, and also needs VPN connectivity to the head office.
Peering Math
Full-mesh peering connections: N × (N - 1) / 2. For 3 VPCs → 3 connections. For 15 VPCs → 105 connections.
Decision
For this growth scenario, Transit Gateway is the more scalable direction. The model becomes:
VPC-A ─┐
VPC-B ─┤
VPC-C ─┼── Transit Gateway ── VPN ── Head Office
VPC-D ─┤
... ┘
Takeaway
The decision was based on connectivity scale and centralized routing, not simply on the number of VPCs that exist today.
VPC Flow Logs — Investigating Suspicious SSH Traffic
Objective
Use Flow Logs as network-level evidence when investigating a suspected unauthorized SSH attempt against an EC2 instance.
Investigation
Enable Flow Logs for the relevant VPC/subnet and send records to CloudWatch. Look for TCP traffic on destination port 22, matching the target EC2's destination IP, from a suspicious source IP, and check whether the action was ACCEPT or REJECT.
The investigation can establish that a source IP attempted network-level SSH traffic.
Important distinction
An ACCEPT Flow Log does not prove that SSH authentication succeeded. If network traffic was accepted, OS-level SSH/authentication logs must be checked to determine whether a login actually occurred.
Problems and Lessons Across the Labs
EC2 Instance Connect vs Network Configuration
We encountered multiple browser-based EC2 Instance Connect issues.
The important lesson was to distinguish "can I manage/connect to the instance?" from "is the network experiment itself working?"
Temporary broad SSH rules were used only to remove management connectivity as a variable during the labs.
NAT Gateway and Private Subnet Design
The NAT Gateway placement initially required clarification. The working architecture established the relationship:
Private EC2 → Private Route Table → NAT Gateway → Public Route Table → IGW → Internet
The NAT Gateway therefore belongs in a subnet with a working path to the IGW.
Route Table Associations
A subnet association means:
The subnet uses this route table for its routing decisions.
This distinction became important when building separate public and private routing paths.
Duplicate Default Route
A private route table already containing 0.0.0.0/0 → IGW could not also contain another 0.0.0.0/0 → NAT Gateway. The existing default route had to be replaced with the correct target.
NACL Return Traffic
The most valuable troubleshooting lesson was that a restrictive NACL can allow the outbound request but still break the connection because the response arrives on an ephemeral port.
This was observed directly through failed commands and then fixed by adding TCP 1024-65535 → ALLOW.
Flow Logs as Evidence
Flow Logs provided actual network records instead of relying only on assumptions about what the network was doing. They were useful for identifying source, destination, ports, protocol, and ACCEPT/REJECT status.
Production-Relevant Takeaways from the Hands-On Work
These are intentionally limited to lessons that came directly from the practical work.
Separate public and private traffic paths
A production design should not expose application/database resources unnecessarily. A common model is:
Internet → ALB / Public tier → Private application tier → Private database tier
Plan NAT for both availability and cost
For multi-AZ workloads, a single NAT Gateway can create a cross-AZ dependency. NAT Gateway hourly and data-processing costs also make unnecessary outbound Internet traffic worth considering.
Use least-privilege Security Groups
The labs temporarily used broad SSH access to troubleshoot EC2 Instance Connect. Production should instead restrict management access to known sources or use an appropriate private management path.
Treat NACLs as an explicit network boundary
If restrictive NACLs are used, both directions of a connection need to be considered because NACLs are stateless.
Use VPC Flow Logs for visibility
Flow Logs can be useful for troubleshooting, rejected-traffic analysis, security investigations, and validating routing/security changes. But they provide network-level evidence, not application-level authentication evidence.
Design from the traffic requirement
A useful sequence is:
What traffic is required? → Where does it originate? → Where does it need to go? → What route provides that path? → What security controls permit it? → How will we observe/troubleshoot it?
Remaining VPC Practicals
The following two practicals remain to be completed:
Connecting VPCs with Peering, Then Scaling the Design with Transit Gateway
Build two VPCs, establish private connectivity using VPC Peering, then introduce a third VPC and move toward a Transit Gateway-based connectivity model.
The focus will be on the route changes and traffic paths required for VPC-to-VPC communication.
Private S3 Access with a Gateway Endpoint and Flow Log Verification
Add an S3 Gateway Endpoint to a VPC and verify the resulting traffic path using VPC Flow Logs.
The goal is to confirm how the endpoint changes the path used for S3 access and to distinguish that traffic from an Internet/NAT-based path.
