Building a chat application may seem simple at first.
A user sends a message, another user receives it, and the conversation continues.
But once your application grows from a few users to millions, the architecture becomes much more interesting.
You need to think about:
- How do users connect in real time?
- How are messages delivered?
- Where are messages stored?
- What happens when a server crashes?
- How do you support millions of concurrent connections?
- How do users receive messages when they're offline?
- How do you monitor the entire system?
This is where all the concepts from system design come together.
In this article, we'll design a scalable chat application from requirements to a production-ready architecture.
Step 1: Understand the Requirements
Before choosing technologies, we need to understand what we're building.
Let's assume we're designing a WhatsApp- or Discord-like chat system.
Functional Requirements
Our application should support:
- One-to-one messaging
- Group chats
- Real-time message delivery
- Message history
- Online/offline status
- Read receipts
- Typing indicators
- Media messages
- Push notifications
Non-Functional Requirements
The system should also provide:
- Low latency
- High availability
- Scalability
- Message durability
- Fault tolerance
- Observability
These requirements will guide our architecture.
Step 2: Estimate the Traffic
Let's assume:
10 Million Daily Active Users
1 Million Concurrent Users
100 Messages per User per Day
That gives us:
10M × 100 = 1 Billion Messages / Day
Average messages per second:
1,000,000,000 / 86,400
≈ 11,500 messages/second
But traffic is not evenly distributed.
Peak traffic could be several times higher.
Let's design for:
50,000+ messages/second
This tells us that a single server or database will eventually become a bottleneck.
Step 3: High-Level Architecture
A scalable chat system might look like this:
┌──────────────┐
│ Users │
└──────┬───────┘
│
WebSocket / HTTPS
│
▼
┌──────────────────┐
│ Load Balancer │
└────────┬─────────┘
│
┌────────────────┼────────────────┐
▼ ▼ ▼
┌───────────┐ ┌───────────┐ ┌───────────┐
│ Chat Node │ │ Chat Node │ │ Chat Node │
└─────┬─────┘ └─────┬─────┘ └─────┬─────┘
│ │ │
└────────────────┼────────────────┘
▼
┌─────────────┐
│ Message Bus │
│ Queue/PubSub│
└──────┬──────┘
│
┌────────────────┼────────────────┐
▼ ▼ ▼
Message Store Presence Notifications
│ Service Service
▼
Chat Database
Each component has a specific responsibility.
This separation makes the system easier to scale independently.
Step 4: Real-Time Communication
Chat applications need a persistent connection between the client and server.
Using traditional HTTP polling would create unnecessary requests.
A better approach is:
Client
│
│ WebSocket Connection
│
▼
Chat Server
Once connected, either side can send data instantly.
User A ─── Message ───→ Chat Server
│
▼
User B receives it
WebSockets are useful for:
- New messages
- Typing indicators
- Read receipts
- Online presence
- Live updates
Step 5: Scaling WebSocket Connections
A single chat server cannot maintain millions of connections.
Instead:
Load Balancer
│
┌─────────────┼─────────────┐
▼ ▼ ▼
Chat Node 1 Chat Node 2 Chat Node 3
Each node manages a subset of active connections.
For example:
Node 1 → 100,000 Connections
Node 2 → 100,000 Connections
Node 3 → 100,000 Connections
As users grow:
More Users
│
▼
Add More Chat Nodes
This provides horizontal scalability.
The load balancer can route new WebSocket connections to healthy nodes.
Step 6: Sending a Message
Let's follow a message from User A to User B.
User A
│
│ "Hello!"
▼
Chat Server
The server first validates:
- Is the user authenticated?
- Is the sender allowed in this conversation?
- Is the message valid?
Then the message enters the processing flow.
Client
│
▼
Chat Gateway
│
▼
Authentication
│
▼
Message Service
A message might look like:
{
"messageId": "msg_123",
"conversationId": "conv_456",
"senderId": "user_A",
"content": "Hello!",
"timestamp": "2026-08-23T12:00:00Z"
}
The messageId is important for deduplication and idempotency.
Step 7: Store the Message First
A message should not be considered successfully sent until the system can reliably persist it.
User A
│
▼
Message Service
│
▼
Message Database
│
▼
Message Stored ✅
After successful storage:
Message Service
│
├──→ Deliver to recipient
│
├──→ Update conversation
│
└──→ Trigger notifications
This improves durability.
If a chat server crashes after the message is stored, the message is not lost.
Step 8: Delivering Messages in Real Time
Now we need to find User B.
What if User B is connected to a different server?
User A
│
▼
Chat Node 1
│
│ Message for User B
▼
?
│
▼
Chat Node 3
│
▼
User B
The system needs a way for chat servers to communicate.
This is where a message broker or Pub/Sub system can help.
Message Broker
/ | \
▼ ▼ ▼
Node 1 Node 2 Node 3
The message can be published to the appropriate channel.
The node holding User B's connection delivers it.
User A
│
▼
Node 1
│
▼
Message Broker
│
▼
Node 3
│
▼
User B
Step 9: Managing User Presence
Presence tells us whether a user is:
- Online
- Offline
- Away
- Last seen recently
A fast in-memory store can manage this information.
user_123 → Chat Node 2
user_456 → Chat Node 3
When a user connects:
user_123 → ONLINE
When the connection closes:
user_123 → OFFLINE
A distributed presence store allows any chat server to find the user's active connection.
Step 10: What If the User Is Offline?
If User B is offline:
User A
│
▼
Message Service
│
├── Message Database ✅
│
└── User B Offline
The message remains stored.
The system can then send a push notification.
Notification Service
│
▼
Push Provider
│
▼
User B 📱
When User B reconnects:
User B
│
▼
Chat Service
│
▼
Fetch Unread Messages
│
▼
Display Conversation
This ensures offline users don't lose messages.
Step 11: Supporting Group Chats
Group chats introduce another challenge.
Imagine a group with:
10,000 Members
If one user sends a message, how should it be delivered?
There are two common approaches.
Fan-Out on Write
When a message is sent:
Message
│
├──→ User 1 Inbox
├──→ User 2 Inbox
├──→ User 3 Inbox
└──→ Thousands More
Advantages
- Fast reads
- Easy message retrieval
Disadvantages
- Expensive writes for large groups
Fan-Out on Read
Store the message once:
Group Message Store
│
▼
Members read when needed
Advantages
- Efficient writes
- Better for very large groups
Disadvantages
- More expensive reads
A real system may use a hybrid approach.
Step 12: Database Design
Chat applications generate massive amounts of data.
A simple message table might look like:
Messages
------------------------------------
message_id
conversation_id
sender_id
content
created_at
status
A useful query is:
SELECT *
FROM messages
WHERE conversation_id = ?
ORDER BY created_at DESC
LIMIT 50;
This suggests that messages should be efficiently organized around:
conversation_id + timestamp
This supports fast retrieval of recent messages.
Step 13: Database Partitioning and Sharding
As the number of messages grows, a single database may become too large.
We can partition messages:
Messages
│
├── Partition 1
├── Partition 2
└── Partition 3
Eventually, we may shard data across multiple database servers.
For example:
hash(conversation_id) % 4
Conversation A → Shard 1
Conversation B → Shard 2
Conversation C → Shard 3
Conversation D → Shard 4
Using conversation_id as a shard key keeps most messages from the same conversation together.
This makes message retrieval easier.
Step 14: Adding Caching
Some data is accessed frequently.
For example:
- User profiles
- Conversation metadata
- Active conversations
- Presence information
Instead of querying the database every time:
Application
│
▼
Cache
│
├── Hit → Return Data ⚡
│
└── Miss → Database
Caching reduces:
- Database load
- Latency
- Infrastructure cost
But the database remains the source of truth.
Step 15: Reliability and Fault Tolerance
Failures are inevitable.
Suppose one chat node crashes:
Chat Node 2 ❌
The load balancer should stop routing new connections to it.
Load Balancer
│
├── Node 1 ✅
├── Node 2 ❌
└── Node 3 ✅
Users connected to the failed node reconnect to healthy servers.
For critical components, we need redundancy.
Load Balancer
│
┌────────────┼────────────┐
▼ ▼ ▼
Node 1 Node 2 Node 3
We also need:
- Health checks
- Timeouts
- Retries
- Circuit breakers
- Database replicas
- Automatic failover
The goal is to minimize the impact of individual failures.
Step 16: Handling Duplicate Messages
Distributed systems can deliver the same message more than once.
For example:
Message Sent
│
Network Timeout
│
Client Doesn't Know if Server Received It
│
Client Retries
Without protection:
Hello!
Hello!
To prevent this, clients can generate a unique messageId.
messageId = msg_123
If the server receives the same message again:
msg_123 already processed
→ Ignore duplicate
This is called idempotency.
It is essential for reliable messaging.
Message Ordering
Users expect:
Hello
How are you?
Not:
How are you?
Hello
Ordering becomes difficult when multiple servers process messages.
A practical approach is to assign a sequence number within each conversation.
Conversation A
Message 1 → Sequence 101
Message 2 → Sequence 102
Message 3 → Sequence 103
Clients can then display messages in the correct order.
Step 17: Security and Authentication
Every connection must be authenticated.
A common flow:
User Login
│
▼
Authentication Service
│
▼
Access Token
│
▼
WebSocket Connection
The server validates the token before allowing the user to connect.
The system should also verify:
- Is the sender part of the conversation?
- Does the user have permission to access this group?
- Is the message within allowed limits?
Security should be enforced on the server—not trusted to the client.
Step 18: Observability
With multiple services and servers, debugging becomes difficult without visibility.
We need:
Logs
Message received
Message stored
Message delivered
Delivery failed
Metrics
Track:
- Messages per second
- Delivery latency
- Active WebSocket connections
- Error rate
- Database latency
- Queue size
Traces
Follow a message across the system:
Client
│
▼
Chat Gateway
│
▼
Message Service
│
▼
Database
│
▼
Message Broker
│
▼
Recipient Chat Node
This helps identify bottlenecks quickly.
Putting Everything Together
Now we can combine everything into one architecture.
USERS
│
WebSocket / HTTPS
│
▼
LOAD BALANCER
│
┌──────────────────┼──────────────────┐
▼ ▼ ▼
CHAT NODE 1 CHAT NODE 2 CHAT NODE 3
│ │ │
└──────────────────┼──────────────────┘
│
┌──────────┴──────────┐
▼ ▼
PRESENCE SERVICE MESSAGE SERVICE
│ │
▼ ▼
IN-MEMORY STORE MESSAGE BROKER
│
┌────────────────┼───────────────┐
▼ ▼ ▼
MESSAGE DB NOTIFICATIONS DELIVERY
│
┌─────┴─────┐
▼ ▼
REPLICA REPLICA
Across the entire architecture:
OBSERVABILITY LAYER
Logs ──────┐
Metrics ───┼──→ Dashboards + Alerts
Traces ────┘
And reliability mechanisms protect the system:
Redundancy
Retries
Timeouts
Circuit Breakers
Failover
Health Checks
Complete Message Flow
Let's follow one message end to end.
1. User A sends a message
User A → WebSocket → Chat Node
2. The server authenticates and validates it
Token Valid? ✅
Conversation Access? ✅
Message Valid? ✅
3. The message is stored
Message Service → Database
4. The message event is published
Database
│
▼
Message Broker
5. Find User B
Presence Store
│
▼
User B → Chat Node 3
6. Deliver in real time
Message Broker
│
▼
Chat Node 3
│
▼
User B 📱
7. If User B is offline
Message Stored ✅
│
▼
Push Notification
8. Track the entire process
Logs + Metrics + Traces
This is the complete journey of a message.
How Does This System Scale?
As usage grows, individual components can scale independently.
More Connections?
→ Add Chat Nodes
More Messages?
→ Add Message Processing Workers
More Database Load?
→ Add Replicas / Partition / Shard
More Cache Load?
→ Expand Cache Cluster
More Notifications?
→ Add Notification Workers
This is one of the biggest advantages of separating responsibilities.
Scale the bottleneck, not the entire system.
Common Trade-Offs
There is no perfect architecture.
Every decision involves trade-offs.
| Decision | Benefit | Trade-Off |
|---|---|---|
| WebSockets | Real-time communication | Connection management complexity |
| Message Broker | Decouples services | Additional infrastructure |
| Caching | Lower latency | Cache consistency |
| Sharding | Horizontal database scale | Operational complexity |
| Replication | Higher availability | Replication lag |
| Retries | Handles temporary failures | Can cause retry storms |
| Fan-out on write | Fast reads | Expensive for large groups |
| Fan-out on read | Efficient writes | Slower reads |
Good system design is about understanding these trade-offs.
Final Takeaways
Designing a scalable chat application brings together nearly every major system design concept.
You start with requirements and traffic estimation.
Then you design for:
- WebSockets for real-time communication
- Load balancing for scalable connections
- Message brokers for cross-server delivery
- Databases for durable message storage
- Caching for low-latency access
- Partitioning and sharding for massive datasets
- Redundancy and failover for reliability
- Retries and idempotency for safe message delivery
- Logs, metrics, and traces for observability
The final architecture is not about using every technology possible.
It's about combining the right components to build a system that is real-time, scalable, reliable, and easy to operate.
The complete system design mindset:
Requirements
↓
Estimate Scale
↓
Choose Architecture
↓
Identify Bottlenecks
↓
Add Caching
↓
Scale Components
↓
Design for Failure
↓
Add Observability
↓
Continuously Improve
*And that's how individual system design concepts come together to build a scalable real-world application. *
