Encryption keys are the crown jewels of your infrastructure, yet many teams still manage them with spreadsheets, environment variables, or worse, hardcoded secrets. A proper Key Management Service (KMS) is the difference between sleeping soundly at night and waking up to a security breach. Today, we're diving into how to architect a system that generates, stores, rotates, and manages encryption keys reliably across multiple services.
Architecture Overview
A production-grade KMS needs to balance several competing concerns: security, availability, performance, and auditability. At its core, the system consists of a few critical layers. The API gateway handles all key requests and enforces authentication and authorization policies. Behind that sits the key store, which is typically a hardware security module (HSM) or a highly secure encrypted vault, ensuring keys never exist in plaintext in memory longer than necessary. A metadata service tracks key versions, rotation policies, and usage metadata without exposing the actual key material.
The architecture also includes a cache layer (with strict TTLs), a key generation service that uses cryptographically secure randomness, and a rotation scheduler that automatically cycles keys based on configurable policies. Each service that needs encryption capabilities doesn't store keys directly. Instead, it calls the KMS to encrypt or decrypt data, or it requests a data encryption key (DEK) to use locally while the master key stays protected in the KMS. This separation of concerns is crucial: if one service is compromised, the attacker gains access to DEKs but not to the master keys that protect everything else.
All requests flow through audit logging before they hit the data store, creating an immutable record of who accessed what keys and when. This isn't just compliance theater, it's your forensic evidence when things go wrong. The system also supports key derivation for different environments and multi-region replication for disaster recovery.
Design Insight: Handling Key Rotation
Here's where things get tricky: when you rotate a key, your old data doesn't magically re-encrypt itself with the new key. The KMS solves this elegantly through versioning. Each key has multiple versions, and the metadata service tags every encrypted piece of data with the key version ID used to encrypt it. When a client requests decryption, it sends the ciphertext and the KMS automatically uses the correct key version to decrypt it, even if that key was rotated years ago.
For proactive re-encryption, the KMS maintains a rotation policy that can trigger background jobs to find all data encrypted with old key versions and re-encrypt them with the current version. This happens asynchronously and doesn't block the service. The system also supports graceful key retirement, where old versions become read-only (decrypt-only) for a period before being archived. This gives you a safe window to migrate away from deprecated keys while maintaining backward compatibility. It's a simple concept with massive implications for operational safety.
Watch the Full Design Process
Want to see how this entire architecture came together in real-time? I generated this system design diagram using AI, exploring how a KMS handles the complete lifecycle of encryption keys. Watch the full design process on your preferred platform:
Try It Yourself
This is Day 163 of our 365-day system design challenge, and each day brings new architectural problems to solve. The beauty of tools like InfraSketch is that you don't need to be a systems engineer to explore these concepts. Head over to InfraSketch and describe your system in plain English. In seconds, you'll have a professional architecture diagram, complete with a design document. Try asking it about your own key management challenges, disaster recovery scenarios, or compliance requirements. You might be surprised at how quickly a solid architecture takes shape.
