In brief: Encryption is only as strong as the identities, keys and operational process that surround it.
The problem
Secrets copied into environment files persist for years, services receive broader access than needed and key rotation is postponed because ciphertext cannot be traced. Multi-tenant requirements then create unclear choices between shared and dedicated keys.
Why the simple answer is not enough
A KMS protects master keys but applications still handle plaintext data keys and decrypted data. Process memory, logs and crash dumps remain part of the threat model.
Dedicated tenant keys reduce blast radius and support revocation, but increase cost and recovery complexity. Offer isolation tiers from explicit risk requirements.
System flow
short-lived authentication→Secret manager
authorised delivery→KMS or HSM
key operation and audit→Ciphertext
versioned key reference
Architecture decisions
Prefer envelope encryption
Encrypt data with generated data keys and wrap those keys through KMS, reducing master-key operations and enabling versioned rotation.
Separate key policy from storage
Ciphertext carries a key reference, not raw key material. Access policy decides which workload may unwrap it.
Design rotation before launch
Readers accept current and previous versions; writers use current only; background re-encryption is resumable and measured.
Going deeper
Deletion and crypto-shredding need evidence
Destroying a dedicated key can make data unrecoverable, but backups, replicas and shared keys change the guarantee. Document exactly what deletion means.
Emergency access is narrowly scoped
Break-glass decrypt needs approval, reason, expiry, alerting and immutable audit; routine support must not depend on it.
Implementation path
- Inventory secret and key use with owners, consumers and expiry.
- Use workload identity to fetch short-lived secrets instead of distributing static credentials.
- Apply envelope encryption and store key reference plus algorithm metadata with ciphertext.
Technical example: Version-aware decrypt and rotate
record = { ciphertext, wrappedDataKey, masterKeyVersion }
dataKey = kms.decrypt(record.wrappedDataKey, context = tenantId)
plaintext = decrypt(dataKey, record.ciphertext)
if record.masterKeyVersion != current:
enqueueRewrap(record.id)Bind tenant and data class as authenticated encryption context so ciphertext cannot be moved silently across boundaries.
Failure modes to design for
- A rotation disables the old key before all readers support the new version.
- Backup restoration lacks historical key versions.
- An operator exports plaintext during debugging.
What to observe
| Signal | Why it matters |
|---|---|
| Secret and key age | Finds overdue rotation and abandoned credentials. |
| Decrypt and access-denied rate | Detects incidents and policy regressions. |
| Records by key version | Shows rotation progress and remaining exposure. |
How to validate the design
- Rotate keys while mixed application versions read traffic.
- Restore an old backup using archived key versions in an isolated environment.
- Attempt cross-tenant decrypt with moved ciphertext and wrong context.
Safe rollout plan
Inventory and remove secrets from source first. Introduce workload identity for one service, then envelope encryption for one high-value data class. Rehearse rotation and recovery before expanding to dedicated tenant-key tiers.
Production checklist
- No secret appears in source, image layers, logs or CI artifacts.
- Rotation supports overlapping versions and rollback.
- Key access is least privilege, auditable and alertable by tenant or workload.
Takeaway
Key management is a lifecycle of identity, access, rotation and recovery, not a one-time encryption call.