๐ Secure Storage and Secrets Anti-Patterns
Intro: Teams usually do not lose secrets because AES is weak. They lose secrets because tokens land in the wrong repository, long-lived credentials are copied into CI variables, backup archives are not encrypted, or too many services can read the same plaintext. This page focuses on the design and review mistakes that repeatedly turn storage and secret management into an incident.
What this page includes
- a practical map of storage and secret-management anti-patterns
- review questions for files, databases, object storage, queues, and CI/CD systems
- secure replacement patterns for common mistakes
- short examples across application, platform, and cloud layers
Start with the data classes
Treat these as different design problems:
| Data class | Typical examples | Better control model |
|---|---|---|
| Secrets | passwords, API keys, refresh tokens, signing keys, DB credentials | secret manager, short-lived issuance, access logs, least privilege |
| Sensitive application data | PII, financial records, regulated fields | application-layer controls, encryption, masking, tokenization |
| Operational evidence | audit logs, session recordings, incident artifacts | immutable or append-only retention, separate archive ownership |
| Bulk recovery data | backups, snapshots, exports | encryption, integrity verification, restore testing, WORM where needed |
Top anti-patterns
1) secrets in source control or images
Examples
.envcommitted to Git;- credentials baked into container images;
- secrets copied into Helm values, Terraform state, or sample configs.
Why it fails
- source and image distribution widen the blast radius permanently;
- rotation becomes chaotic because the secret is now in history and caches.
Better pattern
- inject secrets at runtime from a secret manager or workload identity path;
- keep example files secret-free;
- treat image layers as public to anyone who can pull or inspect them.
2) long-lived credentials where workload identity is available
Examples
- static cloud access keys in CI;
- static database password for every microservice replica;
- shared Redis password across environments.
Why it fails
- long credential lifetime multiplies replay and lateral movement risk;
- it becomes hard to prove who used the secret.
Better pattern
- prefer OIDC, workload identity, SPIFFE/SPIRE, IAM roles, or managed identity;
- issue short-lived creds where direct federation is not possible.
3) one secret store, too many readers
Examples
- broad
GetSecretValuestyle permission for many apps; - โops can read all production secretsโ as default;
- CI runner can read signing keys, DB creds, and cloud admin secrets in one place.
Better pattern
- separate by environment, domain, and privilege level;
- split read, admin, rotation, and audit ownership.
4) encryption at rest used as a substitute for access control
Platform encryption helps, but it does not solve โtoo many principals can still read plaintextโ.
Better pattern
- combine encryption at rest with identity, network, and object-level access control;
- use application-layer encryption for especially sensitive fields or shared platforms.
5) backups protected worse than production
Examples
- prod database encrypted, but dumps land in an open bucket;
- backup keys stored beside the backup archive;
- restore procedures are undocumented and never tested.
Better pattern
- encrypt backups;
- separate archive account / subscription;
- restrict restore rights;
- test restore and decryption regularly.
6) plaintext secrets in logs, traces, crash dumps, and support bundles
Examples
- Authorization header logged;
- connection strings printed on startup;
- stack traces containing decrypted payload fragments.
Better pattern
- redaction-by-default;
- denylist and allowlist rules;
- secure support-bundle generation.
7) environment-variable sprawl without lifecycle governance
Env vars are practical, but not automatically safe.
Risks
- visible in process listings or diagnostics;
- leaked into crash dumps and debugging tools;
- overused instead of short-lived token exchange.
Better pattern
- use env vars as a delivery channel only when lifecycle and exposure are understood;
- prefer file mounts, sidecar injection, or native identity-based fetch where more appropriate.
8) secret rotation treated as a rare project
Symptoms
- โwe rotate once a year if something happensโ;
- application cannot restart cleanly when creds change;
- dual-key windows and phased rollout are not supported.
Better pattern
- make services tolerant of overlap windows and staged rotation;
- track owner, rotation cadence, and rollback path.
9) secrets used for authorization where identity should be used
Examples
- internal API trusts possession of a shared token only;
- microservice-to-microservice access controlled by a copied static header.
Better pattern
- use service identity, mTLS, workload identity, or signed workload tokens;
- keep shared secrets for compatibility edges, not internal default trust.
10) copying secrets into CI/CD just to make tools work
Examples
- private key injected into runner for signing when KMS/HSM signing could be used;
- static registry password rather than short-lived registry token;
- deployment token with broad cloud permissions across many repos.
Better pattern
- use KMS-backed signing, short-lived OIDC federation, environment-scoped approvals, and distinct trust lanes.
Review checklist
| Review question | Good sign | Bad sign |
|---|---|---|
| Where does the plaintext first appear? | only at runtime and close to use | committed, logged, or embedded during build |
| How many principals can read it? | narrowly scoped by service and env | broad ops or CI wildcard access |
| How is it rotated? | documented owner, cadence, overlap window | manual ticket and ad hoc edits |
| Can it be revoked quickly? | yes, with service restart or reissue plan | no, requires code change or image rebuild |
| Is there access logging? | secret reads and admin changes are logged | only success/fail for the workload itself |
Secure replacement patterns
Pattern A โ runtime fetch + workload identity
- service authenticates with workload identity;
- receives short-lived token or directly reads required secret;
- no static cloud keys in repo, image, or CI variables.
Pattern B โ envelope encryption for bulk data
- KMS/HSM-protected key encrypts data keys;
- data keys encrypt objects, backups, or fields;
- key usage is separated from data storage ownership.
Pattern C โ sign instead of encrypt when integrity is the goal
Use signing when the system needs to know who produced a payload or artifact, not to hide the content.
Pattern D โ immutable audit for sensitive secret reads
For especially sensitive keys, retain:
- who requested access;
- what role/path was used;
- whether a human approval or break-glass flow was involved.
Example scenarios
Scenario 1 โ static CI variable becomes a cloud incident
A repository used a long-lived production cloud key in CI to deploy a single service. An unrelated workflow compromise exposed the key, and the attacker used it outside CI hours. The fix was not โmask the variable betterโ; the fix was to replace the key with OIDC federation and environment approval.
Scenario 2 โ backup encryption existed, but the keys were co-located
A team encrypted database backups before upload, but the decryption material lived in the same automation repository and runner context. The design satisfied a checklist but not actual separation of duties.