Monday, 6:00 AM. TruffleHog alert: AWS access key detected in public repo commit history. Developer accidentally committed file with credentials a week ago, then deleted it - but git history remembers. Now that key is in GitGuardian public leaks database. AWS cost explorer shows unknown EC2 instances - crypto mining. Account compromised.
GitGuardian research shows that in 2024 over 12 million secrets were detected in public GitHub repositories. API keys, database passwords, private keys, tokens. Each is a potential attack vector. And that’s just public repos - there are more in private ones.
CI/CD pipeline is particularly risky: build servers have access to everything (code, secrets, production). One compromised CI = full infrastructure access. Secrets management in pipeline isn’t “nice to have” - it’s security foundation.
Why are secrets in CI/CD so problematic?
CI/CD needs secrets to work. Deploy to AWS requires credentials. Push to registry requires token. Test on staging database requires password. Secrets must be available to pipeline - question is how to deliver them securely.
Many places for mistakes. Secrets can leak through: code commits, log output, environment variables in UI, docker images, artifact repositories, shared runners. Each pipeline stage is potential leak.
Persistence of leaks. Secret in git history is there “forever” (without history rewrite). Secret in docker image layer is there even after “deletion” in next layer. Secret in logs may be archived for years.
Sprawling permissions. Pipeline often has more permissions than needed (“because it was easier”). Root access to production, admin on cloud account. Compromised CI = full blast radius.
Shared environments. Self-hosted runners shared between projects. One vulnerability in one project = potential access to other projects’ secrets.
Developer convenience vs. security. Hardcoding secrets is “faster” than proper secrets management. Developers under time pressure choose convenience.
What are the most common secrets management mistakes?
Secrets in code repository. Most obvious and common mistake. .env file in repo. Config file with plaintext password. SQL script with credentials.
Secrets in docker images. COPY .env /app/ or ENV DATABASE_PASSWORD=secret in Dockerfile. Image pushed to registry, secrets are extractable.
Secrets in CI logs. echo $SECRET_TOKEN for debugging. Password appears in stack trace. Verbose mode logs everything.
Environment variables in CI UI. Some CI/CD platforms allow setting env vars visible to everyone with project access. “Secret” that isn’t secret.
Hardcoded secrets in tests. “It’s just a test, we’ll use real API key.” Test is committed, key leaked.
Shared secrets across environments. One API key for dev, staging and production. Dev compromises key = production at risk.
No rotation. Secret set 3 years ago, never rotated. If it leaked - it’s still valid.
How to properly pass secrets to CI/CD pipeline?
CI/CD native secrets (GitHub Secrets, GitLab CI Variables, etc.). Secrets stored in CI/CD platform, encrypted at rest, injected as env variables during build. Basic level security, suitable for less critical secrets.
External secrets manager (HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, GCP Secret Manager). Secrets stored outside CI/CD, pipeline retrieves them at runtime. Better security, centralized management, audit trail.
Just-in-time credentials. Instead of static secrets - short-lived, dynamically generated credentials. Vault dynamic secrets: pipeline requests credentials, Vault generates temporary ones, credentials expire after use.
OIDC federation. Pipeline authenticates to cloud provider without static credentials. GitHub Actions OIDC with AWS: GitHub provides token, AWS verifies, grants temporary credentials. Zero static secrets.
Sealed Secrets / SOPS. Encrypted secrets stored in repo, decryptable only by authorized systems. GitOps-friendly approach.
How to configure HashiCorp Vault for CI/CD?
Vault as central secrets store. All secrets in Vault, applications and pipelines retrieve from Vault. Single source of truth.
Authentication methods for CI/CD:
- AppRole: pipeline has role_id (static) and secret_id (dynamic), uses for auth
- JWT/OIDC: CI platform provides JWT, Vault verifies and grants token
- Kubernetes auth: for K8s-based CI, service account auth
- AWS/GCP/Azure auth: pipeline running in cloud authenticates via cloud IAM
Policies and access control. Principle of least privilege: pipeline for project X has access only to project X secrets. Policies define who can read/write what.
Dynamic secrets. Instead of static database password - Vault generates temporary credentials on request. Pipeline requests DB access → Vault creates user/password with 1h TTL → pipeline uses → credentials auto-expire.
Audit logging. Every secrets access is logged. Who, when, what. Forensics capability.
How to prevent accidental secret commits?
Pre-commit hooks. Tools scanning code before commit:
- TruffleHog: entropy-based detection + known patterns
- git-secrets: AWS-focused, customizable patterns
- detect-secrets: Yelp’s tool, baseline capability
- Gitleaks: fast, comprehensive
CI/CD scanning. Backup for pre-commit - scanning in pipeline:
- GitHub Advanced Security (secret scanning)
- GitLab Secret Detection
- Dedicated tools (TruffleHog, Gitleaks) in CI
.gitignore best practices. .env*, *.pem, *.key, credentials.json - always ignore. Template files (.env.example) without actual values.
How to handle secrets in Docker builds?
Problem: secrets in image. Docker layers are cached and inspectable. Even RUN rm /secrets doesn’t remove secrets from previous layer.
BuildKit secrets (—secret). Secret isn’t in image layer, available only during build.
Multi-stage builds for compile-time secrets. Final image has no secrets.
Runtime secrets injection. Don’t bake secrets into image. Inject at runtime via environment variables, Docker secrets, Kubernetes Secrets, or external secrets manager.
How to manage secrets in Kubernetes?
Native Kubernetes Secrets. Base64 encoded (not encrypted!). Stored in etcd. Basic isolation.
Problem: K8s Secrets are not encrypted at rest by default. Anyone with etcd access or sufficient RBAC can read.
Encryption at rest. Enable encryption provider (aesgcm, kms).
External Secrets Operator. Syncs secrets from external stores (Vault, AWS SM, Azure KV) to K8s Secrets.
Sealed Secrets (Bitnami). Encrypt secrets with public key, commit encrypted version to repo, controller decrypts in cluster. GitOps-friendly.
Table: Secrets management maturity model
| Level | Characteristic | Example | Risk Level | Next Step |
|---|---|---|---|---|
| 0 - Chaos | Secrets in code/repo | Hardcoded passwords in committed .env | Critical | Pre-commit hooks, remove from history |
| 1 - Basic | CI native secrets | GitHub Secrets, GitLab CI Variables | High | Centralize in external secrets manager |
| 2 - Centralized | External secrets manager | Vault or cloud-native (AWS SM) | Medium | Implement rotation, dynamic secrets |
| 3 - Dynamic | Short-lived credentials | Vault dynamic DB creds, OIDC federation | Low | Full audit, zero standing privileges |
| 4 - Zero Trust | Just-in-time, fully audited | Dynamic creds + approval workflow + full audit | Minimal | Continuous improvement, red team testing |
Secrets management in CI/CD is security foundation for software delivery. One credential leak can compromise entire infrastructure. Good practices aren’t hard - they require awareness and consistency.
Key takeaways:
- Never commit secrets - pre-commit hooks are must-have
- CI native secrets is baseline - external secrets manager is better
- OIDC federation eliminates static credentials for cloud
- Docker secrets (BuildKit —secret) protects against layer leaks
- Rotation must be planned - dual-secret period, dynamic secrets
- Audit everything - who, what, when accessed secrets
- Test environment isolation - test secrets ≠ production secrets
- Kubernetes Secrets aren’t encrypted by default - use external solutions
Investment in proper secrets management pays off with first avoided incident. Cost of deploying Vault or AWS Secrets Manager is fraction of breach response cost.
ARDURA Consulting provides DevSecOps specialists through body leasing with experience implementing secrets management for enterprise CI/CD pipelines. Our experts help deploy HashiCorp Vault, configure cloud-native secrets managers, and build secure delivery pipelines. Let’s talk about securing your CI/CD.