The Secrets Problem

Secrets Management for Developers

Every application depends on secrets: API keys, database passwords, encryption keys, OAuth tokens, and TLS certificates. Mishandling these secrets is one of the most common causes of security breaches. A single hardcoded credential committed to a public repository can compromise your entire infrastructure within minutes.

Where Secrets Go Wrong

| Mistake | Consequence | |---------|-------------| | Hardcoded in source code | Credentials exposed in version control | | Stored in environment files committed to git | Accidental exposure in public repos | | Shared via chat or email | Unbounded access, no audit trail | | Stored in config files with wide permissions | Accessible to any process on the machine | | Logged during debugging | Credentials visible in log aggregation systems |

The Vault Pattern

A secrets vault is a centralized service that stores, manages, and audits access to secrets. Applications request secrets at runtime rather than reading them from configuration files.

HashiCorp Vault

Start Vault in development mode

vault server -dev

Store a secret

vault kv put secret/database \

host=db.example.com \

port=5432 \

username=app_user \

password=$(openssl rand -base64 32)

Read a secret

vault kv get secret/database

Vault Integration with Application Code

import hvac

client = hvac.Client(url='https://vault.example.com',

token=get_vault_token())

secret = client.secrets.kv.read_secret_version(

path='database'

)

db_password = secret['data']['data']['password']

Use the secret to connect

conn = psycopg2.connect(

host=secret['data']['data']['host'],

password=db_password

)

Cloud-Native Solutions

AWS Secrets Manager

import boto3

from botocore.exceptions import ClientError

def get_secret(secret_name):

client = boto3.client('secretsmanager')

response = client.get_secret_value(SecretId=secret_name)

return json.loads(response['SecretString'])

Automatic rotation

db_creds = get_secret('prod/database/credentials')

GCP Secret Manager

from google.cloud import secretmanager

def access_secret_version(secret_id, version_id="latest"):

client = secretmanager.SecretManagerServiceClient()

name = f"projects/my-project/secrets/{secret_id}/versions/{version_id}"

response = client.access_secret_version(name=name)

return response.payload.data.decode("UTF-8")

Secrets in CI/CD

Never store secrets in repository CI/CD configuration files that are committed to source control. Use each platform's native secrets store:

| Platform | How to Store Secrets | |----------|---------------------| | GitHub Actions | Settings > Secrets and variables > Actions | | GitLab CI | Settings > CI/CD > Variables | | CircleCI | Project Settings > Environment Variables | | Jenkins | Manage Jenkins > Credentials |

GitHub Actions workflow with secrets

name: Deploy

on: [push]

jobs:

deploy:

runs-on: ubuntu-latest

steps:

\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\- uses: actions/checkout@v4

\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\- name: Deploy to production

env:

DB_PASSWORD: ${{ secrets.DB_PASSWORD }}

API_KEY: ${{ secrets.API_KEY }}

run: ./deploy.sh

Detecting Secret Leaks

Pre-Commit Hooks

Use tools like git-secrets or truffleHog as pre-commit hooks:

Install git-secrets

brew install git-secrets

git secrets --install

git secrets --register-aws

Scan for secrets before commit

!/bin/sh

.git/hooks/pre-commit

git secrets --scan

Scanning Repositories

Scan the entire history

trufflehog git https://github.com/example/repo.git

Scan with Gitleaks

gitleaks detect --source . --verbose

Rotation and Expiration

Secrets should have a limited lifetime. Implement automatic rotation:

  • Database passwords : Rotate every 90 days. Use zero-downtime rotation with two-password windows.

  • API keys : Rotate immediately if compromised, regularly every 180 days.

  • TLS certificates : Automate renewal with Let's Encrypt and cert-manager.

  • SSH keys : Rotate on employee departure or annually.

Example: Zero-downtime DB password rotation

Phase 1: Update app to accept both old and new passwords

Phase 2: Rotate master password in database

Phase 3: Update app to use only new password

Phase 4: Revoke old password

The Principle of Least Privilege

Secrets should be scoped to what a particular service or developer needs:

  • Each microservice gets its own set of secrets, not shared credentials.

  • Developers get ephemeral secrets scoped to development environments.

  • Service accounts have the minimum permissions required for their function.

  • Audit every secret access with timestamp and requester identity.

Summary

Treat secrets as the critical infrastructure they are. Use a dedicated secrets vault in production, store them in platform-native secret stores for CI/CD, never hardcode them or commit them to version control, and implement automatic rotation. Combine these practices with regular scanning for leaked secrets and strict access control to minimize the blast radius of any exposure.