Kubernetes Secrets and Pod Restarts: How to Reload Secrets Safely

Whether a Secret update needs a Pod restart depends on how the application reads it. Environment variables are set when the container starts. A mounted Secret volume can receive updated files, but the application must read them again to use the new values.

The Kubernetes Secret documentation describes eventual updates for mounted volumes and an exception: a Secret mounted with subPath does not receive automatic updates. The examples below use environment variables, so they require replacement containers to load changed credentials.

Regular Kubernetes Secrets

Creating a Basic Secret

The most straightforward way to manage secrets is to define them directly in your cluster. These are base64-encoded which are kind of like writing your password in invisible ink. While it’s hidden from casual glances, anyone who knows how to read base64 can still see you’ve been lazy.

apiVersion: v1
kind: Secret
metadata:
  name: app-credentials
  namespace: default
type: Opaque
data:
  username: YWRtaW4=        # base64 encoded "admin"
  password: cGFzc3dvcmQx    # base64 encoded "password1"

Using the Secret in a Deployment

Once you’ve created your secret, you can inject it into your deployments as environment variables. However, once the pod starts, it reads the secrets… and then never asks again. You can change the secret all you want, but unless you give the pod a firm reboot, it’ll keep using the old values like nothing happened.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web-app
  template:
    metadata:
      labels:
        app: web-app
    spec:
      containers:
      - name: app
        image: nginx:1.21
        env:
        - name: DB_USERNAME
          valueFrom:
            secretKeyRef:
              name: app-credentials
              key: username
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: app-credentials
              key: password

External Secrets

Managing secrets inside Kubernetes works until your setup gets more complex. Most production environments store secrets in external systems like AWS Secrets Manager or HashiCorp Vault. These tools handle rotation, versioning, and access control better than Kubernetes alone.

Installing External Secrets Operator

The External Secrets Operator, or ESO, connects Kubernetes to those systems. It pulls secret values and creates regular Kubernetes Secrets automatically. Your app still reads the secrets as if nothing changed. Once set up, ESO keeps everything in sync. If the secret changes in AWS, Kubernetes gets the update without you needing to do anything. It’s simple, reliable, and lets you focus on building, not babysitting credentials.

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets -n external-secrets-system --create-namespace

ExternalSecret Example (AWS Secrets Manager)

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: vault-secret
  namespace: default
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: app-credentials-external
    creationPolicy: Owner
  data:
  - secretKey: username
    remoteRef:
      key: prod/myapp/credentials
      property: username
  - secretKey: password
    remoteRef:
      key: prod/myapp/credentials
      property: password

SecretStore Configuration

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: aws-secrets-manager
  namespace: default
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        secretRef:
          accessKeyID:
            name: aws-credentials
            key: access-key-id
          secretAccessKey:
            name: aws-credentials
            key: secret-access-key

Using ExternalSecret in Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: external-secret-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: external-secret-app
  template:
    metadata:
      labels:
        app: external-secret-app
    spec:
      containers:
      - name: app
        image: nginx:1.21
        env:
        - name: DB_USERNAME
          valueFrom:
            secretKeyRef:
              name: app-credentials-external  # Created by ExternalSecret
              key: username
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: app-credentials-external
              key: password

Restarting Pods When Secrets Change

Method 1: Manual Restart

This is the simplest way. Just restart the deployment and your pods will reload the new secrets. It works fine for occasional changes but can be a pain if you rotate secrets often.

# Restart deployment when secret is updated
kubectl rollout restart deployment/web-app
kubectl rollout restart deployment/external-secret-app

Method 2: Using Stakater Reloader

If you want Kubernetes to do the heavy lifting, you can use Stakater Reloader. It watches for changes to secrets and automatically restarts pods that use them. Just install it once and add an annotation to your deployments. After that, it handles restarts for you.

Install Reloader:

kubectl apply -f https://raw.githubusercontent.com/stakater/Reloader/master/deployments/kubernetes/reloader.yaml

Add annotation to the deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: auto-restart-app
  annotations:
    reloader.stakater.com/auto: "true"
spec:
  replicas: 2
  selector:
    matchLabels:
      app: auto-restart-app
  template:
    metadata:
      labels:
        app: auto-restart-app
    spec:
      containers:
      - name: app
        image: nginx:1.21
        env:
        - name: SECRET_VAL
          valueFrom:
            secretKeyRef:
              name: app-credentials
              key: password

Method 3: Annotation-Based Restart Trigger

Changing an annotation on the Pod template also triggers a Deployment rollout. Like kubectl rollout restart, this replaces Pods through the Deployment controller; changing an annotation only on the Deployment itself does not change the Pod template.

Update deployment with changing annotation:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: manual-trigger-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: manual-trigger-app
  template:
    metadata:
      labels:
        app: manual-trigger-app
      annotations:
        # Change this value to trigger restart
        restart-trigger: "v1"
    spec:
      containers:
      - name: app
        image: nginx:1.21
        env:
        - name: SECRET_VAL
          valueFrom:
            secretKeyRef:
              name: app-credentials
              key: password

For a healthcare integration, include credential rotation in the application acceptance test. The insurance eligibility workflow describes recoverable jobs and unresolved states that should survive a restart. If a separate calling service is involved, the payer-call evidence and handoff guide helps define what must reach a reviewer after a failed request. Test those outcomes as well as whether the new Pod can read the secret.

Verify the Rotation

After updating a Secret, check the Deployment rollout and confirm that the application can authenticate with the new credential. For an external secret, first confirm that synchronization has updated the Kubernetes Secret; a restart before that point can load the old value again. Verify that in-flight work can recover and that the old credential is retired according to your rotation procedure.

Choose a manual rollout for occasional changes or evaluate Reloader for automated rollouts. In either case, test the application’s readiness and recovery behavior rather than treating a successful restart as proof that rotation completed.