If you’re deploying apps into a Kubernetes cluster, chances are you’re already drowning in YAML wonderland and wondering why the pods aren’t just reading the new secret like you told them to. This article will teach you how to successfully restart a pod in several different ways.

It’s important to remember that secrets are the way to hand over sensitive info like database passwords, API keys, or “definitely-not-12345” credentials to your containers. While Kubernetes has built in secret support, it’s not so great at telling your pods “Hey, you, your password changed.” 

Regular Kubernetes Secrets

Creating a Basic Secret

The most straightforward way to manage secrets is to define them directly in your cluster. These are base64-encoded which are kind of like writing your password in invisible ink. While it’s hidden from casual glances, anyone who knows how to read base64 can still see you’ve been lazy.

apiVersion: v1
kind: Secret
metadata:
  name: app-credentials
  namespace: default
type: Opaque
data:
  username: YWRtaW4=        # base64 encoded "admin"
  password: cGFzc3dvcmQx    # base64 encoded "password1"

Using the Secret in a Deployment

Once you’ve created your secret, you can inject it into your deployments as environment variables. However, once the pod starts, it reads the secrets… and then never asks again. You can change the secret all you want, but unless you give the pod a firm reboot, it’ll keep using the old values like nothing happened.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web-app
  template:
    metadata:
      labels:
        app: web-app
    spec:
      containers:
      - name: app
        image: nginx:1.21
        env:
        - name: DB_USERNAME
          valueFrom:
            secretKeyRef:
              name: app-credentials
              key: username
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: app-credentials
              key: password

External Secrets

Managing secrets inside Kubernetes works until your setup gets more complex. Most production environments store secrets in external systems like AWS Secrets Manager or HashiCorp Vault. These tools handle rotation, versioning, and access control better than Kubernetes alone.

Installing External Secrets Operator

The External Secrets Operator, or ESO, connects Kubernetes to those systems. It pulls secret values and creates regular Kubernetes Secrets automatically. Your app still reads the secrets as if nothing changed. Once set up, ESO keeps everything in sync. If the secret changes in AWS, Kubernetes gets the update without you needing to do anything. It's simple, reliable, and lets you focus on building, not babysitting credentials.

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets -n external-secrets-system --create-namespace

ExternalSecret Example (AWS Secrets Manager)

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: vault-secret
  namespace: default
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: app-credentials-external
    creationPolicy: Owner
  data:
  - secretKey: username
    remoteRef:
      key: prod/myapp/credentials
      property: username
  - secretKey: password
    remoteRef:
      key: prod/myapp/credentials
      property: password

SecretStore Configuration

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: aws-secrets-manager
  namespace: default
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        secretRef:
          accessKeyID:
            name: aws-credentials
            key: access-key-id
          secretAccessKey:
            name: aws-credentials
            key: secret-access-key

Using ExternalSecret in Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: external-secret-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: external-secret-app
  template:
    metadata:
      labels:
        app: external-secret-app
    spec:
      containers:
      - name: app
        image: nginx:1.21
        env:
        - name: DB_USERNAME
          valueFrom:
            secretKeyRef:
              name: app-credentials-external  # Created by ExternalSecret
              key: username
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: app-credentials-external
              key: password

Restarting Pods When Secrets Change

Method 1: Manual Restart

This is the simplest way. Just restart the deployment and your pods will reload the new secrets. It works fine for occasional changes but can be a pain if you rotate secrets often.

# Restart deployment when secret is updated
kubectl rollout restart deployment/web-app
kubectl rollout restart deployment/external-secret-app

Method 2: Using Stakater Reloader

If you want Kubernetes to do the heavy lifting, you can use Stakater Reloader. It watches for changes to secrets and automatically restarts pods that use them. Just install it once and add an annotation to your deployments. After that, it handles restarts for you.

Install Reloader:

kubectl apply -f 
https://raw.githubusercontent.com/stakater/Reloader/master/deployments/kubernetes/reloader.yaml

Add annotation to the deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: auto-restart-app
  annotations:
    reloader.stakater.com/auto: "true"
spec:
  replicas: 2
  selector:
    matchLabels:
      app: auto-restart-app
  template:
    metadata:
      labels:
        app: auto-restart-app
    spec:
      containers:
      - name: app
        image: nginx:1.21
        env:
        - name: SECRET_VAL
          valueFrom:
            secretKeyRef:
              name: app-credentials
              key: password

Method 3: Annotation-Based Restart Trigger

Another trick is to update an annotation on your pod template. Kubernetes sees that as a change and rolls out new pods. It’s manual but cleaner than restarting the whole deployment. You can even script it if you want a DIY automation setup.

Update deployment with changing annotation:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: manual-trigger-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: manual-trigger-app
  template:
    metadata:
      labels:
        app: manual-trigger-app
      annotations:
        # Change this value to trigger restart
        restart-trigger: "v1"
    spec:
      containers:
      - name: app
        image: nginx:1.21
        env:
        - name: SECRET_VAL
          valueFrom:
            secretKeyRef:
              name: app-credentials
              key: password

Conclusion

Secrets are just one of those things that you don’t think about until everything breaks. You could be hardcoding bas64 in YAML or pulling secrets with ExternalSecrets. At the end of the day, the goal is to make sure you keep sensitive stuff safe and make sure your apps can use it.

If you are looking for a simple and reliable solution, you can just use Stakater Reloader. It’s easy to install, works out of the box, and can keep your apps in sync without needing any manual steps. One annotation and your pods are successfully restarting when a secret changes.