An official website of the United States government US flag signifying that this is a United States Federal Government website

Rotating Secrets VI - Kubernetes

Rotating Secrets VI - Kubernetes

This page is primarily for the cloud.gov team. It's public so that you can learn from it. For help using cloud.gov, see the user docs.

Introduction

Cloud.gov uses kubernetes to provided managed services, including elasticsearch and redis. The kubernetes bosh deployment and service broker are deployed from https://github.com/18F/cg-deploy-kubernetes.

Rotating certificates

Generate new certificates for consul and kubernetes using the master bosh certificate authority. Certificates for both consul and kubernetes should be rotated in three steps, as described in https://docs.cloudfoundry.org/deploying/common/consul-security.html#rotating-certs:

  • Append new CA certificate
  • Replace old certificates and keys
  • Drop old CA certificate

Generate consul certificates using https://github.com/18F/cg-deploy-kubernetes/blob/master/generate-consul-certs.sh. Generate kubernetes certificates using the master bosh CA certificate and the IP SANs in https://github.com/18F/kubernetes-release/blob/master/generate-certificates.sh. Note: we should automate this step next time we rotate secrets.

Recreating kubernetes secrets and daemonset-managed pods

Rotating kubernetes certificates breaks the automatically generated secrets. To fix, destroy the default secret in the default and kube-system namespaces, then destroy the daemonset-managed pods that use those secrets. Note: both the secrets and the pods will be recreated automatically by kubernetes.

kubectl delete secret $(kubectl get secrets | grep "default-token" | awk '{print $1}')
for pod in $(kubectl get pod | grep "kube2iam" | awk '{print $1}'); do kubectl delete pod ${pod}; done

kubectl --namespace kube-system delete secret $(kubectl --namespace kube-system get secrets | grep "default-token" | awk '{print $1}')
for pod in $(kubectl --namespace kube-system get pod | grep "fluentd-cloudwatch" | awk '{print $1}'); do kubectl --namespace kube-system delete pod ${pod}; done