An official website of the United States government US flag signifying that this is a United States federal government website

Rotating Secrets V - Cloud Foundry and Diego

Rotating Secrets V - Cloud Foundry and Diego


When rotating secrets for Cloud Foundry and Diego you will need to follow the proper steps in order to perform a rolling update to the platform. There are six deployments that need to be synchronized across two Bosh deployments. These deployments are focused on the Consul components for Cloud Foundry and Diego.

Follow the following documentation to get familiar with which values are going to be replaced.

Prepare a local directory for rotating secret files

Create a directory you can use to generate all the various artifacts that need to be rotated for the secrets files. Create a ./tmp directory inside the cg-deploy-cf repository. Delete this directory at the end of secrets rotation.

Dealing with multiple secret files

The Cloud Foundry secrets are split into two parts, a main.yml and external.yml. Internal secrets can be rotated either all at once or on a rolling schedule of deployments. You can pull the information needed to decrypt these from the cg-deploy-cf pipeline. You can easily grab the information needed to decrypt and encrypt secrets from Concourse using the fly-cli.

# note the target name `fr` here is the output of `fly targets`
$ fly -t fr get-pipeline --pipeline deploy-cf | \
  spruce json | \
  jq -r '.resources | select(.name | test("common-"))'

Internal secrets (main.yml)

  1. Rotate all UAA user passwords with passwords matching the UAA restrictions set in the spec for our uaa-customized-boshrelease. Be sure to match these UAA restrictions as it can cause a stalled deployment when the users cannot be set appropriately.
  2. Rotate all the uaa.clients.* secrets.
    • For uaa.clients.doppler change loggregator.uaa.client_secret as well.
    • For uaa.clients.tcp_emitter change acceptance_tests.oauth_password as well.

Generate all the certificates

Add all the CA certificates signed by the Bosh root CA certificate. These root certificate and key files can be grabbed with the --grab-cert flag for the file in 18F/cg-deploy-cf.

The ci_env environmental variable is the value for the name of the Concourse you would like to target to get the root CA certificate from. (e.g. fly targets). From inside the ./tmp directory, run the following commands:

$ ci_env=fr ../ --grab-cert && \
  mv all-cf-certs staging-cf-diego-certs
$ ci_env=fr ../ --grab-cert && \
  mv all-cert-certs production-cf-diego-certs

This script will generate all of the certificates for both Diego and Cloud Foundry. Since these deployments need to be deployed back-to-back, the certificate generation is consolidated in the cg-deploy-cf repository.

Mind the Security Configuration for Consul schedule

These properties need to be rotated on the same schedule mentioned in the Security Configuration for Consul documentation for Cloud Foundry. The consul VMs need to be updated for both Cloud Foundry and Diego.

  • consul.ca_cert
    • Deploy the new CA certificate after the old one for the first deployment. And remove the old CA certificate for the last deployment.
  • consul.agent_cert
    • Deploy this only after a successful deployment of consul.ca_cert.
  • consul.agent_key
    • Deploy this only after a successful deployment of consul.ca_cert.
  • consul.server_cert
    • Deploy this only after a successful deployment of consul.ca_cert.
  • consul.server_key
    • Deploy this only after a successful deployment of consul.ca_cert.
  • consul.encrypt_keys
    • Deploy the new encrypt key before the old one for the first deployment. And remove the old encrypt key for the last deployment.

Make sure that Diego deploys after each rotation step for the certificate updates from Security Configuration for Consul in order to prevent downtime.

Updating certificates in secrets

Replace all the certificates corresponding to the filename and property matching in the secrets file, e.g. certs_dir/cloud-controller.crt maps to Be sure to include the -----BEGIN CERTIFICATE----- and -----END CERTIFICATE----- and indent the value under the YAML stanza to ensure valid YAML syntax.

Remaining secrets

The remaining secrets in this file besides certificates can all be rotated at once. Do this ahead of the first deployment so that subsequent deployments only need to follow the Security Configuration for Consul.

External secrets (external.yml)

External secrets fall into a spectrum categories which ranges from rotatable with coordination with other deployments on one end and non-rotatable without data loss on the other.

Properties used in other deployments

  1. uaa.clients.*.secrets properties need to be coordinated across various Bosh deployments and apps or service brokers.
  2. login.smtp.* properties for Mandrill.
  3. secret needs to be coordinated across our Service Broker deployments for

New Relic external accounts

  1. collector.newrelic_insights.* properties NewRelic license keys and configuration.
  2. collector.newrelic_plugin.* properties NewRelic license keys and configuration.
  3. uaa.newrelic.* properties NewRelic license keys and configuration.
  4. newrelic.* properties NewRelic license keys and configuration.

Non-rotatable properties

  1. cc.db_encryption_key cannot be rotated once set on a running CF deployment.
    • There is currently no way to rotate this encryption key for the Cloud Controller database.
  2. login.providers.* metadata along with login.saml.* properties for certificates and keys.

Rotating external UAA client secrets

You will need to coordinate these changes with secrets and passphrase changes for various other deployments. You can reference the list of external clients in the cf-properties.yml file in the cg-deploy-cf repository under the properties.uaa.clients.* YAML stanza.

You can search through the cg-deploy-* repositories to find which client secret goes with which deployment. If you have all the repositories cloned locally, you can grep through all of the repositories with the following command or using the GitHub search

open "${client_name}%22&type=Code"
find ${path_to_local_cg_repositories} -d 2 | xargs ack -n -i "(client.+: ${client_name})" -A2 -B2

Recreating the Smoke Tests VMs

If after secrets rotation, the smoke test VMs for staging and production fail to start, you may need to restart the VM. From inside a Concourse jumpbox, run the following command to restart the VM and clear up the old smoke tests VM.

$ bosh-cli restart \
  -d cf-production
  $(bosh-cli vms -d cf-production | grep smoke_tests | cut -d ' ' -f 1)

This should clear up any strange errors that reference

Running errand 'smoke_tests':
  Expected task ${task_number} to succeed but was state is 'error'