US flag signifying that this is a United States Federal Government website An official website of the United States government

Secret key management

Secret key management

This page is primarily for the team. It's public so that you can learn from it. For help using, see the user docs.

Sharing secret keys

For sharing the following types of sensitive information related to, team members must use GSA Google Hangouts. The team member must share the information only with intended recipient(s) who need to know the sensitive information. Team members can use the Hangouts screen-sharing feature or verbally share the information.

  • Passwords
  • Secret keys
  • Sensitive environment variables
  • Other secret authentication information

Maintenance of system secret keys

To meet NIST security control SC-12 (1), we maintain the availability of all information on the platform in the event a cryptographic access key is lost or compromised.

Authorized federal staff rotate, encrypt, and backup keys yearly. Privileged users can access the keys only with two-factor authentication and a decryption passphrase. In the rare case that both the keys and the decryption passphrase for the backup are lost or compromised, new keys can be rotated in by authorized federal staff, while maintaining availability of information.

AWS credentials

If you need to view/update secrets:

  1. Ask in #cg-platform for an account to read/write from the S3 buckets.
  2. Setup aws-vault

Install aws-vault for AWS credentials

aws-vault secures credentials locally and generates temporary credentials to provide an additional layer of security. To install aws-vault use the brew command:

brew cask install aws-vault
aws-vault add cloud-gov-govcloud

Configure MFA for aws-vault

All operators should have MFA enabled, which can be viewed in the AWS console under Services -> IAM -> Users -> firstname.lastname -> Security Credentials, or from the command line with aws iam list-virtual-mfa-devices. The ARN of this MFA device needs to be added to the local Amazon configuration to enable short lived tokens:

me=$(aws iam get-user | jq -r '.User.UserName')
mfa_serial=$(aws iam list-virtual-mfa-devices | jq --arg me "$me" -r '.VirtualMFADevices[]|select(.User.UserName==$me) | .SerialNumber')
echo '[profile cloud-gov-govcloud]' >> ~/.aws/config
echo 'region = us-gov-west-1' >> ~/.aws/config
echo "mfa_serial = $mfa_serial" >> ~/.aws/config

Executing a command with short lived credentials

You can execute any system command with short lived credentials using the aws-vault exec command:

aws-vault exec cloud-gov-govcloud bash

Running env | grep AWS will show you a new set of credentials which are different from the primary IAM role credentials, as they are short lived and issued at runtime. This means that if a malicious script or program attempts to read ~/.aws/credentials or ~/.aws/config all they will be unable to retrieve the primary credentials.

Next Steps

Once this is complete, operators can provision profiles which use only specific resources, or specific permissions such as read only. This scopes the role of the temporary credentials to further reduce the attack surface.

UAA credentials

All UAA clients and users and associated credentials should be created via the Cloud Foundry secrets or the service account or identity provider services. UAA accounts should not be created manually; we reserve the right to drop permissions for or deprovision hand-propped accounts.

Updating Secrets

System secrets fall into one of three categories.

  1. Shared - centrally managed secrets used across multiple deployments
  2. Deployment - secrets managed and stored per bosh deployment and environment
  3. Concourse - secrets consumed as credentials in Concourse CI pipelines

Updating Deployment and Concourse secrets requires first configuring AWS credentials (above).

Updating Shared Secrets

Centrally managed secrets are generated by a Concourse pipeline.

  1. Trigger the relevent build in secrets-rotation. This creates new secrets and stores them in a common file.
  2. Trigger the relevent deployment(s).

Updating Deployment Secrets

Secrets which are not centrally managed must be updated per deployment and environment. Substitute cf.main.yml in the following steps for the relevant file.

View secrets
  1. Clone the cg-pipeline-tasks repository.

    git clone
  2. If you don’t already have the AWS CLI set up with credentials, see the steps above.

  3. Download the credentials file.

    aws-vault exec cloud-gov-govcloud bash
    mkdir -p tmp
    aws s3 cp s3://cloud-gov-varz/cf.main.yml tmp/cf.main.yml.enc
  4. Get the encryption passphrase.

    1. View the deploy-cf pipeline configuration for any of the common-* environments. The following command outputs all the cg-common resources. You can modify the test for test("common-") to grab specific credentials about each resource.

      fly --target <target> get-pipeline --pipeline deploy-cf | \
      spruce json | \
      jq -r '.resources[] | select(.name | test("common-"))'
    2. Grab the secrets_passphrase under the common-prod resource. The following command grabs just the common-prod resource and parses out the secrets_passphrase found in the pipeline.yml file for cg-deploy-cf. This is useful for feeding this value to other commands without visually exposing the secret in your terminal.

      fly --target <target> get-pipeline --pipeline deploy-cf | \
      spruce json | \
      jq -r '.resources[] | select(.name | test("common-prod")) | .sources.secrets_passphrase'
  5. Decrypt the secrets file.

    PASSPHRASE=$(fly --target <target> get-pipeline --pipeline deploy-cf | spruce json | jq -r '.resources[] | select(.name | test("common-prod")) | .sources.secrets_passphrase')
  6. Don’t leave the secrets lying around (for security reasons, and because they get stale).

    rm -rf tmp
Change secrets
  1. Post in #cg-platform saying you’re updating cf.main.yml.
  2. Follow the decryption steps above, up until the cleanup.
  3. Modify the cf.main.yml with the new values.
  4. Re-encrypt the file.

    PASSPHRASE=$(fly --target <target> get-pipeline --pipeline deploy-cf | spruce json | jq -r '.resources[] | select(.name | test("common-prod")) | .sources.secrets_passphrase')
  5. Copy the new file up to S3.

    aws-vault exec cloud-gov-govcloud bash
    aws s3 cp tmp/cf.main.yml.enc-new s3://cloud-gov-varz/cf.main.yml --sse AES256
  6. Clean up the secrets.

    rm -rf tmp

Updating Concourse Secrets

Updating Concourse secrets combines the shared secrets above with local config to change a Concourse pipeline. Note: The steps below are for updating the deploy-bosh pipeline, the same steps can also be applied to the deploy-concourse pipeline.

  1. Clone the cg-scripts repository.

    git clone
  2. If you don’t already have the AWS CLI set up with credentials, see the steps above.

  3. cd to the local clone of the Concourse pipeline you are updating

  4. Create a concourse-environment file.

    aws-vault exec cloud-gov-govcloud bash
    cd $(mktemp -d)
    CI_ENV=cloud-gov-govcloud SECRETS_BUCKET=cloud-gov-varz CG_PIPELINE=~/cg-pipeline-tasks ~/cg-scripts/ ~/cg-deploy-bosh/cg-deploy-bosh.yml

    Where CI_ENV is the environment configured for local Concourse fly CLI, SECRETS_BUCKET is the encrypted S3 bucket where credentials are stored, and CG_PIPELINE is the path to a local clone of the cg-pipeline-tasks git repository.

    Any number of local yml files (cg-deploy-bosh.yml) can also be merged to create the final concourse-environment.yml.

  5. Update the Concourse pipeline with fly.

    fly -t cloud-gov-govcloud set-pipeline -p deploy-bosh -c ~/cg-deploy-bosh/ci/pipeline.yml -l concourse-environment.yml

    If the repository includes a ci/concourse-defaults.yml file, you’ll need to load variables from there as well:

    fly -t cloud-gov-govcloud set-pipeline -p deploy-bosh -c ~/cg-deploy-bosh/ci/pipeline.yml -l concourse-environment.yml -l ~/cg-deploy-bosh/ci/concourse-defaults.yml`
  6. Don’t leave the secrets lying around (for security reasons, and because they get stale).

    rm -rf concourse-environment.yml