Secret key management
Sharing secret keys
For sharing the following types of sensitive information related to cloud.gov, cloud.gov team members must use GSA Google Meet. The team member must share the information only with intended recipient(s) who need to know the sensitive information. Team members can use the Meet screen-sharing feature or verbally share the information.
- Passwords
- Secret keys
- Sensitive environment variables
- Other secret authentication information
AWS credentials
About AWS credential management
Multi-factor authentication is not only important to interactive console logins, but also for command-line interaction. Checking for console MFA is a standard compliance check, e.g. a Chef Inspec check that all AWS users have MFA enabled, but at cloud.gov we go beyond that to ensure we have MFA enabled for all API calls. We do so with the following IAM policy snippet:
{
"Sid": "DenyAllExceptListedIfNoMFA",
"Effect": "Deny",
"NotAction": [
"iam:CreateVirtualMFADevice",
"iam:EnableMFADevice",
"iam:GetUser",
"iam:ListMFADevices",
"iam:ListVirtualMFADevices",
"iam:ResyncMFADevice",
"sts:GetSessionToken"
],
"Resource": "*",
"Condition": {
"BoolIfExists": {"aws:MultiFactorAuthPresent": "false"}
}
}
If operators attempt to use their keys without MFA, an API error is thrown such as:
An error occurred (AccessDenied) when calling the ListUsers operation: User: ... is not authorized to perform: iam:ListUsers on resource: with an explicit deny
To work with AWS credentials on a day-to-day basis, we rely on aws-vault
.
Details on using aws-vault
on our team are in our internal documentation.
Maintenance of system secret keys
To meet NIST security control SC-12 (1), we maintain the availability of all information on the platform in the event a cryptographic access key is lost or compromised.
Authorized federal staff rotate, encrypt, and backup keys yearly. Privileged users can access the keys only with two-factor authentication and a decryption passphrase. In the rare case that both the keys and the decryption passphrase for the backup are lost or compromised, new keys can be rotated in by authorized federal staff, while maintaining availability of information.
Managing legacy cloud.gov credentials
Most credentials are moving to CredHub, but some are still managed in S3.
To work with them you’ll need your AWS account set up, along with aws-vault
(see above).
UAA credentials
All UAA clients and users and associated credentials should be created via the Cloud Foundry secrets or the service account or identity provider services. UAA accounts should not be created manually; we reserve the right to drop permissions for or deprovision hand-propped accounts.
Updating Secrets
System secrets fall into one of three categories.
- Shared - centrally managed secrets used across multiple deployments
- Deployment - secrets managed and stored per bosh deployment and environment
- Concourse - secrets consumed as credentials in Concourse CI pipelines
Updating Deployment and Concourse secrets requires first configuring AWS credentials (above).
Updating Shared Secrets
Centrally managed secrets are generated by a Concourse pipeline.
- Trigger the relevent build in secrets-rotation. This creates new secrets and stores them in a common file.
- Trigger the relevent deployment(s).
Updating Deployment Secrets
Secrets which are not centrally managed must be updated per deployment and environment. Substitute cf.main.yml
in the following steps for the relevant file.
View secrets
-
Clone the cg-pipeline-tasks repository.
git clone https://github.com/18F/cg-pipeline-tasks.git
- If you don’t already have the AWS CLI set up with credentials, see the steps above.
-
Download the credentials file.
aws-vault exec cloud-gov-govcloud bash mkdir -p tmp aws s3 cp s3://cloud-gov-varz/cf.main.yml tmp/cf.main.yml.enc
- Get the encryption passphrase.
-
View the
deploy-cf
pipeline configuration for any of thecommon-*
environments. The following command outputs all thecg-common
resources. You can modify the test fortest("common-")
to grab specific credentials about each resource.fly --target <target> get-pipeline --pipeline deploy-cf | \ spruce json | \ jq -r '.resources[] | select(.name | test("common-"))'
-
Grab the
secrets_passphrase
under the common-prod resource. The following command grabs just the common-prod resource and parses out thesecrets_passphrase
found in thepipeline.yml
file forcg-deploy-cf
. This is useful for feeding this value to other commands without visually exposing the secret in your terminal.fly --target <target> get-pipeline --pipeline deploy-cf | \ spruce json | \ jq -r '.resources[] | select(.name | test("common-prod")) | .sources.secrets_passphrase'
-
-
Decrypt the secrets file.
INPUT_FILE=tmp/cf.main.yml.enc OUTPUT_FILE=tmp/cf.main.yml PASSPHRASE=$(fly --target <target> get-pipeline --pipeline deploy-cf | spruce json | jq -r '.resources[] | select(.name | test("common-prod")) | .sources.secrets_passphrase') ./decrypt.sh
-
Don’t leave the secrets lying around (for security reasons, and because they get stale).
rm -rf tmp
Change secrets
- Post in #cg-platform saying you’re updating
cf.main.yml
. - Follow the decryption steps above, up until the cleanup.
- Modify the
cf.main.yml
with the new values. -
Re-encrypt the file.
INPUT_FILE=tmp/cf.main.yml OUTPUT_FILE=tmp/cf.main.yml.enc PASSPHRASE=$(fly --target <target> get-pipeline --pipeline deploy-cf | spruce json | jq -r '.resources[] | select(.name | test("common-prod")) | .sources.secrets_passphrase') ./encrypt.sh
-
Copy the new file up to S3.
aws-vault exec cloud-gov-govcloud bash aws s3 cp tmp/cf.main.yml.enc-new s3://cloud-gov-varz/cf.main.yml --sse AES256
-
Clean up the secrets.
rm -rf tmp
Updating Concourse Secrets
Updating Concourse secrets combines the shared secrets above with local config to change a Concourse pipeline. Note: The steps below are for updating the deploy-bosh
pipeline, the same steps can also be applied to the deploy-concourse
pipeline.
-
Clone the cg-scripts repository.
git clone https://github.com/18F/cg-scripts.git
-
If you don’t already have the AWS CLI set up with credentials, see the steps above.
-
cd
to the local clone of the Concourse pipeline you are updating -
Create a
concourse-environment
file.aws-vault exec cloud-gov-govcloud bash cd $(mktemp -d) CI_ENV=cloud-gov-govcloud SECRETS_BUCKET=cloud-gov-varz CG_PIPELINE=~/cg-pipeline-tasks ~/cg-scripts/generate-concourse-environment.sh ~/cg-deploy-bosh/cg-deploy-bosh.yml
Where
CI_ENV
is the environment configured for local Concoursefly
CLI,SECRETS_BUCKET
is the encrypted S3 bucket where credentials are stored, andCG_PIPELINE
is the path to a local clone of thecg-pipeline-tasks
git repository.Any number of local yml files (
cg-deploy-bosh.yml
) can also be merged to create the finalconcourse-environment.yml
. -
Update the Concourse pipeline with
fly
.fly -t cloud-gov-govcloud set-pipeline -p deploy-bosh -c ~/cg-deploy-bosh/ci/pipeline.yml -l concourse-environment.yml
If the repository includes a
ci/concourse-defaults.yml
file, you’ll need to load variables from there as well:fly -t cloud-gov-govcloud set-pipeline -p deploy-bosh -c ~/cg-deploy-bosh/ci/pipeline.yml -l concourse-environment.yml -l ~/cg-deploy-bosh/ci/concourse-defaults.yml`
-
Don’t leave the secrets lying around (for security reasons, and because they get stale).
rm -rf concourse-environment.yml