Secret key management
This page is primarily for the cloud.gov team. It's public so that you can learn from it. For help using cloud.gov, see the user docs.
Sharing secret keys
For sharing the following types of sensitive information related to cloud.gov, cloud.gov team members must use GSA Google Hangouts. The team member must share the information only with intended recipient(s) who need to know the sensitive information. Team members can use the Hangouts screen-sharing feature or verbally share the information.
- Secret keys
- Sensitive environment variables
- Other secret authentication information
Maintenance of system secret keys
To meet NIST security control SC-12 (1), we maintain the availability of all information on the platform in the event a cryptographic access key is lost or compromised.
Authorized federal staff rotate, encrypt, and backup keys yearly. Privileged users can access the keys only with two-factor authentication and a decryption passphrase. In the rare case that both the keys and the decryption passphrase for the backup are lost or compromised, new keys can be rotated in by authorized federal staff, while maintaining availability of information.
If you need to view/update secrets:
- Ask in #cg-platform for an account to read/write from the S3 buckets.
- Setup aws-vault
Install aws-vault for AWS credentials
aws-vault secures credentials locally and generates temporary credentials to provide an additional layer of security. To install
aws-vault use the brew command:
brew cask install aws-vault aws-vault add cloud-gov-govcloud
Configure MFA for aws-vault
All operators should have MFA enabled, which can be viewed under
Services -> IAM -> Users -> firstname.lastname -> Security Credentials. This MFA device needs to be added to the Amazon configuration to enable short lived tokens:
echo '[profile cloud-gov-govcloud]' >> ~/.aws/config echo 'region = us-gov-west-1' >> ~/.aws/config echo 'mfa_serial = arn:aws-us-gov:iam::1234567890:mfa/firstname.lastname' >> ~/.aws/config
Executing a command with short lived credentials
You can execute any system command with short lived credentials using the
aws-vault exec command:
aws-vault exec cloud-gov-govcloud bash
env | grep AWS will show you a new set of credentials which are different from the primary IAM role credentials, as they are short lived and issued at runtime. This means that if a malicious script or program attempts to read
~/.aws/config all they will be unable to retrieve the primary credentials.
Once this is complete, operators can provision profiles which use only specific resources, or specific permissions such as read only. This scopes the role of the temporary credentials to further reduce the attack surface.
Generate and upload keys
Use this script to generate keys and upload them as access keys to AWS EC2 and an encrypted backup to AWS S3. You just need to provide a
BUCKET destination for the backup and a decryption
PASSPHRASE. In your terminal:
BUCKET=my-bucket PASSPHRASE=somethingorother ./generate_key.sh
Once the key is uploaded to AWS, the cf-secrets.yml file you use for deployment needs to be updated. Modify
key_name to use the new key, then start a new production deployment.
Dealing with secrets
cf.main.yml in the following steps for the relevant file.
All UAA clients and users and associated credentials should be created via the Cloud Foundry secrets or the service account or identity provider services. UAA accounts should not be created manually; we reserve the right to drop permissions for or deprovision hand-propped accounts.
Clone the cg-pipeline-tasks repository.
git clone https://github.com/18F/cg-pipeline-tasks.git
If you don’t already have the AWS CLI set up with credentials, see the steps above.
Download the credentials file.
aws-vault exec cloud-gov-govcloud bash mkdir -p tmp aws s3 cp s3://cloud-gov-varz/cf.main.yml tmp/cf.main.yml.enc
Get the encryption passphrase.
deploy-cfpipeline configuration for any of the
common-*environments. The following command outputs all the
cg-commonresources. You can modify the test for
test("common-")to grab specific credentials about each resource.
fly --target <target> get-pipeline --pipeline deploy-cf | \ spruce json | \ jq -r '.resources | select(.name | test("common-"))'
secrets_passphraseunder the common-prod resource. The following command grabs just the common-prod resource and parses out the
secrets_passphrasefound in the
cg-deploy-cf. This is useful for feeding this value to other commands without visually exposing the secret in your terminal.
fly --target <target> get-pipeline --pipeline deploy-cf | \ spruce json | \ jq -r '.resources | select(.name | test("common-prod")) | .sources.secrets_passphrase'
Decrypt the secrets file.
INPUT_FILE=tmp/cf.main.yml.enc OUTPUT_FILE=tmp/cf.main.yml PASSPHRASE=$(fly --target <target> get-pipeline --pipeline deploy-cf | spruce json | jq -r '.resources | select(.name | test("common-prod")) | .sources.secrets_passphrase') ./decrypt.sh
Don’t leave the secrets lying around (for security reasons, and because they get stale).
rm -rf tmp
- Post in #cg-platform saying you’re updating
- Follow the decryption steps above, up until the cleanup.
- Modify the
cf.main.ymlwith the new values.
Re-encrypt the file.
INPUT_FILE=tmp/cf.main.yml OUTPUT_FILE=tmp/cf.main.yml.enc PASSPHRASE=$(fly --target <target> get-pipeline --pipeline deploy-cf | spruce json | jq -r '.resources | select(.name | test("common-prod")) | .sources.secrets_passphrase') ./encrypt.sh
Copy the new file up to S3.
aws-vault exec cloud-gov-govcloud bash aws s3 cp tmp/cf.main.yml.enc-new s3://cloud-gov-varz/cf.main.yml --sse AES256
Clean up the secrets.
rm -rf tmp