US flag signifying that this is a United States Federal Government website An official website of the United States government

Configuration management

Configuration management

This page is primarily for the cloud.gov team. It's public so that you can learn from it. For help using cloud.gov, see the user docs.

This document describes how the cloud.gov team approaches configuration management of the core platform. Before configuration changes go into production, they need to pass our significant change rubric, as described in our Feature Lifecycle and Story Lifecycle.

What goes into configuration management?

In short, everything needed to run and operate the platform that is not a secret. (See Secret Key Management for that.)

Here are some examples that should be in configuration management:

  • CI pipelines (Concourse)
  • Infrastructure/network configuration (Terraform)
  • VM setup and quantity (BOSH)
  • Software configuration (BOSH)
  • 18F-developed code

Where should all this configuration go?

All configuration must be stored in GitHub using the following “Change Workflow” unless it is a secret.

How do we test these changes?

If possible, first test the changes locally. After that, upload them to a development environment where either manual or automated testing needs to be run. Security tests need to be executed in the development environment where changes are applied.

Change workflow

  1. All configuration changes must flow through a git repository, centrally managed through GitHub, unless they contain sensitive information. In these cases, sensitive information should be stored in an S3 bucket with a proper security policy and encryption, versioned such that changes can be easily rolled back.
  2. A change is initiated and discussed, following the steps in our Story Lifecycle.
  3. In the appropriate GitHub repository for the component, a pull request (PR) against the master branch is created that addresses the change.
  4. If the repository contains 18F-developed code, the PR must have an automated Code Climate check, which must pass before the PR can be merged.
  5. The PR is reviewed by someone other than the committer. Pairing via screen-sharing is encouraged and qualifies as a review. Review should include assessment of architectural design, DRY principles, security and code quality. The reviewer approves the PR via GitHub.
  6. The reviewer merges the approved PR. The committer may merge an approved PR if the changes made are time-sensitive.
  7. A continuous integration (CI) server handles automated tests and continuous deployment (CD) of the merged changes.

    • All changes are deployed to a testing environment, such as development.
    • Any and all automated tests are run.
    • If all tests pass, changes can be promoted for deployment to production in the pipeline.
  8. The CI/CD tool uses GitHub repositories and S3-stored sensitive content as the canonical source of truth for what the platform should look like. If there are manual changes, it will reset the state of all systems to match.

Checklist for new repositories

Before we put a new repository into production:

  • Give it a name starting with cg- or cf- (to make clear that it’s part of our work, unless we have a good reason to name it something else).
  • Add LICENSE, CONTRIBUTING, and README files (to support open source reuse of our work).
  • Set up Code Climate (SI-3).
  • Configure a protected master branch with required reviews (CM-9).
  • Configure permissions (CM-3):
    • If it’s a platform configuration repo, restrict permissions to Cloud Ops, as follows:
    • If it’s not a platform configuration repo, configure as follows:
  • Set up CI/CD for changes (CM-3).
  • Open a PR to add it to the repos list (to help us keep track of our repos and support open source reuse of our work).

What if a configuration changed and it is not in Configuration Management?

If possible, Configuration Management tools need to be set up to always roll back to a known state. Other than that, these tools need to be able to “recreate” all settings from the known configurations.

GitHub contribution guidelines

Because cloud.gov is built and maintained by 18F, we follow the 18F requirements for using GitHub. These are our team practices within those requirements.

Forking vs. branching

Both forking and branching are welcome in our repositories. Contributors inside 18F can use forking or branching according to their personal preferences, and contributors outside 18F can fork repositories.

The team often practices branching. The rationale for branching within a team is that paired collaboration on a single branch avoids certain types of friction:

  • Having to create multiple forked PRs in order to contribute to the branch
  • Having to add new users to forked repositories as collaborators in order to have people directly contribute on short-lived forked branches

When contributing directly on a branch, we’re able to modify work-in-progress (WIP) pull requests and encourage collaboration across the Cloud Operations team.

Squashing commits

Squashing commits is allowed but discouraged, except in rare instances.

Rebase or merge

The team prefers rebasing over merging.

When should a PR be created?

Work-in-progress PRs are encouraged. When a PR is ready for review, it should be tagged in GitHub with the review-needed label. If you create a work-in-progress PR, you might also make it plain in the PR name with a [WIP] prefix.

Should PRs be assigned?

PRs are typically not assigned in GitHub, unless someone specifically needs to sign off on the change.

You can request a review using GitHub’s built-in tools, mention someone in the PR with the @ notation, or contact them outside the GitHub context to request a review.

When reviewing a PR, should the change be tested locally?

Whenever possible, the proposed changes should be tested locally. Because of the nature of many of the cloud.gov repositories and deployment environments, local testing is not always possible or practical. Visual code review, however, is always required.