An official website of the United States government US flag signifying that this is a United States federal government website

Configuration management

Configuration management

This document describes how the cloud.gov team approaches configuration management of the core platform.

What goes into configuration management?

In short, everything needed to run and operate the platform that is not a secret. (See Secret Key Management for that.)

Here are some examples that should be in configuration management:

  • CI pipelines (Concourse)
  • Infrastructure/network configuration (Terraform)
  • VM setup and quantity (BOSH)
  • Software configuration (BOSH)
  • 18F-developed code

Where should all this configuration go?

All configuration must be stored in GitHub using the following “Change Workflow” unless it is a secret.

How do we test these changes?

If possible, first test the changes locally. After that, upload them to a development environment where either manual or automated testing needs to be run. Security tests need to be executed in the development environment where changes are applied.

Change workflow

  1. All configuration changes must flow through a git repository, centrally managed through GitHub, unless they contain sensitive information. In these cases, sensitive information should be stored in an S3 bucket with a proper security policy and encryption, versioned such that changes can be easily rolled back.
  2. A change is initiated and discussed as a GitHub issue in the relevant 18F repository.
  3. A pull request (PR) is created that addresses the change and references the created issue. If there is a staging environment available for a given repository, the PR should be created against the staging branch. Otherwise, the PR should be created against the master branch on the canonical repository.
  4. If the repository contains 18F-developed code, the PR must have an automated Code Climate check, which must pass before the PR can be merged.
  5. The PR is reviewed by someone other than the committer. Pairing via screen-sharing is encouraged and qualifies as a review. Review should include assessment of architectural design, DRY principles, security and code quality.
  6. The reviewer merges the PR.
  7. A continuous integration (CI) server handles automated tests and continuous deployment (CD) of the merged changes.

    • All changes are deployed to a testing environment, such as development.
    • Any and all automated tests are run.
    • If all tests pass, changes can be promoted for deployment to production in the pipeline.
  8. The CI/CD tool uses GitHub repositories and S3-stored sensitive content as the canonical source of truth for what the platform should look like. If there are manual changes, it will reset the state of all systems to match.

Pipeline Example

A more detailed example of this process can be seen in Updating Cloud Foundry.

What if a configuration changed and it is not in Configuration Management?

If possible, Configuration Management tools need to be set up to always roll back to a known state. Other than that, these tools need to be able to “recreate” all settings from the known configurations.

GitHub contribution guidelines

Because cloud.gov is built and maintained by 18F, we follow the 18F requirements for using GitHub. These are our team practices within those requirements.

Forking vs. branching

Both forking and branching are welcome in our repositories. Contributors inside 18F can use forking or branching according to their personal preferences, and contributors outside 18F can fork repositories.

The team often practices branching. The rationale for branching within a team is that paired collaboration on a single branch avoids certain types of friction:

  • Having to create multiple forked PRs in order to contribute to the branch
  • Having to add new users to forked repositories as collaborators in order to have people directly contribute on short-lived forked branches

When contributing directly on a branch, we’re able to modify work-in-progress (WIP) pull requests and encourage collaboration across the Cloud Operations team.

Squashing commits

Squashing commits is allowed but discouraged, except in rare instances.

Rebase or merge

The team prefers rebasing over merging.

When should a PR be created?

Work-in-progress PRs are encouraged. When a PR is ready for review, it should be tagged in GitHub with the review-needed label. If you create a work-in-progress PR, you might also make it plain in the PR name with a [WIP] prefix.

Should PRs be assigned?

PRs are typically not assigned in GitHub, unless someone specifically needs to sign off on the change.

You can request a review using GitHub’s built-in tools, mention someone in the PR with the @ notation, or contact them outside the GitHub context to request a review.

When reviewing a PR, should the change be tested locally?

Whenever possible, the proposed changes should be tested locally. Because of the nature of many of the cloud.gov repositories and deployment environments, local testing is not always possible or practical. Visual code review, however, is always required.