Creating a Dev Environment on AWS
This page is primarily for the cloud.gov team. It's public so that you can learn from it. For help using cloud.gov, see the user docs.
BOSH Lite is designed to provide a local development environment for BOSH and by extension Cloud Foundry. Though BOSH Lite can be run locally via Virtualbox, this guide is primarily concerned with bringing up a single-instance environment on Amazon EC2.
The bosh-lite repo provides a Vagrantfile and box for quickly standing up a bosh lite director on AWS.
brew install vagrant vagrant plugin install vagrant-aws
(Amazon Web Services Command Line Interface)
While not strictly necessary, but this guide assumes a recent version it. Install or update it with:
sudo pip install awscli -U
Needed to clone the bosh-lite repo.
brew install git
Cloud Foundry CLI
cf command is our gateway to the Cloud Foundry environment, it handles everything from the creation of users and organizations to pushing and scaling applications.
brew tap pivotal/tap brew install cloudfoundry-cli
Bosh IAM User
You’ll need a recent version of the AWS CLI for some of these commands. The current release is 1.7.12 as of this writing. Earlier versions may not have all the required commands.
Create the bosh user.
aws iam create-user --user-name bosh
Apply a policy that will allow bosh to perform actions in the AWS environment.
aws iam attach-user-policy --user-name bosh --policy-arn 'arn:aws:iam::aws:policy/AmazonEC2FullAccess'
Create the bosh keypair.
aws ec2 create-key-pair --key-name bosh --query 'KeyMaterial' --output text > bosh.pem
Create the bosh access key and secret key.
aws iam create-access-key --user-name bosh
Export the required environment variables using key file and credentials create above.
This is the path to the private portion of the bosh keypair (bosh.pem).
This is AWS access key to be used by bosh.
This is AWS secret key to be used by bosh.
Bosh EC2 Security Group
A EC2 security group which allows SSH and HTTP/S into your AWS environment:
Again the AWS CLI is nice to have here.
Create the security group. By default, the bosh-lite Vagrantfile expects this security group to be named inception. You can call it something else, but you’ll need to make the new name available to vagrant with
aws ec2 create-security-group --group-name inception --description 'Access to Cloud Foundry.'
Add your own public IP at minimum.
aws ec2 authorize-security-group-ingress --group-name bosh-inception --protocol tcp --port 22 --cidr $(curl icanhazip.com)/32
With the prerequisites out of the way, we can use Vagrant to deploy the instance which will run the BOSH Lite directory and wardenized Cloud Foundry services.
Grab the bosh-lite repo.
git clone https://github.com/cloudfoundry/bosh-lite.git cd bosh-lite
Add the bosh-lite vagrant box.
vagrant box add cloudfoundry/bosh-lite
Bring the vagrant machine up.
vagrant up includes provisioning scripts which set up port forwarding rules via iptables to expose Cloud Foundry service VMs running as warden containers on the instance. If the machine is restarted or halted for any reason we’ll need to follow the steps in reprovisioning below.
Grab the public IP of the resulting instance and keep it handy. We’ll need it later in the deployment and when accessing the resulting environment with the Cloud Foundry CLI.
aws ec2 describe-instances \ --instance-id $(cat .vagrant/machines/default/aws/id) \ --query 'Reservations.Instances.PublicIpAddress'
Finish Building the EC2 Instance:
Connect to the AWS instance via SSH.
There are a few utilities that we’ll need to configure the Cloud Foundry environment which aren’t included with the BOSH Lite public AMI.
sudo apt-get update && apt-get -y install git unzip
Add the spiff utility.
wget https://github.com/cloudfoundry-incubator/spiff/releases/download/v1.0.7/spiff_linux_amd64.zip unzip spiff_linux_amd64.zip -d /usr/local/bin
Create a workspace directory.
mkdir workspace cd workspace
Target the local bosh director.
bosh target 127.0.0.1
Clone the bosh bosh-lite repo.
git clone https://github.com/cloudfoundry/bosh-lite.git cd bosh-lite
Configure Cloud Foundry:
Update the deployment stub to reflect the public IP of your instance as recorded above by adding a
domain: key to the properties section of
manifests/cf-stubb-spiff.yml. The resulting file should look something like this:
... properties: domain: INSTANCE-PUBLIC-IP.xip.io ...
Ensure the Cloud Foundry services will be able to access each other via the external address.
aws ec2 authorize-security-group-ingress --group-name bosh-inception --protocol tcp --port 0-65536 --cidr INSTANCE-PUBLIC-IP/32
Provision Cloud Foundry:
Run the automated provisioning script.
This script automates the process of:
- Recursively cloning the cf-release repository.
- Uploading the latest release to the director.
- Uploading a public stemcell to the director.
- Generating a deployment manifest from the included templates.
- Deploying the generated manifest to a set of warden containers inside the instance.
This process takes quite a while, up to an hour on the default
m3.xlarge instance type.
When complete, use bosh vms to have a look at the resulting environment.
ubuntu@agent-id-bosh-0:~/workspace/bosh-lite$ bosh vms Deployment 'cf-warden' Director task 4 Task 4 done +------------------------------------+---------+---------------+--------------+ | Job/index | State | Resource Pool | IPs | +------------------------------------+---------+---------------+--------------+ | api_z1/0 | running | large_z1 | 192.0.2.138 | | etcd_z1/0 | running | medium_z1 | 192.0.2.42 | | ha_proxy_z1/0 | running | router_z1 | 192.0.2.34 | | hm9000_z1/0 | running | medium_z1 | 192.0.2.142 | | loggregator_trafficcontroller_z1/0 | running | small_z1 | 192.0.2.150 | | loggregator_z1/0 | running | medium_z1 | 192.0.2.146 | | login_z1/0 | running | medium_z1 | 192.0.2.134 | | nats_z1/0 | running | medium_z1 | 192.0.2.6 | | postgres_z1/0 | running | medium_z1 | 192.0.2.30 | | router_z1/0 | running | router_z1 | 192.0.2.22 | | runner_z1/0 | running | runner_z1 | 192.0.2.26 | | uaa_z1/0 | running | medium_z1 | 192.0.2.130 | +------------------------------------+---------+---------------+--------------+ VMs total: 12
We’re done provisioning, now we’ll disconnect and access the environment via the Cloud Foundry CLI.
If the Vagrant machine is halted or restarted for any reason there are several steps we’ll need to take to recreate the environment.
1.) Recreate forwarding rules.
2.) Repair the VMs.
bosh cck cf-warden
Choose option 2 “Recreate VM using last known apply spec” for each VM then confirm the choices. A new director task will be created to recreate each VM in turn. This will take several minutes to complete.
3.) If necessary, re-deploy using the new public IP address.
manifests/cf-stub-spiff.yml to reflect the new public IP address, then run:
./bin/make_manifest_spiff bosh deploy
If the machine will be regularly restarted, consider associating an elastic IP to avoid having to repeat the third step above.
Initial Cloud Foundry Configuration:
Connect using the Cloud Foundry CLI.
cf api --skip-ssl-validation https://api.INSTANCE-PUBLIC-IP.xip.io cf auth admin admin
Note: This step will fail with a 500 error if the VMs cannot reach each other via
Create and target an organization.
cf create-org MY-ORGANIZATION cf target -o MY-ORGANIZATION
Create and target a space.
cf create-space MY-SPACE cf target -s MY-SPACE