AWS Elasticsearch

cloud.gov offers aws-elasticsearch 7.4 as a service hosted in AWS Elasticsearch

Changes

This service is currently running Elasticsearch 7.4. Our prior Elasticsearch service ran version 5.8. There has been a good deal of changes including breaking changes between the 5.X and 7.X releases. Customers are encouraged to read the following links for more information on ES API changes:

• ES Release notes for 6.0
• ES Breaking Changes for 7.0
• AWS Supported ES Operations

Plans

Pricing

$200/month per node. Six nodes included for customers using the FISMA Moderate plan. More information on the pricing page.

How to create an instance

To create a service instance run the following command:

cf create-service aws-elasticsearch es-medium my-elastic-service

Note: AWS Elasticsearch creation times will vary and is outside of Cloud.gov’s control. AWS says approximately 15-30 mins per node.

Shard/replica configuration for high availability

When using the medium and medium-ha plans, please read Scalability and resilience: clusters, nodes, and shards for the elasticsearch basics on clusters as well as the AWS specific Developer Guide. The medium plan is mainly focused for customers that need a single index and have coverage with 2 data nodes. For customers wanting more coverage and more indexes, then the medium-ha plan scales the cluster to 4 data nodes to offer high availability (HA).

AWS Signing requests requirement

In order to use the Elasticsearch service hosted on AWS you will need to use AWS signed HTTP headers

Managing backups

By default AWS will create hourly snapshot backups of your Elasticsearch service instance and retain those snapshots for 14 days - 336 total snapshots are retained. These snapshots are only accessible to cluster restore purposes only and can not be exported.

For customers that would like to import or export their Elasticsearch data, this can be done following the AWS Elasticsearch Manual Snapshot Process

As part of the process of using manual snapshots, customers will need to assign an existing S3 bucket and the broker will provide a AWS policy ARN to use with the manual S3 snapshot process

• Create an instance of the S3 service:

    cf create-service s3 basic my-s3-bucket
  

• Create a service key to access S3 credentials:

    cf create-service-key my-s3-bucket my-key
  

• Get your s3 bucket name:

   s3_credentials=$(cf service-key my-s3-bucket my-key | tail -n +3)
   s3_bucket=$(echo "${s3_credentials}" | jq -r '.bucket')
  

• Assign your new bucket to your Elasticsearch instance:

   cf create-service-key my-elastic-service my-key -c '{"bucket":"${s3_bucket}"}'
  

• Find your AWS ARN for your AWS policy to reference for your backup:

   es_arn=$(cf service-key my-elastic-service my-key | tail -n +3)
   snapshotRoleARN=$(echo "${es_arn}" | jq -r '.snapshotRoleARN')
  

Once you have your s3 bucket connected and have the snapshotRoleARN you can then inside your application connect to the AWS ES host and register your s3 repository endpoint and then perform your snapshot export/import operations using AWS signed HTTP headers. Due to the nature of AWS Signature Calculations there is no direct way with curl to perform these operations and it’s best left to client libraries in a programing language your applications are written in.

cloud.gov does offer a code sample repository on Github - aws-elasticsearch-example that shows an example in Python on how to interact with the new ES service using signed headers. Our customers are encouraged to submit PRs of other examples to share with fellow customers.

Rotating credentials

You can rotate credentials by creating a new instance and deleting the existing instance. If this is not an option, email cloud.gov support to request rotating the credentials manually.

The broker in GitHub

You can find the broker here: https://github.com/cloud-gov/aws-broker.