AWS Elasticsearch

cloud.gov offers aws-elasticsearch 7.4 as a service hosted in AWS Elasticsearch

Changes

This service is currently running Elasticsearch 7.4. Our prior Elasticsearch service ran version 5.8. There has been a good deal of changes including breaking changes between the 5.X and 7.X releases. Customers are encouraged to read the following links for more information on ES API changes:

Plans

Service Name	Plan Name	Description	Number of nodes
`aws-elasticsearch`	`es-dev`	Single data node for non-prod use only	1
`aws-elasticsearch`	`es-medium`	3 Primary and 2 Data node cluster	5
`aws-elasticsearch`	`es-medium-ha`	3 Primary and 4 Data node cluster	7

How to create an instance

To create a service instance run the following command:

cf create-service aws-elasticsearch es-medium my-elastic-service

Shard/replica configuration for high availability

When using the medium and medium-ha plans, please read Scalability and resilience: clusters, nodes, and shards for the elasticsearch basics on clusters as well as the AWS specific Developer Guide. The medium plan is mainly focused for customers that need a single index and have coverage with 2 data nodes. For customers wanting more coverage and more indexes, then the medium-ha plan scales the cluster to 4 data nodes to offer high availability (HA).

AWS Signing requests requirement

In order to use the Elasticsearch service hosted on AWS you will need to use AWS signed HTTP headers

Managing backups

By default AWS will create hourly snapshot backups of your Elasticsearch service instance and retain those snapshots for 14 days - 336 total snapshots are retained. These snapshots are only accessible to cluster restore purposes only and can not be exported.

For customers that would like to import or export their Elasticsearch data, this can be done following the AWS Elasticsearch Manual Snapshot Process

As part of the process of using manual snapshots, customers will need to assign an existing S3 bucket and the broker will provide a AWS policy ARN to use with the manual S3 snapshot process

Create an instance of the S3 service:

  cf create-service s3 basic my-s3-bucket

Create a service key to access S3 credentials:

  cf create-service-key my-s3-bucket my-key

Get your s3 bucket name:

 s3_credentials=$(cf service-key my-s3-bucket my-key | tail -n +3)
 s3_bucket=$(echo "${s3_credentials}" | jq -r '.bucket')

Assign your new bucket to your Elasticsearch instance:

 cf create-service-key my-elastic-service my-key -c '{"bucket":"${s3_bucket}"}'

Find your AWS ARN for your AWS policy to reference for your backup:

 es_arn=$(cf service-key my-elastic-service my-key | tail -n +3)
 snapshotRoleARN=$(echo "${es_arn}" | jq -r '.snapshotRoleARN')

Once you have your s3 bucket connected and have the snapshotRoleARN you can then inside your application connect to the AWS ES host and register your s3 repository endpoint and then perform your snapshot export/import operations using AWS signed HTTP headers. Due to the nature of AWS Signature Calculations there is no direct way with curl to perform these operations and it’s best left to client libraries in a programing language your applications are written in.

cloud.gov does offer a code sample repository on Github - aws-elasticsearch-example that shows an example in Python on how to interact with the new ES service using signed headers. Our customers are encouraged to submit PRs of other examples to share with fellow customers.

Migration from Kubernetes Elasticsearch 5.6 to AWS Elasticsearch 7.4

This content is only applicable to customers using the pre-AWS Elasticsearch service on cloud.gov during the transition period before that service is fully deprecated in 2021.

We recommend that you build new document indices in AWS Elasticsearch from your source documents instead of migrating the data and indices from the existing service to the new service. The migration is experimental could result in more time and effort than a rebuild would entail.

The overall process looks like this:

Ask Cloud.gov Support to access the AWS Elasticsearch plan named es-dev-6.8-migration
Take a snapshot of your ES 5.6 index and store in an s3 bucket.
Create ES 6.8 service instance and create a service key with s3-bucket info attached (See Above Section)
Restore ES 5.6 Snapshot into your ES 6.8
Upgrade your ES 6.8 to 7.4

Note: Steps 2-4 is done by the user

Prepare for Migration

Have a S3 service instance created with service key - S3 Service Instructions
Have an existing ES 5.6 instance with snapshots to S3 - ES 5.6 Backups Instructions
Ask Cloud.gov Support to have access to migration plan es-dev-6.8-migration
Create new AWS Elasticsearch service instance with es-dev-6.8-migration plan
Create Service Key with attached S3 - ES 7.4 Service Key

Backup your ES 5.6 to S3 Bucket

Connect to your Elasticsearch service using port forwarding.

  es_credentials=$(cf service-key my-elastic-service my-key | tail -n +3)

  es_hostname=$(echo "${es_credentials}" | jq -r '.hostname')
  es_port=$(echo "${es_credentials}" | jq -r '.port')
  es_username=$(echo "${es_credentials}" | jq -r '.username')
  es_password=$(echo "${es_credentials}" | jq -r '.password')

  cf ssh my-app -L "9200:${es_hostname}:${es_port}"

Note: You’ll need to leave this cf ssh command running and perform the following steps a different terminal so that you can access the remote Elasticsearch instance from your local environment.

Create a snapshot repository:

  s3_credentials=$(cf service-key my-s3-bucket my-key | tail -n +3)

  s3_bucket=$(echo "${s3_credentials}" | jq -r '.bucket')
  s3_region=$(echo "${s3_credentials}" | jq -r '.region')
  s3_access_key=$(echo "${s3_credentials}" | jq -r '.access_key_id')
  s3_secret_key=$(echo "${s3_credentials}" | jq -r '.secret_access_key')

  curl -X PUT -u "${es_username}:${es_password}" "localhost:9200/_snapshot/my_s3_repository" -d @<(cat <<EOF
  {
    "type": "s3",
    "settings": {
      "bucket": "${s3_bucket}",
      "region": "${s3_region}",
      "access_key": "${s3_access_key}",
      "secret_key": "${s3_secret_key}"
    }
  }
  EOF
  )

Create a snapshot:

  curl -X PUT -u "${es_username}:${es_password}" "localhost:9200/_snapshot/my_s3_repository/my_s3_snapshot"

Restore from S3 to ES 6.8 and reindex

This part will vary greatly depending on which language your application is written in and which library is used.

You need to add the following ES repository to your AWS Elasticsearch Domain

  {
 "type": "s3",
 "settings": {
   "bucket": "${s3_bucket}",
   "region": "${s3_region}",
   "role_arn" : "${snapshotRoleARN}"
 }
  }

s3_bucket and s3_region is same as previous step but the snapshotRoleARN is given from service-key credentials for the AWS ES 7.4.

Restore using above repository and matching snapshot name in previous step. ES 6.8 Restore
Reindex your index - You will need to do this so the index will be the correct version for when upgrading. ES 6.8 Reindexing API Example reindexing body:
```
{
  "source": {
 "index": "your_index"
  },
  "dest": {
 "index": "your_index_in_6.8"
  }
}
```
Once your indexes are at least version 6.8, you can move on to the final part!
Upgrading ES 6.8 to 7.4
Contact Cloud.gov Support to perform a ES 6.8 to ES 7.4 Upgrade Eligibility
Cloud.gov Support will let you know if your ES 6.8 instance is eligible to upgrade. We will be able to let you know which index is incompatible with ES 7.4. Cloud.gov Support will not have the ability to see why the index is incompatible.
If the instance is ready to upgrade, then support can do the upgrade to ES 7.4. This will result in some downtime. Please note that these upgrades and only be done during normal support hours and not schedulable after hours or weekends.

Rotating credentials

You can rotate credentials by creating a new instance and deleting the existing instance. If this is not an option, email cloud.gov support to request rotating the credentials manually.

The broker in GitHub

You can find the broker here: https://github.com/cloud-gov/aws-broker.

Suggest edits