AWS Elasticsearch
cloud.gov offers aws-elasticsearch 7.4 as a service hosted in AWS Elasticsearch
Changes
This service is currently running Elasticsearch 7.4. Our prior Elasticsearch service ran version 5.8. There has been a good deal of changes including breaking changes between the 5.X and 7.X releases. Customers are encouraged to read the following links for more information on ES API changes:
Plans
Service Name | Plan Name | Description | Number of nodes |
---|---|---|---|
aws-elasticsearch |
es-dev |
Single data node for non-prod use only | 1 |
aws-elasticsearch |
es-medium |
3 Primary and 2 Data node cluster | 5 |
aws-elasticsearch |
es-medium-ha |
3 Primary and 4 Data node cluster | 7 |
How to create an instance
To create a service instance run the following command:
cf create-service aws-elasticsearch es-medium my-elastic-service
Shard/replica configuration for high availability
When using the medium
and medium-ha
plans, please read Scalability and resilience: clusters, nodes, and shards for the elasticsearch basics on clusters as well as the AWS specific Developer Guide. The medium
plan is mainly focused for customers that need a single index and have coverage with 2 data nodes. For customers wanting more coverage and more indexes, then the medium-ha
plan scales the cluster to 4 data nodes to offer high availability (HA).
AWS Signing requests requirement
In order to use the Elasticsearch service hosted on AWS you will need to use AWS signed HTTP headers
Managing backups
By default AWS will create hourly snapshot backups of your Elasticsearch service instance and retain those snapshots for 14 days - 336 total snapshots are retained. These snapshots are only accessible to cluster restore purposes only and can not be exported.
For customers that would like to import or export their Elasticsearch data, this can be done following the AWS Elasticsearch Manual Snapshot Process
As part of the process of using manual snapshots, customers will need to assign an existing S3 bucket and the broker will provide a AWS policy ARN to use with the manual S3 snapshot process
-
Create an instance of the S3 service:
cf create-service s3 basic my-s3-bucket
-
Create a service key to access S3 credentials:
cf create-service-key my-s3-bucket my-key
-
Get your s3 bucket name:
s3_credentials=$(cf service-key my-s3-bucket my-key | tail -n +3) s3_bucket=$(echo "${s3_credentials}" | jq -r '.bucket')
-
Assign your new bucket to your Elasticsearch instance:
cf create-service-key my-elastic-service my-key -c '{"bucket":"${s3_bucket}"}'
-
Find your AWS ARN for your AWS policy to reference for your backup:
es_arn=$(cf service-key my-elastic-service my-key | tail -n +3) snapshotRoleARN=$(echo "${es_arn}" | jq -r '.snapshotRoleARN')
Once you have your s3 bucket connected and have the snapshotRoleARN
you can then inside your application connect to the AWS ES host and register your s3 repository endpoint and then perform your snapshot export/import operations using AWS signed HTTP headers. Due to the nature of AWS Signature Calculations there is no direct way with curl to perform these operations and it’s best left to client libraries in a programing language your applications are written in.
cloud.gov does offer a code sample repository on Github - aws-elasticsearch-example that shows an example in Python on how to interact with the new ES service using signed headers. Our customers are encouraged to submit PRs of other examples to share with fellow customers.
Migration from Kubernetes Elasticsearch 5.6 to AWS Elasticsearch 7.4
This content is only applicable to customers using the pre-AWS Elasticsearch service on cloud.gov during the transition period before that service is fully deprecated in 2021.
We recommend that you build new document indices in AWS Elasticsearch from your source documents instead of migrating the data and indices from the existing service to the new service. The migration is experimental could result in more time and effort than a rebuild would entail.
The overall process looks like this:
- Ask Cloud.gov Support to access the AWS Elasticsearch plan named
es-dev-6.8-migration
- Take a snapshot of your ES 5.6 index and store in an s3 bucket.
- Create ES 6.8 service instance and create a service key with
s3-bucket
info attached (See Above Section) - Restore ES 5.6 Snapshot into your ES 6.8
- Upgrade your ES 6.8 to 7.4
Note: Steps 2-4 is done by the user
Prepare for Migration
- Have a S3 service instance created with service key - S3 Service Instructions
- Have an existing ES 5.6 instance with snapshots to S3 - ES 5.6 Backups Instructions
- Ask Cloud.gov Support to have access to migration plan
es-dev-6.8-migration
- Create new AWS Elasticsearch service instance with
es-dev-6.8-migration
plan - Create Service Key with attached S3 - ES 7.4 Service Key
Backup your ES 5.6 to S3 Bucket
-
Connect to your Elasticsearch service using port forwarding.
es_credentials=$(cf service-key my-elastic-service my-key | tail -n +3) es_hostname=$(echo "${es_credentials}" | jq -r '.hostname') es_port=$(echo "${es_credentials}" | jq -r '.port') es_username=$(echo "${es_credentials}" | jq -r '.username') es_password=$(echo "${es_credentials}" | jq -r '.password') cf ssh my-app -L "9200:${es_hostname}:${es_port}"
Note: You’ll need to leave this
cf ssh
command running and perform the following steps a different terminal so that you can access the remote Elasticsearch instance from your local environment. -
Create a snapshot repository:
s3_credentials=$(cf service-key my-s3-bucket my-key | tail -n +3) s3_bucket=$(echo "${s3_credentials}" | jq -r '.bucket') s3_region=$(echo "${s3_credentials}" | jq -r '.region') s3_access_key=$(echo "${s3_credentials}" | jq -r '.access_key_id') s3_secret_key=$(echo "${s3_credentials}" | jq -r '.secret_access_key') curl -X PUT -u "${es_username}:${es_password}" "localhost:9200/_snapshot/my_s3_repository" -d @<(cat <<EOF { "type": "s3", "settings": { "bucket": "${s3_bucket}", "region": "${s3_region}", "access_key": "${s3_access_key}", "secret_key": "${s3_secret_key}" } } EOF )
-
Create a snapshot:
curl -X PUT -u "${es_username}:${es_password}" "localhost:9200/_snapshot/my_s3_repository/my_s3_snapshot"
Restore from S3 to ES 6.8 and reindex
This part will vary greatly depending on which language your application is written in and which library is used.
- You need to add the following ES repository to your AWS Elasticsearch Domain
{ "type": "s3", "settings": { "bucket": "${s3_bucket}", "region": "${s3_region}", "role_arn" : "${snapshotRoleARN}" } }
s3_bucket
and s3_region
is same as previous step but the snapshotRoleARN
is given from service-key credentials for the AWS ES 7.4.
- Restore using above repository and matching snapshot name in previous step. ES 6.8 Restore
- Reindex your index - You will need to do this so the index will be the correct version for when upgrading. ES 6.8 Reindexing API
Example reindexing body:
{ "source": { "index": "your_index" }, "dest": { "index": "your_index_in_6.8" } }
- Once your indexes are at least version
6.8
, you can move on to the final part!Upgrading ES 6.8 to 7.4
- Contact Cloud.gov Support to perform a ES 6.8 to ES 7.4 Upgrade Eligibility
- Cloud.gov Support will let you know if your ES 6.8 instance is eligible to upgrade. We will be able to let you know which index is incompatible with ES 7.4. Cloud.gov Support will not have the ability to see why the index is incompatible.
- If the instance is ready to upgrade, then support can do the upgrade to ES 7.4. This will result in some downtime. Please note that these upgrades and only be done during normal support hours and not schedulable after hours or weekends.
Rotating credentials
You can rotate credentials by creating a new instance and deleting the existing instance. If this is not an option, email cloud.gov support to request rotating the credentials manually.
The broker in GitHub
You can find the broker here: https://github.com/cloud-gov/aws-broker.