About Migration of ElasticSearch

General
, ,
1

Hello

I’m looking to migrate Pega from EC2-based 8.7 to EKS-hosted.
For migration, we are considering migrating data such as schema from RDS. RDS is intended to use PSQL to export and import schemas.

ElasticSearch will be externalized for use with K8s.
Is it necessary to migrate index information from the existing ElasticSearch when externalizing?

If so, can I follow the steps in the link below?
Migrate internal indices | Elasticsearch Service Documentation | Elastic

2

@keisukek17309988 When externalizing Elasticsearch, it is necessary to consider migrating index information from the existing Elasticsearch. You can follow the steps outlined in the external Elasticsearch migration documentation to assist with this process. It is important to ensure that the version compatibility and configurations are properly addressed during the migration.

External Elasticsearch in your deployment

Externalization of services in your deployment

:warning: This is a GenAI-powered answer. All generated answers require validation against the provided references.

Yes, it is necessary to migrate the index information when externalizing ElasticSearch for use with K8s. This is crucial for maintaining search functionality after the migration. Here’s why and how:

  1. Why Index Migration is Necessary:
    • To maintain search functionality in the new EKS environment
    • To ensure proper association with the Search node type in the externalized setup
    • To preserve existing search configurations and data
  2. Migration Approach: The steps in the ElasticSearch documentation you referenced are applicable, but need to be adapted for the EKS context. You have two main options:

a) Reindexing Method:

  • Use remote reindexing for live migration
  • Reset search indexing to use the appropriate Search node type
  • Remove any legacy command-line search settings that modify index host nodes

b) Snapshot and Restore Method:

  • Take a snapshot of your existing ElasticSearch indices
  • Register the snapshot repository in read-only mode in the new environment
  • Restore the indices with special handling for internal indices (.kibana, .security)

For your specific case, I recommend following these steps:

  1. First, remove any command-line Search settings that use JVM startup (like -Dindex-directory)
  2. Reset your search indexing to use the Search node type for EKS compatibility
  3. Follow the migration steps, paying special attention to internal indices
  4. Verify the search functionality after migration

Important Considerations:

  • When handling internal indices (.kibana, .security), ensure the include_aliases parameter is set to false during restoration
  • Test the migration process in a non-production environment first
  • Validate search functionality after migration
  • Keep the original indices until the new setup is fully verified

Sources: Pega Cloud Migration Documentation ElasticSearch Migration Guide