1. Overview
An index in Elasticsearch is a grouping of data shards that usually have some kind of logical connection between each other.
In this tutorial, we walk through effectively renaming an index in ElasticSearch. We also learn how to use index aliases, create a new index with the desired name, reindex the data, update aliases atomically, and clean up an old index.
2. Index Aliases
In essence, an alias in ElasticSearch acts as a secondary name for an index. In other words, this feature makes it possible to reference an index by a different name.
To illustrate the concept, let’s look at a quick example of how an alias works:
$ curl -X POST "localhost:9200/_aliases" -H 'Content-Type: application/json' -d'
{
"actions": [
{ "add": { "index": "customers", "alias": "current_customers" }}
]
}'
This command creates an alias named current_customers that points to the customers index. Consequently, we can now use current_customers in place of customers in most ElasticSearch operations, providing a layer of abstraction that we leverage in the renaming process.
3. The Renaming Process
Now that we’ve laid the groundwork, let’s break down the process of renaming an index into manageable steps.
3.1. Creating a New Index
To begin with, let’s create a new index with the desired name. To maintain consistency, we copy the settings and mappings from the old index.
So, let’s provide a fairly simple example to accomplish this:
import elasticsearch
es = elasticsearch.Elasticsearch("http://localhost:9200")
# Define the old and new index names
old_index = "customers"
new_index = "customers_v2"
# Get the settings and mappings of the old index
index_info = es.indices.get(index=old_index)
settings = index_info[old_index]["settings"]["index"]
mappings = index_info[old_index]["mappings"]
# Remove auto-managed settings
auto_managed_settings = [
"creation_date", "uuid", "version", "provided_name"
]
filtered_settings = {k: v for k, v in settings.items() if k not in auto_managed_settings}
# Create the new index with the same settings and mappings
es.indices.create(index=new_index, body={
"settings": filtered_settings,
"mappings": mappings
})
This script first retrieves the settings and mappings from the customers index. Then, it filters out auto-managed settings that can’t be directly set when creating a new index.
Subsequently, the script uses this filtered information to create the new customers_v2 index, ensuring that the structure remains consistent while avoiding potential errors.
3.2. Reindexing Data
After having the new index in place, it’s time to transfer the data.
For this task, we utilize the Reindex API:
reindex_body = {
"source": {
"index": old_index
},
"dest": {
"index": new_index
}
}
reindex_response = es.reindex(body=reindex_body, wait_for_completion=False)
task_id = reindex_response['task']
# Monitor the reindexing progress
while True:
task_status = es.tasks.get(task_id=task_id)
if task_status['completed']:
print("Reindexing completed!")
break
else:
progress = task_status['task']['status']['created'] / task_status['task']['status']['total']
print(f"Reindexing progress: {progress:.2%}")
time.sleep(5) # Wait for 5 seconds before checking again
In particular, this section initiates the reindexing process from customers to customers_v2 and then monitors its progress. By using the wait_for_completion=False parameter, we ensure that large datasets are handled asynchronously. Additionally, the while loop checks the task status every 5 seconds and prints the progress.
3.3. Updating Aliases
Once the reindexing process is complete, we update the aliases. It’s important to perform this step in a single, atomic operation to ensure zero downtime:
update_aliases_body = {
"actions": [
{"remove": {"index": old_index, "alias": "current_customers"}},
{"add": {"index": new_index, "alias": "current_customers"}}
]
}
es.indices.update_aliases(body=update_aliases_body)
# Verify the alias update
alias_info = es.indices.get_alias(name="current_customers")
if new_index in alias_info:
print("Alias updated successfully!")
else:
print("Alias update failed.")
This section of the code simultaneously removes the current_customers alias from the old customers index and adds it to the new customers_v2 index. Following this, it verifies that the update was successful.
4. Specific Cases
While the process we outlined works for most scenarios, let’s consider a few specific cases.
4.1. Time-Based Indices
When working with time-based indices, we might need to rename multiple indices. In this case, we can use the Reindex API with a wildcard in the source index:
# Get all indices matching the pattern
old_indices = es.indices.get(index="logs-2023-*")
for old_index in old_indices:
# Extract the date part from the old index name
date_part = old_index.split('-', 2)[-1]
new_index = f"new-logs-2023-{date_part}"
# Reindex data from old index to new index
reindex_body = {
"source": {
"index": old_index
},
"dest": {
"index": new_index
}
}
es.reindex(body=reindex_body, wait_for_completion=True)
To start, this code snippet retrieves all indices matching the pattern logs-2023-*. Then, for each matching index, it extracts the date part from the old index name and creates a new index name with a new- prefix.
4.2. Index Patterns in Kibana
If using Kibana, we also update any index patterns that reference the old index name.
We can do this through the Kibana UI or via the Kibana API:
import requests
kibana_url = "http://localhost:5601"
headers = {"kbn-xsrf": "true", "Content-Type": "application/json"}
pattern_body = {
"attributes": {
"title": "customers_v2*"
}
}
response = requests.put(f"{kibana_url}/api/saved_objects/index-pattern/pattern-id",
headers=headers, json=pattern_body)
if response.status_code == 200:
print("Kibana index pattern updated successfully!")
else:
print("Failed to update Kibana index pattern.")
Notably, we replace pattern-id with the actual ID of the index pattern.
5. Conclusion
In this article, we covered the entire process of renaming an ElasticSearch index.
We started by explaining the concept of index aliases. Then, we walked through each step of the renaming procedure.
In conclusion, index aliases can be a helpful tool for data organization and management purposes.