Implementing a Hot-Warm-Cold Architecture in Elasticsearch
A node in Elasticsearch can be categorized as “hot,” “warm,” or “cold” depending on its intended use case and hardware characteristics.
High-speed data ingestion and high query throughput are supported by hot nodes. Often, these nodes include powerful Processors, lots of memory, and high-performance storage like solid-state drives (SSDs). Hot nodes are perfect for real-time data processing and analysis since they are utilized to index new data and handle a lot of search requests.
Warm nodes are designed to handle data that is still searchable but is no longer being actively written to. To store data that isn’t used regularly, these nodes frequently include slower, higher-capacity storage, including hard disk drives (HDDs). Warm nodes are typically used to hold historical information or archives that are needed for analysis but may not be routinely searched.
For long-term data storage and retrieval, cold nodes are created. In contrast to hot and warm nodes, these nodes often have the lowest performance but the most storage capacity. Cold nodes are frequently used to store data that is rarely used but must still be accessible for compliance or regulatory purposes, like financial or medical information.
Users of Elasticsearch can tailor their cluster for their unique data processing and storage requirements by combining hot, warm, and cold nodes.
Elasticsearch’s hot-warm-cold architecture is implemented by deploying numerous nodes and customizing each one according to its intended use. The general procedures are as follows:
Hardware specifications
We must establish the appropriate CPU, memory, and storage parameters for each node based on the anticipated workload and the needs for data storage. Warm and cold nodes could be able to employ lesser performance but higher capacity hardware, but hot nodes will normally need the greatest performance hardware.
What is the better specification that my hardware must have? It depends.
- CPU: Elasticsearch is CPU-bound, so having a powerful CPU is crucial for handling high query throughput.
- Memory: Elasticsearch relies heavily on memory for caching frequently accessed data and speeding up queries.
- Storage: Fast solid-state drives (SSDs) are recommended for hot nodes. This will ensure that the node can handle high-speed data ingestion and indexing without experiencing disk I/O bottlenecks.
- Network: A fast network connection is recommended to handle the high volume of requests.
Shard Allocation Awareness
With the help of this functionality, you may specify how and under what circumstances index shards are distributed among various cluster nodes. You can decide how Elasticsearch distributes data among the cluster’s nodes depending on hardware characteristics, geographic location, or other considerations.
Shard distribution in Elasticsearch is by default random across all cluster nodes. This strategy might not be the best one, though, in some circumstances. To improve performance and lower network traffic, you might wish to distribute shards more carefully if your nodes have varying hardware requirements or are situated in various geographical areas.
You can set up Elasticsearch to allocate shards based on specific attributes or situations for cluster nodes by configuring it with shard allocation awareness. You may, for instance, provide CPU, RAM, disk type, location, or other attributes, and then configure Elasticsearch to assign shards to nodes that satisfy specific requirements based on these attributes.
Configure Shard Allocation Awareness in a Elasticsearch Cluster running on AWS
For example, you can use the AWS CLI using something like
aws ec2 create-tags — resources i-13274628736— tags Key=node.type,Value=hot
In the “Value” field, enter the value for the tag (e.g., “hot”, “warm”, or “cold”). It’s a good idea to configure the data tiering — configuring the nodes to use different storage tiers based on their role in the hot-warm-cold architecture. For example, you can configure hot and warm nodes to use SSDs for fast data ingestion and querying, while cold nodes can use slower HDDs.
ILM policy example
PUT _ilm/policy/hot_warm_cold
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_size": "50gb",
"max_age": "1d"
},
"set_priority": {
"priority": 100
}
}
},
"warm": {
"min_age": "1d",
"actions": {
"forcemerge": {
"max_num_segments": 1
},
"allocate": {
"require": {
"data": "warm"
}
},
"set_priority": {
"priority": 50
}
}
},
"cold": {
"min_age": "7d",
"actions": {
"freeze": {},
"allocate": {
"require": {
"data": "cold"
}
},
"set_priority": {
"priority": 0
}
}
},
"delete": {
"min_age": "30d",
"actions": {
"delete": {}
}
}
}
}
}This policy specifies the following phases:
- Hot phase: This phase rolls over the index when it reaches 50 GB or 1 day old (whichever comes first). It sets the index priority to 100, which means it will be assigned to the most powerful nodes available in the cluster.
- Warm phase: This phase starts after 1 day and applies a force merge to optimize the index. It then allocates the index to warm nodes and sets the index priority to 50.
- Cold phase: This phase starts after 7 days and freezes the index, preventing any changes from being made to it. It then allocates the index to cold nodes and sets the index priority to 0, which means it will be assigned to the least powerful nodes available in the cluster.
- Delete phase: This phase starts after 30 days and deletes the index.
Those are some tips that you can use to setup hot-warm-cold architecture on Elasticsearch in AWS. If you are using Elastic Cloud, you can define this architecture in the advanced settings of your nodes:
