last day (14 days later) » 

3:08 PM
0
Q: Elasticsearch - Deciding number of nodes/shards based on heap usage

Utkarsh MishraWe have 2 major indices and both are updated/incremented with high frequency. One of the index (product_index_v10) undergoes bulk update every midnight, updating 3-4 fields for all documents. Index Doc count Pri Rep Store.size Pri.store.size aggregations 18708399 5 0 49.2...

 
Elasticsearch 1.5.2
 
Where do you have those stats numbers from? I find it odd to see Completion and Segments with same values. Something must be wrong: if your heap size is 16GB for the first node just adding up Completion and Segments and you are already over 16GB.
 
From the _nodes/stats API, while heap_committed: "15.9gb"
@AndreiStefan: Is it possible that segments are using swap memory ?
 
No. Provide the _nodes/stats api output in a gist, please.
 
@AndreiStefan Here
 
3:08 PM
First of all, if you only have those two indices, I would configure them to so that each has 6 primary shards. So that the data and load is spread evenly on all nodes. Secondly, how are you doing your bulk updates? Have you tested with different bulk batch sizes? Also, if you are updating all documents, it's far better to delete the index and create a new one with the new values.
 
@AndreiStefan: Agree with your sharding advice. Plus, our bulk update is a scheduled script sending batches of 250 docs with a sleep time of 1.25s between each batch hit. We've tried multiple variations of batch sizes and sleep time and this works properly for now. Also, one clarification. We do not update all the fields in each documents daily, and instead we update only 4-5 fields per doc in that index.
 
The dangerous part is updating all documents. It doesn't matter how many fields you update, but that you are updating all documents. Don't do that! Create a new index, index updated documents in this new index, then delete the old index (like DELETE /product_index_v10). And use aliases to switch between the old product_index_v10 index and the new product_index_v10_new index.
well?
 
@AndreiStefan Full data reindex would be pretty time taking. We've had data corruption incidents in the past, and full data reindex took multiple days since we have to maintain uptime and stable response time. Updating complete docs is like update 80% more than what we do right now.
 
you said in the post that you are updating all the documents
you are or not?
 
Will add a correction there, only few fields in all documents
I actually have that mentioned already :)
 
4:02 PM
@AndreiStefan ?
Let me know if you need more info.
 
so you are updating all documents
it's better to reindex all the docs than updating all the docs
elasticsearch is anyway deleting and inserting the document when there is an update
 
Okay. Anything else I should change at infra/config level ?
If there is a chance to have more nodes in future, should we change number of shards only when node is added or any scale ready shard count can be achieved ?
 
4:35 PM
It depends on how easy is to reindex all the data. Changing the number of shards requires reindex. If you go with delete+reindex for existing data then you can add a node and then reindex using the new shards count. Otherwise you need to think ahead a bit and it depends in the future plans.
 
Exactly. The problem with reindexing all data in our case is that we fetch all data from MySQL db, which uses multiple joins, its going to take a lot more time than the expected SLA. Even if we use scan and scroll to reindex with updated data, the time required would be much more as far as I know. Should we come up with a faster reindexing code ?
 
5:06 PM
To see what's using memory, look at the memory usage during bulk update. Oreven taking a heap dump and analyzing it.
 

last day (14 days later) »