- elasticsearch/
elasticsearch ElasticSearch-无法加载包含2.1 B以上文档的索引
elasticsearch ElasticSearch-无法加载包含2.1 B以上文档的索引
我停止了elasticSearch服务并重新启动了它。现在，无法加载其中一个索引。以下是索引的状态：
yellow open test-index _taPEhm3ReSwCYjb1Y5hQA 1 1 3 0 7.5kb 7.5kb
yellow open wiki-index-000001 aaZvlpgJSuO43uMGKuyqKw 1 1 0 0 208b 208b
yellow open wiki-index-unique s1tU2HpnStWNobkZjQ7KQA 1 1 118098273 51827014 36.8gb 36.8gb
yellow open corpora-index ruArLOJoSv6HkKVBv-o1MA 1 1 289045137 0 47.3gb 47.3gb
red open corpora -86nIPPwS8K5IOpFYNcXBQ 1 1
yellow open simple_bulk 6N8NCLd5S5qOKf9o6R6YhA 1 1 6 0 9.6kb 9.6kb
yellow open test1 SN8ViALMRNGHkBF7o8-3zw 1 1 2 1 5.3kb 5.3kb
yellow open simple-index zYctGNhNRGWrnCOYpKHBcQ 1 1 1 0 4.5kb 4.5kb
[2020-11-20T23:52:54,498][INFO ][o.e.i.s.IndexShard ] [ilcompn0] [corpora][0] ignoring recovery of a corrupt translog entry
java.lang.IllegalArgumentException: number of documents in the index cannot exceed 2147483519
at org.apache.lucene.index.DocumentsWriterPerThread.reserveOneDoc(DocumentsWriterPerThread.java:211) ~[lucene-core-8.6.2.jar:8.6.2 016993b65e393b58246d54e8dd$
at org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:232) ~[lucene-core-8.6.2.jar:8.6.2 016993b65e393b58246d54e8$
at org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:419) ~[lucene-core-8.6.2.jar:8.6.2 016993b65e393b58246d54e8ddda9f56a453eb0e -$
at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1333) ~[lucene-core-8.6.2.jar:8.6.2 016993b65e393b58246d54e8ddda9f56a453eb0e - ivera $
at org.apache.lucene.index.IndexWriter.softUpdateDocument(IndexWriter.java:1661) ~[lucene-core-8.6.2.jar:8.6.2 016993b65e393b58246d54e8ddda9f56a453eb0e - ive$
at org.elasticsearch.index.engine.InternalEngine.updateDocs(InternalEngine.java:1260) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1091) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:935) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:819) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:791) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1526) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.index.shard.IndexShard.runTranslogRecovery(IndexShard.java:1557) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.index.shard.IndexShard.lambda$openEngineAndRecoverFromTranslog$9(IndexShard.java:1605) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.index.engine.InternalEngine.recoverFromTranslogInternal(InternalEngine.java:488) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.index.engine.InternalEngine.recoverFromTranslog(InternalEngine.java:463) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.index.engine.InternalEngine.recoverFromTranslog(InternalEngine.java:125) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1610) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:436) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:98) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:325) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:96) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1883) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:710) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.9.1.jar:7.9.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
bin/elastisearch-shard remove-corrupted-data --index corpora --shard-id 0 --truncate-clean-translog
bin/elastisearch-shard remove-corrupted-data --index corpora --shard-id 0 --truncate-clean-translog
您将丢失所有尚未写入索引的挂起操作，但至少可以打开索引并将其拆分。您运行的是哪个版本？碎片不能包含多个Integer.MAX文档。在最新版本中，他们添加了一个检查，如果超过该限制，则拒绝写入文档。但是，在您的情况下，您的版本似乎没有进行该检查，因此允许您超出限制。@val：我使用的是7.9.1版。那么，有没有办法从索引中删除一些文档，以便我可以恢复其余的索引文档？您只有一个碎片？索引的大小是多少？@dadoonet:是的，我只有一个碎片，其大小约为700GB。@Val:这是另一个名为“语料库索引”的索引的大小，只有280m个文档。“corpora”索引包含超过2.1b的文档，其大小约为700GB。您正在运行哪个版本？碎片不能包含多个Integer.MAX文档。在最新版本中，他们添加了一个检查，如果超过该限制，则拒绝写入文档。但是，在您的情况下，您的版本似乎没有进行该检查，因此允许您超出限制。@val：我使用的是7.9.1版。那么，有没有办法从索引中删除一些文档，以便我可以恢复其余的索引文档？您只有一个碎片？索引的大小是多少？@dadoonet:是的，我只有一个碎片，其大小约为700GB。@Val:这是另一个名为“语料库索引”的索引的大小，只有280m个文档。“corpora”索引包含超过2.1b的文档，其大小约为700GB。看来应该行得通。我的问题是，要运行此命令，我需要通过
elasticsearchelasticsearchelasticsearchrunuser-leasticsearch-c'…命令…'elasticsearchelasticsearchelasticsearchrunuser-leasticsearch-c'…命令…'