Lucene:indexsearch.search()在非常大的数据库上导致java堆空间错误
我有一个非常大的数据库(大约3000万条记录,每个记录至少有26个字段),我用ApacheLuceneJava对其进行了索引 我正在从两个字段构造一个查询。每个搜索词可以出现在九个字段中的任何一个字段中,如果两个搜索词都出现在文档中的任何相关字段中,我希望我的查询返回一个文档。查询的结构如下所示:Lucene:indexsearch.search()在非常大的数据库上导致java堆空间错误,java,memory-management,lucene,Java,Memory Management,Lucene,我有一个非常大的数据库(大约3000万条记录,每个记录至少有26个字段),我用ApacheLuceneJava对其进行了索引 我正在从两个字段构造一个查询。每个搜索词可以出现在九个字段中的任何一个字段中,如果两个搜索词都出现在文档中的任何相关字段中,我希望我的查询返回一个文档。查询的结构如下所示: Private Query CreateQuery(String theSearchTerm, String theField) throws ParseException { Standar
Private Query CreateQuery(String theSearchTerm, String theField) throws ParseException
{
StandardAnalyzer theAnalyzer = new StandardAnalyzer(Version.LUCENE_35);
Query q;
QueryParser qp = new QueryParser(Version.LUCENE_35, theField, theAnalyzer);
qp.setDefaultOperator(QueryParser.Operator.AND);
qp.setAllowLeadingWildcard = true;
q = qp.parse(theSearchTerm);
return q;
}
Public ScoreDoc[] RunTheQuery(String searchTerm1, String searchTerm2)
{
Directory theIndex = new SimpleFSDirectory(new File("C:\\MyDirectory");
IndexSearcher theSearcher = new IndexSearcher(InderReader.open(theIndex));
BooleanQuery theTopLevelBooleanQuery = new BooleanQuery();
BooleanQuery fields1 = new BooleanQuery();
BooleanQuery fields2 = new BooleanQuery();
BooleanQuery fields3 = new BooleanQuery();
BooleanQuery fields4 = new BooleanQuery();
BooleanQuery fields5 = new BooleanQuery();
BooleanQuery fields6 = new BooleanQuery();
BooleanQuery fields7 = new BooleanQuery();
BooleanQuery fields8 = new BooleanQuery();
BooleanQuery fields9 = new BooleanQuery();
BooleanQuery innerQuery = new BooleanQuery();
fields1.add(CreateQuery(searchTerm1, param1), BooleanClause.Occur.MUST);
fields1.add(CreateQuery(searchTerm2, param2), BooleanClause.Occur.MUST);
fields2.add(CreateQuery(searchTerm1, param3), BooleanClause.Occur.MUST);
fields2.add(CreateQuery(searchTerm2, param4), BooleanClause.Occur.MUST);
fields3.add(CreateQuery(searchTerm1, param5), BooleanClause.Occur.MUST);
fields3.add(CreateQuery(searchTerm2, param6), BooleanClause.Occur.MUST);
fields4.add(CreateQuery(searchTerm1, param7), BooleanClause.Occur.MUST);
fields4.add(CreateQuery(searchTerm2, param8), BooleanClause.Occur.MUST);
fields5.add(CreateQuery(searchTerm1, param9), BooleanClause.Occur.MUST);
fields5.add(CreateQuery(searchTerm2, param10), BooleanClause.Occur.MUST);
fields6.add(CreateQuery(searchTerm1, param11), BooleanClause.Occur.MUST);
fields6.add(CreateQuery(searchTerm2, param12), BooleanClause.Occur.MUST);
fields7.add(CreateQuery(searchTerm1, param13), BooleanClause.Occur.MUST);
fields7.add(CreateQuery(searchTerm2, param14), BooleanClause.Occur.MUST);
fields8.add(CreateQuery(searchTerm1, param15), BooleanClause.Occur.MUST);
fields8.add(CreateQuery(searchTerm2, param16), BooleanClause.Occur.MUST);
fields9.add(CreateQuery(searchTerm1, param17), BooleanClause.Occur.MUST);
fields9.add(CreateQuery(searchTerm2, param18), BooleanClause.Occur.MUST);
innerQuery.add(fields1, BooleanClause.Occur.SHOULD);
innerQuery.add(fields2, BooleanClause.Occur.SHOULD);
innerQuery.add(fields3, BooleanClause.Occur.SHOULD);
innerQuery.add(fields4, BooleanClause.Occur.SHOULD);
innerQuery.add(fields5, BooleanClause.Occur.SHOULD);
innerQuery.add(fields6, BooleanClause.Occur.SHOULD);
innerQuery.add(fields7, BooleanClause.Occur.SHOULD);
innerQuery.add(fields8, BooleanClause.Occur.SHOULD);
innerQuery.add(fields9, BooleanClause.Occur.SHOULD);
theTopLevelBooleanQuery.add(innerQuery, BooleanClause.Occur.MUST);
TopDocScoreCollector collector = TopDocScoreCollector.create(200, true);
//Heap space error occurs here
theSearcher.search(theTopLevelBooleanQuery, collector);
ScoreDoc[] hits = collector.topDocs().scoreDocs;
return hits;
}
我的问题是,当我调用IndexSearcher.search()方法时,服务器(Windows server 2003 R2)上的java.exe进程消耗的内存超过540 MB,这会导致java堆空间错误。为了完整起见,java应用程序运行在一个web服务器上(目前是Oracle Glassfish,不过我希望迁移到ApacheTomcat)
有人知道如何停止这个堆空间错误吗?堆垛式溢流桩(http://stackoverflow.com/questions/7259736/cant-open-lucene-index-java-heap-space)似乎解决了类似的问题,但没有给出详细的答案
增加Java进程可以使用的内存量的唯一答案是什么?是写一个新的搜索者的唯一答案,在这种情况下,谁能推荐一篇关于轻量级搜索者的好文章
有没有办法通过修改上述代码来解决这个问题
任何帮助都将受到感激,
谢谢
Rik您可以这样增加java堆空间:
java -Xmx6g myprogram
或参阅以下帖子:
或:
您能否提供堆栈跟踪,以便我们了解是什么分配导致OutOfMemoryError?