Solr FileListenityProcessor+；java堆异常

我试图从文件系统中索引大量数据，数据大小只有几TB。我正在与TikaEntityProcessor一起使用FileListenityProcessor

data-config.xml

<dataConfig>
    <dataSource name="bin" type="BinFileDataSource" />
    <document>
        <entity name="f" dataSource="null" rootEntity="true"
            processor="FileListEntityProcessor" transformer="TemplateTransformer"
            baseDir="//mathworks/devel/bat/A/logs/66048/"
            fileName=".*\.*" onError="skip" recursive="true">

            <field column="fileAbsolutePath" name="path" />
            <field column="fileSize" name="size"/>
            <field column="fileLastModified" name="lastmodified" />

            <entity name="file" dataSource="bin" processor="TikaEntityProcessor" url="${f.fileAbsolutePath}" format="text" onError="skip"
           rootEntity="true">
                <field column="text" name="text"/>   
            </entity>
        </entity>
    </document>
</dataConfig>

非常感谢。

---

Prerna

您能看到索引过程在哪个特定文档上失败吗？你能发布文件类型（PDF，Word，…）和文件大小吗？最好的办法是，如果你能把医生放在某个地方。我想重现这个问题。我正在尝试为所有文本文件的日志文件编制索引。我没有看到任何特定的文档无法完成此过程。一个半小时后，它只显示异常。我编辑了这个问题以提供更多信息。

INFO: [tika] webapp=null path=null params={event=newSearcher&q=static+newSearcher+warming+query+from+solrconfig.xml} hits=872 status=0 QTime=1
Oct 8, 2013 6:04:27 PM org.apache.solr.core.QuerySenderListener newSearcher
INFO: QuerySenderListener done.
Oct 8, 2013 6:04:27 PM org.apache.solr.core.SolrCore registerSearcher
INFO: [tika] Registered new searcher Searcher@15cab4c0 main
Oct 8, 2013 6:04:27 PM org.apache.solr.search.SolrIndexSearcher close
INFO: Closing Searcher@67071180 main
        fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
        filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
        queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=3,evictions=0,size=3,warmupTime=5,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
        documentCache{lookups=0,hits=0,hitratio=0.00,inserts=10,evictions=0,size=10,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Oct 8, 2013 6:04:33 PM org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: {deleteByQuery=*:*,add=[(null), (null), (null), (null), (null), (null), (null), (null), ... (20157 adds)]} 0 4
Oct 8, 2013 6:04:33 PM org.apache.solr.common.SolrException log
SEVERE: Full Import failed:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.solr.handler.dataimport.DataImportHandlerException: java.lang.OutOfMemoryError: Java heap space
        at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:264)
        at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:375)
        at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:445)
        at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:426)
Caused by: java.lang.RuntimeException: org.apache.solr.handler.dataimport.DataImportHandlerException: java.lang.OutOfMemoryError: Java heap space
        at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:621)
        at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:327)
        at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:225)
        ... 3 more
Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException: java.lang.OutOfMemoryError: Java heap space
        at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:759)
        at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:619)
        ... 5 more
Caused by: java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Unknown Source)
        at java.lang.AbstractStringBuilder.expandCapacity(Unknown Source)
        at java.lang.AbstractStringBuilder.append(Unknown Source)
        at java.lang.StringBuilder.append(Unknown Source)
        at org.apache.solr.common.SolrInputField.toString(SolrInputField.java:200)
        at java.lang.String.valueOf(Unknown Source)
        at java.lang.StringBuilder.append(Unknown Source)
        at java.util.AbstractMap.toString(Unknown Source)
        at java.lang.String.valueOf(Unknown Source)
        at java.lang.StringBuilder.append(Unknown Source)
        at org.apache.solr.common.SolrInputDocument.toString(SolrInputDocument.java:182)
        at java.lang.String.valueOf(Unknown Source)
        at java.lang.StringBuilder.append(Unknown Source)
        at org.apache.solr.handler.dataimport.SolrWriter.upload(SolrWriter.java:68)
        at org.apache.solr.handler.dataimport.DataImportHandler$1.upload(DataImportHandler.java:293)
        at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:723)
        ... 6 more

Oct 8, 2013 6:04:33 PM org.apache.solr.update.DirectUpdateHandler2 rollback
INFO: start rollback
Oct 8, 2013 6:04:33 PM org.apache.solr.update.DirectUpdateHandler2 rollback
INFO: end_rollback