Indexing 将数据从Nutch传递到Solr_Indexing_Solr_Web Crawler_Nutch

Indexing 将数据从Nutch传递到Solr

indexing solr web-crawler

Indexing 将数据从Nutch传递到Solr,indexing,solr,web-crawler,nutch,Indexing,Solr,Web Crawler,Nutch,我正在尝试使用以下命令将Nutch web cralwer爬网的数据传递到Solr搜索和索引平台： bin/nutch index -Dsolr.server.url=http://localhost:8983/solr crawl/crawldb/ -linkdb crawl/linkdb/ crawl/segments/ -dir crawl/segments/20161124145935/ crawl/segments/20161124150145/ -filter -normalize

我正在尝试使用以下命令将Nutch web cralwer爬网的数据传递到Solr搜索和索引平台：

bin/nutch index -Dsolr.server.url=http://localhost:8983/solr crawl/crawldb/ -linkdb crawl/linkdb/ crawl/segments/ -dir crawl/segments/20161124145935/ crawl/segments/20161124150145/ -filter -normalize

但我得到了以下错误：

The input path at segments is not a segment... skipping
The input path at content is not a segment... skipping
The input path at crawl_fetch is not a segment... skipping
Skipping segment: file:/Users/cell/Desktop/usi/information-retrieval/project/apache-nutch-1.12/crawl/segments/20161124145935/crawl_generate. Missing sub directories: parse_data, parse_text, crawl_parse, crawl_fetch
The input path at crawl_parse is not a segment... skipping
The input path at parse_data is not a segment... skipping
The input path at parse_text is not a segment... skipping
Segment dir is complete: crawl/segments/20161124150145.
Indexer: starting at 2016-11-25 05:02:17
Indexer: deleting gone documents: false
Indexer: URL filtering: true
Indexer: URL normalizing: true
Active IndexWriters :
SOLRIndexWriter
    solr.server.url : URL of the SOLR instance
    solr.zookeeper.hosts : URL of the Zookeeper quorum
    solr.commit.size : buffer size when sending to SOLR (default 1000)
    solr.mapping.file : name of the mapping file for fields (default solrindex-mapping.xml)
    solr.auth : use authentication (default false)
    solr.auth.username : username for authentication
    solr.auth.password : password for authentication


Indexing 250/250 documents
Deleting 0 documents
Indexing 250/250 documents
Deleting 0 documents
Indexer: java.io.IOException: Job failed!
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:836)
    at org.apache.nutch.indexer.IndexingJob.index(IndexingJob.java:145)
    at org.apache.nutch.indexer.IndexingJob.run(IndexingJob.java:228)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.nutch.indexer.IndexingJob.main(IndexingJob.java:237)

这是Nutch的日志：

2016-11-25 06:05:03,378 WARN  util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-11-25 06:05:03,500 WARN  segment.SegmentChecker - The input path at segments is not a segment... skipping
2016-11-25 06:05:03,506 WARN  segment.SegmentChecker - The input path at content is not a segment... skipping
2016-11-25 06:05:03,506 WARN  segment.SegmentChecker - The input path at crawl_fetch is not a segment... skipping
2016-11-25 06:05:03,507 WARN  segment.SegmentChecker - Skipping segment: file:/Users/cell/Desktop/usi/information-retrieval/project/apache-nutch-1.12/crawl/segments/20161124145935/crawl_generate. Missing sub directories: parse_data, parse_text, crawl_parse, crawl_fetch
2016-11-25 06:05:03,507 WARN  segment.SegmentChecker - The input path at crawl_parse is not a segment... skipping
2016-11-25 06:05:03,507 WARN  segment.SegmentChecker - The input path at parse_data is not a segment... skipping
2016-11-25 06:05:03,507 WARN  segment.SegmentChecker - The input path at parse_text is not a segment... skipping
2016-11-25 06:05:03,509 INFO  segment.SegmentChecker - Segment dir is complete: crawl/segments/20161124150145.
2016-11-25 06:05:03,510 INFO  indexer.IndexingJob - Indexer: starting at 2016-11-25 06:05:03
2016-11-25 06:05:03,512 INFO  indexer.IndexingJob - Indexer: deleting gone documents: false
2016-11-25 06:05:03,512 INFO  indexer.IndexingJob - Indexer: URL filtering: true
2016-11-25 06:05:03,512 INFO  indexer.IndexingJob - Indexer: URL normalizing: true
2016-11-25 06:05:03,614 INFO  indexer.IndexWriters - Adding org.apache.nutch.indexwriter.solr.SolrIndexWriter
2016-11-25 06:05:03,615 INFO  indexer.IndexingJob - Active IndexWriters :
SOLRIndexWriter
    solr.server.url : URL of the SOLR instance
    solr.zookeeper.hosts : URL of the Zookeeper quorum
    solr.commit.size : buffer size when sending to SOLR (default 1000)
    solr.mapping.file : name of the mapping file for fields (default solrindex-mapping.xml)
    solr.auth : use authentication (default false)
    solr.auth.username : username for authentication
    solr.auth.password : password for authentication


2016-11-25 06:05:03,616 INFO  indexer.IndexerMapReduce - IndexerMapReduce: crawldb: crawl/crawldb
2016-11-25 06:05:03,616 INFO  indexer.IndexerMapReduce - IndexerMapReduce: linkdb: crawl/linkdb
2016-11-25 06:05:03,617 INFO  indexer.IndexerMapReduce - IndexerMapReduces: adding segment: crawl/segments/20161124150145
2016-11-25 06:05:04,006 WARN  conf.Configuration - file:/tmp/hadoop-cell/mapred/staging/cell1463380038/.staging/job_local1463380038_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2016-11-25 06:05:04,010 WARN  conf.Configuration - file:/tmp/hadoop-cell/mapred/staging/cell1463380038/.staging/job_local1463380038_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2016-11-25 06:05:04,088 WARN  conf.Configuration - file:/tmp/hadoop-cell/mapred/local/localRunner/cell/job_local1463380038_0001/job_local1463380038_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2016-11-25 06:05:04,090 WARN  conf.Configuration - file:/tmp/hadoop-cell/mapred/local/localRunner/cell/job_local1463380038_0001/job_local1463380038_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2016-11-25 06:05:04,258 INFO  anchor.AnchorIndexingFilter - Anchor deduplication is: off
2016-11-25 06:05:04,272 INFO  regex.RegexURLNormalizer - can't find rules for scope 'indexer', using default
2016-11-25 06:05:08,950 INFO  regex.RegexURLNormalizer - can't find rules for scope 'indexer', using default
2016-11-25 06:05:09,344 INFO  regex.RegexURLNormalizer - can't find rules for scope 'indexer', using default
2016-11-25 06:05:09,734 INFO  regex.RegexURLNormalizer - can't find rules for scope 'indexer', using default
2016-11-25 06:05:10,908 INFO  regex.RegexURLNormalizer - can't find rules for scope 'indexer', using default
2016-11-25 06:05:11,376 INFO  regex.RegexURLNormalizer - can't find rules for scope 'indexer', using default
2016-11-25 06:05:11,686 INFO  indexer.IndexWriters - Adding org.apache.nutch.indexwriter.solr.SolrIndexWriter
2016-11-25 06:05:11,775 INFO  solr.SolrMappingReader - source: content dest: content
2016-11-25 06:05:11,775 INFO  solr.SolrMappingReader - source: title dest: title
2016-11-25 06:05:11,775 INFO  solr.SolrMappingReader - source: host dest: host
2016-11-25 06:05:11,775 INFO  solr.SolrMappingReader - source: segment dest: segment
2016-11-25 06:05:11,775 INFO  solr.SolrMappingReader - source: boost dest: boost
2016-11-25 06:05:11,775 INFO  solr.SolrMappingReader - source: digest dest: digest
2016-11-25 06:05:11,775 INFO  solr.SolrMappingReader - source: tstamp dest: tstamp
2016-11-25 06:05:11,940 INFO  solr.SolrIndexWriter - Indexing 250/250 documents
2016-11-25 06:05:11,940 INFO  solr.SolrIndexWriter - Deleting 0 documents
2016-11-25 06:05:12,139 INFO  solr.SolrIndexWriter - Indexing 250/250 documents
2016-11-25 06:05:12,139 INFO  solr.SolrIndexWriter - Deleting 0 documents
2016-11-25 06:05:12,207 WARN  mapred.LocalJobRunner - job_local1463380038_0001
java.lang.Exception: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://localhost:8983/solr: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Not Found</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/update. Reason:
<pre>    Not Found</pre></p>
</body>
</html>

    at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:529)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://localhost:8983/solr: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Not Found</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/update. Reason:
<pre>    Not Found</pre></p>
</body>
</html>

    at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:543)
    at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
    at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
    at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)
    at org.apache.nutch.indexwriter.solr.SolrIndexWriter.push(SolrIndexWriter.java:209)
    at org.apache.nutch.indexwriter.solr.SolrIndexWriter.write(SolrIndexWriter.java:173)
    at org.apache.nutch.indexer.IndexWriters.write(IndexWriters.java:85)
    at org.apache.nutch.indexer.IndexerOutputFormat$1.write(IndexerOutputFormat.java:50)
    at org.apache.nutch.indexer.IndexerOutputFormat$1.write(IndexerOutputFormat.java:41)
    at org.apache.hadoop.mapred.ReduceTask$OldTrackingRecordWriter.write(ReduceTask.java:493)
    at org.apache.hadoop.mapred.ReduceTask$3.collect(ReduceTask.java:422)
    at org.apache.nutch.indexer.IndexerMapReduce.reduce(IndexerMapReduce.java:367)
    at org.apache.nutch.indexer.IndexerMapReduce.reduce(IndexerMapReduce.java:56)
    at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
    at org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:319)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2016-11-25 06:05:12,293 ERROR indexer.IndexingJob - Indexer: java.io.IOException: Job failed!
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:836)
    at org.apache.nutch.indexer.IndexingJob.index(IndexingJob.java:145)
    at org.apache.nutch.indexer.IndexingJob.run(IndexingJob.java:228)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.nutch.indexer.IndexingJob.main(IndexingJob.java:237)

2016-11-25 06:05:03378警告util.NativeCodeLoader-无法为您的平台加载本机hadoop库。。。在适用的情况下使用内置java类
2016-11-25 06:05:03500警告段。段检查器-段处的输入路径不是段。。。跳过
2016-11-25 06:05:03506警告segment.SegmentChecker-内容处的输入路径不是段。。。跳过
2016-11-25 06:05:03506 WARN segment.SegmentChecker-抓取时的输入路径不是段。。。跳过
2016-11-25 06:05:03507 WARN segment.SegmentChecker-跳过段：文件：/Users/cell/Desktop/usi/information retrieval/project/apache-nutch-1.12/crawl/segments/20161124145935/crawl\u生成。缺少子目录：解析数据、解析文本、爬网解析、爬网获取
2016-11-25 06:05:03507 WARN segment.SegmentChecker-爬网解析处的输入路径不是段。。。跳过
2016-11-25 06:05:03507警告段.SegmentChecker-parse_数据处的输入路径不是段。。。跳过
2016-11-25 06:05:03507 WARN segment.SegmentChecker-parse_text处的输入路径不是段。。。跳过
2016-11-25 06:05:03509信息段。段检查器-段目录已完成：爬网/段/20161124150145。
2016-11-25 06:05:03510信息索引器。索引作业-索引器：从2016-11-25 06:05:03开始
2016-11-25 06:05:03512信息索引器。索引作业-索引器：删除丢失的文档：false
2016-11-25 06:05:03512信息索引器。索引作业-索引器：URL筛选：true
2016-11-25 06:05:03512信息索引器。索引作业-索引器：URL规范化：true
2016-11-25 06:05:03614 INFO indexer.IndexWriters-添加org.apache.nutch.indexwriter.solr.SolrIndexWriter
2016-11-25 06:05:03615信息索引器。索引作业-活动索引编写器：
SOLRIndexWriter
solr.server.url:solr实例的url
solr.zookeer.hosts:zookeer仲裁的URL
solr.commit.size：发送到solr时的缓冲区大小（默认为1000）
solr.mapping.file：字段映射文件的名称（默认为solrindex mapping.xml）
solr.auth：使用身份验证（默认为false）
solr.auth.username：用于身份验证的用户名
solr.auth.password：用于身份验证的密码
2016-11-25 06:05:03616信息索引器。索引器MapReduce-索引器MapReduce:crawldb:crawl/crawldb
2016-11-25 06:05:03616信息索引器。索引器MapReduce-索引器MapReduce:linkdb:crawl/linkdb
2016-11-25 06:05:03617信息索引器。索引器MapReduce-索引器MapReduces：添加段：爬网/段/20161124150145
2016-11-25 06:05:04006警告配置文件：/tmp/hadoop cell/mapred/staging/cell146338038/.staging/job\u local146338038\u 0001/job.xml：试图覆盖最终参数：mapreduce.job.end-notification.max.retry.interval；忽略。
2016-11-25 06:05:04010警告配置文件：/tmp/hadoop cell/mapred/staging/cell146338038/.staging/job\u local146338038\u 0001/job.xml：试图覆盖最终参数：mapreduce.job.end-notification.max.truments；忽略。
2016-11-25 06:05:04088警告配置-文件：/tmp/hadoop cell/mapred/local/localRunner/cell/job\u local146338038\u 0001/job\u local146338038\u 0001.xml：试图覆盖最终参数：mapreduce.job.end-notification.max.retry.interval；忽略。
2016-11-25 06:05:04090警告配置文件：/tmp/hadoop cell/mapred/local/localRunner/cell/job\u local146338038\u 0001/job\u local146338038\u 0001.xml：试图覆盖最终参数：mapreduce.job.end-notification.max.truments；忽略。
2016-11-25 06:05:04258 INFO anchor.AnchorIndexingFilter-锚点重复数据消除是：关闭
2016-11-25 06:05:04272 INFO regex.RegexURLNormalizer-使用默认值找不到作用域“索引器”的规则
2016-11-25 06:05:08950 INFO regex.RegexURLNormalizer-使用默认值找不到作用域“索引器”的规则
2016-11-25 06:05:09344信息regex.RegexURLNormalizer-使用默认值找不到作用域“索引器”的规则
2016-11-25 06:05:09734 INFO regex.RegexURLNormalizer-使用默认值找不到作用域“索引器”的规则
2016-11-25 06:05:10908 INFO regex.RegexURLNormalizer-使用默认值找不到作用域“索引器”的规则
2016-11-25 06:05:11376 INFO regex.RegexURLNormalizer-使用默认值找不到作用域“索引器”的规则
2016-11-25 06:05:11686 INFO indexer.IndexWriters-添加org.apache.nutch.indexwriter.solr.SolrIndexWriter
2016-11-25 06:05:11775信息solr.SolrMappingReader-来源：内容目的：内容
2016-11-25 06:05:11775信息solr.SolrMappingReader-来源：标题目的：标题
2016-11-25 06:05:11775信息solr.SolrMappingReader-来源：主机目的地：主机
2016-11-25 06:05:11775信息solr.SolrMappingReader-来源：段目的地：段
2016-11-25 06:05:11775信息solr.SolrMappingReader-来源：boost dest:boost
2016-11-25 06:05:11775信息solr.SolrMappingReader-来源：摘要目的：摘要
2016-11-25 06:05:11775信息solr.SolrMappingReader-来源：tstamp目的地：tstamp
2016-11-25 06:05:11940信息solr.SolrIndexWriter-索引250/250个文档
2016-11-25 06:05:11940信息solr.SolrIndexWriter-删除0个文档
2016-11-25 06:05:12139信息solr.SolrIndexWriter-索引250/250个文档
2016-11-25 06:05:12139信息solr.SolrIndexWriter-删除0个文档
2016-11-25 06:05:12207 WARN mapred.LocalJobRunner-job_local146338038_0001
java.lang.Exception:org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:服务器上的错误http://localhost:8983/solr: 应为mime类型的应用程序/八位字节流，但得到文本/html。
找不到错误404
HTTP错误404
访问/solr/update时出现问题。原因:
找不到
位于org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks（LocalJobRunner.java:462）
在org.apache.hadoop.mapred.LocalJobRunner$J上