在Cloudera SolrCloud中加载solr配置

我们尝试使用MapReduce批量索引将数据导入SolrCloud。我们在reduce阶段面临一个问题，即无法找到solr.xml。我们创建了一个“twitter”集合，但查看日志，在solr.xml中加载失败后，它使用默认集合并尝试创建“collection1”失败和“core1”成功SolrCore。我不确定我们是否需要创建自己的solr.xml以及将它放在哪里，我们尝试将它放在几个地方，但似乎没有加载。以下是日志：

2022 [main] INFO  org.apache.solr.hadoop.HeartBeater  - Heart beat reporting class is org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
2025 [main] INFO  org.apache.solr.hadoop.SolrRecordWriter  - Using this unpacked directory as solr home: /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip
2025 [main] INFO  org.apache.solr.hadoop.SolrRecordWriter  - Creating embedded Solr server with solrHomeDir: /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip, fs: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-1828461666_1, ugi=nguyen (auth:SIMPLE)]], outputShardDir: hdfs://master.hadoop:8020/user/nguyen/twitter/outdir/reducers/_temporary/_attempt_201311191613_0320_r_000014_0/part-r-00014
2029 [Thread-64] INFO  org.apache.solr.hadoop.HeartBeater  - HeartBeat thread running
2030 [Thread-64] INFO  org.apache.solr.hadoop.HeartBeater  - Issuing heart beat for 1 threads
2083 [main] INFO  org.apache.solr.core.SolrResourceLoader  - new SolrResourceLoader for directory: '/data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/'
2259 [main] INFO  org.apache.solr.hadoop.SolrRecordWriter  - Constructed instance information solr.home /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip (/data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip), instance dir /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/, conf dir /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/conf/, writing index to solr.data.dir hdfs://master.hadoop:8020/user/nguyen/twitter/outdir/reducers/_temporary/_attempt_201311191613_0320_r_000014_0/part-r-00014/data, with permdir hdfs://master.hadoop:8020/user/nguyen/twitter/outdir/reducers/_temporary/_attempt_201311191613_0320_r_000014_0/part-r-00014
2266 [main] INFO  org.apache.solr.core.ConfigSolr  - Loading container configuration from /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/solr.xml
2267 [main] INFO  org.apache.solr.core.ConfigSolr  - /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/solr.xml does not exist, using default configuration
2505 [main] INFO  org.apache.solr.core.CoreContainer  - New CoreContainer 696103669
2505 [main] INFO  org.apache.solr.core.CoreContainer  - Loading cores into CoreContainer [instanceDir=/data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/]
2515 [main] INFO  org.apache.solr.handler.component.HttpShardHandlerFactory  - Setting socketTimeout to: 0
2515 [main] INFO  org.apache.solr.handler.component.HttpShardHandlerFactory  - Setting urlScheme to: http://
2515 [main] INFO  org.apache.solr.handler.component.HttpShardHandlerFactory  - Setting connTimeout to: 0
2515 [main] INFO  org.apache.solr.handler.component.HttpShardHandlerFactory  - Setting maxConnectionsPerHost to: 20
2516 [main] INFO  org.apache.solr.handler.component.HttpShardHandlerFactory  - Setting corePoolSize to: 0
2516 [main] INFO  org.apache.solr.handler.component.HttpShardHandlerFactory  - Setting maximumPoolSize to: 2147483647
2516 [main] INFO  org.apache.solr.handler.component.HttpShardHandlerFactory  - Setting maxThreadIdleTime to: 5
2516 [main] INFO  org.apache.solr.handler.component.HttpShardHandlerFactory  - Setting sizeOfQueue to: -1
2516 [main] INFO  org.apache.solr.handler.component.HttpShardHandlerFactory  - Setting fairnessPolicy to: false
2527 [main] INFO  org.apache.solr.client.solrj.impl.HttpClientUtil  - Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
2648 [main] INFO  org.apache.solr.logging.LogWatcher  - Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
2676 [coreLoadExecutor-3-thread-1] INFO  org.apache.solr.core.CoreContainer  - Creating SolrCore 'collection1' using instanceDir: /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/collection1
2677 [coreLoadExecutor-3-thread-1] INFO  org.apache.solr.core.SolrResourceLoader  - new SolrResourceLoader for directory: '/data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/collection1/'
2691 [coreLoadExecutor-3-thread-1] ERROR org.apache.solr.core.CoreContainer  - Failed to load file /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/collection1/solrconfig.xml
2693 [coreLoadExecutor-3-thread-1] ERROR org.apache.solr.core.CoreContainer  - Unable to create core: collection1
org.apache.solr.common.SolrException: Could not load config for solrconfig.xml
    at org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:596)
    at org.apache.solr.core.CoreContainer.create(CoreContainer.java:661)
    at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:368)
    at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:360)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:724)
Caused by: java.io.IOException: Can't find resource 'solrconfig.xml' in classpath or '/data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/collection1/conf/', cwd=/data/05/mapred/local/taskTracker/nguyen/jobcache/job_201311191613_0320/attempt_201311191613_0320_r_000014_0/work
    at org.apache.solr.core.SolrResourceLoader.openResource(SolrResourceLoader.java:322)
    at org.apache.solr.core.SolrResourceLoader.openConfig(SolrResourceLoader.java:287)
    at org.apache.solr.core.Config.<init>(Config.java:116)
    at org.apache.solr.core.Config.<init>(Config.java:86)
    at org.apache.solr.core.SolrConfig.<init>(SolrConfig.java:120)
    at org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:593)
    ... 11 more
2695 [coreLoadExecutor-3-thread-1] ERROR org.apache.solr.core.CoreContainer  - null:org.apache.solr.common.SolrException: Unable to create core: collection1
    at org.apache.solr.core.CoreContainer.recordAndThrow(CoreContainer.java:1158)
    at org.apache.solr.core.CoreContainer.create(CoreContainer.java:670)
    at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:368)
    at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:360)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:724)
Caused by: org.apache.solr.common.SolrException: Could not load config for solrconfig.xml
    at org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:596)
    at org.apache.solr.core.CoreContainer.create(CoreContainer.java:661)
    ... 10 more
Caused by: java.io.IOException: Can't find resource 'solrconfig.xml' in classpath or '/data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/collection1/conf/', cwd=/data/05/mapred/local/taskTracker/nguyen/jobcache/job_201311191613_0320/attempt_201311191613_0320_r_000014_0/work
    at org.apache.solr.core.SolrResourceLoader.openResource(SolrResourceLoader.java:322)
    at org.apache.solr.core.SolrResourceLoader.openConfig(SolrResourceLoader.java:287)
    at org.apache.solr.core.Config.<init>(Config.java:116)
    at org.apache.solr.core.Config.<init>(Config.java:86)
    at org.apache.solr.core.SolrConfig.<init>(SolrConfig.java:120)
    at org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:593)
    ... 11 more

2697 [main] INFO  org.apache.solr.core.CoreContainer  - Creating SolrCore 'core1' using instanceDir: /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip
2697 [main] INFO  org.apache.solr.core.SolrResourceLoader  - new SolrResourceLoader for directory: '/data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/'
2751 [main] INFO  org.apache.solr.core.SolrConfig  - Adding specified lib dirs to ClassLoader
2752 [main] WARN  org.apache.solr.core.SolrResourceLoader  - Can't find (or read) directory to add to classloader: ../../../contrib/extraction/lib (resolved as: /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/../../../contrib/extraction/lib).
2752 [main] WARN  org.apache.solr.core.SolrResourceLoader  - Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/../../../dist).
2752 [main] WARN  org.apache.solr.core.SolrResourceLoader  - Can't find (or read) directory to add to classloader: ../../../contrib/clustering/lib/ (resolved as: /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/../../../contrib/clustering/lib).
2753 [main] WARN  org.apache.solr.core.SolrResourceLoader  - Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/../../../dist).
2753 [main] WARN  org.apache.solr.core.SolrResourceLoader  - Can't find (or read) directory to add to classloader: ../../../contrib/langid/lib/ (resolved as: /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/../../../contrib/langid/lib).
2753 [main] WARN  org.apache.solr.core.SolrResourceLoader  - Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/../../../dist).
2753 [main] WARN  org.apache.solr.core.SolrResourceLoader  - Can't find (or read) directory to add to classloader: ../../../contrib/velocity/lib (resolved as: /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/../../../contrib/velocity/lib).
2753 [main] WARN  org.apache.solr.core.SolrResourceLoader  - Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/../../../dist).
2785 [main] INFO  org.apache.solr.update.SolrIndexConfig  - IndexWriter infoStream solr logging is enabled
2790 [main] INFO  org.apache.solr.core.SolrConfig  - Using Lucene MatchVersion: LUCENE_44
2869 [main] INFO  org.apache.solr.core.Config  - Loaded SolrConfig: solrconfig.xml
2879 [main] INFO  org.apache.solr.schema.IndexSchema  - Reading Solr Schema from schema.xml
2937 [main] INFO  org.apache.solr.schema.IndexSchema  - [core1] Schema name=twitter
3352 [main] INFO  org.apache.solr.schema.IndexSchema  - unique key field: id
3471 [main] INFO  org.apache.solr.schema.FileExchangeRateProvider  - Reloading exchange rates from file currency.xml
3478 [main] INFO  org.apache.solr.schema.FileExchangeRateProvider  - Reloading exchange rates from file currency.xml
3635 [main] INFO  org.apache.solr.core.HdfsDirectoryFactory  - Solr Kerberos Authentication disabled
3636 [main] INFO  org.apache.solr.core.JmxMonitoredMap  - No JMX servers found, not exposing Solr information with JMX.
3652 [main] INFO  org.apache.solr.core.HdfsDirectoryFactory  - creating directory factory for path hdfs://master.hadoop:8020/user/nguyen/twitter/outdir/reducers/_temporary/_attempt_201311191613_0320_r_000014_0/part-r-00014/data
3686 [main] INFO  org.apache.solr.core.CachingDirectoryFactory  - return new directory for hdfs://master.hadoop:8020/user/nguyen/twitter/outdir/reducers/_temporary/_attempt_201311191613_0320_r_000014_0/part-r-00014/data
3711 [main] WARN  org.apache.solr.core.SolrCore  - [core1] Solr index directory 'hdfs:/master.hadoop:8020/user/nguyen/twitter/outdir/reducers/_temporary/_attempt_201311191613_0320_r_000014_0/part-r-00014/data/index' doesn't exist. Creating new index...
3719 [main] INFO  org.apache.solr.core.HdfsDirectoryFactory  - creating directory factory for path hdfs://master.hadoop:8020/user/nguyen/twitter/outdir/reducers/_temporary/_attempt_201311191613_0320_r_000014_0/part-r-00014/data/index
3719 [main] INFO  org.apache.solr.core.HdfsDirectoryFactory  - Number of slabs of block cache [1] with direct memory allocation set to [true]
3720 [main] INFO  org.apache.solr.core.HdfsDirectoryFactory  - Block cache target memory usage, slab size of [134217728] will allocate [1] slabs and use ~[134217728] bytes
3721 [main] INFO  org.apache.solr.store.blockcache.BufferStore  - Initializing the 1024 buffers with [8192] buffers.
3740 [main] INFO  org.apache.solr.store.blockcache.BufferStore  - Initializing the 8192 buffers with [8192] buffers.
3891 [main] INFO  org.apache.solr.core.CachingDirectoryFactory  - return new directory for hdfs://master.hadoop:8020/user/nguyen/twitter/outdir/reducers/_temporary/_attempt_201311191613_0320_r_000014_0/part-r-00014/data/index
3988 [main] INFO  org.apache.solr.update.LoggingInfoStream  - [IFD][main]: init: current segments file is "null"; deletionPolicy=org.apache.solr.core.IndexDeletionPolicyWrapper@65b01d5d
3992 [main] INFO  org.apache.solr.update.LoggingInfoStream  - [IFD][main]: now checkpoint "" [0 segments ; isCommit = false]
3992 [main] INFO  org.apache.solr.update.LoggingInfoStream  - [IFD][main]: 0 msec to checkpoint
3992 [main] INFO  org.apache.solr.update.LoggingInfoStream  - [IW][main]: init: create=true
3992 [main] INFO  org.apache.solr.update.LoggingInfoStream  - [IW][main]: 
dir=NRTCachingDirectory(org.apache.solr.store.hdfs.HdfsDirectory@17e5a6d8 lockFactory=org.apache.solr.store.hdfs.HdfsLockFactory@7f117668; maxCacheMB=192.0 maxMergeSizeMB=16.0)

---

solr查找solr.home参数并在那里搜索solrConfig.xml文件。如果没有，它将尝试加载默认配置

看起来你的家很好

/data/06/mapred/local/taskTracker/distcache/3866561797898787678-1754062477-512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/collection1/

检查该文件夹中的solrconfig.xml文件 如果没有，则从solr的示例目录复制一个

---

如果有，请将文件/文件夹权限与服务器实例相匹配

通常情况下，它是这样工作的，但对于SolrCloud，配置文件是通过Zookeeper分发的。不知何故，solr.xml不在solr zip文件中。我登记了zookeeper服务器和它的服务器。问题是如何将这个solr.xml放入分布式缓存solr.zip/solr.xml是一个更具体的问题。是否有人经历过这种情况？可能需要更多的输入，例如，关于问题涉及的软件版本。