Apache spark Flume在使用客户端提交群集上的作业时无法发送事件

我使用Horton Works群集（2节点群集）运行spark和flume，因此当我使用--master“local[*]”运行作业时，flume能够发送事件，spark也能够接收并在localhost上检查：4040我可以看到正在从flume接收事件。（我们使用flume ng sql源每秒从flume泵送100个事件，每个事件的大小约为1KB）

当我使用--master“纱线客户端”运行同一个示例时，我在flume中得到以下错误，spark也没有得到任何事件

2015-08-13 18:24:24,927 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:160)] Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to send events
        at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:403)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.flume.FlumeException: NettyAvroRpcClient { host: localhost, port: 55555 }: RPC connection error
        at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:182)
        at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:121)
        at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:638)
        at org.apache.flume.api.RpcClientFactory.getInstance(RpcClientFactory.java:88)
        at org.apache.flume.sink.AvroSink.initializeRpcClient(AvroSink.java:127)
        at org.apache.flume.sink.AbstractRpcSink.createConnection(AbstractRpcSink.java:222)
        at org.apache.flume.sink.AbstractRpcSink.verifyConnection(AbstractRpcSink.java:283)
        at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:360)
        ... 3 more
Caused by: java.io.IOException: Error connecting to localhost/127.0.0.1:55555
        at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
        at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
        at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
        at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:168)
        ... 10 more
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
        at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:496)
        at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:452)
        at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:365)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        ... 1 more
^

火花：

sudo -u hdfs spark-submit --master "yarn-client" --class "org.paladion.atm.FlumeEventCount" target/atm-1.1-jar-with-dependencies.jar > sparklog.txt

sudo -u hdfs spark-submit --master "local[*]" --class "org.paladion.atm.FlumeEventCount" target/atm-1.1-jar-with-dependencies.jar > sparklog.txt

---

好心的l；让我知道这里可能出了什么问题？

结果如下：

1-如果以本地方式运行，则给出水槽中本地机器的IP以及火花

2-如果作为集群（纱线客户端或纱线集群）运行，则在Flume和spark中为集群中要发送事件的机器（执行程序的机器除外）提供IP，因此可能为不是主节点的节点提供IP

---

如果我错了，请告诉我，这可能是由于其他原因而起作用的，并且有更好的解决方案。

解决方案如下：

1-如果以本地方式运行，则给出水槽中本地机器的IP以及火花

2-如果作为集群（纱线客户端或纱线集群）运行，则在Flume和spark中为集群中要发送事件的机器（执行程序的机器除外）提供IP，因此可能为不是主节点的节点提供IP

---

如果我错了，请告诉我，这可能是因为其他原因，有更好的解决办法。

大家好，有没有人也遇到过这样的问题？大家好，有没有人也遇到过这样的问题

sudo -u hdfs spark-submit --master "yarn-client" --class "org.paladion.atm.FlumeEventCount" target/atm-1.1-jar-with-dependencies.jar > sparklog.txt

sudo -u hdfs spark-submit --master "local[*]" --class "org.paladion.atm.FlumeEventCount" target/atm-1.1-jar-with-dependencies.jar > sparklog.txt