Java 为什么可选水槽通道会导致非可选水槽通道出现问题？
我有一个似乎是一个简单的水槽配置，这给了我很多问题。让我首先描述问题，然后列出配置文件 我有3台服务器：服务器1、服务器2、服务器3 服务器1： Netcat源代码/Syslogtcp源代码（我在没有acks的Netcat和Syslogtcp上都测试了这一点） 2个存储通道 2个Avro接收器（每个通道一个） 复制选择器，第二个内存通道可选 服务器2,3： Avro源 存储通道 卡夫卡水槽 在我的模拟中，Server2模拟“生产”，因此不会发生任何数据丢失，而Server3模拟“开发”，数据丢失是正常的。 我的假设是，使用2个通道和2个源将使两台服务器彼此解耦，如果Server3停机，则不会影响Sever2（特别是使用可选配置选项！）。然而，情况并非如此。当我运行模拟并使用CTRL-C终止Server3时，我在Server2上遇到了减速，从Server2到Kafka接收器的输出变成了爬行。当我恢复Server3上的Flume代理时，一切都恢复正常 我没想到会有这种行为。我所期望的是，因为我有两个通道和两个接收器，如果一个通道和/或接收器坏了，另一个通道和/或接收器应该不会有问题。这是水槽的限制吗？这是对我的源、汇或通道的限制吗？当我使用一个代理和多个相互解耦的通道和接收器时，有没有办法让Flume发挥作用？我真的不想在一台机器上为每个“环境”（生产和开发）设置多个Flume代理。附件是我的配置文件，因此您可以通过更专业的方式查看我所做的工作： SERVER1（第一层代理） SERVER2“PROD”水槽代理Java 为什么可选水槽通道会导致非可选水槽通道出现问题？,java,apache-kafka,flume,avro,flume-ng,Java,Apache Kafka,Flume,Avro,Flume Ng,我有一个似乎是一个简单的水槽配置，这给了我很多问题。让我首先描述问题，然后列出配置文件 我有3台服务器：服务器1、服务器2、服务器3 服务器1： Netcat源代码/Syslogtcp源代码（我在没有acks的Netcat和Syslogtcp上都测试了这一点） 2个存储通道 2个Avro接收器（每个通道一个） 复制选择器，第二个内存通道可选 服务器2,3： Avro源 存储通道 卡夫卡水槽 在我的模拟中，Server2模拟“生产”，因此不会发生任何数据丢失，而Server3模拟“开发”，数据丢失
#Describe the top level configuration
agent.sources = mySource
agent.channels = defaultChannel1 defaultChannel2
agent.sinks = mySink1 mySink2
#Describe/configure the source
agent.sources.mySource.type = netcat
agent.sources.mySource.port = 6666
agent.sources.mySource.bind = 0.0.0.0
agent.sources.mySource.max-line-length = 150000
agent.sources.mySource.ack-every-event = false
#agent.sources.mySource.type = syslogtcp
#agent.sources.mySource.host = 0.0.0.0
#agent.sources.mySource.port = 7103
#agent.sources.mySource.eventSize = 150000
agent.sources.mySource.channels = defaultChannel1 defaultChannel2
agent.sources.mySource.selector.type = replicating
agent.sources.mySource.selector.optional = defaultChannel2
#Describe/configure the channel
agent.channels.defaultChannel1.type = memory
agent.channels.defaultChannel1.capacity = 5000
agent.channels.defaultChannel1.transactionCapacity = 200
agent.channels.defaultChannel2.type = memory
agent.channels.defaultChannel2.capacity = 5000
agent.channels.defaultChannel2.transactionCapacity = 200
#Avro Sink
agent.sinks.mySink1.channel = defaultChannel1
agent.sinks.mySink1.type = avro
agent.sinks.mySink1.hostname = Server2
agent.sinks.mySink1.port = 6666
agent.sinks.mySink2.channel = defaultChannel2
agent.sinks.mySink2.type = avro
agent.sinks.mySink2.hostname = Server3
agent.sinks.mySink2.port = 6666
#Describe the top level configuration
agent.sources = mySource
agent.channels = defaultChannel
agent.sinks = mySink
#Describe/configure the source
agent.sources.mySource.type = avro
agent.sources.mySource.port = 6666
agent.sources.mySource.bind = 0.0.0.0
agent.sources.mySource.max-line-length = 150000
agent.sources.mySource.channels = defaultChannel
#Describe/configure the interceptor
agent.sources.mySource.interceptors = myInterceptor
agent.sources.mySource.interceptors.myInterceptor.type = myInterceptor$Builder
#Describe/configure the channel
agent.channels.defaultChannel.type = memory
agent.channels.defaultChannel.capacity = 5000
agent.channels.defaultChannel.transactionCapacity = 200
#Describe/configure the sink
agent.sinks.mySink.type = org.apache.flume.sink.kafka.KafkaSink
agent.sinks.mySink.topic = Server2-topic
agent.sinks.mySink.brokerList = broker1:9092, broker2:9092
agent.sinks.mySink.requiredAcks = -1
agent.sinks.mySink.batchSize = 100
agent.sinks.mySink.channel = defaultChannel
#Describe the top level configuration
agent.sources = mySource
agent.channels = defaultChannel
agent.sinks = mySink
#Describe/configure the source
agent.sources.mySource.type = avro
agent.sources.mySource.port = 6666
agent.sources.mySource.bind = 0.0.0.0
agent.sources.mySource.max-line-length = 150000
agent.sources.mySource.channels = defaultChannel
#Describe/configure the interceptor
agent.sources.mySource.interceptors = myInterceptor
agent.sources.mySource.interceptors.myInterceptor.type = myInterceptor$Builder
#Describe/configure the channel
agent.channels.defaultChannel.type = memory
agent.channels.defaultChannel.capacity = 5000
agent.channels.defaultChannel.transactionCapacity = 200
#Describe/configure the sink
agent.sinks.mySink.type = org.apache.flume.sink.kafka.KafkaSink
agent.sinks.mySink.topic = Server3-topic
agent.sinks.mySink.brokerList = broker1:9092, broker2:9092
agent.sinks.mySink.requiredAcks = -1
agent.sinks.mySink.batchSize = 100
agent.sinks.mySink.channel = defaultChannel
#Describe the top level configuration
agent.sources = mySource
agent.channels = defaultChannel1 defaultChannel2
agent.sinks = mySink1 mySink2
#Describe/configure the source
agent.sources.mySource.type = netcat
agent.sources.mySource.port = 6666
agent.sources.mySource.bind = 0.0.0.0
agent.sources.mySource.max-line-length = 150000
agent.sources.mySource.ack-every-event = false
#agent.sources.mySource.type = syslogtcp
#agent.sources.mySource.host = 0.0.0.0
#agent.sources.mySource.port = 7103
#agent.sources.mySource.eventSize = 150000
agent.sources.mySource.channels = defaultChannel1 defaultChannel2
agent.sources.mySource.selector.type = replicating
agent.sources.mySource.selector.optional = defaultChannel2
#Describe/configure the channel
agent.channels.defaultChannel1.type = memory
agent.channels.defaultChannel1.capacity = 5000
agent.channels.defaultChannel1.transactionCapacity = 200
agent.channels.defaultChannel2.type = memory
agent.channels.defaultChannel2.capacity = 5000
agent.channels.defaultChannel2.transactionCapacity = 200
#Avro Sink
agent.sinks.mySink1.channel = defaultChannel1
agent.sinks.mySink1.type = avro
agent.sinks.mySink1.hostname = Server2
agent.sinks.mySink1.port = 6666
agent.sinks.mySink2.channel = defaultChannel2
agent.sinks.mySink2.type = avro
agent.sinks.mySink2.hostname = Server3
agent.sinks.mySink2.port = 6666
#Describe the top level configuration
agent.sources = mySource
agent.channels = defaultChannel
agent.sinks = mySink
#Describe/configure the source
agent.sources.mySource.type = avro
agent.sources.mySource.port = 6666
agent.sources.mySource.bind = 0.0.0.0
agent.sources.mySource.max-line-length = 150000
agent.sources.mySource.channels = defaultChannel
#Describe/configure the interceptor
agent.sources.mySource.interceptors = myInterceptor
agent.sources.mySource.interceptors.myInterceptor.type = myInterceptor$Builder
#Describe/configure the channel
agent.channels.defaultChannel.type = memory
agent.channels.defaultChannel.capacity = 5000
agent.channels.defaultChannel.transactionCapacity = 200
#Describe/configure the sink
agent.sinks.mySink.type = org.apache.flume.sink.kafka.KafkaSink
agent.sinks.mySink.topic = Server2-topic
agent.sinks.mySink.brokerList = broker1:9092, broker2:9092
agent.sinks.mySink.requiredAcks = -1
agent.sinks.mySink.batchSize = 100
agent.sinks.mySink.channel = defaultChannel
#Describe the top level configuration
agent.sources = mySource
agent.channels = defaultChannel
agent.sinks = mySink
#Describe/configure the source
agent.sources.mySource.type = avro
agent.sources.mySource.port = 6666
agent.sources.mySource.bind = 0.0.0.0
agent.sources.mySource.max-line-length = 150000
agent.sources.mySource.channels = defaultChannel
#Describe/configure the interceptor
agent.sources.mySource.interceptors = myInterceptor
agent.sources.mySource.interceptors.myInterceptor.type = myInterceptor$Builder
#Describe/configure the channel
agent.channels.defaultChannel.type = memory
agent.channels.defaultChannel.capacity = 5000
agent.channels.defaultChannel.transactionCapacity = 200
#Describe/configure the sink
agent.sinks.mySink.type = org.apache.flume.sink.kafka.KafkaSink
agent.sinks.mySink.topic = Server3-topic
agent.sinks.mySink.brokerList = broker1:9092, broker2:9092
agent.sinks.mySink.requiredAcks = -1
agent.sinks.mySink.batchSize = 100
agent.sinks.mySink.channel = defaultChannel
谢谢你的帮助 我将考虑调整此配置参数，因为它与内存通道有关： agent.channels.defaultChannel.capacity=5000 agent.channels.defaultChannel.transactionCapacity=200 可能先尝试加倍，然后再次执行测试，您会看到改进： agent.channels.defaultChannel.capacity=10000 agent.channels.defaultChannel.transactionCapacity=400
在测试过程中，观察Apache Flume实例的JVM也会很好，我将考虑调整此配置参数，因为这与内存通道有关： agent.channels.defaultChannel.capacity=5000 agent.channels.defaultChannel.transactionCapacity=200 可能先尝试加倍，然后再次执行测试，您会看到改进： agent.channels.defaultChannel.capacity=10000 agent.channels.defaultChannel.transactionCapacity=400
在您的测试过程中，当我发现使用故障转移接收器来解决这个问题时，观察Apache Flume实例的JVM也会很好。我还大大增加了通道容量。谢谢你的帮助。@PhillipAMann我希望你也能把它标记为答案：）但这并不是真正的答案。。。这只是一个建议。我最终想出了一个非常酷的方法，使用故障转移接收器来解决这个问题。我还大大增加了通道容量。谢谢你的帮助。@PhillipAMann我希望你也能把它标记为答案：）但这并不是真正的答案。。。这只是一个建议。