Apache spark 火花流+运动：违反接收器最大速率
我打电话给spark submit passing maxRate，我有一个kinesis接收器和一批1 spark submit-conf spark.streaming.receiver.maxRate=10 但是，单个批次可能会大大超过既定的最大速率。i、 e:我有300张唱片Apache spark 火花流+运动：违反接收器最大速率,apache-spark,spark-streaming,amazon-kinesis,Apache Spark,Spark Streaming,Amazon Kinesis,我打电话给spark submit passing maxRate，我有一个kinesis接收器和一批1 spark submit-conf spark.streaming.receiver.maxRate=10 但是，单个批次可能会大大超过既定的最大速率。i、 e:我有300张唱片 我是否缺少任何设置？我觉得这好像是个bug。在代码中到处乱翻，看起来Kinesis完全忽略了spark.streaming.receiver.maxRate配置 如果查看kinisReceiver.onStart内
我是否缺少任何设置？我觉得这好像是个bug。在代码中到处乱翻，看起来Kinesis完全忽略了spark.streaming.receiver.maxRate配置 如果查看kinisReceiver.onStart内部，您会看到：
val kinesisClientLibConfiguration =
new KinesisClientLibConfiguration(checkpointAppName, streamName, awsCredProvider, workerId)
.withKinesisEndpoint(endpointUrl)
.withInitialPositionInStream(initialPositionInStream)
.withTaskBackoffTimeMillis(500)
.withRegionName(regionName)
public KinesisClientLibConfiguration(String applicationName,
String streamName,
AWSCredentialsProvider kinesisCredentialsProvider,
AWSCredentialsProvider dynamoDBCredentialsProvider,
AWSCredentialsProvider cloudWatchCredentialsProvider,
String workerId) {
this(applicationName, streamName, null, DEFAULT_INITIAL_POSITION_IN_STREAM, kinesisCredentialsProvider,
dynamoDBCredentialsProvider, cloudWatchCredentialsProvider, DEFAULT_FAILOVER_TIME_MILLIS, workerId,
DEFAULT_MAX_RECORDS, DEFAULT_IDLETIME_BETWEEN_READS_MILLIS,
DEFAULT_DONT_CALL_PROCESS_RECORDS_FOR_EMPTY_RECORD_LIST, DEFAULT_PARENT_SHARD_POLL_INTERVAL_MILLIS,
DEFAULT_SHARD_SYNC_INTERVAL_MILLIS, DEFAULT_CLEANUP_LEASES_UPON_SHARDS_COMPLETION,
new ClientConfiguration(), new ClientConfiguration(), new ClientConfiguration(),
DEFAULT_TASK_BACKOFF_TIME_MILLIS, DEFAULT_METRICS_BUFFER_TIME_MILLIS, DEFAULT_METRICS_MAX_QUEUE_SIZE,
DEFAULT_VALIDATE_SEQUENCE_NUMBER_BEFORE_CHECKPOINTING, null);
}