Apache kafka 卡夫卡主题的每个分区中的提交数和偏移量
如何在已知卡夫卡主题的每个分区中查找提交数和当前偏移量。我使用的是kafka v0.8.1.1您的问题不清楚您对哪种补偿感兴趣。 实际上有三种类型的偏移：Apache kafka 卡夫卡主题的每个分区中的提交数和偏移量,apache-kafka,Apache Kafka,如何在已知卡夫卡主题的每个分区中查找提交数和当前偏移量。我使用的是kafka v0.8.1.1您的问题不清楚您对哪种补偿感兴趣。 实际上有三种类型的偏移： 主题分区中第一条可用消息的偏移量。使用-2 （最早）as--GetOffsetShell工具的时间参数 主题分区中最后一条可用消息的偏移量。使用-1（最新）作为--time 参数 由维护的上次读取/处理的邮件偏移量 卡夫卡消费者。高级消费者将每个消费者组的此信息存储在 卡夫卡的一个内部话题（以前是动物园管理员）并关注 在调用commit（）或
Get Offset Shell
获取主题的偏移量
bin/kafka-run-class.sh kafka.tools.GetOffsetShell
必需参数[代理列表]，[主题]
选项说明
------ -----------
--要连接到的服务器的代理列表端口。
--max wait ms每个提取请求等待的最大时间。（默认值：1000）
--偏移返回的偏移数（默认值：1）
--分区以逗号分隔的分区ID列表。如果未指定，将查找所有分区的偏移量（默认）
--时间
--所需主题：要从中获取偏移量的主题。
游戏）：
我将获得games:0:47841
，这意味着对于主题games
和0
分区，我有最新未使用的偏移量47841
（最新可用消息）
您可以使用-2
查看第一条可用消息。此信息还有助于创建脚本，以查看某个主题分区上的消息数（从命令行）。虽然像Kafka Web控制台这样的工具很不错，但我们中的一些人生活在一个非GUI的世界中
这是剧本。。。使用和修改它的风险由您自己承担：-）
#/bin/bash
主题=$1
如果[[-z“${topic}”]]；然后
echo“用法：${0}”
出口1
fi
如果[[-z“${KAFKA_HOME}”]；然后
#$KAFKA\u未设置主页，使用默认值/KAFKA
卡夫卡之家=“/KAFKA”
fi
如果[！-d${KAFKA_HOME}]；然后
echo“\$KAFKA\u HOME未指向有效目录[$KAFKA\u HOME]”
出口1
fi
cd$KAFKA_主页
回声
echo“主题：${Topic}：”
#
printf“分区计数\n”
printf“~~~~~~~~~~~~~~~~~~~~~~~~~\n”
idx=0
对于'bin/kafka-run-class.sh kafka.tools.GetOffsetShell--topic${topic}--代理列表localhost:9092--time-1'中的消息；做
name=`echo${msg}|awk-F:“{print$1}”`
分区=`echo${msg}|awk-F:“{print$2}”`
总计=`echo${msg}|awk-F:“{print$3}”`
printf“%10d%12d\n”${partition}${total}
idx=$（（idx+1））
完成
如果[${idx}-eq 0]；然后
echo“未找到主题名称！”
出口1
fi
回声
退出${rc}
从0.9.0.x版开始，您应该开始使用kafka.admin.ConsumerGroup命令工具。下面是该工具采用的参数
List all consumer groups, describe a consumer group, or delete consumer group info.
Option Description
------ -----------
--bootstrap-server <server to connect REQUIRED (only when using new-
to> consumer): The server to connect to.
--command-config <command config Property file containing configs to be
property file> passed to Admin Client and Consumer.
--delete Pass in groups to delete topic
partition offsets and ownership
information over the entire consumer
group. For instance --group g1 --
group g2
Pass in groups with a single topic to
just delete the given topic's
partition offsets and ownership
information for the given consumer
groups. For instance --group g1 --
group g2 --topic t1
Pass in just a topic to delete the
given topic's partition offsets and
ownership information for every
consumer group. For instance --topic
t1
WARNING: Group deletion only works for
old ZK-based consumer groups, and
one has to use it carefully to only
delete groups that are not active.
--describe Describe consumer group and list
offset lag related to given group.
--group <consumer group> The consumer group we wish to act on.
--list List all consumer groups.
--new-consumer Use new consumer.
--topic <topic> The topic whose consumer group
information should be deleted.
--zookeeper <urls> REQUIRED (unless new-consumer is
used): The connection string for the
zookeeper connection in the form
host:port. Multiple URLS can be
given to allow fail-over.
说“支持”明天我们将有一个名为的主题27
我们的要求是
请求1：想知道主题的分区和偏移详细信息。
Ans：我们可以使用GetOffsetShell命令，如下面的屏幕截图所示
请求2：想知道消费者群体消费的抵消数量。
答：我们可以使用ConsumerGroupCommand，如下面的屏幕截图所示
我需要两个功能1。监视每个分区2的延迟。在系统完全重新启动的情况下（zookeeper、broker、producer和consumer），如何从高级使用者中上次读取/处理的消息偏移量恢复每个分区的最新偏移量不一定等于由于日志保留而每个分区中当前有多少条消息。如果未启用日志压缩，您可以通过从最新的偏移量中减去最早的偏移量来实现该计数。需要注意的是：如果启用日志压缩，这将不准确。是否有任何方法可以通过编程方式获得相同的值？是否有人已经整理出如何使用bin/kafka consumer groups.sh
和--command config
和较新版本修改偏移量卡夫卡0.10.x？
bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic games --time -1
#!/bin/bash
topic=$1
if [[ -z "${topic}" ]] ; then
echo "Usage: ${0} <topic>"
exit 1
fi
if [[ -z "${KAFKA_HOME}" ]] ; then
# $KAFKA_HOME not set, using default /kafka
KAFKA_HOME="/kafka"
fi
if [ ! -d ${KAFKA_HOME} ] ; then
echo "\$KAFKA_HOME does not point to a valid directory [$KAFKA_HOME]"
exit 1
fi
cd $KAFKA_HOME
echo
echo "Topic: ${topic}: "
#
printf "Partition Count\n"
printf "~~~~~~~~~~ ~~~~~~~~~~~~\n"
idx=0
for msg in `bin/kafka-run-class.sh kafka.tools.GetOffsetShell --topic ${topic} --broker-list localhost:9092 --time -1` ; do
name=`echo ${msg} | awk -F ":" '{print $1}'`
partition=`echo ${msg} | awk -F ":" '{print $2}'`
total=`echo ${msg} | awk -F ":" '{print $3}'`
printf "%10d %12d\n" ${partition} ${total}
idx=$((idx + 1))
done
if [ ${idx} -eq 0 ] ; then
echo "Topic name not found!"
exit 1
fi
echo
exit ${rc}
List all consumer groups, describe a consumer group, or delete consumer group info.
Option Description
------ -----------
--bootstrap-server <server to connect REQUIRED (only when using new-
to> consumer): The server to connect to.
--command-config <command config Property file containing configs to be
property file> passed to Admin Client and Consumer.
--delete Pass in groups to delete topic
partition offsets and ownership
information over the entire consumer
group. For instance --group g1 --
group g2
Pass in groups with a single topic to
just delete the given topic's
partition offsets and ownership
information for the given consumer
groups. For instance --group g1 --
group g2 --topic t1
Pass in just a topic to delete the
given topic's partition offsets and
ownership information for every
consumer group. For instance --topic
t1
WARNING: Group deletion only works for
old ZK-based consumer groups, and
one has to use it carefully to only
delete groups that are not active.
--describe Describe consumer group and list
offset lag related to given group.
--group <consumer group> The consumer group we wish to act on.
--list List all consumer groups.
--new-consumer Use new consumer.
--topic <topic> The topic whose consumer group
information should be deleted.
--zookeeper <urls> REQUIRED (unless new-consumer is
used): The connection string for the
zookeeper connection in the form
host:port. Multiple URLS can be
given to allow fail-over.
bin/kafka-run-class.sh kafka.admin.ConsumerGroupCommand --zookeeper <zookeeper urls> --describe --group consumerGroup_Y
GROUP, TOPIC, PARTITION, CURRENT OFFSET, LOG END OFFSET, LAG, OWNER
consumerGroup, Topic_X, 0, 3030460, 3168412, 137952, none
consumerGroup, Topic_X, 1, 3030903, 3168884, 137981, none
consumerGroup, Topic_X, 2, 801564, 939540, 137976, none
consumerGroup, Topic_X, 3, 737290, 875262, 137972, none
consumerGroup, Topic_X, 4, 737288, 875254, 137966, none
consumerGroup, Topic_X, 5, 737276, 875241, 137965, none
consumerGroup, Topic_X, 6, 737290, 875251, 137961, none
consumerGroup, Topic_X, 7, 737290, 875248, 137958, none
consumerGroup, Topic_X, 8, 737288, 875246, 137958, none
consumerGroup, Topic_X, 9, 737293, 875251, 137958, none
consumerGroup, Topic_X, 10, 737289, 875244, 137955, none
consumerGroup, Topic_X, 11, 737273, 875226, 137953, none