devbox
你好赵伟
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

kafka监控

作者：你好赵伟 | 2024-08-09 09:16:03

赞

踩

jmx["kafka.log:type=logflushstats,name=logflushrateandtimems","95thperce

为什么80%的码农都做不了架构师？>>>   hot3.png

    kafka使用也很久了，如何细粒度的监控kafka，目前还找不到一款比较合适的开源监控工具，但是不妨碍总结一下如何监控kafka，最官方的方法就是使用metrics的值来监控kafka，目前我们就是使用jmxtrans来获取metrics值监控kafka的。kafak监控主要分为三个方面：broker监控、consumer监控、producer监控。三者的监控项可以通过jconsole来查看具体的mbean。

    在具体查看mbean之前，我们先了解一下metrics(kafka使用Yammer metrics )，metrics是一个度量工具包，提供多种度量类型来统计程序的各项指标，在JAVA代码中嵌入Metrics代码，可以方便的对业务代码的各个指标进行监控。metrics默认支持并开启了jmx的方式暴露监控数据，开发人员可以使用jmx的方式轻松的获取度量数据，metrics还支持其他类型的reporter，如csv、console，metrics提供了对Ehcache、Apache HttpClient、JDBI、Jersey、Jetty、Log4J、Logback、JVM等的集成，可以方便地将Metrics输出到Ganglia、Graphite中，供用户图形化展示。metrics提供了五种类型的度量类型：

  1.         gauge：是一个最简单的计量，一般用来统计瞬时状态的数据信息，比如系统中处于pending状态的job
  2.         counter：是gauge的一个特例，维护一个计数器，可以通过inc()和dec()方法对计数器做修改。一般用来记录某个事件发生的次数或者请求的个数
  3.         meters：用来度量某个时间段的平均处理次数（request per second）。统计结果有总的请求数，平均每秒的请求数，以及最近的1、5、15分钟的平均TPS。
  4.         histograms：主要使用来统计数据的分布情况，最大值、最小值、平均值、中位数，百分比（75%、90%、95%、98%、99%和99.9%）。例如，需要统计某个页面的请求响应时间分布情况，可以使用该种类型的Metrics进行统计
  5.         timers：主要是用来统计某一块代码段的执行时间以及其分布情况，具体是基于Histograms和Meters来实现的
  6.         health checks：用于对Application、其子模块或者关联模块的运行是否正常做检测。该模块是独立metrics-core模块的，使用时则导入metrics-healthchecks包    

broker监控

    broker metrics

  1. kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions	Number of under-replicated partitions (ISR < all replicas ). Alert if value is greater than 0.
  2. kafka.controller:type=KafkaController,name=OfflinePartitionsCount	Number of partitions that don’t have an active leader and are hence not writable or readable. Alert if value is greater than 0.
  3. kafka.controller:type=KafkaController,name=ActiveControllerCount	Number of active controllers in the cluster. Alert if value is anything other than 1.
  4. kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec	Aggregate incoming message rate.
  5. kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec	Aggregate incoming byte rate.
  6. kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec	Aggregate outgoing byte rate.
  7. kafka.network:type=RequestMetrics,name=RequestsPerSec,request={Produce or FetchConsumer or FetchFollower}	Request rate.
  8. kafka.log:type=LogFlushStats,name=LogFlushRateAndTimeMs	Log flush rate and time.
  9. kafka.controller:type=ControllerStats,name=LeaderElectionRateAndTimeMs	Leader election rate and latency.
  10. kafka.controller:type=ControllerStats,name=UncleanLeaderElectionsPerSec	Unclean leader election rate.
  11. kafka.server:type=ReplicaManager,name=PartitionCount	Number of partitions on this broker. This should be mostly even across all brokers.
  12. kafka.server:type=ReplicaManager,name=LeaderCount	Number of leaders on this broker. This should be mostly even across all brokers. If not, set auto.leader.rebalance.enable to true on all brokers in the cluster.
  13. kafka.server:type=ReplicaManager,name=IsrShrinksPerSec	If a broker goes down, ISR for some of the partitions will shrink. When that broker is up again, ISR will be expanded once the replicas are fully caught up. Other than that, the expected value for both ISR shrink rate and expansion rate is 0.
  14. kafka.server:type=ReplicaManager,name=IsrExpandsPerSec	When a broker is brought up after a failure, it starts catching up by reading from the leader. Once it is caught up, it gets added back to the ISR.
  15. kafka.server:type=ReplicaFetcherManager,name=MaxLag,clientId=Replica	Maximum lag in messages between the follower and leader replicas. This is controlled by the replica.lag.max.messages config.
  16. kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=([-.\w]+),topic=([-.\w]+),partition=([0-9]+)	Lag in number of messages per follower replica. This is useful to know if the replica is slow or has stopped replicating from the leader.
  17. kafka.network:type=RequestMetrics,name=TotalTimeMs,request={Produce or FetchConsumer or FetchFollower}	Total time in ms to serve the specified request.
  18. kafka.server:type=ProducerRequestPurgatory,name=PurgatorySize	Number of requests waiting in the producer purgatory. This should be non-zero acks=-1 is used on the producer.
  19. kafka.server:type=FetchRequestPurgatory,name=PurgatorySize	Number of requests waiting in the fetch purgatory. This is high if consumers use a large value for fetch.wait.max.ms .

    生产jmxtrans-agent.xml样例

  1. <jmxtrans-agent>
  2.     <queries>
  3.  
  4.         <!-- Message in rate -->
  5.         <query objectName="kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec" attributes="MeanRate,OneMinuteRate" resultAlias="Kafka.BrokerTopicMetrics.MessagesInPerSec.#attribute#"/>
  6.  
  7.         <!-- Byte in rate -->
  8.         <query objectName="kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec" attributes="MeanRate,OneMinuteRate" resultAlias="Kafka.BrokerTopicMetrics.BytesInPerSec.#attribute#"/>
  9.  
  10.         <!-- Byte out rate -->
  11.         <query objectName="kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec" attributes="MeanRate,OneMinuteRate" resultAlias="Kafka.BrokerTopicMetrics.BytesOutPerSec.#attribute#"/>
  12.  
  13.         <!-- Log flush rate and time -->
  14.         <query objectName="kafka.log:type=LogFlushStats,name=LogFlushRateAndTimeMs" attributes="OneMinuteRate" resultAlias="Kafka.LogFlushStats.FlushRateAndTimeMs.#attribute#"/>
  15.  
  16.         <!-- Request rate -->
  17.         <query objectName="kafka.network:type=RequestMetrics,name=RequestsPerSec,request=Produce" attributes="OneMinuteRate" resultAlias="Kafka.RequestsPerSec.Produce.#attribute#"/>
  18.         <query objectName="kafka.network:type=RequestMetrics,name=RequestsPerSec,request=FetchConsumer" attributes="OneMinuteRate" resultAlias="Kafka.RequestsPerSec.FetchConsumer.#attribute#"/>
  19.         <query objectName="kafka.network:type=RequestMetrics,name=RequestsPerSec,request=FetchFollower" attributes="OneMinuteRate" resultAlias="Kafka.RequestsPerSec.FetchFollower.#attribute#"/>
  20.  
  21.         <!-- Log flush rate and time -->
  22.         <query objectName="kafka.log:type=LogFlushStats,name=LogFlushRateAndTimeMs" resultAlias="Kafka.LogFlushStats.LogFlushRateAndTimeMs.#attribute#"/>
  23.  
  24.         <!-- Partition counts -->
  25.         <query objectName="kafka.server:type=ReplicaManager,name=PartitionCount" attribute="Value" resultAlias="Kafka.Topic.PartitionCount.#attribute#"/>
  26.  
  27.         <!-- of under replicated partitions (|ISR| < |all replicas|)  -->
  28.         <query objectName="kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions" resultAlias="Kafka.ReplicaManager.UnderReplicatedPartitions.#attribute#"/>
  29.  
  30.         <!-- Is controller active on broker -->
  31.         <query objectName="kafka.controller:type=KafkaController,name=ActiveControllerCount" resultAlias="Kafka.KafkaController.ActiveControllerCount.#attribute#"/>
  32.  
  33.         <!-- Leader election rate -->
  34.         <query objectName="kafka.controller:type=ControllerStats,name=LeaderElectionRateAndTimeMs" attributes="OneMinuteRate,Max,Min,75thPercentile,95thPercentile" resultAlias="Kafka.ControllerStats.LeaderElectionRateAndTimeMs.#attribute#"/>
  35.  
  36.         <!-- Leader replica counts -->
  37.         <query objectName="kafka.server:type=ReplicaManager,name=LeaderCount" resultAlias="Kafka.ReplicaManager.LeaderCount.#attribute#"/>
  38.  
  39.         <!-- Max lag in messages btw follower and leader replicas -->
  40.         <query objectName="kafka.server:type=ReplicaFetcherManager,name=MaxLag,clientId=Replica" resultAlias="Kafka.ReplicaFetcherManager.MaxLag.#attribute#"/>
  41.         <!-- Lag in messages per follower replica -->
  42.         <query objectName="kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=([-.\w]+)" resultAlias="Kafka.ConsumerLag.clientId.#attribute#"/>
  43.         <query objectName="kafka.server:type=FetcherLagMetrics,name=ConsumerLag,topic=([-.\w]+)" resultAlias="Kafka.ConsumerLag.topic.#attribute#"/>
  44.         <query objectName="kafka.server:type=FetcherLagMetrics,name=ConsumerLag,partition=([0-9]+)" resultAlias="Kafka.ConsumerLag.partition.#attribute#"/>
  45.  
  46.         <!-- Requests waiting in the producer purgatory -->
  47.         <query objectName="kafka.server:type=ProducerRequestPurgatory,name=PurgatorySize" resultAlias="Kafka.ProducerRequestPurgatory.PurgatorySize.#attribute#"/>
  48.         <!-- Requests waiting in the fetch purgatory -->
  49.         <query objectName="kafka.server:type=FetchRequestPurgatory,name=PurgatorySize" resultAlias="Kafka.FetchRequestPurgatory.PurgatorySize.#attribute#"/>
  50.  
  51.         <!-- Request total time -->
  52.         <query objectName="kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce" attributes="Max,Min,75thPercentile,95thPercentile" resultAlias="Kafka.RequestMetrics.TotalTimeMs.Produce.#attribute#"/>
  53.         <query objectName="kafka.network:type=RequestMetrics,name=TotalTimeMs,request=FetchConsumer" attributes="Max,Min,75thPercentile,95thPercentile" resultAlias="Kafka.RequestMetrics.TotalTimeMs.FetchConsumer.#attribute#"/>
  54.         <query objectName="kafka.network:type=RequestMetrics,name=TotalTimeMs,request=FetchFollower" attributes="Max,Min,75thPercentile,95thPercentile" resultAlias="Kafka.RequestMetrics.TotalTimeMs.FetchFollower.#attribute#"/>
  55.  
  56.         <!--Time the request waiting in the request queue -->
  57.         <query objectName="kafka.network:type=RequestMetrics,name=QueueTimeMs,request=Produce" attributes="OneMinuteRate,Max,Min,75thPercentile,95thPercentile" resultAlias="Kafka.RequestMetrics.QueueTimeMs.Produce.#attribute#"/>
  58.         <query objectName="kafka.network:type=RequestMetrics,name=QueueTimeMs,request=FetchConsumer" attributes="OneMinuteRate,Max,Min,75thPercentile,95thPercentile" resultAlias="Kafka.RequestMetrics.QueueTimeMs.FetchConsumer.#attribute#"/>
  59.         <query objectName="kafka.network:type=RequestMetrics,name=QueueTimeMs,request=Produce" attributes="OneMinuteRate,Max,Min,75thPercentile,95thPercentile" resultAlias="Kafka.RequestMetrics.QueueTimeMs.Produce.#attribute#"/>
  60.  
  61.         <!-- Time the request being processed at the leader -->
  62.         <query objectName="kafka.network:type=RequestMetrics,name=LocalTimeMs,request=Produce" attributes="Max,Min,75thPercentile,95thPercentile" resultAlias="Kafka.RequestMetrics.LocalTimeMs.Produce.#attribute#"/>
  63.         <query objectName="kafka.network:type=RequestMetrics,name=LocalTimeMs,request=FetchConsumer" attributes="Max,Min,75thPercentile,95thPercentile" resultAlias="Kafka.RequestMetrics.LocalTimeMs.FetchConsumer.#attribute#"/>
  64.         <query objectName="kafka.network:type=RequestMetrics,name=LocalTimeMs,request=FetchFollower" attributes="Max,Min,75thPercentile,95thPercentile" resultAlias="Kafka.RequestMetrics.LocalTimeMs.FetchFollower.#attribute#"/>
  65.  
  66.         <!-- Time the request waits for the follower -->
  67.         <query objectName="kafka.network:type=RequestMetrics,name=RemoteTimeMs,request=Produce" attributes="Max,Min,75thPercentile,95thPercentile" resultAlias="Kafka.RequestMetrics.RemoteTimeMs.Produce.#attribute#"/>
  68.         <query objectName="kafka.network:type=RequestMetrics,name=RemoteTimeMs,request=FetchConsumer" attributes="Max,Min,75thPercentile,95thPercentile" resultAlias="Kafka.RequestMetrics.RemoteTimeMs.FetchConsumer.#attribute#"/>
  69.         <query objectName="kafka.network:type=RequestMetrics,name=RemoteTimeMs,request=FetchFollower" attributes="Max,Min,75thPercentile,95thPercentile" resultAlias="Kafka.RequestMetrics.RemoteTimeMs.FetchFollower.#attribute#"/>
  70.  
  71.         <!-- Time to send the response -->
  72.         <query objectName="kafka.network:type=RequestMetrics,name=ResponseSendTimeMs,request=Produce" attributes="Max,Min,75thPercentile,
  73. 95thPercentile" resultAlias="Kafka.RequestMetrics.ResponseSendTimeMs.Produce.#attribute#"/>
  74.         <query objectName="kafka.network:type=RequestMetrics,name=ResponseSendTimeMs,request=FetchConsumer" attributes="Max,Min,75thPercentile,
  75. 95thPercentile" resultAlias="Kafka.RequestMetrics.ResponseSendTimeMs.FetchConsumer.#attribute#"/>
  76.         <query objectName="kafka.network:type=RequestMetrics,name=ResponseSendTimeMs,request=FetchFollower" attributes="Max,Min,75thPercentile,95thPercentile" resultAlias="Kafka.RequestMetrics.ResponseSendTimeMs.FetchFollower.#attribute#"/>
  77.  
  78.         <!-- Number of messages the consumer lags behind the producer by -->
  79.         <query objectName="kafka.consumer:type=ConsumerFetcherManager,name=MaxLag,clientId=([-.\w]+)" resultAlias="Kafka.ConsumerFetcherManager.MaxLag.clientId.#attribute#"/>
  80.  
  81.         <!-- The average fraction of time the network processors are idle -->
  82.         <query objectName="kafka.network:type=SocketServer,name=NetworkProcessorAvgIdlePercent" resultAlias="Kafka.SocketServer.NetworkProcessorAvgIdlePercent.#attribute#"/>
  83.  
  84.         <!-- The average fraction of time the request handler threads are idle -->
  85.         <query objectName="kafka.server:type=KafkaRequestHandlerPool,name=RequestHandlerAvgIdlePercent" attributes="OneMinuteRate" resultAlias="Kafka.KafkaRequestHandlerPool.RequestHandlerAvgIdlePercent.#attribute#"/>
  86.  
  87.         <!-- Quota metrics per client-id -->
  88.         <query objectName="kafka.server:type=Produce,client-id=([-.\w]+)" resultAlias="Kafka.Produce.client-id.#attribute#"/>
  89.         <query objectName="kafka.server:type=Fetch,client-id=([-.\w]+)" resultAlias="Kafka.Fetch.client-id.#attribute#"/>
  90.  
  91.         </queries>
  92.  
  93.         <!--
  94.                 <outputWriter class="org.jmxtrans.agent.RollingFileOutputWriter">
  95.                 <fileName>/tmp/roll-jmxing.log</fileName>
  96.                 <maxFileSize>1024</maxFileSize>
  97.                 <maxBackupIndex>10</maxBackupIndex>
  98.         </outputWriter>
  99.         -->
  100.  
  101.         <outputWriter class="org.jmxtrans.agent.FileOverwriterOutputWriter">
  102.                 <fileName>/tmp/jmxing.log</fileName>
  103.                 <showTimeStamp>false</showTimeStamp>
  104.         </outputWriter>
  105.  
  106.         <collectIntervalInSeconds>60</collectIntervalInSeconds>
  107. </jmxtrans-agent>

 

consumer监控

    Fetch Metrics: kafka.consumer:type=consumer-fetch-manager-metrics,client-id=([-.w]+)

  1. records-lag-max			The maximum lag in terms of number of records for any partition in this window. An increasing value over time is your best indication that the consumer group is not keeping up with the producers.
  2. fetch-size-avg			The average number of bytes fetched per request.
  3. fetch-size-max			The max number of bytes fetched per request.
  4. bytes-consumed-rate		The average number of bytes consumed per second.
  5. records-per-request-avg	The average number of records in each request.
  6. records-consumed-rate	The average number of records consumed per second
  7. fetch-rate				The number of fetch requests per second.
  8. fetch-latency-avg		The average time taken for a fetch request.
  9. fetch-latency-max		The max time taken for a fetch request.
  10. fetch-throttle-time-avg	The average throttle time in ms. When quotas are enabled, the broker may delay fetch requests in order to throttle a consumer which has exceeded its limit. This metric indicates how throttling time has been added to fetch requests on average.
  11. fetch-throttle-time-avg	The maximum throttle time in ms

    Topic-level Fetch Metrics: kafka.consumer:type=consumer-fetch-manager-metrics,client-id=([-.w]+),topic=([-.w]+)

  1. fetch-size-avg			The average number of bytes fetched per request for a specific topic.
  2. fetch-size-max			The maximum number of bytes fetched per request for a specific topic.
  3. bytes-consumed-rate		The average number of bytes consumed per second for a specific topic.
  4. records-per-request-avg	The average number of records in each request for a specific topic.
  5. records-consumed-rate	The average number of records consumed per second for a specific topic.

    Consumer Group Metrics: kafka.consumer:type=consumer-coordinator-metrics,client-id=([-.w]+)

  1. assigned-partitions		The number of partitions currently assigned to this consumer.
  2. commit-latency-avg		The average time taken for a commit request.
  3. commit-latency-max		The max time taken for a commit request
  4. commit-rate				The number of commit calls per second.
  5. join-rate				The number of group joins per second. Group joining is the first phase of the rebalance protocol. A large value indicates that the consumer group is unstable and will likely be coupled with increased lag.
  6. join-time-avg			The average time taken for a group rejoin. This value can get as high as the configured session timeout for the consumer, but should usually be lower.
  7. join-time-max			The max time taken for a group rejoin. This value should not get much higher than the configured session timeout for the consumer.
  8. sync-rate				The number of group syncs per second. Group synchronization is the second and last phase of the rebalance protocol. Similar to join-rate, a large value indicates group instability.
  9. sync-time-avg			The average time taken for a group sync.
  10. sync-time-max			The max time taken for a group sync.
  11. heartbeat-rate			The average number of heartbeats per second. After a rebalance, the consumer sends heartbeats to the coordinator to keep itself active in the group. You can control this using the heartbeat.interval.ms setting for the consumer. You may see a lower rate than configured if the processing loop is taking more time to handle message batches. Usually this is OK as long as you see no increase in the join rate.
  12. heartbeat-response-time-max	The max time taken to receive a response to a heartbeat request.
  13. last-heartbeat-seconds-ago	The number of seconds since the last controller heartbeat.

    Global Request Metrics: kafka.consumer:type=consumer-metrics,client-id=([-.w]+)

  1. request-latency-avg   The average request latency in ms.
  2. request-latency-max	  The maximum request latency in ms.
  3. request-rate		  The average number of requests sent per second.
  4. response-rate		  The average number of responses received per second.
  5. incoming-byte-rate	  The average number of incoming bytes received per second from all servers.
  6. outgoing-byte-rate	  The average number of outgoing bytes sent per second to all servers.

    Global Connection Metrics: kafka.consumer:type=consumer-metrics,client-id=([-.w]+)

  1. connection-count		    The current number of active connections.
  2. connection-creation-rate	New connections established per second in the window.
  3. connection-close-rate		Connections closed per second in the window.
  4. io-ratio					The fraction of time the I/O thread spent doing I/O.
  5. io-time-ns-avg				The average length of time for I/O per select call in nanoseconds.
  6. io-wait-ratio				The fraction of time the I/O thread spent waiting.
  7. select-rate					Number of times the I/O layer checked for new I/O to perform per second.
  8. io-wait-time-ns-avg			The average length of time the I/O thread spent waiting for a socket ready for reads or writes in nanoseconds.

    Per-Broker Metrics: kafka.consumer:type=consumer-node-metrics,client-id=([-.w]+),node-id=([0-9]+)

  1. request-size-max	The maximum size of any request sent in the window for a broker.
  2. request-size-avg	The average size of all requests in the window for a broker.
  3. request-rate		The average number of requests sent per second to the broker.
  4. response-rate		The average number of responses received per second from the broker.
  5. incoming-byte-rate	The average number of bytes received per second from the broker.
  6. outgoing-byte-rate	The average number of bytes sent per second to the broker.

 

producer监控

    Global Request Metrics: kafka.producer:type=producer-metrics,client-id=([-.w]+)

  1. request-latency-avg		The average request latency in ms.
  2. request-latency-max		The maximum request latency in ms.
  3. request-rate			The average number of requests sent per second.
  4. response-rate			The average number of responses received per second.
  5. incoming-byte-rate		The average number of incoming bytes received per second from all servers.
  6. outgoing-byte-rate		The average number of outgoing bytes sent per second to all servers.

    Global Connection Metrics: kafka.producer:type=producer-metrics,client-id=([-.w]+)

  1. connection-count			The current number of active connections.
  2. connection-creation-rate	New connections established per second in the window.
  3. connection-close-rate		Connections closed per second in the window.
  4. io-ratio					The fraction of time the I/O thread spent doing I/O.
  5. io-time-ns-avg				The average length of time for I/O per select call in nanoseconds.
  6. io-wait-ratio				The fraction of time the I/O thread spent waiting.
  7. select-rate					Number of times the I/O layer checked for new I/O to perform per second.
  8. io-wait-time-ns-avg			The average length of time the I/O thread spent waiting for a socket ready for reads or writes in nanoseconds.

    Per-Broker Metrics: kafka.producer:type=producer-node-metrics,client-id=([-.w]+),node-id=([0-9]+)

  1. request-size-max	The maximum size of any request sent in the window for a broker.
  2. request-size-avg	The average size of all requests in the window for a broker.
  3. request-rate		The average number of requests sent per second to the broker.
  4. response-rate		The average number of responses received per second from the broker.
  5. incoming-byte-rate	The average number of bytes received per second from the broker.
  6. outgoing-byte-rate	The average number of bytes sent per second to the broker.

    Per-Topic Metrics: kafka.producer:type=producer-topic-metrics,client-id=([-.w]+),topic=([-.w]+)

  1. byte-rate			The average number of bytes sent per second for a topic.
  2. record-send-rate	The average number of records sent per second for a topic.
  3. compression-rate	The average compression rate of record batches for a topic.
  4. record-retry-rate	The average per-second number of retried record sends for a topic.
  5. record-error-rate	The average per-second number of record sends that resulted in errors for a topic.

 

参照：confluent

 

转载于:https://my.oschina.net/guol/blog/1560547

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/你好赵伟/article/detail/952484
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号