devbox
盐析白兔
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

KafkaProducer使用介绍、参数配置以及核心源码解析_org.apache.kafka.clients.producer.kafkaproducer

作者：盐析白兔 | 2024-05-24 17:19:46

赞

踩

org.apache.kafka.clients.producer.kafkaproducer

前言

Kafka，作为目前在大数据领域应用最为广泛的消息队列，其内部实现和设计有很多值得深入研究和分析的地方，使用kafka首先需要接触到producer的开发，然后是consumer开发，自0.8.2.x版本以后，kafka提供了java 版本的producer以代替以前scala版本的producer，下面进行producer的解析。

Producer 概要设计

发送简略流程图

KafkaProducer首先使用序列化器将需要发送的数据进行序列化，让后通过分区器（partitioner）确定该数据需要发送的Topic的分区，kafka提供了一个默认的分区器，如果消息指定了key，那么partitioner会根据key的hash值来确定目标分区，如果没有指定key，那么将使用轮询的方式确定目标分区，这样可以最大程度的均衡每个分区的消息，确定分区之后，将会进一步确认该分区的leader节点(处理该分区消息读写的主节点)，消息会进入缓冲池进行缓冲，然后等消息到达一定数量或者大小后进行批量发送

Producer 程序开发

  • producer同步发送消息
  1. package com.huawei.kafka.producer;
  2.  
  3. import org.apache.kafka.clients.producer.KafkaProducer;
  4. import org.apache.kafka.clients.producer.ProducerRecord;
  5. import org.apache.kafka.clients.producer.RecordMetadata;
  6.  
  7. import java.util.Properties;
  8. import java.util.concurrent.ExecutionException;
  9.  
  10. /**
  11.  * @author: xuqiangnj@163.com
  12.  * @date: 2019/4/15 21:59
  13.  * @description:
  14.  */
  15. public class SynKafkaProducer {
  16.  
  17.     public static final Properties props = new Properties();
  18.  
  19.     static {
  20.         props.put("bootstrap.servers", "192.168.142.139:9092");
  21.         // 0:producer不会等待broker发送ack
  22.         // 1:当leader接收到消息后发送ack
  23.         // -1:当所有的follower都同步消息成功后发送ack
  24.         props.put("acks", "-1");//
  25.         props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
  26.         props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
  27.     }
  28.  
  29.     final KafkaProducer<String, String> kafkaProducer = new KafkaProducer<String, String>(props);
  30.  
  31.     private String topicName;
  32.  
  33.     public SynKafkaProducer(String topicName) {
  34.         this.topicName = topicName;
  35.     }
  36.  
  37.     public RecordMetadata send(String key, String value) {
  38.         RecordMetadata recordMetadata = null;
  39.         try {
  40.             recordMetadata = kafkaProducer.send(new ProducerRecord<String, String>(topicName,
  41.                     key, value)).get();//get方法将阻塞，直到返回结果RecordMetadata
  42.         } catch (InterruptedException e) {
  43.             //进行日志记录或者异常处理
  44.         } catch (ExecutionException e) {
  45.             //进行日志记录或者异常处理
  46.         }
  47.         return recordMetadata;
  48.     }
  49.  
  50.     public void close() {
  51.         if (kafkaProducer != null) {
  52.             kafkaProducer.close();
  53.         }
  54.     }
  55.  
  56.     public static void main(String[] args) {
  57.         SynKafkaProducer synKafkaProducer = new SynKafkaProducer("kafka-test");
  58.         for (int i = 0; i < 10; i++) {
  59.             RecordMetadata metadata = synKafkaProducer.send(String.valueOf(i), "This is " + i +
  60.                     " message");
  61.             System.out.println("TopicName : " + metadata.topic() + " Partiton : " + metadata
  62.                     .partition() + " Offset : " + metadata.offset());
  63.         }
  64.         synKafkaProducer.close();
  65.     }
  66. }

运行结果

  • producer异步发送消息
  1. package com.huawei.kafka.producer;
  2.  
  3. import org.apache.kafka.clients.producer.Callback;
  4. import org.apache.kafka.clients.producer.KafkaProducer;
  5. import org.apache.kafka.clients.producer.ProducerRecord;
  6. import org.apache.kafka.clients.producer.RecordMetadata;
  7.  
  8. import java.util.Properties;
  9. import java.util.concurrent.ExecutionException;
  10.  
  11. /**
  12.  * @author: xuqiangnj@163.com
  13.  * @date: 2019/4/15 21:59
  14.  * @description:
  15.  */
  16. public class AsynKafkaProducer {
  17.  
  18.     public static final Properties props = new Properties();
  19.  
  20.     static {
  21.         props.put("bootstrap.servers", "192.168.142.139:9092");
  22.         // 0:producer不会等待broker发送ack
  23.         // 1:当leader接收到消息后发送ack
  24.         // -1:当所有的follower都同步消息成功后发送ack
  25.         props.put("acks", "1");//
  26.         props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
  27.         props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
  28.     }
  29.  
  30.     final KafkaProducer<String, String> kafkaProducer = new KafkaProducer<String, String>(props);
  31.  
  32.     private String topicName;
  33.  
  34.     public AsynKafkaProducer(String topicName) {
  35.         this.topicName = topicName;
  36.     }
  37.  
  38.     public void send(String key, String value) {
  39.         kafkaProducer.send(new ProducerRecord<String, String>(topicName,
  40.                 key, value), new Callback() {
  41.             public void onCompletion(RecordMetadata metadata, Exception exception) {
  42.                 if (exception == null){
  43.                     System.out.println("TopicName : " + metadata.topic() + " Partiton : " + metadata
  44.                             .partition() + " Offset : " + metadata.offset());
  45.                 }
  46.                 else {
  47.                     //进行异常处理
  48.                 }
  49.             }
  50.         });
  51.     }
  52.  
  53.     public void close() {
  54.         if (kafkaProducer != null) {
  55.             kafkaProducer.close();
  56.         }
  57.     }
  58.  
  59.     public static void main(String[] args) {
  60.         AsynKafkaProducer synKafkaProducer = new AsynKafkaProducer("kafka-test");
  61.         for (int i = 0; i < 10; i++) {
  62.             synKafkaProducer.send(String.valueOf(i), "This is " + i + " message");
  63.         }
  64.         try {
  65.             Thread.sleep(2000);//这里阻塞两秒钟 等待回调函数打印结果
  66.         } catch (InterruptedException e) {
  67.             e.printStackTrace();
  68.         }
  69.         synKafkaProducer.close();
  70.     }
  71.  
  72. }

运行结果

同步发送消息

优点：可以保证每条消息准确无误的写入了broker，对于立即需要发送结果的情况非常适用，在producer故障或者宕机的情况也可以保证结果的正确性

缺点：由于同步发送需要每条消息都需要及时发送到broker，没有缓冲批量操作，性能较低

异步发送消息

优点：可以通过缓冲池对消息进行缓冲，然后进行消息的批量发送，大量减少了和broker的交互频率，性能极高，可以通过回调机制获取发送结果

缺点：在producer直接断电或者重启等故障，将有可能丢失消息发送结果，对消息准确性要求很高的场景不适用

producer使用非常简单

1.配置producer参数

2.构造ProducerRecord消息

3.调用send方法进行发送

4.最后关闭producer资源

Producer参数说明

  • bootstrap.servers：Kafka群集信息列表，用于连接kafka集群，如果集群中机器数很多，只需要指定部分的机器主机的信息即可，不管指定多少台，producer都会通过其中一台机器发现集群中所有的broker，一般指定3台机器的信息即可，格式为 hostname1:port,hostname2:port,hostname3:port
  • key.serializer：任何消息发送到broker格式都是字节数组，因此在发送到broker之前需要进行序列化，该参数用于对ProducerRecord中的key进行序列化
  • value.serializer：该参数用于对ProducerRecord中的value进行序列化

kafka默认提供了常用的序列化类，也可以通过实现org.apache.kafka.common.serialization.Serializer实现定义的序列化，在下面说明消息序列化详细列出 

  • acks：ack具有3个取值 0、1和-1(all)

acks=0：发送消息时，完全不用等待broker的处理结果，即可进行下一次发送，吞吐量最高

acks=-1或all：不仅需要leader broker将消息持久化，同时还需要等待ISR副本（in-sync replica 即与leader节点保持消息同步的副本集合）全部都成功持久化消息后，才返回响应结果，吞吐量最低

ack=1：是前面两个的折中，只需要leader broker持久化后，便响应结果

  • buffer.memory：该参数用于设置发送缓冲池的内存大小，单位是字节。默认值33554432KB（32M）
  • compression.type：压缩类型，目前kafka支持四种种压缩类型gzip、snappy、lz4、zstd，性能依次递增。默认值none（不压缩）
  • retries：重试次数，0表示不进行重试。默认值2147483647
  • batch.size：批次处理大小，通常增加该参数，可以提升producer的吞吐量
  • linger.ms：发送时延，和batch.size任满足其一条件，就会进行发送
  • max.block.ms：当producer缓冲满了之后，阻塞的时间
  • max.request.size：控制producer发送请求的大小
  • receive.buffer.bytes：读取数据时要使用的TCP接收缓冲区（SO_RCVBUF）的大小。如果值为-1，则将使用OS默认值。默认值32768
  • send. buffer.bytes：发送数据时使用的TCP发送缓冲区（SO_SNDBUF）的大小。如果值为-1，则将使用OS默认值。默认值131072
  • request.timeout.ms：生产者发送数据时等待服务器返回响应的时间。默认值30000ms

消息分区机制

  • 消息分区策略：kafka发送消息到broker之前需要确定该消息的具体分区信息，kafka默认提供分区器，该分区机制为如果在ProducerRecord中指定了key，那么将根据key的值进行hash计算，如果没有则采用轮询方式进行，除此之外，还可以在构造ProducerRecord的时候指定分区信息，Kafka将优先使用指定的分区信息，避免进行计算提升了性能，但是客户端在指定分区信息时需要考虑数据均衡问题。
  • 自定义分区机制：可以通过实现org.apache.kafka.clients.producer.Partitioner自定分区策略，在构造KafkaProducer是配置参数partitioner.class为自定义的分区类即可

消息序列化

  • 默认序列化：下图为kafka实现的默认序列化与反序列器，主要为常用类型序列化器

  • 自定义序列化：可以通过org.apache.kafka.common.serialization.Serializer来自定义序列化器

消息压缩

  • 压缩类型：kafka压缩类型有5个有效值 none（默认值）、gzip、snappy、lz4、zstd
  • 性能比较(压缩比和压缩速率)： gzip << snappy << lz4 << zstd

多线程处理

producer实例是线程安全的，在多线程环境，一般有两种使用方式

  • 多线程单实例：实现简单，所有线程共用一个KafkaProducer实例，内存开销小（使用一个内存缓冲区），但是如果某个线程发送阻塞或则崩溃将会影响到所有线程的正常发送。
  • 多线程多实例：可以使用一对一（即一个线程一个实例）或者M对N（即M个线程共同使用N个实例）两种方式，一对一方式每个线程维护自己的KafkaProducer实例，管理简单，一个线程崩溃，不会影响到其他，M对N方式可以达到资源利用率最大化，根据实际线程数来调整KafkaProducer实例资源的个数，而且在某个KafkaProducer崩溃，还可以切换到其他实例上，总体上更佳，它们相比于单实例需要消耗更多的内存资源并且实现更为复杂。

Producer发送流程源码解析

从上面使用producer代码示例可以看出，producer发送消息就是调用send方法

producer send()方法实现

  1. // 异步向一个 topic 发送数据
  2. @Override
  3. public Future<RecordMetadata> send(ProducerRecord<K, V> record) {
  4. 	return send(record, null);
  5. }
  6.  
  7. // 异步向一个 topic 发送数据，并注册回调函数，在回调函数中接受发送响应
  8. @Override
  9. public Future<RecordMetadata> send(ProducerRecord<K, V> record, Callback callback) {
  10. 	// intercept the record, which can be potentially modified; this method does not throw exceptions
  11. 	ProducerRecord<K, V> interceptedRecord = this.interceptors.onSend(record);
  12. 	return doSend(interceptedRecord, callback);
  13. }

send() 方法通过重载实现带回调和不带回调的参数，最终调用 doSend() 方法

producer doSend()方法实现

  1. //实现异步发送消息到对应topic
  2. private Future<RecordMetadata> doSend(ProducerRecord<K, V> record, Callback callback) {
  3. 	TopicPartition tp = null;
  4. 	try {
  5. 		//1、检查producer实例是否已关闭,如果关闭则抛出异常
  6. 		throwIfProducerClosed();
  7. 		ClusterAndWaitTime clusterAndWaitTime;
  8. 		try {
  9. 			//2、确保topic的元数据(metadata)是可用的
  10. 			clusterAndWaitTime = waitOnMetadata(record.topic(), record.partition(), maxBlockTimeMs);
  11. 		} catch (KafkaException e) {
  12. 			if (metadata.isClosed())
  13. 				throw new KafkaException("Producer closed while send in progress", e);
  14. 			throw e;
  15. 		}
  16. 		
  17. 		long remainingWaitMs = Math.max(0, maxBlockTimeMs - clusterAndWaitTime.waitedOnMetadataMs);
  18. 		Cluster cluster = clusterAndWaitTime.cluster;
  19. 		//3、序列化key
  20. 		byte[] serializedKey;
  21. 		try {
  22. 			serializedKey = keySerializer.serialize(record.topic(), record.headers(), record.key());
  23. 		} catch (ClassCastException cce) {
  24. 			throw new SerializationException("Can't convert key of class " + record.key().getClass().getName() +
  25. 					" to class " + producerConfig.getClass(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG).getName() +
  26. 					" specified in key.serializer", cce);
  27. 		}
  28. 		//4、序列化value
  29. 		byte[] serializedValue;
  30. 		try {
  31. 			serializedValue = valueSerializer.serialize(record.topic(), record.headers(), record.value());
  32. 		} catch (ClassCastException cce) {
  33. 			throw new SerializationException("Can't convert value of class " + record.value().getClass().getName() +
  34. 					" to class " + producerConfig.getClass(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG).getName() +
  35. 					" specified in value.serializer", cce);
  36. 		}
  37. 		//5、获取分区信息,如果ProducerRecord指定了分区信息就使用该指定分区否则通过计算获取
  38. 		int partition = partition(record, serializedKey, serializedValue, cluster);
  39. 		tp = new TopicPartition(record.topic(), partition);
  40.  
  41. 		setReadOnly(record.headers());
  42. 		Header[] headers = record.headers().toArray();
  43. 		//6、估算消息的字节大小
  44. 		int serializedSize = AbstractRecords.estimateSizeInBytesUpperBound(apiVersions.maxUsableProduceMagic(),
  45. 				compressionType, serializedKey, serializedValue, headers);
  46. 		//7、确保消息大小不超过发送请求最大值(max.request.size)或者发送缓冲池发小(buffer.memory)
  47. 		ensureValidRecordSize(serializedSize);
  48. 		
  49. 		long timestamp = record.timestamp() == null ? time.milliseconds() : record.timestamp();
  50. 		log.trace("Sending record {} with callback {} to topic {} partition {}", record, callback, record.topic(), partition);
  51. 		// producer callback will make sure to call both 'callback' and interceptor callback
  52. 		Callback interceptCallback = new InterceptorCallback<>(callback, this.interceptors, tp);
  53.                 //8、是否使用事务
  54. 		if (transactionManager != null && transactionManager.isTransactional())
  55. 			transactionManager.maybeAddPartitionToTransaction(tp);
  56. 		//9、向 accumulator 中追加数据
  57. 		RecordAccumulator.RecordAppendResult result = accumulator.append(tp, timestamp, serializedKey,
  58. 				serializedValue, headers, interceptCallback, remainingWaitMs);
  59. 		//10、如果 batch 已经满了,唤醒 sender 线程发送数据
  60. 		if (result.batchIsFull || result.newBatchCreated) {
  61. 			log.trace("Waking up the sender since topic {} partition {} is either full or getting a new batch", record.topic(), partition);
  62. 			this.sender.wakeup();
  63. 		}
  64. 		return result.future;
  65. 		// handling exceptions and record the errors;
  66. 		// for API exceptions return them in the future,
  67. 		// for other exceptions throw directly
  68. 	} catch (ApiException e) {
  69. 		.............
  70. 	}
  71.         ..........
  72. }

doSend()方法主要分为10步完成：

1.检查producer实例是否已关闭,如果关闭则抛出异常

2.确保topic的元数据(metadata)是可用的

3.序列化ProducerRecord中的key

4.序列化ProducerRecord中的value

5.确定分区信息,如果构造ProducerRecord指定了分区信息就使用该指定分区否则通过计算获取

6.估算整条消息的字节大小

7.确保消息大小不超过发送请求最大值(max.request.size)或者发送缓冲池发小(buffer.memory)，如果超过则抛出异常

8.是否使用事务，如果使用则按照事务流程进行

9.向 accumulator 中追加数据

10.如果 batch 已经满了,唤醒 sender 线程发送数据

发送过程详解

获取 topic 的 metadata 信息

Producer 通过 waitOnMetadata() 方法来获取对应 topic 的 metadata 信息，具体实现如下

  1. //如果元数据Topic列表中还没有该Topic，则将其添加到元数据Topic列表中，并重置过期时间
  2. private ClusterAndWaitTime waitOnMetadata(String topic, Integer partition, long maxWaitMs) throws InterruptedException {
  3. 	//首先从元数据中获取集群信息
  4. 	Cluster cluster = metadata.fetch();
  5. 	//集群的无效topic列表包含该topic那么抛出异常
  6. 	if (cluster.invalidTopics().contains(topic))
  7. 		throw new InvalidTopicException(topic);
  8. 	//否则将topic添加到Topic列表中
  9. 	metadata.add(topic);
  10. 	
  11. 	//获取该topic的分区数
  12. 	Integer partitionsCount = cluster.partitionCountForTopic(topic);
  13. 	//如果存在缓存的元数据，并且未指定分区的或者在已知分区范围内，那么返回缓存的元数据
  14. 	if (partitionsCount != null && (partition == null || partition < partitionsCount))
  15. 		return new ClusterAndWaitTime(cluster, 0);
  16.  
  17. 	long begin = time.milliseconds();
  18. 	long remainingWaitMs = maxWaitMs;
  19. 	long elapsed;
  20. 	// Issue metadata requests until we have metadata for the topic and the requested partition,
  21. 	// or until maxWaitTimeMs is exceeded. This is necessary in case the metadata
  22. 	// is stale and the number of partitions for this topic has increased in the meantime.
  23. 	//发出元数据请求，直到主题和所请求分区的元数据出现，或者超过maxWaitTimeMs为止。
  24. 	//同时可以有效检测元数据已经过期或者主题的分区数量增加。
  25. 	do {
  26. 		if (partition != null) {
  27. 			log.trace("Requesting metadata update for partition {} of topic {}.", partition, topic);
  28. 		} else {
  29. 			log.trace("Requesting metadata update for topic {}.", topic);
  30. 		}
  31. 		metadata.add(topic);
  32. 		int version = metadata.requestUpdate();
  33. 		sender.wakeup();
  34. 		try {
  35. 			metadata.awaitUpdate(version, remainingWaitMs);
  36. 		} catch (TimeoutException ex) {
  37. 			// Rethrow with original maxWaitMs to prevent logging exception with remainingWaitMs
  38. 			throw new TimeoutException(
  39. 					String.format("Topic %s not present in metadata after %d ms.",
  40. 							topic, maxWaitMs));
  41. 		}
  42. 		cluster = metadata.fetch();
  43. 		elapsed = time.milliseconds() - begin;
  44. 		if (elapsed >= maxWaitMs) {
  45. 			throw new TimeoutException(partitionsCount == null ?
  46. 					String.format("Topic %s not present in metadata after %d ms.",
  47. 							topic, maxWaitMs) :
  48. 					String.format("Partition %d of topic %s with partition count %d is not present in metadata after %d ms.",
  49. 							partition, topic, partitionsCount, maxWaitMs));
  50. 		}
  51. 		if (cluster.unauthorizedTopics().contains(topic))
  52. 			throw new TopicAuthorizationException(topic);
  53. 		if (cluster.invalidTopics().contains(topic))
  54. 			throw new InvalidTopicException(topic);
  55. 		remainingWaitMs = maxWaitMs - elapsed;
  56. 		partitionsCount = cluster.partitionCountForTopic(topic);
  57. 	} while (partitionsCount == null || (partition != null && partition >= partitionsCount));
  58. 	//返回集群信息和获取信息等待的时间
  59. 	return new ClusterAndWaitTime(cluster, elapsed);
  60. }

key 和 value 的序列化

对于发送到broker的信息都需要进行序列化，根据在构造Producer的参数中指定或者构造方法中指定序列化方式

确定partition值

关于 partition 值的计算，分为三种情况：

  1. 指明 partition 的情况下，直接将指定的值直接作为 partiton 值；
  2. 没有指明 partition 值但有 key 的情况下，将 key 的 hash 值与 topic 的 partition 数进行取余得到 partition 值；
  3. 既没有 partition 值又没有 key 值的情况下，第一次调用时随机生成一个整数（后面每次调用在这个整数上自增），将这个值与 topic 可用的 partition 总数取余得到 partition 值，也就是常说的轮询算法。

具体实现如下

  1. private int partition(ProducerRecord<K, V> record, byte[] serializedKey, byte[] serializedValue, Cluster cluster) {
  2. 	Integer partition = record.partition();
  3. 	//如果指定了partition,那么直接使用该值 否则使用分区器计算
  4. 	return partition != null ? partition : partitioner.partition(
  5. 			record.topic(), record.key(), serializedKey, record.value(), serializedValue, cluster);
  6. }
  7.  
  8. //默认使用的 partitioner是org.apache.kafka.clients.producer.internals.DefaultPartitioner,分区实现为
  9. public int partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster) {
  10. 	List<PartitionInfo> partitions = cluster.partitionsForTopic(topic);
  11. 	int numPartitions = partitions.size();
  12. 	if (keyBytes == null) {
  13. 		//不指定key时,根据 topic 获取对应的整数变量
  14. 		int nextValue = nextValue(topic);
  15. 		List<PartitionInfo> availablePartitions = cluster.availablePartitionsForTopic(topic);
  16. 		if (availablePartitions.size() > 0) {
  17. 			int part = Utils.toPositive(nextValue) % availablePartitions.size();
  18. 			return availablePartitions.get(part).partition();
  19. 		} else {
  20. 			// no partitions are available, give a non-available partition
  21. 			return Utils.toPositive(nextValue) % numPartitions;
  22. 		}
  23. 	} else {
  24. 		//指定key时，通过key进行hash运算然后对该topic分区总数取余
  25. 		return Utils.toPositive(Utils.murmur2(keyBytes)) % numPartitions;
  26. 	}
  27. }
  28.  
  29. private int nextValue(String topic) {
  30. 	AtomicInteger counter = topicCounterMap.get(topic);
  31. 	if (null == counter) {
  32.         //第一次随机生成一个数
  33. 		counter = new AtomicInteger(ThreadLocalRandom.current().nextInt());
  34. 		AtomicInteger currentCounter = topicCounterMap.putIfAbsent(topic, counter);
  35. 		if (currentCounter != null) {
  36. 			counter = currentCounter;
  37. 		}
  38. 	}
  39.     //以后每次递增
  40. 	return counter.getAndIncrement();
  41. }

估算消息大小并检查

校验消息是为了防止消息过大或者数量过多，导致内存异常，具体实现

  1. //估算消息大小
  2. int serializedSize = AbstractRecords.estimateSizeInBytesUpperBound(apiVersions.maxUsableProduceMagic(),
  3.                     compressionType, serializedKey, serializedValue, headers);
  4. //确保消息大小在有效范围内
  5. ensureValidRecordSize(serializedSize);
  6.  
  7. private void ensureValidRecordSize(int size) {
  8. 	//是否超过最大请求的限制,如果超过抛出异常
  9. 	if (size > this.maxRequestSize)
  10. 		throw new RecordTooLargeException("The message is " + size +
  11. 				" bytes when serialized which is larger than the maximum request size you have configured with the " +
  12. 				ProducerConfig.MAX_REQUEST_SIZE_CONFIG +
  13. 				" configuration.");
  14. 	//是否超过最大内存缓冲池大小,如果超过抛出异常
  15. 	if (size > this.totalMemorySize)
  16. 		throw new RecordTooLargeException("The message is " + size +
  17. 				" bytes when serialized which is larger than the total memory buffer you have configured with the " +
  18. 				ProducerConfig.BUFFER_MEMORY_CONFIG +
  19. 				" configuration.");
  20. }

向 accumulator 写数据

  1. public RecordAppendResult append(TopicPartition tp,
  2.                                      long timestamp,
  3.                                      byte[] key,
  4.                                      byte[] value,
  5.                                      Header[] headers,
  6.                                      Callback callback,
  7.                                      long maxTimeToBlock) throws InterruptedException {
  8. 	// We keep track of the number of appending thread to make sure we do not miss batches in
  9. 	// abortIncompleteBatches().
  10. 	appendsInProgress.incrementAndGet();
  11. 	ByteBuffer buffer = null;
  12. 	if (headers == null) headers = Record.EMPTY_HEADERS;
  13. 	try {
  14. 		// 当前tp有对应queue时直接返回，否则新建一个返回
  15. 		Deque<ProducerBatch> dq = getOrCreateDeque(tp);
  16. 		// 在对一个 queue 进行操作时,会保证线程安全
  17. 		synchronized (dq) {
  18. 			if (closed)
  19. 				throw new KafkaException("Producer closed while send in progress");
  20. 			RecordAppendResult appendResult = tryAppend(timestamp, key, value, headers, callback, dq);
  21. 			if (appendResult != null)
  22. 				return appendResult;
  23. 		}
  24.  
  25. 		// 为 topic-partition 创建一个新的 RecordBatch
  26. 		byte maxUsableMagic = apiVersions.maxUsableProduceMagic();
  27. 		int size = Math.max(this.batchSize, AbstractRecords.estimateSizeInBytesUpperBound(maxUsableMagic, compression, key, value, headers));
  28. 		log.trace("Allocating a new {} byte message buffer for topic {} partition {}", size, tp.topic(), tp.partition());
  29. 		// 给这个 RecordBatch 初始化一个 buffer
  30. 		buffer = free.allocate(size, maxTimeToBlock);
  31. 		synchronized (dq) {
  32. 			if (closed)
  33. 				throw new KafkaException("Producer closed while send in progress");
  34.  
  35. 			RecordAppendResult appendResult = tryAppend(timestamp, key, value, headers, callback, dq);
  36. 			if (appendResult != null) {
  37. 				// 如果突然发现这个 queue 已经存在，那么就释放这个已经分配的空间
  38. 				return appendResult;
  39. 			}
  40. 			// 给 topic-partition 创建一个 RecordBatch
  41. 			MemoryRecordsBuilder recordsBuilder = recordsBuilder(buffer, maxUsableMagic);
  42. 			ProducerBatch batch = new ProducerBatch(tp, recordsBuilder, time.milliseconds());
  43. 			// 向新的 RecordBatch 中追加数据
  44. 			FutureRecordMetadata future = Utils.notNull(batch.tryAppend(timestamp, key, value, headers, callback, time.milliseconds()));
  45. 			// 将 RecordBatch 添加到对应的 queue 中
  46. 			dq.addLast(batch);
  47. 			// 向未 ack 的 batch 集合添加这个 batch
  48. 			incomplete.add(batch);
  49.             
  50. 			buffer = null;
  51. 			// 如果 dp.size()>1 就证明这个 queue 有一个 batch 是可以发送了
  52. 			return new RecordAppendResult(future, dq.size() > 1 || batch.isFull(), true);
  53. 		}
  54. 	} finally {
  55. 		if (buffer != null)
  56. 			free.deallocate(buffer);
  57. 		appendsInProgress.decrementAndGet();
  58. 	}
  59. }

使用sender线程批量发送

  1. //开始启动KafkaProducer I/O线程
  2. public void run() {
  3. 	while (running) {
  4. 		try {
  5. 			//执行IO线程逻辑
  6. 			run(time.milliseconds());
  7. 		} catch (Exception e) {
  8. 			log.error("Uncaught error in kafka producer I/O thread: ", e);
  9. 		}
  10. 	}
  11.  
  12. 	// 累加器中可能仍然有请求，或者等待确认,等待他们完成就停止接受请求
  13. 	while (!forceClose && (this.accumulator.hasUndrained() || this.client.inFlightRequestCount() > 0)) {
  14. 		try {
  15. 			run(time.milliseconds());
  16. 		} catch (Exception e) {
  17. 			log.error("Uncaught error in kafka producer I/O thread: ", e);
  18. 		}
  19. 	}
  20. 	if (forceClose) {
  21. 		this.accumulator.abortIncompleteBatches();
  22. 	}
  23. 	try {
  24. 		this.client.close();
  25. 	} catch (Exception e) {
  26. 		log.error("Failed to close network client", e);
  27. 	}
  28. }
  29.  
  30. void run(long now) {
  31. 	//首先还是判断是否使用事务发送
  32. 	if (transactionManager != null) {
  33. 		try {
  34. 			if (transactionManager.shouldResetProducerStateAfterResolvingSequences())
  35. 				transactionManager.resetProducerId();
  36. 			if (!transactionManager.isTransactional()) {
  37. 				maybeWaitForProducerId();
  38. 			} else if (transactionManager.hasUnresolvedSequences() && !transactionManager.hasFatalError()) {
  39. 				transactionManager.transitionToFatalError(
  40. 					new KafkaException("The client hasn't received acknowledgment for " +
  41. 						"some previously sent messages and can no longer retry them. It isn't safe to continue."));
  42. 			} else if (transactionManager.hasInFlightTransactionalRequest() || maybeSendTransactionalRequest(now)) {
  43. 				// as long as there are outstanding transactional requests, we simply wait for them to return
  44. 				client.poll(retryBackoffMs, now);
  45. 				return;
  46. 			}
  47.  
  48. 			//如果事务管理器转态错误 或者producer没有id 将停止进行发送
  49. 			if (transactionManager.hasFatalError() || !transactionManager.hasProducerId()) {
  50. 				RuntimeException lastError = transactionManager.lastError();
  51. 				if (lastError != null)
  52. 					maybeAbortBatches(lastError);
  53. 				client.poll(retryBackoffMs, now);
  54. 				return;
  55. 			} else if (transactionManager.hasAbortableError()) {
  56. 				accumulator.abortUndrainedBatches(transactionManager.lastError());
  57. 			}
  58. 		} catch (AuthenticationException e) {
  59. 			transactionManager.authenticationFailed(e);
  60. 		}
  61. 	}
  62. 	//发送Producer数据
  63. 	long pollTimeout = sendProducerData(now);
  64. 	client.poll(pollTimeout, now);
  65. }
  66.  
  67. private long sendProducerData(long now) {
  68. 	Cluster cluster = metadata.fetch();
  69. 	// 获取准备发送数据的分区列表
  70. 	RecordAccumulator.ReadyCheckResult result = this.accumulator.ready(cluster, now);
  71.  
  72. 	//如果有分区leader信息是未知的,那么就强制更新metadata
  73. 	if (!result.unknownLeaderTopics.isEmpty()) {
  74. 		for (String topic : result.unknownLeaderTopics)
  75. 			this.metadata.add(topic);
  76. 		this.metadata.requestUpdate();
  77. 	}
  78.  
  79. 	// 移除没有准备好发送的Node
  80. 	Iterator<Node> iter = result.readyNodes.iterator();
  81. 	long notReadyTimeout = Long.MAX_VALUE;
  82. 	while (iter.hasNext()) {
  83. 		Node node = iter.next();
  84. 		if (!this.client.ready(node, now)) {
  85. 			iter.remove();
  86. 			notReadyTimeout = Math.min(notReadyTimeout, this.client.pollDelayMs(node, now));
  87. 		}
  88. 	}
  89.  
  90. 	//创建Producer请求内容
  91. 	Map<Integer, List<ProducerBatch>> batches = this.accumulator.drain(cluster, result.readyNodes, this.maxRequestSize, now);
  92. 	addToInflightBatches(batches);
  93. 	if (guaranteeMessageOrder) {
  94. 		// Mute all the partitions drained
  95. 		for (List<ProducerBatch> batchList : batches.values()) {
  96. 			for (ProducerBatch batch : batchList)
  97. 				this.accumulator.mutePartition(batch.topicPartition);
  98. 		}
  99. 	}
  100. 	// 累加器充值下一批次过期时间
  101. 	accumulator.resetNextBatchExpiryTime();
  102. 	List<ProducerBatch> expiredInflightBatches = getExpiredInflightBatches(now);
  103. 	List<ProducerBatch> expiredBatches = this.accumulator.expiredBatches(now);
  104. 	expiredBatches.addAll(expiredInflightBatches);
  105.  
  106. 	
  107. 	if (!expiredBatches.isEmpty())
  108. 		log.trace("Expired {} batches in accumulator", expiredBatches.size());
  109. 	for (ProducerBatch expiredBatch : expiredBatches) {
  110. 		String errorMessage = "Expiring " + expiredBatch.recordCount + " record(s) for " + expiredBatch.topicPartition
  111. 			+ ":" + (now - expiredBatch.createdMs) + " ms has passed since batch creation";
  112. 		failBatch(expiredBatch, -1, NO_TIMESTAMP, new TimeoutException(errorMessage), false);
  113. 		if (transactionManager != null && expiredBatch.inRetry()) {
  114. 			// This ensures that no new batches are drained until the current in flight batches are fully resolved.
  115. 			transactionManager.markSequenceUnresolved(expiredBatch.topicPartition);
  116. 		}
  117. 	}
  118. 	
  119. 	sensors.updateProduceRequestMetrics(batches);
  120.  
  121. 	long pollTimeout = Math.min(result.nextReadyCheckDelayMs, notReadyTimeout);
  122. 	pollTimeout = Math.min(pollTimeout, this.accumulator.nextExpiryTimeMs() - now);
  123. 	pollTimeout = Math.max(pollTimeout, 0);
  124. 	if (!result.readyNodes.isEmpty()) {
  125. 		pollTimeout = 0;
  126. 	}
  127. 	//发送ProduceRequest
  128. 	sendProduceRequests(batches, now);
  129. 	return pollTimeout;
  130. }
  131.  
  132. private void sendProduceRequests(Map<Integer, List<ProducerBatch>> collated, long now) {
  133. 	for (Map.Entry<Integer, List<ProducerBatch>> entry : collated.entrySet())
  134. 		sendProduceRequest(now, entry.getKey(), acks, requestTimeoutMs, entry.getValue());
  135. }
  136.  
  137. private void sendProduceRequest(long now, int destination, short acks, int timeout, List<ProducerBatch> batches) {
  138. 	if (batches.isEmpty())
  139. 		return;
  140.  
  141. 	Map<TopicPartition, MemoryRecords> produceRecordsByPartition = new HashMap<>(batches.size());
  142. 	final Map<TopicPartition, ProducerBatch> recordsByPartition = new HashMap<>(batches.size());
  143.  
  144. 	// 找到创建记录集时使用的最小版本
  145. 	byte minUsedMagic = apiVersions.maxUsableProduceMagic();
  146. 	for (ProducerBatch batch : batches) {
  147. 		if (batch.magic() < minUsedMagic)
  148. 			minUsedMagic = batch.magic();
  149. 	}
  150.  
  151. 	for (ProducerBatch batch : batches) {
  152. 		TopicPartition tp = batch.topicPartition;
  153. 		MemoryRecords records = batch.records();
  154. 		if (!records.hasMatchingMagic(minUsedMagic))
  155. 			records = batch.records().downConvert(minUsedMagic, 0, time).records();
  156. 		produceRecordsByPartition.put(tp, records);
  157. 		recordsByPartition.put(tp, batch);
  158. 	}
  159.  
  160. 	String transactionalId = null;
  161. 	if (transactionManager != null && transactionManager.isTransactional()) {
  162. 		transactionalId = transactionManager.transactionalId();
  163. 	}
  164. 	ProduceRequest.Builder requestBuilder = ProduceRequest.Builder.forMagic(minUsedMagic, acks, timeout,
  165. 			produceRecordsByPartition, transactionalId);
  166. 	RequestCompletionHandler callback = new RequestCompletionHandler() {
  167. 		public void onComplete(ClientResponse response) {
  168. 			handleProduceResponse(response, recordsByPartition, time.milliseconds());
  169. 		}
  170. 	};
  171.  
  172. 	String nodeId = Integer.toString(destination);
  173. 	ClientRequest clientRequest = client.newClientRequest(nodeId, requestBuilder, now, acks != 0,
  174. 			requestTimeoutMs, callback);
  175. 	//发送数据
  176. 	client.send(clientRequest, now);
  177. 	log.trace("Sent produce request to {}: {}", nodeId, requestBuilder);
  178. }

总结

kafka在发送端的设计非常的精妙，这也是为什么kafka能够达到如此高的性能，我们可以在实际项目中借鉴其中的思想，提高整体业务的效率

参考 ：

   kafka官方文档

《Apache Kafka实战》

(转载请注明出处)

 

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/盐析白兔/article/detail/618465
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号