devbox
从前慢现在也慢
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

26.RocketMQ之生产者发送消息源码

作者：从前慢现在也慢 | 2024-03-17 15:55:26

赞

踩

rocketmq发送消息源码

highlight: arduino-light

生产者发送消息的流程图

image.png

生产者发送消息的调用链路如下:

java org.apache.rocketmq.client.producer.DefaultMQProducer#send(org.apache.rocketmq.common.message.Message) ​ org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl#send(org.apache.rocketmq.common.message.Message) ​ org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl#send(org.apache.rocketmq.common.message.Message, long) ​ org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl#sendDefaultImpl

接下来通过源码一步一步分析生产者是如何发送消息的,注释都在代码里。

同步消息发送

DefaultMQProducerImpl#send

java //发送消息 public SendResult send(Message msg) { //this.defaultMQProducer.getSendMsgTimeout(),发送超时时间默认3秒 return send(msg, this.defaultMQProducer.getSendMsgTimeout()); }

DefaultMQProducerImpl#send

java //发送消息,默认超时时间为3s public SendResult send(Message msg,long timeout){ //发送消息的模式是CommunicationMode.SYNC,即同步 return this.sendDefaultImpl(msg, CommunicationMode.SYNC, null, timeout); } 第一个参数是要发送的消息。

第二个参数是发送消息的模式,同步、异步、单向等。

第三个参数是回调,由于是同步发送消息,不需要回调,只有异步发送消息才需要回调。

第四个参数是发送消息的超时时间,默认3秒。

具体发送消息的源码先贴出来,接下来一步一步分析。

DefaultMQProducerImpl#sendDefaultImpl

```java private SendResult sendDefaultImpl( Message msg, final CommunicationMode communicationMode, final SendCallback sendCallback, final long timeout ) throws MQClientException, RemotingException, MQBrokerException, InterruptedException { this.makeSureStateOK(); //1.校验消息 Validators.checkMessage(msg, this.defaultMQProducer);

  1. final long invokeID = random.nextLong();
  2.         //第一次发送消息的时间
  3.         long beginTimestampFirst = System.currentTimeMillis();
  4.         //上一次发送消息的时间
  5.         //第一次发送消息的时间-上一次发送消息的时间等于发送消息的花费时间
  6.         long beginTimestampPrev = beginTimestampFirst;
  7.         long endTimestamp = beginTimestampFirst;
  8.         //2.根据topic查找路由
  9.         //从本地的topicPublishInfoTable查找TopicPublishInfo。
  10.         //(TopicPublishInfo为空)或者(messageQueueList为empty或者null)
  11.         //那么从nameServer拉取。
  12.         //TopicPublishInfo主要包括messageQueueList,全局的sendWhichQueue。
  13.         //判断是否发生改变 若发生改变需要更新原来的路由信息
  14.         TopicPublishInfo topicPublishInfo=this.tryToFindTopicPublishInfo(msg.getTopic());
  15.  
  16.         //如果路由信息不为空 && 路由信息ok指的是topic对应的队列不为空
  17.         if (topicPublishInfo != null && topicPublishInfo.ok()) {
  18.             boolean callTimeout = false;
  19.             MessageQueue mq = null;
  20.             Exception exception = null;
  21.             SendResult sendResult = null;
  22.  
  23.             //private int retryTimesWhenSendFailed = 2;
  24.             //如果是同步模式 那么timesTotal是 1+2 =3,即会重试2次
  25.             //如果是异步发送 那么timesTotal是 1,既不会重试
  26.             //发送超时即花费时间超过超时时间不会再去重试,而是直接抛出异常。
  27.             //重试是在一个for循环里立刻去重试         
  28.             //总次数
  29.             int timesTotal = 
  30.                 communicationMode == CommunicationMode.SYNC ? 
  31.                     1 + this.defaultMQProducer.getRetryTimesWhenSendFailed() : 1;
  32.  
  33.             //已使用次数             
  34.             int times = 0;
  35.  
  36.             String[] brokersSent = new String[timesTotal];
  37.  
  38.             //for循环开始
  39.             for (; times < timesTotal; times++) {
  40.  
  41.                 //lastBrokerName是故障的brokerNmae,初始值是null
  42.                 //只有发送失败了才会再次进入循环
  43.                 //等第二次进入时lastBrokerName是上一次发送失败的brokerNmae
  44.                 String lastBrokerName = null == mq ? null : mq.getBrokerName();
  45.  
  46.                 // 3.选择队列
  47.                 // 这里的lastBrokerName第一次是null
  48.                 // 第二次lastBrokerName是上一次发送失败的brokerNmae
  49.                 MessageQueue mqSelected = 
  50.                         this.selectOneMessageQueue(topicPublishInfo, lastBrokerName);
  51.  
  52.                 if (mqSelected != null) {
  53.                     mq = mqSelected;
  54.                     //保存每次发送时的BrokerName到对应的数组下标
  55.                     brokersSent[times] = mq.getBrokerName();
  56.  
  57.                     try {
  58.                         beginTimestampPrev = System.currentTimeMillis();
  59.                      if (times > 0) {
  60.                       //Reset topic with namespace during resend.
  61.                       msg.setTopic(this.defaultMQProducer.withNamespace(msg.getTopic()));
  62.                     }
  63.                     //第一次发送消息的时间-上一次发送消息的时间等于发送消息的花费时间
  64.                     long costTime = beginTimestampPrev - beginTimestampFirst;
  65.                     //如果总的花费时间超过3秒那么认为超时 超时不在重试跳出循环然后直接报错
  66.                     //private int sendMsgTimeout = 3000;
  67.                     if (timeout < costTime) {
  68.                       callTimeout = true;
  69.                       break;
  70.                    }
  71.                         //核心代码:发送消息
  72.                          sendResult = this.sendKernelImpl(
  73.                          msg, 
  74.                          mq, 
  75.                          communicationMode, 
  76.                          sendCallback, 
  77.                          topicPublishInfo,
  78.                          timeout - costTime);
  79.  
  80.                         endTimestamp = System.currentTimeMillis();
  81.                         //更新不可用的broker
  82.                         this.updateFaultItem
  83.                                 (mq.getBrokerName(), 
  84.                                 endTimestamp - beginTimestampPrev, false);
  85.  
  86.  
  87.                         switch (communicationMode) {
  88.                             case ASYNC:
  89.                                 return null;
  90.                             case ONEWAY:
  91.                                 return null;
  92.                             case SYNC:
  93.                                 //同步模式返回    
  94.                                 if (sendResult.getSendStatus()!= SendStatus.SEND_OK) {
  95.                                     //isRetryAnotherBrokerWhenNotStoreOK默认是true
  96.                                     //代表当消息发送失败尝试发送到其他broker
  97.                                     if (this.defaultMQProducer
  98.                                     .isRetryAnotherBrokerWhenNotStoreOK()) {
  99.                                         continue;
  100.                                     }
  101.                                 }
  102.                                 return sendResult;
  103.                             default:
  104.                                 break;
  105.                         }
  106.                     } catch (RemotingException e) {
  107.                         endTimestamp = System.currentTimeMillis();
  108.                         this.updateFaultItem
  109.                             (mq.getBrokerName(), endTimestamp - beginTimestampPrev, true);
  110.                         exception = e;
  111.                         continue;
  112.                     } catch (MQClientException e) {
  113.                         endTimestamp = System.currentTimeMillis();
  114.                         this.updateFaultItem
  115.                             (mq.getBrokerName(), endTimestamp - beginTimestampPrev, true);
  116.                         exception = e;
  117.                         continue;
  118.                     } catch (MQBrokerException e) {
  119.                         endTimestamp = System.currentTimeMillis();
  120.                         this.updateFaultItem
  121.                             (mq.getBrokerName(), endTimestamp - beginTimestampPrev, true);
  122.  
  123.                         exception = e;
  124.                         switch (e.getResponseCode()) {
  125.                             case ResponseCode.TOPIC_NOT_EXIST:
  126.                             case ResponseCode.SERVICE_NOT_AVAILABLE:
  127.                             case ResponseCode.SYSTEM_ERROR:
  128.                             case ResponseCode.NO_PERMISSION:
  129.                             case ResponseCode.NO_BUYER_ID:
  130.                             case ResponseCode.NOT_IN_CURRENT_UNIT:
  131.                                 continue;
  132.                             default:
  133.                                 if (sendResult != null) {
  134.                                     return sendResult;
  135.                                 }
  136.                                 throw e;
  137.                         }
  138.                     } catch (InterruptedException e) {
  139.                         endTimestamp = System.currentTimeMillis();
  140.                         this.updateFaultItem
  141.                             (mq.getBrokerName(), endTimestamp - beginTimestampPrev, false);
  142.                         throw e;
  143.                     }
  144.                 } else {
  145.                     break;
  146.                 }
  147.             }
  148.             //for循环结束
  149.  
  150.             //返回发送结果
  151.             if (sendResult != null) {
  152.                 return sendResult;
  153.             }
  154.  
  155.             //如果发送结果为空
  156.             String info = String.format
  157.                 ("Send [%d] times, still failed, cost [%d]ms, Topic: %s, BrokersSent: %s",
  158.                 times,
  159.                 System.currentTimeMillis() - beginTimestampFirst,
  160.                 msg.getTopic(),
  161.                 Arrays.toString(brokersSent));
  162.  
  163.             info += FAQUrl.suggestTodo(FAQUrl.SEND_MSG_FAILED);
  164.  
  165.             MQClientException mqClientException = new MQClientException(info, exception);
  166.             //如果是发送超时直接报错
  167.             if (callTimeout) {
  168.                 throw new RemotingTooMuchRequestException("sendDefaultImpl call timeout");
  169.             }
  170.  
  171.             if (exception instanceof MQBrokerException) {
  172.                 mqClientException.setResponseCode(((MQBrokerException) exception).getResponseCode());
  173.             } else if (exception instanceof RemotingConnectException) {
  174.                 mqClientException.setResponseCode(ClientErrorCode.CONNECT_BROKER_EXCEPTION);
  175.             } else if (exception instanceof RemotingTimeoutException) {
  176.                 mqClientException.setResponseCode(ClientErrorCode.ACCESS_BROKER_TIMEOUT);
  177.             } else if (exception instanceof MQClientException) {
  178.                 mqClientException.setResponseCode(ClientErrorCode.BROKER_NOT_EXIST_EXCEPTION);
  179.             }
  180.             throw mqClientException;
  181.         //if (topicPublishInfo != null && topicPublishInfo.ok()) 判断结束
  182.         }
  183.  
  184.         //路由信息为空 或者 路由信息不ok即topic对应的队列为空
  185.         //获取nameserverList
  186.         List<String> nsList=this.getmQClientFactory()
  187.                                 .getMQClientAPIImpl()
  188.                                 .getNameServerAddressList();
  189.  
  190.         //如果nameserverList为空 抛出异常 没有nameserver的地址,请配置 
  191.         if (null == nsList || nsList.isEmpty()) {
  192.             throw new MQClientException(
  193.                 "No name server address, please set it.");
  194.         }
  195.         //直接抛出没有topic对应的路由信息的异常
  196.         throw new MQClientException("No route info of this topic, ");
  197.     }

```

验证消息大小

Validators#checkMessage

```java public static void checkMessage(Message msg, DefaultMQProducer defaultMQProducer) throws MQClientException { //判断是否为空 if (null == msg) { throw new MQClientException(ResponseCode.MESSAGE_ILLEGAL, "the message is null"); } // 校验Topic Validators.checkTopic(msg.getTopic());

  1. // 校验消息体
  2.     if (null == msg.getBody()) {
  3.         throw new 
  4.             MQClientException(ResponseCode.MESSAGE_ILLEGAL, "the message is null");
  5.     }
  6.  
  7.     if (0 == msg.getBody().length) {
  8.         throw new 
  9.             MQClientException(ResponseCode.MESSAGE_ILLEGAL, "the messagelength is zero");
  10.     }
  11.     //默认是4M 4194304=1024 * 1024 * 4
  12.     if (msg.getBody().length > defaultMQProducer.getMaxMessageSize()) {
  13.         throw new MQClientException(ResponseCode.MESSAGE_ILLEGAL,
  14.                         "the message body size over max value, MAX: ");
  15.     }
  16. }
  1. public static void checkTopic(String topic) throws MQClientException {
  2.         if (UtilAll.isBlank(topic)) {
  3.             throw new MQClientException("The specified topic is blank", null);
  4.         }
  5.  
  6.         if (!regularExpressionMatcher(topic, PATTERN)) {
  7.             throw new MQClientException
  8.                 ("The specified topic[%s] contains illegal characters, allowing only");
  9.         }
  10.  
  11.         if (topic.length() > CHARACTER_MAX_LENGTH) {
  12.             throw new MQClientException
  13.                         ("The specified topic is longer than topic max length 255.");
  14.         }
  15.  
  16.         //whether the same with system reserved keyword
  17.         if (topic.equals(MixAll.AUTO_CREATE_TOPIC_KEY_TOPIC)) {
  18.             throw new MQClientException
  19.                         ("The topic[%s] is conflict with AUTO_CREATE_TOPIC_KEY_TOPIC.");
  20.         }
  21.     }

```

缓存查找Topic||从服务器拉取Topic

DefaultMQProducerImpl#tryToFindTopicPublishInfo

```java private TopicPublishInfo tryToFindTopicPublishInfo(final String topic) { //从缓存中获得主题的路由信息 TopicPublishInfo topicPublishInfo = this.topicPublishInfoTable.get(topic); //缓存的路由信息为空,则从NameServer获取路由 if (null == topicPublishInfo || !topicPublishInfo.ok()) { this.topicPublishInfoTable.putIfAbsent(topic, new TopicPublishInfo()); this.mQClientFactory.updateTopicRouteInfoFromNameServer(topic); topicPublishInfo = this.topicPublishInfoTable.get(topic); }

  1. if (topicPublishInfo.isHaveTopicRouterInfo() || topicPublishInfo.ok()) {
  2.         return topicPublishInfo;
  3.     } else {
  4.         //MQClientInstance#startScheduledTask
  5.         //调用updateTopicRouteInfoFromNameServer起来一个定时器 30秒执行一次
  6.         //如果未找到当前主题的路由信息,则用默认主题继续查找 
  7.         //#进入下一个方法
  8.         this.mQClientFactory.updateTopicRouteInfoFromNameServer
  9.                                     (topic, true, this.defaultMQProducer);
  10.         topicPublishInfo = this.topicPublishInfoTable.get(topic);
  11.         return topicPublishInfo;
  12.     }
  13. }

**代码：返回值TopicPublishInfo** java public class TopicPublishInfo { private boolean orderTopic = false; //是否是顺序消息 private boolean haveTopicRouterInfo = false; //该主题对应的所有的消息队列 private List messageQueueList = new ArrayList ();
//每选择一次消息队列,该值+1 默认值是一个随机数 private volatile ThreadLocalIndex sendWhichQueue = new ThreadLocalIndex(); //关联Topic路由元信息 private TopicRouteData topicRouteData; } **返回值:TopicRouteData** java public class TopicRouteData extends RemotingSerializable { private String orderTopicConf; private List queueDatas; private List brokerDatas; private HashMap /* Filter Server */> filterServerTable; 

  1. public TopicRouteData cloneTopicRouteData() {
  2.         TopicRouteData topicRouteData = new TopicRouteData();
  3.         topicRouteData.setQueueDatas(new ArrayList<QueueData>());
  4.         topicRouteData.setBrokerDatas(new ArrayList<BrokerData>());
  5.         topicRouteData.setFilterServerTable(new HashMap<String, List<String>>());
  6.         topicRouteData.setOrderTopicConf(this.orderTopicConf);
  7.  
  8.         if (this.queueDatas != null) {
  9.             topicRouteData.getQueueDatas().addAll(this.queueDatas);
  10.         }
  11.  
  12.         if (this.brokerDatas != null) {
  13.             topicRouteData.getBrokerDatas().addAll(this.brokerDatas);
  14.         }
  15.  
  16.         if (this.filterServerTable != null) {
  17.             topicRouteData.getFilterServerTable().putAll(this.filterServerTable);
  18.         }
  19.  
  20.         return topicRouteData;
  21.     }

```

MQClientInstance#updateTopicRouteInfoFromNameServer

```java //使用默认主题从NameServer获取路由信息 返回的是TopicRouteData if (isDefault && defaultMQProducer != null) { topicRouteData = this.mQClientAPIImpl.getDefaultTopicRouteInfoFromNameServer (defaultMQProducer.getCreateTopicKey(),1000 * 3);

  1. if (topicRouteData != null) {
  2.         for (QueueData data : topicRouteData.getQueueDatas()) {
  3.             //defaultMQProducer.getDefaultTopicQueueNums() = 4
  4.             int queueNums 
  5.                 = Math.min(defaultMQProducer.getDefaultTopicQueueNums(), 
  6.                                                         data.getReadQueueNums());
  7.             data.setReadQueueNums(queueNums);
  8.             data.setWriteQueueNums(queueNums);
  9.         }
  10.     }
  11. } else {
  12.     //使用指定主题从NameServer获取路由信息
  13.     topicRouteData = 
  14.         this.mQClientAPIImpl.getTopicRouteInfoFromNameServer(topic, 1000 * 3);
  15. }
  16.  
  17. //判断路由是否发生变化 如果发生变化 那么需要更改路由信息
  18. TopicRouteData old = this.topicRouteTable.get(topic);
  19. boolean changed = topicRouteDataIsChange(old, topicRouteData);
  20. if (!changed) {
  21.     changed = this.isNeedUpdateTopicRouteInfo(topic);
  22. } else {
  23.     log.info("the topic[{}] route info changed, old[{}] ,new[{}]", topic, old, topicRouteData);
  24. }
  25.  
  26. if (changed) {
  27.     //将topicRouteData转换为发布队列
  28.     TopicPublishInfo publishInfo = topicRouteData2TopicPublishInfo(topic, topicRouteData);
  29.     publishInfo.setHaveTopicRouterInfo(true);
  30.     //遍历生产
  31.     Iterator<Entry<String, MQProducerInner>> it = this.producerTable.entrySet().iterator();
  32.     while (it.hasNext()) {
  33.         Entry<String, MQProducerInner> entry = it.next();
  34.         MQProducerInner impl = entry.getValue();
  35.         if (impl != null) {
  36.             //生产者不为空时,更新publishInfo信息
  37.             impl.updateTopicPublishInfo(topic, publishInfo);
  38.         }
  39.     }
  40. }

```

MQClientInstance#topicRouteData2TopicPublishInfo

```java public static TopicPublishInfo topicRouteData2TopicPublishInfo(final String topic, final TopicRouteData route) { //创建TopicPublishInfo对象 TopicPublishInfo info = new TopicPublishInfo(); //关联topicRoute info.setTopicRouteData(route); //顺序消息,更新TopicPublishInfo if (route.getOrderTopicConf() != null && route.getOrderTopicConf().length() > 0) { String[] brokers = route.getOrderTopicConf().split(";"); for (String broker : brokers) { String[] item = broker.split(":"); int nums = Integer.parseInt(item[1]); for (int i = 0; i < nums; i++) { MessageQueue mq = new MessageQueue(topic, item[0], i); info.getMessageQueueList().add(mq); } }

  1. info.setOrderTopic(true);
  2.         } else {
  3.             //非顺序消息更新TopicPublishInfo
  4.             List<QueueData> qds = route.getQueueDatas();
  5.             Collections.sort(qds);
  6.             //遍历topic队列信息
  7.             for (QueueData qd : qds) {
  8.                 //是否是写队列
  9.                 if (PermName.isWriteable(qd.getPerm())) {
  10.                     BrokerData brokerData = null;
  11.                     //遍历写队列Broker
  12.                     for (BrokerData bd : route.getBrokerDatas()) {
  13.                         //根据名称获得读队列对应的Broker
  14.                         if (bd.getBrokerName().equals(qd.getBrokerName())) {
  15.                         brokerData = bd;
  16.                         break;
  17.                     }
  18.                 }
  19.  
  20.                 if (null == brokerData) {
  21.                     continue;
  22.                 }
  23.  
  24.                 if (!brokerData.getBrokerAddrs().containsKey(MixAll.MASTER_ID)) {
  25.                     continue;
  26.                 }
  27.                 //封装TopicPublishInfo写队列
  28.                 for (int i = 0; i < qd.getWriteQueueNums(); i++) {
  29.                     MessageQueue mq = new MessageQueue(topic, qd.getBrokerName(), i);
  30.                     info.getMessageQueueList().add(mq);
  31.                 }
  32.             }
  33.         }
  34.         info.setOrderTopic(false);
  35.     }
  36.     //返回TopicPublishInfo对象
  37.     return info;
  38. }

```

选择队列:规避不可用队列,递增取模选择队列

DefaultMQProducerImpl#selectOneMessageQueue

java public MessageQueue selectOneMessageQueue(final TopicPublishInfo tpInfo, final String lastBrokerName) { return this.mqFaultStrategy.selectOneMessageQueue(tpInfo, lastBrokerName); } MQFaultStrategy#selectOneMessageQueue ```java public MessageQueue selectOneMessageQueue(final TopicPublishInfo tpInfo, final String lastBrokerName) { //Broker故障延迟机制 //默认不启用Broker故障延迟机制 if (this.sendLatencyFaultEnable) { try { //累加 int index = tpInfo.getSendWhichQueue().getAndIncrement(); //循环遍历规避不可用broker for (int i = 0; i < tpInfo.getMessageQueueList().size(); i++) { int pos = Math.abs(index++) % tpInfo.getMessageQueueList().size(); if (pos < 0) pos = 0; MessageQueue mq = tpInfo.getMessageQueueList().get(pos); //从不可用brokerMap中根据brokerName获取value //如果isAvailable是true 说明是可用的broker
if (latencyFaultTolerance.isAvailable(mq.getBrokerName())) { //如果lastBrokerName 说明是第一次进入该方法 if (null == lastBrokerName || mq.getBrokerName().equals(lastBrokerName)) return mq; } }

  1. //如果遍历完所有的broker发现都是出过故障的 
  2.                 //只能尝试从规避的Broker中选择一个可用的Broker
  3.                 //具体策略是从存放不可用的broker的Map中的元素放入List
  4.                 //shuffle以后再sort 再对size 取半 记为half
  5.                 // 如果 half 小于0 取0 
  6.                 // 否则  tmpList.get(whichItemWorst.getAndIncrement() % half).getName()
  7.  
  8.                 final String notBestBroker = latencyFaultTolerance.pickOneAtLeast();
  9.                 int writeQueueNums = tpInfo.getQueueIdByBroker(notBestBroker);
  10.                 if (writeQueueNums > 0) {
  11.                     final MessageQueue mq = tpInfo.selectOneMessageQueue();
  12.                     if (notBestBroker != null) {
  13.                         //此处设置了name selectOneMessageQueue没设置name
  14.                         //个人猜测设置了name 是根据name找到对应的queueList 再根据下标获取
  15.                         //没设置name 直接从全局的messageQueueList根据下标获取
  16.                         mq.setBrokerName(notBestBroker);
  17.                         //对该broker上的queueSize 取模选择队列 
  18.                         mq.setQueueId
  19.                           (tpInfo.getSendWhichQueue().getAndIncrement()%writeQueueNums);
  20.                     }
  21.                     return mq;
  22.                 } else {
  23.                     //写队列小于0 意味着没必要参与发送消息 所以从故障的brokerMap中移除notBestBroker
  24.                     latencyFaultTolerance.remove(notBestBroker);
  25.                 }
  26.             } catch (Exception e) {
  27.                 log.error("Error occurred when selecting message queue", e);
  28.             }
  29.  
  30.             return tpInfo.selectOneMessageQueue();
  31.         }
  32.  
  33.  
  34.         //进入TopicPublishInfo#selectOneMessageQueue(lastBrokerName)
  35.         return tpInfo.selectOneMessageQueue(lastBrokerName);
  36.     }
  37.  
  38.  
  39.     // 验证该broker是否可用 ,维护了一个存放不可用的broker的Map faultItemTable
  40.    //如果map里有这个broker 那么就是不可用 返回false 否则返回true
  41.     @Override
  42.     public boolean isAvailable(final String name) {
  43.         final FaultItem faultItem = this.faultItemTable.get(name);
  44.         if (faultItem != null) {
  45.             return faultItem.isAvailable();
  46.         }
  47.         return true;
  48.     }

```

这种规避Broker发送失败的方法有一个缺陷，如果broker宕机,如果上一次选择的队列是宕机broker的第一个队列，那么下次选择的是宕机broker的第二个队列，导致消息再次发送失败。那么我们就需要利用broker故障延迟机制，在第一次broker消息发送失败后就将该broker暂时排除。

参考:原文链接：https://blog.csdn.net/weixin_45003125/article/details/105093907

```java

  1. //存放不可用的broker的Map 根据brokerName获取
  2. public boolean isAvailable(final String name) {
  3.         final FaultItem faultItem = this.faultItemTable.get(name);
  4.         if (faultItem != null) {
  5.             return faultItem.isAvailable();
  6.         }
  7.         return true;
  8.     }
  9. //old.setStartTimestamp(System.currentTimeMillis() + notAvailableDuration);
  10. //因为startTimestamp 等于 update时的当前系统时间 + 规避时长
  11. //所以这里直接用当前的系统时间和startTimestamp 比较
  12.  public boolean isAvailable() { 
  13.      return (System.currentTimeMillis() - startTimestamp) >= 0;
  14.  }

```

TopicPublishInfo#selectOneMessageQueue(lastBrokerName)

java public MessageQueue selectOneMessageQueue(final String lastBrokerName) { //第一次选择队列 if (lastBrokerName == null) { return selectOneMessageQueue(); } else { //sendWhichQueue int index = this.sendWhichQueue.getAndIncrement(); //遍历消息队列集合 for (int i = 0; i < this.messageQueueList.size(); i++) { //sendWhichQueue自增后 对所有的queueSize取模 int pos = Math.abs(index++) % this.messageQueueList.size(); if (pos < 0) pos = 0; //尽量规避上次已经发送过的Broker队列 MessageQueue mq = this.messageQueueList.get(pos); if (!mq.getBrokerName().equals(lastBrokerName)) { return mq; } } //如果以上情况都不满足,返回sendWhichQueue取模后的队列 return selectOneMessageQueue(); } } 代码：TopicPublishInfo#selectOneMessageQueue() java //第一次选择队列 public MessageQueue selectOneMessageQueue() { //sendWhichQueue自增 int index = this.sendWhichQueue.getAndIncrement(); //对队列大小取模 int pos = Math.abs(index) % this.messageQueueList.size(); if (pos < 0) pos = 0; //返回对应的队列 return this.messageQueueList.get(pos); } image.png

  • 延迟机制接口规范

java public interface LatencyFaultTolerance<T> { //更新失败条目 void updateFaultItem(final T name, final long currentLatency, final long notAvailableDuration); //判断Broker是否可用 boolean isAvailable(final T name); //移除Fault条目 void remove(final T name); //尝试从规避的Broker中选择一个可用的Broker T pickOneAtLeast(); } * FaultItem：失败条目

java class FaultItem implements Comparable<FaultItem> { //条目唯一键,这里为brokerName private final String name; //本次消息发送延迟 private volatile long currentLatency; //故障规避开始时间 private volatile long startTimestamp; } * 消息失败策略

java public class MQFaultStrategy { //根据currentLatency本地消息发送延迟,从latencyMax尾部向前找到第一个比currentLatency小的索引,如果没有找到,返回0 private long[] latencyMax = {50L, 100L, 550L, 1000L, 2000L, 3000L, 15000L}; //根据这个索引从notAvailableDuration取出对应的时间,在该时长内,Broker设置为不可用 private long[] notAvailableDuration = {0L, 0L, 30000L, 60000L, 120000L, 180000L, 600000L}; } 原理分析

代码：DefaultMQProducerImpl#sendDefaultImpl java sendResult = this.sendKernelImpl(msg, mq, communicationMode, sendCallback, topicPublishInfo, timeout - costTime); endTimestamp = System.currentTimeMillis(); this.updateFaultItem(mq.getBrokerName(), endTimestamp - beginTimestampPrev, false); 如果上述发送过程出现异常，则调用DefaultMQProducerImpl#updateFaultItem java public void updateFaultItem(final String brokerName, final long currentLatency, boolean isolation) { //参数一：broker名称 //参数二:本次消息发送延迟时间 //参数三:是否隔离 this.mqFaultStrategy.updateFaultItem(brokerName, currentLatency, isolation); } 代码：MQFaultStrategy#updateFaultItem ```java public void updateFaultItem(final String brokerName, final long currentLatency, boolean isolation) { if (this.sendLatencyFaultEnable) { //计算broker规避的时长 long duration = computeNotAvailableDuration(isolation ? 30000 : currentLatency); //更新该FaultItem规避时长 this.latencyFaultTolerance.updateFaultItem(brokerName, currentLatency, duration); } }

代码：MQFaultStrategy#computeNotAvailableDuration

  1. private long computeNotAvailableDuration(final long currentLatency) {
  2.     //遍历latencyMax
  3.     for (int i = latencyMax.length - 1; i >= 0; i--) {
  4.         //找到第一个比currentLatency的latencyMax值
  5.         if (currentLatency >= latencyMax[i])
  6.             return this.notAvailableDuration[i];
  7.     }
  8.     //没有找到则返回0
  9.     return 0;
  10. }

**代码：LatencyFaultToleranceImpl#updateFaultItem** java public void updateFaultItem(final String name, final long currentLatency, final long notAvailableDuration) { //获得原FaultItem FaultItem old = this.faultItemTable.get(name); //为空新建faultItem对象,设置规避时长和开始时间 if (null == old) { final FaultItem faultItem = new FaultItem(name); faultItem.setCurrentLatency(currentLatency); faultItem.setStartTimestamp(System.currentTimeMillis() + notAvailableDuration);

  1. old = this.faultItemTable.putIfAbsent(name, faultItem);
  2.         if (old != null) {
  3.             old.setCurrentLatency(currentLatency);
  4.             old.setStartTimestamp(System.currentTimeMillis() + notAvailableDuration);
  5.         }
  6.     } else {
  7.         //更新规避时长和开始时间
  8.         old.setCurrentLatency(currentLatency);
  9.         old.setStartTimestamp(System.currentTimeMillis() + notAvailableDuration);
  10.     }
  11. }

```

发送消息

java org.apache.rocketmq.client.producer.DefaultMQProducer#send(org.apache.rocketmq.common.message.Message) ​ org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl#send(org.apache.rocketmq.common.message.Message) ​ org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl#send(org.apache.rocketmq.common.message.Message, long) ​ org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl#sendDefaultImpl    

DefaultMQProducerImpl#sendKernelImpl

```java private SendResult sendKernelImpl(    final Message msg, //待发送消息    final MessageQueue mq, //消息发送队列    final CommunicationMode communicationMode, //消息发送模式    final SendCallback sendCallback, //异步消息回调函数    final TopicPublishInfo topicPublishInfo, //主题路由信息    final long timeout //超时时间   ){ throws MQClientException, RemotingException, MQBrokerException, InterruptedException {            long beginStartTime = System.currentTimeMillis();            //获得broker网络地址信息 172.18.95.215:10911 //首先从缓存的map中找        String brokerAddr = this.mQClientFactory           .findBrokerAddressInPublish(mq.getBrokerName());       //没有找到从NameServer更新broker网络地址信息放入缓存map        if (null == brokerAddr) {            tryToFindTopicPublishInfo(mq.getTopic());            //从缓存的map中找            brokerAddr = this.mQClientFactory               .findBrokerAddressInPublish(mq.getBrokerName());       }

  1. SendMessageContext context = null;
  2.         if (brokerAddr != null) {
  3.             brokerAddr = MixAll
  4.                 .brokerVIPChannel(this.defaultMQProducer
  5.                                       .isSendMessageWithVIPChannel(), brokerAddr);
  6. ​
  7.             byte[] prevBody = msg.getBody();
  8.             try {
  9.                  
  10.                 //判断不是批量发送消息 为消息分配唯一ID
  11.                 if (!(msg instanceof MessageBatch)) {
  12.                     MessageClientIDSetter.setUniqID(msg);
  13.                 }
  14.  
  15.                 boolean topicWithNamespace = false;
  16.                 if (null != this.mQClientFactory.getClientConfig().getNamespace()) {
  17.                     msg.setInstanceId(this.mQClientFactory.getClientConfig()
  18.                                       .getNamespace());
  19.                     topicWithNamespace = true;
  20.                 }
  21.  
  22.                 //消息大小超过4K,启用消息压缩
  23.                 int sysFlag = 0;
  24.                 boolean msgBodyCompressed = false;
  25.                 if (this.tryToCompressMessage(msg)) {
  26.                     sysFlag |= MessageSysFlag.COMPRESSED_FLAG;
  27.                     msgBodyCompressed = true;
  28.                 }
  29.                 
  30.                 //如果是事务消息,设置消息标记MessageSysFlag.TRANSACTION_PREPARED_TYPE
  31.                 final String tranMsg = msg
  32.                     .getProperty(MessageConst.PROPERTY_TRANSACTION_PREPARED);
  33.                 
  34.                 if (tranMsg != null && Boolean.parseBoolean(tranMsg)) {
  35.                     sysFlag |= MessageSysFlag.TRANSACTION_PREPARED_TYPE;
  36.                 }
  37.                 
  38.                 //如果注册了消息发送钩子函数,在执行消息发送前的增强逻辑
  39.                 if (hasCheckForbiddenHook()) {
  40.                     
  41.                     CheckForbiddenContext 
  42.                         checkForbiddenContext = new CheckForbiddenContext();
  43.                     
  44.                     checkForbiddenContext
  45.                         .setNameSrvAddr(this.defaultMQProducer.getNamesrvAddr());
  46.                     
  47.                     checkForbiddenContext
  48.                         .setGroup(this.defaultMQProducer.getProducerGroup());
  49.                     
  50.                     checkForbiddenContext.
  51.                         setCommunicationMode(communicationMode);
  52.                     
  53.                     checkForbiddenContext.setBrokerAddr(brokerAddr);
  54.                     checkForbiddenContext.setMessage(msg);
  55.                     checkForbiddenContext.setMq(mq);
  56.                     checkForbiddenContext.setUnitMode(this.isUnitMode());
  57.                     //可以通过 registerCheckForbiddenHook 注册 CheckForbiddenHook 
  58.                     this.executeCheckForbiddenHook(checkForbiddenContext);
  59.                 }
  60.  
  61.                 
  62.                 if (this.hasSendMessageHook()) {
  63.                     context = new SendMessageContext();
  64.                     context.setProducer(this);
  65.                     context.setProducerGroup(this.defaultMQProducer.getProducerGroup());
  66.                     context.setCommunicationMode(communicationMode);
  67.                     context.setBornHost(this.defaultMQProducer.getClientIP());
  68.                     context.setBrokerAddr(brokerAddr);
  69.                     context.setMessage(msg);
  70.                     context.setMq(mq);
  71.                     context.setNamespace(this.defaultMQProducer.getNamespace());
  72.                     String isTrans = msg.
  73.                         getProperty(MessageConst.PROPERTY_TRANSACTION_PREPARED);
  74.                     if (isTrans != null && isTrans.equals("true")) {
  75.                         context.setMsgType(MessageType.Trans_Msg_Half);
  76.                     }
  77. ​
  78.                     if (msg.getProperty("__STARTDELIVERTIME") != null 
  79.                         || msg.getProperty(MessageConst.PROPERTY_DELAY_TIME_LEVEL) 
  80.                         != null) {
  81.                         context.setMsgType(MessageType.Delay_Msg);
  82.                     }
  83.                     //可以通过 registerSendMessageHook(final SendMessageHook hook) 注册hook
  84.                     //执行钩子前置函数
  85.                     this.executeSendMessageHookBefore(context);
  86.                 }
  87.                 //构建消息发送请求包
  88.                 SendMessageRequestHeader requestHeader = new SendMessageRequestHeader();
  89.                 //生产者组
  90.                 requestHeader.setProducerGroup(this.defaultMQProducer.getProducerGroup());
  91.                 //主题
  92.                 requestHeader.setTopic(msg.getTopic());
  93.                 //默认创建主题Key
  94.                 requestHeader.setDefaultTopic(this.defaultMQProducer.getCreateTopicKey());
  95.                 //该主题在单个Broker默认队列树
  96.                 requestHeader.setDefaultTopicQueueNums
  97.                             (this.defaultMQProducer.getDefaultTopicQueueNums());
  98.                 //队列ID
  99.                 requestHeader.setQueueId(mq.getQueueId());
  100.                 //消息系统标记
  101.                 requestHeader.setSysFlag(sysFlag);
  102.                 //消息发送时间
  103.                 requestHeader.setBornTimestamp(System.currentTimeMillis());
  104.                 //消息标记
  105.                 requestHeader.setFlag(msg.getFlag());
  106.                 //消息扩展信息
  107.                 requestHeader.setProperties
  108.                             (MessageDecoder.messageProperties2String(msg.getProperties()));
  109.                 //消息重试次数
  110.                 requestHeader.setReconsumeTimes(0);
  111.                 requestHeader.setUnitMode(this.isUnitMode());
  112.                 //是否是批量消息等
  113.                 requestHeader.setBatch(msg instanceof MessageBatch);
  114.                 
  115.                 //如果topic以%RETRY% 开头
  116.                 if (requestHeader.getTopic()
  117.                         .startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)){ 
  118.                     
  119.                     //获取属性 RECONSUME_TIME
  120.                     String reconsumeTimes = MessageAccessor.getReconsumeTime(msg);
  121.                     if (reconsumeTimes != null) {
  122.                         requestHeader.
  123.                             setReconsumeTimes(Integer.valueOf(reconsumeTimes));
  124.                         //清除属性 RECONSUME_TIME
  125.                         MessageAccessor.
  126.                             clearProperty(msg, MessageConst.PROPERTY_RECONSUME_TIME);
  127.                     }
  128.                 
  129.                      //获取属性 MAX_RECONSUME_TIMES
  130.                     String maxReconsumeTimes = MessageAccessor.getMaxReconsumeTimes(msg);
  131.                     if (maxReconsumeTimes != null) {
  132.                         requestHeader
  133.                             .setMaxReconsumeTimes(Integer.valueOf(maxReconsumeTimes));
  134.                         //清除属性 MAX_RECONSUME_TIMES
  135.                         MessageAccessor.clearProperty
  136.                             (msg,MessageConst.PROPERTY_MAX_RECONSUME_TIMES);
  137.                     }
  138.                 }
  139. ​
  140.                 SendResult sendResult = null;
  141.                 switch (communicationMode) {
  142.                      //异步发送消息
  143.                     case ASYNC:
  144.                         Message tmpMessage = msg;
  145.                         boolean messageCloned = false;
  146.                         //如果消息被压缩了
  147.                         if (msgBodyCompressed) {
  148.                             tmpMessage = MessageAccessor.cloneMessage(msg);
  149.                             messageCloned = true;
  150.                             msg.setBody(prevBody);
  151.                         }
  152. ​
  153.                         if (topicWithNamespace) {
  154.                             if (!messageCloned) {
  155.                                 tmpMessage = MessageAccessor.cloneMessage(msg);
  156.                                 messageCloned = true;
  157.                             }
  158.                             msg.setTopic(
  159.                                 NamespaceUtil.withoutNamespace(msg.getTopic(),                                                         this.defaultMQProducer.getNamespace()));
  160.                         }
  161. ​
  162.                         long costTimeAsync = System.currentTimeMillis() - beginStartTime;
  163.                         if (timeout < costTimeAsync) {
  164.                             throw new RemotingTooMuchRequestException ("sendKernelImpl call timeout");
  165.                         }
  166.                         //发送消息
  167.                         sendResult = this.mQClientFactory.getMQClientAPIImpl()
  168.                             .sendMessage(
  169.                             brokerAddr,
  170.                             mq.getBrokerName(),
  171.                             tmpMessage,
  172.                             requestHeader,
  173.                             timeout - costTimeAsync,
  174.                             communicationMode,
  175.                             sendCallback,
  176.                             topicPublishInfo,
  177.                             this.mQClientFactory,
  178.                             this.defaultMQProducer.getRetryTimesWhenSendAsyncFailed(),
  179.                             context,
  180.                             this);
  181.                         break;
  182.                     case ONEWAY:
  183.                     case SYNC:
  184.                           //同步发送消息
  185.                         long costTimeSync = System.currentTimeMillis() - beginStartTime;
  186.                         if (timeout < costTimeSync) {
  187.                              throw new RemotingTooMuchRequestException("sendKernelImpl call timeout");
  188.                         }
  189.                         sendResult = this.mQClientFactory.getMQClientAPIImpl().sendMessage(
  190.                             brokerAddr,
  191.                             mq.getBrokerName(),
  192.                             msg,
  193.                             requestHeader,
  194.                             timeout - costTimeSync,
  195.                             communicationMode,
  196.                             context,
  197.                             this);
  198.                         break;
  199.                     default:
  200.                         assert false;
  201.                         break;
  202.                 }
  203.                 
  204.                 //可以通过 registerSendMessageHook(final SendMessageHook hook) 注册hook
  205.                 //执行后置hook函数
  206.                 if (this.hasSendMessageHook()) {
  207.                     context.setSendResult(sendResult);
  208.                     this.executeSendMessageHookAfter(context);
  209.                 }
  210.                 return sendResult;
  211.             } catch (RemotingException e) {
  212.                 if (this.hasSendMessageHook()) {
  213.                     context.setException(e);
  214.                     this.executeSendMessageHookAfter(context);
  215.                 }
  216.                 throw e;
  217.             } catch (MQBrokerException e) {
  218.                 if (this.hasSendMessageHook()) {
  219.                     context.setException(e);
  220.                     this.executeSendMessageHookAfter(context);
  221.                 }
  222.                 throw e;
  223.             } catch (InterruptedException e) {
  224.                 if (this.hasSendMessageHook()) {
  225.                     context.setException(e);
  226.                     this.executeSendMessageHookAfter(context);
  227.                 }
  228.                 throw e;
  229.             } finally {
  230.                 msg.setBody(prevBody);
  231.                 msg.setTopic(NamespaceUtil.withoutNamespace(msg.getTopic(),                 
  232.                              this.defaultMQProducer.getNamespace()));
  233.             }
  234.         }
  235. ​
  236.         throw new MQClientException("The broker[" + mq.getBrokerName() + "] not exist", null);
  237.     }
  238. ​
  239.     public MQClientInstance getmQClientFactory() {
  240.         return mQClientFactory;
  241.     }
  242.  
  243.     //单条消息大于4K需要压缩
  244.     //private int compressMsgBodyOverHowmuch = 1024 * 4;
  245.     private boolean tryToCompressMessage(final Message msg) {
  246.         if (msg instanceof MessageBatch) {
  247.             //batch dose not support compressing right now
  248.             return false;
  249.         }
  250.         byte[] body = msg.getBody();
  251.         if (body != null) {
  252.             //
  253.             if (body.length >= this.defaultMQProducer.getCompressMsgBodyOverHowmuch()) {
  254.                 try {
  255.                     byte[] data = UtilAll.compress(body, zipCompressLevel);
  256.                     if (data != null) {
  257.                         msg.setBody(data);
  258.                         return true;
  259.                     }
  260.                 } catch (IOException e) {
  261.                     log.error("tryToCompressMessage exception", e);
  262.                     log.warn(msg.toString());
  263.                 }
  264.             }
  265.         }
  266. ​
  267.         return false;
  268.     }

```

SendMessageHook

java //在producer中可以通过 registerSendMessageHook(final SendMessageHook hook) 注册hook public interface SendMessageHook {    String hookName(); ​    void sendMessageBefore(final SendMessageContext context); ​    void sendMessageAfter(final SendMessageContext context); } 一般只有异步发消息才需要回调。

批量消息发送

image.png

批量消息发送是将同一个主题的多条消息一起打包发送到消息服务端，减少网络调用次数，提高网络传输效率。当然，并不是在同一批次中发送的消息数量越多越好，其判断依据是单条消息的长度，如果单条消息内容比较长，则打包多条消息发送会影响其他线程发送消息的响应时间，并且单批次消息总长度不能超过DefaultMQProducer#maxMessageSize,默认4M。

批量消息发送要解决的问题是如何将这些消息编码以便服务端能够正确解码出每条消息的消息内容。

DefaultMQProducer#send

java public SendResult send(Collection<Message> msgs)    throws MQClientException, RemotingException, MQBrokerException, InterruptedException {    //压缩消息集合成一条消息,然后发送出去    return this.defaultMQProducerImpl.send(batch(msgs)); }

DefaultMQProducer#batch

java private MessageBatch batch(Collection<Message> msgs) throws MQClientException {    MessageBatch msgBatch;    try {        //将集合消息封装到MessageBatch        msgBatch = MessageBatch.generateFromList(msgs);        //遍历消息集合,检查消息合法性,设置消息ID,设置Topic        for (Message message : msgBatch) {            Validators.checkMessage(message, this);            MessageClientIDSetter.setUniqID(message);            message.setTopic(withNamespace(message.getTopic()));       }        //压缩消息,设置消息body        msgBatch.setBody(msgBatch.encode());   } catch (Exception e) {        throw new MQClientException("Failed to initiate the MessageBatch", e);   }    //设置msgBatch的topic    msgBatch.setTopic(withNamespace(msgBatch.getTopic()));    return msgBatch; }
生产者发送消息就讲这么多,其中同步/异步发送消息大同小异,异步消息多了1个发送的回调。至于发送消息的请求到达broker端,broker如何处理我们在讲broker的时候细说。

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/从前慢现在也慢/article/detail/256901
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号