RabbitMQ脑裂「建议收藏」

RabbitMQ脑裂「建议收藏」在RabbitMQ3.4.x中会出现错误的网络分区检测(某种意义上可以称之为脑裂)的现象,本文通过实验验证此现象,愿小伙伴们少走弯路。Preview网上有两篇帖子(需要翻墙)https://groups.google.com/forum/#!topic/rabbitmq-users/dt8VFhMb2zMhttps://groups.google.com/forum/#!top…

大家好,又见面了,我是你们的朋友全栈君。如果您正在找激活码,请点击查看最新教程,关注关注公众号 “全栈程序员社区” 获取激活教程,可能之前旧版本教程已经失效.最新Idea2022.1教程亲测有效,一键激活。

Jetbrains全家桶1年46,售后保障稳定


欢迎支持笔者新作:《深入理解Kafka:核心设计与实践原理》和《RabbitMQ实战指南》,同时欢迎关注笔者的微信公众号:朱小厮的博客。
RabbitMQ脑裂「建议收藏」


欢迎跳转到本文的原文链接:https://honeypps.com/mq/rabbitmq-network-partition-1/

在RabbitMQ3.4.x中会出现错误的网络分区检测(某种意义上可以称之为脑裂)的现象,本文通过实验验证此现象,愿小伙伴们少走弯路。

Preview

网上有两篇帖子(需要翻墙)
https://groups.google.com/forum/#!topic/rabbitmq-users/dt8VFhMb2zM
https://groups.google.com/forum/#!topic/rabbitmq-users/06OQkYtLJd8
陈述了脑裂的现象。

帖子中描述现象:

Hey Folk,

i just set up a rabbitmq cluster:

Three Nodes:
Node A | Node B | Node C

All three nodes see each other (same erlang-cookie, mode: pause_minority).
 rabbitmqctl cluster_status => shows status of all nodes on every instance.

Every queue is mirrored to the other nodes.

If i shutdown Node B, the following is happening:
* Node A realizes Node B is offline.
* Node A asks Node C for Node B status.
* Node C answers: "I still have connection to Node B."
* Node A shuts down itself.
* Node C realizes some seconds later, that the connection to Node B is no more possible.

From three Nodes only one is left in case of an unexpected outage.

I would like to realize a setup where Node A and C keep the connection even if Node B goes offline.
Is there any way to do this?

Jetbrains全家桶1年46,售后保障稳定

Michael Klishin(rabbitmq-server第二贡献者)回复:

A known issue which is partially resolve in 3.4.x releases. 26474 can be related. 

(根据RabbitMQ 3.4.2 Release日志:26474 prevent false positive detection of partial partitions (since 3.4.0)) ====》错误的网络分区检测。

Simon MacMullen(也是rabbitmq-server的contributor):

So this is caused by the new partial partition detection in 3.4.x. It 
looks like it is too sensitive - C should only reply "yes" if it has 
positive confirmation that it can still talk to B, not if the connection 
just hasn't failed yet. 

This will be fixed in 3.4.2. 

假设

自此可以假设:rabbitmq3.4.0存在错误的网络分区检测,rabbitmq3.4.2修复了此bug。
论证过程:分别对rabbitmq3.4.0, rabbitmq3.4.1, rabbitmq3.4.2, rabbitmq3.6.0进行实验, 分别配置A B C三个节点组成一个cluster,然后通过停止C的网络来验证A和B是否出现错误的网络分区检测.


论证

论证1

rabbitmq版本:3.4.0
rabbitmq节点配置
共三个节点:A B C,分别为:
A:rabbit@zhuzhonghua2-fqawb
B:rabbit@hiddenzhu-8drd
C:rabbit@hidden-local
B join_cluster A; C join_cluster A

查看cluster_status:(rabbitmqctl cluster_status)

Cluster status of node 'rabbit@zhuzhonghua2-fqawb' ...
[{nodes,[{disc,['rabbit@hidden-local','rabbit@hiddenzhu-8drdc',
                'rabbit@zhuzhonghua2-fqawb']}]},
 {running_nodes,['rabbit@hidden-local','rabbit@hiddenzhu-8drdc',
                 'rabbit@zhuzhonghua2-fqawb']},
 {cluster_name,<<"rabbit@zhuzhonghua2-fqawb">>},
 {partitions,[]}]

在C节点执行service network stop
在A节点查看cluster_status

[{nodes,[{disc,['rabbit@hidden-local','rabbit@hiddenzhu-8drdc',
                'rabbit@zhuzhonghua2-fqawb']}]},
 {running_nodes,['rabbit@zhuzhonghua2-fqawb']},
 {cluster_name,<<"rabbit@zhuzhonghua2-fqawb">>},
 {partitions,[]}]

再次在A节点查看cluster_status

Cluster status of node 'rabbit@zhuzhonghua2-fqawb' ...
[{nodes,[{disc,['rabbit@hidden-local','rabbit@hiddenzhu-8drdc',
                'rabbit@zhuzhonghua2-fqawb']}]},
 {running_nodes,['rabbit@zhuzhonghua2-fqawb']},
 {cluster_name,<<"rabbit@zhuzhonghua2-fqawb">>},
 {partitions,[{'rabbit@zhuzhonghua2-fqawb',['rabbit@hiddenzhu-8drdc']}]}]
在B节点查看cluster_status
[{nodes,[{disc,['rabbit@hidden-local','rabbit@hiddenzhu-8drdc',
                'rabbit@zhuzhonghua2-fqawb']}]},
 {running_nodes,['rabbit@hiddenzhu-8drdc']},
 {cluster_name,<<"rabbit@zhuzhonghua2-fqawb">>},
 {partitions,[{'rabbit@hiddenzhu-8drdc',['rabbit@zhuzhonghua2-fqawb']}]}]

结论:【这里出现了网络分区,但是真正的网络分区是要在网络恢复连通之后才能检测】

在C节点执行service network start
查看A节点cluster_status

[{nodes,
     [{disc,
          ['rabbit@hidden-local','rabbit@hiddenzhu-8drdc',
           'rabbit@zhuzhonghua2-fqawb']}]},
 {running_nodes,['rabbit@zhuzhonghua2-fqawb']},
 {cluster_name,<<"rabbit@zhuzhonghua2-fqawb">>},
 {partitions,
     [{'rabbit@zhuzhonghua2-fqawb',
          ['rabbit@hidden-local','rabbit@hiddenzhu-8drdc']}]}]

查看B节点cluster_status

[{nodes,[{disc,['rabbit@hidden-local','rabbit@hiddenzhu-8drdc',
                'rabbit@zhuzhonghua2-fqawb']}]},
 {running_nodes,['rabbit@hiddenzhu-8drdc']},
 {cluster_name,<<"rabbit@zhuzhonghua2-fqawb">>},
 {partitions,[{'rabbit@hiddenzhu-8drdc',['rabbit@zhuzhonghua2-fqawb']}]}]

查看C节点cluster_status

[{nodes,[{disc,['rabbit@hidden-local','rabbit@hiddenzhu-8drdc',
                'rabbit@zhuzhonghua2-fqawb']}]},
 {running_nodes,['rabbit@hidden-local']},
 {cluster_name,<<"rabbit@zhuzhonghua2-fqawb">>},
 {partitions,[{'rabbit@hidden-local',['rabbit@zhuzhonghua2-fqawb']}]}]

论证2

rabbitmq版本:3.4.1
节点配置如上(B join_cluster A, C join_cluster A)
查看节点状态:

[{nodes,[{disc,['rabbit@hidden-local','rabbit@hiddenzhu-8drdc',
                'rabbit@zhuzhonghua2-fqawb']}]},
 {running_nodes,['rabbit@hiddenzhu-8drdc','rabbit@zhuzhonghua2-fqawb',
                 'rabbit@hidden-local']},
 {cluster_name,<<"rabbit@zhuzhonghua2-fqawb">>},
 {partitions,[]}]

在C节点执行service network stop
查看A节点cluster_status

Cluster status of node 'rabbit@zhuzhonghua2-fqawb' ...
[{nodes,[{disc,['rabbit@hidden-local','rabbit@hiddenzhu-8drdc',
                'rabbit@zhuzhonghua2-fqawb']}]},
 {running_nodes,['rabbit@zhuzhonghua2-fqawb']},
 {cluster_name,<<"rabbit@zhuzhonghua2-fqawb">>},
 {partitions,[{'rabbit@zhuzhonghua2-fqawb',['rabbit@hiddenzhu-8drdc']}]}]

查看B节点cluster_status

[{nodes,[{disc,['rabbit@hidden-local','rabbit@hiddenzhu-8drdc',
                'rabbit@zhuzhonghua2-fqawb']}]},
 {running_nodes,['rabbit@hiddenzhu-8drdc']},
 {cluster_name,<<"rabbit@zhuzhonghua2-fqawb">>},
 {partitions,[{'rabbit@hiddenzhu-8drdc',['rabbit@zhuzhonghua2-fqawb']}]}]

结论:【复现】

在C节点执行service network start
查看A节点cluster_status

[{nodes,
     [{disc,
          ['rabbit@hidden-local','rabbit@hiddenzhu-8drdc',
           'rabbit@zhuzhonghua2-fqawb']}]},
 {running_nodes,['rabbit@zhuzhonghua2-fqawb']},
 {cluster_name,<<"rabbit@zhuzhonghua2-fqawb">>},
 {partitions,
     [{'rabbit@zhuzhonghua2-fqawb',
          ['rabbit@hidden-local','rabbit@hiddenzhu-8drdc']}]}]

查看B节点cluster_status

[{nodes,[{disc,['rabbit@hidden-local','rabbit@hiddenzhu-8drdc',
                'rabbit@zhuzhonghua2-fqawb']}]},
 {running_nodes,['rabbit@hiddenzhu-8drdc']},
 {cluster_name,<<"rabbit@zhuzhonghua2-fqawb">>},
 {partitions,[{'rabbit@hiddenzhu-8drdc',['rabbit@zhuzhonghua2-fqawb']}]}]

查看C节点cluster_status

[{nodes,[{disc,['rabbit@hidden-local','rabbit@hiddenzhu-8drdc',
                'rabbit@zhuzhonghua2-fqawb']}]},
 {running_nodes,['rabbit@hidden-local']},
 {cluster_name,<<"rabbit@zhuzhonghua2-fqawb">>},
 {partitions,[{'rabbit@hidden-local',['rabbit@zhuzhonghua2-fqawb']}]}]

论证3

rabbitmq版本:3.4.2 (版本3.6.0与此相同)
节点配置如上(B join_cluster A, C join_cluster A)
查看节点状态

[{nodes,[{disc,['rabbit@hidden-local','rabbit@hiddenzhu-8drdc',
                'rabbit@zhuzhonghua2-fqawb']}]},
 {running_nodes,['rabbit@hiddenzhu-8drdc','rabbit@zhuzhonghua2-fqawb',
                 'rabbit@hidden-local']},
 {cluster_name,<<"rabbit@zhuzhonghua2-fqawb">>},
 {partitions,[]}]

在C节点执行service network stop
查看A节点cluster_status

[{nodes,[{disc,['rabbit@hidden-local','rabbit@hiddenzhu-8drdc',
                'rabbit@zhuzhonghua2-fqawb']}]},
 {running_nodes,['rabbit@hiddenzhu-8drdc','rabbit@zhuzhonghua2-fqawb']},
 {cluster_name,<<"rabbit@zhuzhonghua2-fqawb">>},
 {partitions,[]}]

查看B节点cluster_status

[{nodes,[{disc,['rabbit@hidden-local','rabbit@hiddenzhu-8drdc',
                'rabbit@zhuzhonghua2-fqawb']}]},
 {running_nodes,['rabbit@zhuzhonghua2-fqawb','rabbit@hiddenzhu-8drdc']},
 {cluster_name,<<"rabbit@zhuzhonghua2-fqawb">>},
 {partitions,[]}]

结论:【未复现】

在C节点执行service network start
查看A节点cluster_status

[{nodes,[{disc,['rabbit@hidden-local','rabbit@hiddenzhu-8drdc',
                'rabbit@zhuzhonghua2-fqawb']}]},
 {running_nodes,['rabbit@hiddenzhu-8drdc','rabbit@zhuzhonghua2-fqawb']},
 {cluster_name,<<"rabbit@zhuzhonghua2-fqawb">>},
 {partitions,[{'rabbit@zhuzhonghua2-fqawb',['rabbit@hidden-local']}]}]

查看B节点cluster_status

[{nodes,[{disc,['rabbit@hidden-local','rabbit@hiddenzhu-8drdc',
                'rabbit@zhuzhonghua2-fqawb']}]},
 {running_nodes,['rabbit@zhuzhonghua2-fqawb','rabbit@hiddenzhu-8drdc']},
 {cluster_name,<<"rabbit@zhuzhonghua2-fqawb">>},
 {partitions,[{'rabbit@zhuzhonghua2-fqawb',['rabbit@hidden-local']}]}]

查看C节点cluster_status

[{nodes,[{disc,['rabbit@hidden-local','rabbit@hiddenzhu-8drdc',
                'rabbit@zhuzhonghua2-fqawb']}]},
 {running_nodes,['rabbit@hidden-local']},
 {cluster_name,<<"rabbit@zhuzhonghua2-fqawb">>},
 {partitions,[{'rabbit@hidden-local',['rabbit@zhuzhonghua2-fqawb']}]}]

结论

版本问题基本得到验证,为了防止错误的网络分区检测现象,建议正在使用rabbitmq的小伙伴升级,避免使用3.4.0和3.4.1这两个版本。

网络分区

有关网络分区有篇文章(RabbitMQ 网络分区问题)这样介绍:

RabbitMQ 集群的网络分区容错性并不是非常高,在网络经常发生分区时会有些问题,最明显的就是脑裂问题。

官方文档是这样介绍的:

RabbitMQ clusters do not tolerate network partitions well. If you are thinking of clustering across a WAN, don't. You should use federation or the shovel instead.

从中我们可以看出,在广域网环境下不应该使用集群,而应该使用 federation 或者 shovel 来解决。

不过即使是在局域网环境下,网络分区也不可能完全避免,网络设备(比如中继设备、网卡)出现故障也会导致网络分区。

Network partition detected

Mnesia reports that this RabbitMQ cluster has experienced a network partition. This is a dangerous situation. RabbitMQ clusters should not be installed on networks which can experience partitions. 

当出现网络分区时,不同分区里的节点会认为不属于自身所在分区的节点都已经挂了,对 queue、exchange、binding 的操作仅对当前分区有效。在 RabbitMQ 的默认配置下,即使网络恢复了也不会自动处理网络分区带来的问题从而恢复集群。RabbitMQ(3.1+)会自动探测网络分区,并且提供了配置来解决这个问题。

[
 {rabbit,
 [{tcp_listeners,[5672]},
 {cluster_partition_handling, ignore}]
 }
].

RabbitMQ 提供了4种配置(详细参考:http://blog.csdn.net/u013256816/article/details/73757884):

  1. ignore:默认配置,发生网络分区时不作处理,当认为网络是可靠时选用该配置
  2. autoheal:各分区协商后重启客户端连接最少的分区节点,恢复集群(CAP 中保证 AP,有状态丢失)
  3. pause_if_all_down。
  4. pause_minority:分区发生后判断自己所在分区内节点是否超过集群总节点数一半,如果没有超过则暂停这些节点(保证 CP,总节点数为奇数个)

参考:
RabbitMQ 官方文档
网络分区
脑裂问题

欢迎跳转到本文的原文链接:https://honeypps.com/mq/rabbitmq-network-partition-1/


欢迎支持笔者新作:《深入理解Kafka:核心设计与实践原理》和《RabbitMQ实战指南》,同时欢迎关注笔者的微信公众号:朱小厮的博客。
RabbitMQ脑裂「建议收藏」


版权声明:本文内容由互联网用户自发贡献,该文观点仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 举报,一经查实,本站将立刻删除。

发布者:全栈程序员-用户IM,转载请注明出处:https://javaforall.cn/219186.html原文链接:https://javaforall.cn

【正版授权,激活自己账号】: Jetbrains全家桶Ide使用,1年售后保障,每天仅需1毛

【官方授权 正版激活】: 官方授权 正版激活 支持Jetbrains家族下所有IDE 使用个人JB账号...

(0)
blank

相关推荐

  • Windows平台及服务器部署安装多个Tomcat服务(详细版)

    Windows平台及服务器部署安装多个Tomcat服务(详细版)Tomcat安装有两种方式,一是下载安装版,这个也是在部署多个Tomcat服务的时候最方便的,也是最快的,二是就是下载免安装版,解压即可,单个Tomcat免安装版是最快的。一、下载Tomcat,我用的是Tomcat8.5到Tomcat官网http://tomcat.apache.org/,根据需求下载所需要的版本我下载的是Tomcat8.5二、安装二.一安装版下载…

  • how tomcat works 札记(两)———-一个简单的servlet集装箱

    how tomcat works 札记(两)———-一个简单的servlet集装箱

  • QT之QFile[通俗易懂]

    QT之QFile[通俗易懂]1、QFile::QFile()构造一个没有名字的QFile对象2、QFile::QFile(constQString&name)构造一个以name为文件名的QFile对象。注:也可以QFile::QFile(),然后调用setName()方法来实现类似动作。3、bootQFile::atEnd()const[虚函数]如果已经到达文件末尾则返回TR

  • DropDownList的常用属性和事件「建议收藏」

    DropDownList的常用属性和事件「建议收藏」SelectedItem属性设置或获取下拉菜单的选中项,该属性的类型为System.Web.UI.WebControls.ListItem.所有列表控件(ListControl)中的项都是该类型,它

  • java 工作流 详解

    java 工作流 详解工作流基本概念:什么是工作流? 工作流:两个或两个以上的人,为了共同的目标,连续的以串行或并行的方式去完成某一业务。 业务:工作流所指业务涵盖了与经营相关的活动。   串行或并行:业务中的步骤也许以一步接着一步的方式进行,我们称之为串行;或者由不同的人或组合根据不同的情况处理,我们称之为并行。   两个或两个以上的人:如工作流的名称所表达的含义,一个人处理的业务

发表回复

您的电子邮箱地址不会被公开。

关注全栈程序员社区公众号