OpenStack HA集群3-Pacemake Corosync

节点间主机名必须能解析

[root@controller1 ~]# cat /etc/hosts

192.168.17.149  controller1

192.168.17.141  controller2

192.168.17.166  controller3

192.168.17.111  demo.open-stack.cn

各节点间要互信，无密码能登录

[root@controller1 ~]# ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key (/root/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /root/.ssh/id_rsa.

Your public key has been saved in /root/.ssh/id_rsa.pub.

The key fingerprint is:

20:79:d4:a4:9f:8b:75:cf:12:58:f4:47:a4:c1:29:f3 root@controller1

The key’s randomart image is:

+–[ RSA 2048]—-+

|      .o. …oo  |

|     o …o.o+   |

|    o +   .+o .  |

|     o o +  E.   |

|        S o      |

|       o o +     |

|      . . . o    |

|           .     |

|                 |

+—————–+

[root@controller1 ~]# ssh-copy-id controller2

[root@controller1 ~]# ssh-copy-id controller3

配置YUM源

# vim /etc/yum.repos.d/ha-clustering.repo

[network_ha-clustering_Stable]

name=Stable High Availability/Clustering packages (CentOS-7)

type=rpm-md

baseurl=http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-7/

gpgcheck=0

gpgkey=http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-7/repodata/repomd.xml.key

enabled=1

这个YUM源可能会冲突，先enabled=0，如果剩下一个crmsh包，再enabled=1打开后安装

Corosync下载地址，目前最新版本2.4.2

http://build.clusterlabs.org/corosync/releases/

http://build.clusterlabs.org/corosync/releases/corosync-2.4.2.tar.gz

[root@controller1 ~]# ansible controller -m copy -a “src=/etc/yum.repos.d/ha-cluster.repo dest=/etc/yum.repos.d/”

安装软件包

# yum install pacemaker pcs resource-agents -y cifs-utils quota psmisc corosync fence-agents-all lvm2 resource-agents

#  yum install crmsh  -y

启动pcsd，并确认启动正常

# systemctl enable pcsd

# systemctl enable corosync

# systemctl start pcsd

# systemctl status pcsd

[root@controller2 ~]# pacemakerd -$

Pacemaker 1.1.15-11.el7_3.2

Written by Andrew Beekhof

[root@controller1 ~]# ansible controller -m command -a “pacemakerd -$”

修改hacluster密码

【all】# echo zoomtech | passwd –stdin hacluster

[root@controller1 ~]# ansible controller -m command -a “echo zoomtech | passwd –stdin hacluster”

# passwd hacluster

编辑corosync.conf

[root@controller3 ~]# vim /etc/corosync/corosync.conf

totem {

        version: 2

        secauth: off

        cluster_name: openstack-cluster

        transport: udpu

}

nodelist {

        node {

                ring0_addr: controller1

                nodeid: 1

        }

        node {

                ring0_addr: controller2

                nodeid: 2

        }

        node {

                ring0_addr: controller3

                nodeid: 3

        }

}

logging {

        to_logfile: yes

        logfile: /var/log/cluster/corosync.log

        to_syslog: yes

}

quorum {

        provider: corosync_votequorum

}

[root@controller1 ~]# scp /etc/corosync/corosync.conf controller2:/etc/corosync/

[root@controller1 ~]# scp /etc/corosync/corosync.conf controller3:/etc/corosync/

[root@controller1 corosync]# ansible controller -m copy -a “src=corosync.conf dest=/etc/corosync”

创建集群

使用pcs设置集群身份认证

[root@controller1 ~]# pcs cluster auth controller1 controller2 controller3 -u hacluster -p zoomtech –force

controller3: Authorized

controller2: Authorized

controller1: Authorized

现在我们创建一个集群并添加一些节点。注意,这个名字不能超过15个字符

[root@controller1 ~]# pcs cluster setup –force –name openstack-cluster controller1 controller2 controller3

Destroying cluster on nodes: controller1, controller2, controller3…

controller3: Stopping Cluster (pacemaker)…

controller2: Stopping Cluster (pacemaker)…

controller1: Stopping Cluster (pacemaker)…

controller2: Successfully destroyed cluster

controller1: Successfully destroyed cluster

controller3: Successfully destroyed cluster

Sending cluster config files to the nodes…

controller1: Succeeded

controller2: Succeeded

controller3: Succeeded

Synchronizing pcsd certificates on nodes controller1, controller2, controller3…

controller3: Success

controller2: Success

controller1: Success

Restarting pcsd on the nodes in order to reload the certificates…

controller3: Success

controller2: Success

controller1: Success

启动集群

[root@controller1 ~]# pcs cluster enable –all

controller1: Cluster Enabled

controller2: Cluster Enabled

controller3: Cluster Enabled

[root@controller1 ~]# pcs cluster start –all

controller2: Starting Cluster…

controller1: Starting Cluster…

controller3: Starting Cluster…

查看集群状态

[root@controller1 corosync]# ansible controller -m command -a “pcs cluster status”

[root@controller1 ~]# pcs cluster status

Cluster Status:

 Stack: corosync

 Current DC: controller3 (version 1.1.15-11.el7_3.2-e174ec8) – partition with quorum

 Last updated: Fri Feb 17 10:39:38 2017        Last change: Fri Feb 17 10:39:29 2017 by hacluster via crmd on controller3

 3 nodes and 0 resources configured

PCSD Status:

  controller2: Online

  controller3: Online

  controller1: Online

[root@controller1 corosync]# ansible controller -m command -a “pcs status”

[root@controller1 ~]# pcs status

Cluster name: openstack-cluster

Stack: corosync

Current DC: controller2 (version 1.1.15-11.el7_3.2-e174ec8) – partition with quorum

Last updated: Thu Mar  2 17:07:34 2017        Last change: Thu Mar  2 01:44:44 2017 by root via cibadmin on controller1

3 nodes and 1 resource configured

Online: [ controller1 controller2 controller3 ]

Full list of resources:

 vip    (ocf::heartbeat:IPaddr2):    Started controller2

Daemon Status:

  corosync: active/enabled

  pacemaker: active/enabled

  pcsd: active/enabled

查看集群状态

[root@controller1 corosync]# ansible controller -m command -a “crm_mon -1”

[root@controller1 ~]# crm_mon -1

Stack: corosync

Current DC: controller2 (version 1.1.15-11.el7_3.2-e174ec8) – partition with quorum

Last updated: Wed Mar  1 17:54:04 2017          Last change: Wed Mar  1 17:44:38 2017 by root via cibadmin on controller1

3 nodes and 1 resource configured

Online: [ controller1 controller2 controller3 ]

Active resources:

vip     (ocf::heartbeat:IPaddr2):    Started controller1

查看pacemaker进程状态

[root@controller1 ~]# ps aux | grep pacemaker

root      75900  0.2  0.5 132632  9216 ?        Ss   10:39   0:00 /usr/sbin/pacemaked -f

haclust+  75901  0.3  0.8 135268 15376 ?        Ss   10:39   0:00 /usr/libexec/pacemaker/cib

root      75902  0.1  0.4 135608  7920 ?        Ss   10:39   0:00 /usr/libexec/pacemaker/stonithd

root      75903  0.0  0.2 105092  5020 ?        Ss   10:39   0:00 /usr/libexec/pacemaker/lrmd

haclust+  75904  0.0  0.4 126924  7636 ?        Ss   10:39   0:00 /usr/libexec/pacemaker/attrd

haclust+  75905  0.0  0.2 117040  4560 ?        Ss   10:39   0:00 /usr/libexec/pacemaker/pengine

haclust+  75906  0.1  0.5 145328  8988 ?        Ss   10:39   0:00 /usr/libexec/pacemaker/crmd

root      75997  0.0  0.0 112648   948 pts/0    R+   10:40   0:00 grep –color=auto pacemaker

查看集群状态

[root@controller1 ~]# corosync-cfgtool -s

Printing ring status.

Local node ID 1

RING ID 0

    id    = 192.168.17.132

    status    = ring 0 active with no faults

[root@controller2 corosync]# corosync-cfgtool -s

Printing ring status.

Local node ID 2

RING ID 0

    id    = 192.168.17.146

    status    = ring 0 active with no faults

[root@controller3 ~]# corosync-cfgtool -s

Printing ring status.

Local node ID 3

RING ID 0

    id    = 192.168.17.138

    status    = ring 0 active with no faults

[root@controller1 ~]# corosync-cmapctl | grep members

runtime.totem.pg.mrp.srp.members.1.config_version (u64) = 0

runtime.totem.pg.mrp.srp.members.1.ip (str) = r(0) ip(192.168.17.132)

runtime.totem.pg.mrp.srp.members.1.join_count (u32) = 1

runtime.totem.pg.mrp.srp.members.1.status (str) = joined

runtime.totem.pg.mrp.srp.members.2.config_version (u64) = 0

runtime.totem.pg.mrp.srp.members.2.ip (str) = r(0) ip(192.168.17.146)

runtime.totem.pg.mrp.srp.members.2.join_count (u32) = 1

runtime.totem.pg.mrp.srp.members.2.status (str) = joined

runtime.totem.pg.mrp.srp.members.3.config_version (u64) = 0

runtime.totem.pg.mrp.srp.members.3.ip (str) = r(0) ip(192.168.17.138)

runtime.totem.pg.mrp.srp.members.3.join_count (u32) = 1

runtime.totem.pg.mrp.srp.members.3.status (str) = joined

查看集群状态

[root@controller1 ~]# pcs status corosync

Membership information

———————-

    Nodeid      Votes Name

         1          1 controller1 (local)

         3          1 controller3

         2          1 controller2

[root@controller2 corosync]# pcs status corosync

Membership information

———————-

    Nodeid      Votes Name

         1          1 controller1

         3          1 controller3

         2          1 controller2 (local)

[root@controller3 ~]# pcs status corosync

Membership information

———————-

    Nodeid      Votes Name

         1          1 controller1

         3          1 controller3 (local)

         2          1 controller2

[root@controller1 ~]# crm_verify -L -V

   error: unpack_resources:    Resource start-up disabled since no STONITH resources have been defined

   error: unpack_resources:    Either configure some or disable STONITH with the stonith-enabled option

   error: unpack_resources:    NOTE: Clusters with shared data need STONITH to ensure data integrity

Errors found during check: config not valid

[root@controller1 ~]#

[root@controller1 ~]# pcs property set stonith-enabled=false

[root@controller1 ~]# pcs property set no-quorum-policy=ignore

[root@controller1 ~]# crm_verify -L -V

[root@controller1 corosync]# ansible controller -m command -a “pcs property set stonith-enabled=false

[root@controller1 corosync]# ansible controller -m command -a “pcs property set no-quorum-policy=ignore”

[root@controller1 corosync]# ansible controller -m command -a “crm_verify -L -V”

配置 VIP

[root@controller1 ~]# crm

crm(live)# configure

crm(live)configure# show

node 1: controller1

node 2: controller2

node 3: controller3

property cib-bootstrap-options: \

    have-watchdog=false \

    dc-version=1.1.15-11.el7_3.2-e174ec8 \

    cluster-infrastructure=corosync \

    cluster-name=openstack-cluster \

    stonith-enabled=false \

    no-quorum-policy=ignore

crm(live)configure# primitive vip ocf:heartbeat:IPaddr2 params ip=192.168.17.111 cidr_netmask=24 nic=ens37 op start interval=0s timeout=20s op stop interval=0s timeout=20s monitor interval=30s meta priority=100

crm(live)configure# show

node 1: controller1

node 2: controller2

node 3: controller3

primitive vip IPaddr2 \

    params ip=192.168.17.111 cidr_netmask=24 nic=ens37 \

    op start interval=0s timeout=20s \

    op stop interval=30s timeout=20s monitor \

    meta priority=100

property cib-bootstrap-options: \

    have-watchdog=false \

    dc-version=1.1.15-11.el7_3.2-e174ec8 \

    cluster-infrastructure=corosync \

    cluster-name=openstack-cluster \

    stonith-enabled=false \

    no-quorum-policy=ignore

crm(live)configure# commit

crm(live)configure# exit

查看VIP已绑定在ens37网卡上

[root@controller1 ~]# ip a

4: ens37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 00:0c:29:ff:8b:4b brd ff:ff:ff:ff:ff:ff

    inet 192.168.17.141/24 brd 192.168.17.255 scope global dynamic ens37

       valid_lft 2388741sec preferred_lft 2388741sec

    inet 192.168.17.111/24 brd 192.168.17.255 scope global secondary ens37

       valid_lft forever preferred_lft forever

上面指定的网卡名称3个节点必须是同一个名称，否则飘移会出现问题，切换不过去

[root@controller1 ~]# crm status

Stack: corosync

Current DC: controller1 (version 1.1.15-11.el7_3.2-e174ec8) – partition with quorum

Last updated: Wed Feb 22 11:42:07 2017        Last change: Wed Feb 22 11:22:56 2017 by root via cibadmin on controller1

3 nodes and 1 resource configured

Online: [ controller1 controller2 controller3 ]

Full list of resources:

 vip    (ocf::heartbeat:IPaddr2):    Started controller1

查看corosync引擎是否正常启动

[root@controller1 ~]# grep -e “Corosync Cluster Engine” -e “configuration file” /var/log/cluster/corosync.log

[51405] controller1 corosyncnotice  [MAIN  ] Corosync Cluster Engine (‘2.4.0’): started and ready to provide service.

Mar 01 17:35:20 [51425] controller1        cib:     info: retrieveCib:    Reading cluster configuration file /var/lib/pacemaker/cib/cib.xml (digest: /var/lib/pacemaker/cib/cib.xml.sig)

Mar 01 17:35:20 [51425] controller1        cib:  warning: cib_file_read_and_verify:    Could not verify cluster configuration file /var/lib/pacemaker/cib/cib.xml: No such file or directory (2)

Mar 01 17:35:20 [51425] controller1        cib:  warning: cib_file_read_and_verify:    Could not verify cluster configuration file /var/lib/pacemaker/cib/cib.xml: No such file or directory (2)

Mar 01 17:35:20 [51425] controller1        cib:     info: cib_file_write_with_digest:    Reading cluster configuration file /var/lib/pacemaker/cib/cib.Apziws (digest: /var/lib/pacemaker/cib/cib.0ZxsVW)

Mar 01 17:35:21 [51425] controller1        cib:     info: cib_file_write_with_digest:    Reading cluster configuration file /var/lib/pacemaker/cib/cib.ObYehI (digest: /var/lib/pacemaker/cib/cib.O8Rntg)

Mar 01 17:35:42 [51425] controller1        cib:     info: cib_file_write_with_digest:    Reading cluster configuration file /var/lib/pacemaker/cib/cib.eqrhsF (digest: /var/lib/pacemaker/cib/cib.6BCfNj)

Mar 01 17:35:42 [51425] controller1        cib:     info: cib_file_write_with_digest:    Reading cluster configuration file /var/lib/pacemaker/cib/cib.riot2E (digest: /var/lib/pacemaker/cib/cib.SAqtzj)

Mar 01 17:35:42 [51425] controller1        cib:     info: cib_file_write_with_digest:    Reading cluster configuration file /var/lib/pacemaker/cib/cib.Q8H9BL (digest: /var/lib/pacemaker/cib/cib.MBljlq)

Mar 01 17:38:29 [51425] controller1        cib:     info: cib_file_write_with_digest:    Reading cluster configuration file /var/lib/pacemaker/cib/cib.OTIiU4 (digest: /var/lib/pacemaker/cib/cib.JnHr1v)

Mar 01 17:38:36 [51425] controller1        cib:     info: cib_file_write_with_digest:    Reading cluster configuration file /var/lib/pacemaker/cib/cib.2cK9Yk (digest: /var/lib/pacemaker/cib/cib.JSqEH8)

Mar 01 17:44:38 [51425] controller1        cib:     info: cib_file_write_with_digest:    Reading cluster configuration file /var/lib/pacemaker/cib/cib.aPFtr3 (digest: /var/lib/pacemaker/cib/cib.E3Ve7X)

[root@controller1 ~]#

查看初始化成员节点通知是否正常发出 

[root@controller1 ~]# grep  TOTEM /var/log/cluster/corosync.log 

[51405] controller1 corosyncnotice  [TOTEM ] Initializing transport (UDP/IP Unicast).

[51405] controller1 corosyncnotice  [TOTEM ] Initializing transmit/receive security (NSS) crypto: none hash: none

[51405] controller1 corosyncnotice  [TOTEM ] The network interface [192.168.17.149] is now up.

[51405] controller1 corosyncnotice  [TOTEM ] adding new UDPU member {192.168.17.149}

[51405] controller1 corosyncnotice  [TOTEM ] adding new UDPU member {192.168.17.141}

[51405] controller1 corosyncnotice  [TOTEM ] adding new UDPU member {192.168.17.166}

[51405] controller1 corosyncnotice  [TOTEM ] A new membership (192.168.17.149:4) was formed. Members joined: 1

[51405] controller1 corosyncnotice  [TOTEM ] A new membership (192.168.17.141:12) was formed. Members joined: 2 3

检查启动过程中是否有错误产生

[root@controller1 ~]# grep ERROR: /var/log/cluster/corosync.log

本文转自 OpenStack2015 51CTO博客，原文链接：http://blog.51cto.com/andyliu/1917391，如需转载请自行联系原作者