目录
- 集群操作系统的安装
- 必要补丁的安装
- 网络配置
- IPSAN 的iscsi initiator配置
- 分区的创建
- 其余集群节点配置和同步
系统基础软件的安装和配置
本集群，我们假设有两台服务器，分别是racnode1,racnode2,分别有两张网卡，一张用于对外提供服务，一张用于私网，提供到IPSAN的连接，并提供集群的Cache Fusion的私有连接。
1.集群节点操作系统的安装
最新安装包是7.6：
V980739-01.iso Oracle Linux Release 7 Update 6 for x86 (64 bit), 4.3 GB
安装过程此处省略。
2.必要的补丁安装
如下是手册中列出的补丁清单，我们需要首先将这些包全部安装完毕。这里把安装命令重复执行了一边。
[root@racnode1 Packages]# yum install binutils elfutils-libelf glibc glibc-common ksh libaio libgcc libstdc++ make elfutils-libelf-devel gcc gcc-c++ glibc-devel glibc-headers libstdc++-devel unixODBC compat-libstdc++ libaio-devel libaio-devel unixODBC-devel sysstat
Loaded plugins: langpacks, ulninfo
Package binutils-2.27-34.base.0.1.el7.x86_64 already installed and latest version
Package elfutils-libelf-0.172-2.el7.x86_64 already installed and latest version
Package glibc-2.17-260.0.9.el7.x86_64 already installed and latest version
Package glibc-common-2.17-260.0.9.el7.x86_64 already installed and latest version
Package ksh-20120801-139.0.1.el7.x86_64 already installed and latest version
Package libaio-0.3.109-13.el7.x86_64 already installed and latest version
Package libgcc-4.8.5-36.0.1.el7.x86_64 already installed and latest version
Package libstdc++-4.8.5-36.0.1.el7.x86_64 already installed and latest version
Package 1:make-3.82-23.el7.x86_64 already installed and latest version
Package elfutils-libelf-devel-0.172-2.el7.x86_64 already installed and latest version
Package gcc-4.8.5-36.0.1.el7.x86_64 already installed and latest version
Package gcc-c++-4.8.5-36.0.1.el7.x86_64 already installed and latest version
Package glibc-devel-2.17-260.0.9.el7.x86_64 already installed and latest version
Package glibc-headers-2.17-260.0.9.el7.x86_64 already installed and latest version
Package libstdc++-devel-4.8.5-36.0.1.el7.x86_64 already installed and latest version
Package unixODBC-2.3.1-11.el7.x86_64 already installed and latest version
No package compat-libstdc++ available.
Package libaio-devel-0.3.109-13.el7.x86_64 already installed and latest version
Package libaio-devel-0.3.109-13.el7.x86_64 already installed and latest version
Package unixODBC-devel-2.3.1-11.el7.x86_64 already installed and latest version
Package sysstat-10.1.5-17.el7.x86_64 already installed and latest version
Nothing to do
compat-libstdc++没有安装上，我们需要进入/mnt/cdrom/Package ,单独安装下。
[root@racnode1 Packages]# ls compat-libstdc*
compat-libstdc++-33-3.2.3-72.el7.i686.rpm compat-libstdc++-33-3.2.3-72.el7.x86_64.rpm
[root@racnode1 Packages]# rpm -ivh compat-libstdc++-33-3.2.3-72.el7.*
Preparing... ################################# [100%]
package compat-libstdc++-33-3.2.3-72.el7.x86_64 is already installed
package compat-libstdc++-33-3.2.3-72.el7.i686 is already installed
3.网络配置
每个节点服务器有两张网卡，一张对外提供服务，一张私网的用于IPSAN和节点之间互联。
公网的网卡配置如下：
[root@racnode1 Packages]# cat /etc/sysconfig/network-scripts/ifcfg-ens33
TYPE=Ethernet
BOOTPROTO=static
IPADDR=192.168.1.8
NETMASK=255.255.255.0
DEFROUTE=yes
NAME=ens33
UUID=7f29b95d-7854-4069-acb9-cb60afbc88f4
DEVICE=ens33
ONBOOT=yes
HWADDR=00:0C:29:53:4D:2D
ZONE=public
PREFIX=24
GATEWAY=192.168.1.1
DNS1=192.168.1.1
私网的网卡配置如下：
[root@racnode1 Packages]# cat /etc/sysconfig/network-scripts/ifcfg-ens37
TYPE=Ethernet
BOOTPROTO=static
IPADDR=192.168.237.126
NETMASK=255.255.255.0
NAME=ens37
DEVICE=ens37
ONBOOT=yes
HWADDR=00:0C:29:53:4D:37
节点主机名配置：
[root@racnode1 Packages]# cat /etc/hostname
racnode1
节点的hosts文件配置：
[root@racnode1 Packages]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
#public network
192.168.1.8 racnode1
192.168.1.9 racnode2
#private network
192.168.237.126 racnode1-priv
192.168.237.127 racnode2-priv
#virtual network
192.168.1.18 racnode1-vip
192.168.1.19 racnode2-vip
#scan
192.168.1.22 racnode-cluster-scan
#storage
192.168.237.128 storage-priv
预览一下配置生效后的结果：
[root@racnode1 Packages]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.1.8 netmask 255.255.255.0 broadcast 192.168.1.255
inet6 240e:39a:344:3b10:18f0:d699:a52:715a prefixlen 64 scopeid 0x0<global>
inet6 fe80::6fb5:dd8b:90f:3edb prefixlen 64 scopeid 0x20<link>
inet6 fe80::7395:ca88:671:f19f prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:53:4d:2d txqueuelen 1000 (Ethernet)
RX packets 4360 bytes 348580 (340.4 KiB)
RX errors 0 dropped 1302 overruns 0 frame 0
TX packets 2519 bytes 427636 (417.6 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ens37: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.237.126 netmask 255.255.255.0 broadcast 192.168.237.255
inet6 fe80::3431:565:9a78:4692 prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:53:4d:37 txqueuelen 1000 (Ethernet)
RX packets 6057 bytes 6882440 (6.5 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2349 bytes 215629 (210.5 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
4.IPSAN 的iscsi initiator配置
命令如下：
[root@racnode1 Packages]# iscsiadm -m discovery -t sendtargets -p storage-priv
192.168.237.128:3260,1 iqn.2003-01.org.linux-iscsi.storage.x8664:sn.b0a7e05a08de
[root@racnode1 Packages]# iscsiadm -m node -T iqn.2003-01.org.linux-iscsi.storage.x8664:sn.b0a7e05a08de -p storage-priv --login
Logging in to [iface: default, target: iqn.2003-01.org.linux-iscsi.storage.x8664:sn.b0a7e05a08de, portal: 192.168.237.128,3260] (multiple)
Login to [iface: default, target: iqn.2003-01.org.linux-iscsi.storage.x8664:sn.b0a7e05a08de, portal: 192.168.237.128,3260] successful.
[root@racnode1 Packages]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdd 8:48 0 2G 0 disk
sdb 8:16 0 20G 0 disk
sr0 11:0 1 4.3G 0 rom /mnt/cdrom
sdc 8:32 0 15G 0 disk
sda 8:0 0 40G 0 disk
├─sda2 8:2 0 39G 0 part
│ ├─ol-swap 252:1 0 2G 0 lvm [SWAP]
│ └─ol-root 252:0 0 37G 0 lvm /
└─sda1 8:1 0 1G 0 part /boot
[root@racnode1 Packages]# iscsiadm -m node
192.168.237.128:3260,1 iqn.2003-01.org.linux-iscsi.storage.x8664:sn.b0a7e05a08de
最后确认下，是否将防火墙关闭，selinux 是否关闭。
[root@racnode1 ~]# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
[root@racnode1 ~]#
[root@racnode1 ~]# getenforce
Disabled
在系统启动后，自动启动iscsi initiator 登录
[root@racnode1 ~]# iscsiadm -m node
192.168.237.128:3260,1 iqn.2003-01.org.linux-iscsi.storage.x8664:sn.b0a7e05a08de
[root@racnode1 ~]# iscsiadm -m node -T iqn.2003-01.org.linux-iscsi.storage.x8664:sn.b0a7e05a08de -p storage-priv --op update -n node.startup -v automatic
[root@racnode1 ~]#
本小节需要注意以下，安装节点时如果是采用直接克隆虚拟机方式的，会有出现initiator-name相同的情况，后台会出现大量错误。
Jun 8 17:57:50 racnode2 iscsid: connection1:0 is operational after recovery (1 attempts)
Jun 8 17:57:52 racnode2 kernel: connection1:0: detected conn error (1020)
Jun 8 17:57:52 racnode2 iscsid: Kernel reported iSCSI connection 1:0 error (1020 - ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (3)
Jun 8 17:57:54 racnode2 iscsid: connection1:0 is operational after recovery (1 attempts)
Jun 8 17:57:56 racnode2 kernel: connection1:0: detected conn error (1020)
Jun 8 17:57:56 racnode2 iscsid: Kernel reported iSCSI connection 1:0 error (1020 - ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (3)
Jun 8 17:57:58 racnode2 iscsid: connection1:0 is operational after recovery (1 attempts)
Jun 8 17:58:00 racnode2 kernel: connection1:0: detected conn error (1020)
Jun 8 17:58:00 racnode2 iscsid: Kernel reported iSCSI connection 1:0 error (1020 - ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) sta
我们需要避免这样的情况发生。所以需要在节点上修改IQN，避免重名。
[root@racnode1 ~]# cat /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.1988-12.com.oracle:53507fdd1974
5.创建分区
依次创建分区：
首先是/dev/sdb
[root@racnode1 ~]# fdisk /dev/sdb
Welcome to fdisk (util-linux 2.23.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0xb72735dd.
Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (8192-41943039, default 8192):
Using default value 8192
Last sector, +sectors or +size{K,M,G} (8192-41943039, default 41943039):
Using default value 41943039
Partition 1 of type Linux and of size 20 GiB is set
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
其次是/dev/sdc：
[root@racnode1 ~]# fdisk /dev/sdc
Welcome to fdisk (util-linux 2.23.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0xa2733da1.
Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p):
Using default response p
Partition number (1-4, default 1):
First sector (8192-31457279, default 8192):
Using default value 8192
Last sector, +sectors or +size{K,M,G} (8192-31457279, default 31457279):
Using default value 31457279
Partition 1 of type Linux and of size 15 GiB is set
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
最后是/dev/sdd：
[root@racnode1 ~]# fdisk /dev/sdd
Welcome to fdisk (util-linux 2.23.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x74090209.
Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p):
Using default response p
Partition number (1-4, default 1):
First sector (8192-4194303, default 8192):
Using default value 8192
Last sector, +sectors or +size{K,M,G} (8192-4194303, default 4194303):
Using default value 4194303
Partition 1 of type Linux and of size 2 GiB is set
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
6.其余集群节点配置和同步
在另外一个节点服务器上重复1-4的步骤（可以直接采用虚拟机的克隆方式，这样更快）。需要注意的是相应的主机名和IP需要同步按照上文中的集群架构图中标识的那样进行修改。
这里的第5步不用在做。只需要同步即可。
[root@racnode2 ~]# partprobe
Warning: Unable to open /dev/sr0 read-write (Read-only file system). /dev/sr0 has been opened read-only.
[root@racnode2 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdd 8:48 0 2G 0 disk
└─sdd1 8:49 0 2G 0 part
sdb 8:16 0 20G 0 disk
└─sdb1 8:17 0 20G 0 part
sr0 11:0 1 4.3G 0 rom
sdc 8:32 0 15G 0 disk
└─sdc1 8:33 0 15G 0 part
sda 8:0 0 40G 0 disk
├─sda2 8:2 0 39G 0 part
│ ├─ol-swap 252:1 0 2G 0 lvm [SWAP]
│ └─ol-root 252:0 0 37G 0 lvm /
└─sda1 8:1 0 1G 0 part /boot
[root@racnode2 ~]#
下一节，我们继续配置工作。
