分布式键值存储系统 etcd 集群部署指南

2025-11-27 12:38:56 分类：分布式 热度：20 评论： 0

分布式键值存储系统 etcd 集群部署指南

一、环境描述

192.168.183.10 server1.linux.***
192.168.183.11 server2.linux.***
192.168.183.12 serve3.linux.***

二、基础配置

1、关闭SELinux, 防火墙、配置时间同步
2、添加所有主机名解析
3、配置所有节点免密SSH

三、在所有节点安装docker

[root@server1 ~]# cat /etc/docker/daemon.json 
{"registry-mirrors": ["http://f1361db2.m.daocloud.io"]}

[root@server1 ~]# docker version 
Client: Docker Engine - ***munity
 Version:           20.10.6
 API version:       1.41
 Go version:        go1.13.15
 Git ***mit:        370c289
 Built:             Fri Apr  9 22:45:33 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - ***munity
 Engine:
  Version:          20.10.6
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git ***mit:       8728dd2
  Built:            Fri Apr  9 22:43:57 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.4
  Git***mit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc:
  Version:          1.0.0-rc93
  Git***mit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
 docker-init:
  Version:          0.19.0
  Git***mit:        de40ad0

四、配置etcd集群

  1. 安装etcd数据库
[root@server1 src]# tar xf etcd-v3.4.15-linux-amd64.tar.gz 
[root@server1 src]# cd etcd-v3.4.15-linux-amd64/
[root@server1 etcd-v3.4.15-linux-amd64]# cp etcd etcdctl /usr/local/bin/
  1. 编辑etcd配置文件
[root@server1 ~]# mkdir /etc/etcd
[root@server1 ~]# mkdir /var/lib/etcd
[root@server1 ~]# vim /etc/etcd/etcd.conf
ETCD_NAME=etcd1
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_LISTEN_CLIENT_URLS="http://192.168.183.10:2379,http://127.0.0.1:2379"
ETCD_LISTEN_PEER_URLS="http://192.168.183.10:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.183.10:2380"
ETCD_INITIAL_CLUSTER="etcd1=http://192.168.183.10:2380,etcd2=http://192.168.183.11:2380,etcd3=http://192.168.183.12:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.183.10:2379"
  1. 编写etcd启动脚本
[root@server2 ~]# cat /etc/systemd/system/etcd.service 
[Unit]
Description=Etcd Server
After=***work.target

[Service]
Type=simple
WorkingDirectory=/var/lib/etcd
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/local/bin/etcd --enable-v2"
Type=notify
[Install]
WantedBy=multi-user.target

GOMAXPROCS=设置etcd的进程数
–enable-v2
作用: 兼容v2版本的API
此etcd数据库为3.4版本，其API版本为V3，而后续flannel写入网络信息时，需要连接V2版本的API写入

  1. 启动etcd服务
[root@server1 ~]# systemctl daemon-reload
[root@server1 ~]# systemctl enable etcd.service
[root@server1 ~]# systemctl start etcd.service

此时，第一个节点启动etcd服务时会卡住，无法正常启动；因为我们是以集群的方式部署etcd，它启动时会尝试连接配置文件中的其他节点，当连接不到时，会无法正常启动。可以按ctrl + c终止启动，通过查看进程可以看到etcd进程存在；等待其他节点配置完成，再次启动即可！

[root@server1 ~]# ***stat -antp | grep etcd
tcp        0      0 192.168.183.10:2379     0.0.0.0:*               LISTEN      2191/etcd           

[root@server1 ~]# ps -elf | grep etcd
4 S root       2191      1  3  80   0 - 2653208 futex_ 20:30 ?      00:00:36 /usr/local/bin/etcd

另外两个节点按照相同的配置即可，仅需要注意修改etcd.conf配置文件中的监听地址、节点名称即可

  1. 查看etcd集群运行状态

查看集群状态

[root@server1 ~]# etcdctl member list
594853835b20098, started, etcd3, http://192.168.183.12:2380, http://192.168.183.12:2379, false
bce1def4364b82f9, started, etcd1, http://192.168.183.10:2380, http://192.168.183.10:2379, false
e1d1c6fc7809991b, started, etcd2, http://192.168.183.11:2380, http://192.168.183.11:2379, false

[root@server1 ~]# etcdctl --endpoints=http://192.168.183.10:2379,http://192.168.183.11:2379,192.168.183.12:2379 endpoint status --write-out=table
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|          ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| http://192.168.183.10:2379 | bce1def4364b82f9 |  3.4.15 |   25 kB |      true |      false |        65 |          9 |                  9 |        |
| http://192.168.183.11:2379 | e1d1c6fc7809991b |  3.4.15 |   20 kB |     false |      false |        65 |          9 |                  9 |        |
|        192.168.183.12:2379 |  594853835b20098 |  3.4.15 |   25 kB |     false |      false |        65 |          9 |                  9 |        |
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

[root@server1 ~]# etcdctl --endpoints=http://192.168.183.10:2379,http://192.168.183.11:2379,192.168.183.12:2379 endpoint health
http://192.168.183.10:2379 is healthy: su***essfully ***mitted proposal: took = 5.499123ms
192.168.183.12:2379 is healthy: su***essfully ***mitted proposal: took = 5.767108ms
http://192.168.183.11:2379 is healthy: su***essfully ***mitted proposal: took = 5.437299ms

查看etcd数据库版本

[root@server1 ~]# etcd --version
etcd Version: 3.4.15
Git SHA: aa7126864
Go Version: go1.12.17
Go OS/Arch: linux/amd64

查看etcd API版本

[root@server1 ~]# etcdctl version
etcdctl version: 3.4.15
API version: 3.4

早期的etcd数据库API版本为V2， 本文档使用的数据库API版本为V3， V3版本与早期的V2版本操作指令具有很大的不同，可以通过etcdctl --help查看相应指令帮助
早期数据库API V2版本时，可以通过定义环境变量的方式修改API版本为3

# export ETCDCTL_API=3

五、在所有节点配置flannel网络

安装flannel

[root@server1 src]# tar xf flannel-v0.13.0-linux-amd64.tar.gz 
[root@server1 src]# cp flanneld mk-docker-opts.sh /usr/local/bin/

向etcd数据库写入flannel网络信息

[root@server1 src]# export ETCDCTL_API=2
[root@server1 src]# etcdctl set /coreos.***/***work/config '{"***work": "172.16.0.0/16"}'

编写flanneld启动脚本

[root@server1 src]# cat /etc/systemd/system/flannel.service 
[Unit]
Description=Flanneld
Documentation=https://github.***/coreos/flannel
After=***work.target
Before=docker.service

[Service]
User=root
ExecStartPost=/usr/local/bin/mk-docker-opts.sh
ExecStart=/usr/local/bin/flanneld \
--etcd-endpoints=http://192.168.183.10:2379,http://192.168.183.11:2379,http://192.168.183.12:2379 \
--iface=192.168.183.10 \
--ip-masq=true \
--etcd-prefix=/coreos.***/***work
Restart=on-failure
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

# systemctl daemon-reload
# systemctl enable flannel.service 
# systemctl start flannel.service

修改docker启动脚本，重启docker

[root@server1 src]# vim/usr/lib/systemd/system/docker.service 
EnvironmentFile=-/run/docker_opts.env
ExecStart=/usr/bin/dockerd $DOCKER_OPTS -H fd:// --containerd=/run/containerd/containerd.sock

[root@server1 src]# systemctl daemon-reload
[root@server1 src]# systemctl restart docker

验证docker网络被flannel接管

[root@server1 src]# ifconfig docker0
docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        i*** 172.16.57.1  ***mask 255.255.255.0  broadcast 172.16.57.255
        i***6 fe80::42:c0ff:fe16:8875  prefixlen 64  scopeid 0x20<link>
        ether 02:42:c0:16:88:75  txqueuelen 0  (Ether***)
        RX packets 13  bytes 924 (924.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 18  bytes 1544 (1.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@server1 src]# ifconfig flannel0
flannel0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>  mtu 1472
        i*** 172.16.57.0  ***mask 255.255.255.255  destination 172.16.57.0
        i***6 fe80::768c:9099:95c0:6ad  prefixlen 64  scopeid 0x20<link>
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 500  (UNSPEC)
        RX packets 10  bytes 840 (840.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 13  bytes 984 (984.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

将flanneld相关命令、管理脚本拷贝到另外两个节点

[root@server1 src]# rsync -av /etc/systemd/system/flannel.service root@192.168.183.11:/etc/systemd/system/flannel.service 
[root@server1 src]# rsync -av /etc/systemd/system/flannel.service root@192.168.183.12:/etc/systemd/system/flannel.service
 
[root@server1 src]# rsync -av /usr/local/bin/flanneld root@192.168.183.11:/usr/local/bin/
[root@server1 src]# rsync -av /usr/local/bin/mk-docker-opts.sh root@192.168.183.11:/usr/local/bin/

[root@server1 src]# rsync -av /usr/local/bin/flanneld root@192.168.183.12:/usr/local/bin/
[root@server1 src]#  rsync -av /usr/local/bin/mk-docker-opts.sh root@192.168.183.12:/usr/local /bin/

将另外两个节点中flannel脚本–iface=修改为对应的IP地址，启动flanneld；
同样的方法修改docker启动脚本，添加flannel选项，重启docker即可！！！！

验证容器间的通信

在三个节点分别创建容器，测试容器间可相互通信即可，到此，flannel网络部署完毕！

# docker run -tid busybox 
# docker exec -ti b283 /bin/sh

/ # ping 172.16.69.2
PING 172.16.69.2 (172.16.69.2): 56 data bytes
64 bytes from 172.16.69.2: seq=0 ttl=60 time=0.860 ms
64 bytes from 172.16.69.2: seq=1 ttl=60 time=1.877 ms
64 bytes from 172.16.69.2: seq=2 ttl=60 time=0.962 ms

/ # ping 172.16.87.2
PING 172.16.87.2 (172.16.87.2): 56 data bytes
 64 bytes from 172.16.87.2: seq=0 ttl=60 time=1.294 ms
64 bytes from 172.16.87.2: seq=1 ttl=60 time=1.066 ms

六、生成k8s组件间通信需要的证书

证书说明

  • ca.pem、ca-key.pem
    CA的证书、密钥
  • server.pem server-key.pem
    kube-apiserver组件的证书，基于https提供服务
  • admin.pem、admin-key.pem
    kubectl客户端工具与kube-apiserver通信时使用的证书
  • kube-proxy.pem、kube-proxy-key.pem
    kube-proxy组件与kube-apiserver通信时使用的证书
  • kubelet
    kubelet启动时，需要向kube-apiserver进行注册，注册时需要token认证
    同时kube-apiserver会向kubelet自动颁发证书
转载请说明出处内容投诉
CSS教程网 » 分布式键值存储系统 etcd 集群部署指南
心舞飞扬

心舞飞扬

分享到：

发表评论

欢迎 访客 发表评论

一个令你着迷的主题！

查看演示 官网购买