etcd集群配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
etcd  命令含义
`--name` 
etcd集群中的节点名，这里可以随意，可区分且不重复就行
`--listen-peer-urls`
监听的用于节点之间通信的url，可监听多个，集群内部将通过这些url进行数据交互(如选举，数据同步等) 
 `--initial-advertise-peer-urls`
建议用于节点之间通信的url，节点间将以该值进行通信。
`--listen-client-urls`
监听的用于客户端通信的url,同样可以监听多个。
`--advertise-client-urls`
建议使用的客户端通信url,该值用于etcd代理或etcd成员与etcd节点通信。
`--initial-cluster-token etcd-cluster-1`
节点的token值，设置该值后集群将生成唯一id,并为每个节点也生成唯一id,当使用相同配置文件再启动一个集群时，只要该token值不一样，etcd集群就不会相互影响。
-`-initial-cluster`
也就是集群中所有的initial-advertise-peer-urls 的合集
`--initial-cluster-state new`
新建集群的标志，初始化状态使用 new，建立之后改此值为 existing

ETCD安装配置

1.部署架构

1
2
3
192.168.108.128 节点1
192.168.108.129 节点2
192.168.108.130 节点3

2.解压安装：

1
2
3
4
5
$ tar -zxvf  etcd-v3.2.6-linux-amd64.tar.gz -C /opt/
$ cd /opt
$ mv etcd-v3.2.6-linux-amd64  etcd-v3.2.6
# 创建etcd配置文件目录
$ mkdir /etc/etcd

3.创建etcd配置文件：

1
$ vi /etc/etcd/conf.yml

节点1，添加如下内容：

1
2
3
4
5
6
7
8
9
name: etcd-1
data-dir: /opt/etcd-v3.2.6/data
listen-client-urls: http://192.168.108.128:2379,http://127.0.0.1:2379
advertise-client-urls: http://192.168.108.128:2379,http://127.0.0.1:2379
listen-peer-urls: http://192.168.108.128:2380
initial-advertise-peer-urls: http://192.168.108.128:2380
initial-cluster: etcd-1=http://192.168.108.128:2380,etcd-2=http://192.168.108.129:2380,etcd-3=http://192.168.108.130:2380
initial-cluster-token: etcd-cluster-token
initial-cluster-state: new

节点2，添加如下内容：

1
2
3
4
5
6
7
8
9
name: etcd-2
data-dir: /opt/etcd-v3.2.6/data
listen-client-urls: http://192.168.108.129:2379,http://127.0.0.1:2379
advertise-client-urls: http://192.168.108.129:2379,http://127.0.0.1:2379
listen-peer-urls: http://192.168.108.129:2380
initial-advertise-peer-urls: http://192.168.108.129:2380
initial-cluster: etcd-1=http://192.168.108.128:2380,etcd-2=http://192.168.108.129:2380,etcd-3=http://192.168.108.130:2380
initial-cluster-token: etcd-cluster-token
initial-cluster-state: new

节点3，添加如下内容：

1
2
3
4
5
6
7
8
9
name: etcd-3
data-dir: /opt/etcd-v3.2.6/data
listen-client-urls: http://192.168.108.130:2379,http://127.0.0.1:2379
advertise-client-urls: http://192.168.108.130:2379,http://127.0.0.1:2379
listen-peer-urls: http://192.168.108.130:2380
initial-advertise-peer-urls: http://192.168.108.130:2380
initial-cluster: etcd-1=http://192.168.108.128:2380,etcd-2=http://192.168.108.129:2380,etcd-3=http://192.168.108.130:2380
initial-cluster-token: etcd-cluster-token
initial-cluster-state: new

4.更新etcd系统默认配置：

当前使用的是etcd v3版本，系统默认的是v2，通过下面命令修改配置。

1
$ vi /etc/profile

在文件末尾追加：

1
export ETCDCTL_API=3

使文件生效：

1
$ source /etc/profile

ETCD命令

1
$ cd  /opt/etcd-v3.2.6

1.创建etcd配置文件：

1
2
3
4
$ ./etcdctl version
 
etcdctl version: 3.2.6
API version: 3.2

2.启动命令：

1
$ ./etcd --config-file=/etc/etcd/conf.yml

3.查看集群成员信息：

1
2
3
4
5
$ ./etcdctl member list
 
2618ce5cd761aa8e: name=etcd-3 peerURLs=http://192.168.108.130:2380 clientURLs=http://127.0.0.1:2379,http://192.168.108.130:2379 isLeader=false
9c359d48a2f34938: name=etcd-1 peerURLs=http://192.168.108.128:2380 clientURLs=http://127.0.0.1:2379,http://192.168.108.128:2379 isLeader=false
f3c45714407d68f3: name=etcd-2 peerURLs=http://192.168.108.129:2380 clientURLs=http://127.0.0.1:2379,http://192.168.108.129:2379 isLeader=true

4.查看集群状态（Leader节点）：

1
2
3
4
5
6
$ ./etcdctl cluster-health
 
member 2618ce5cd761aa8e is healthy: got healthy result from http://127.0.0.1:2379
member 9c359d48a2f34938 is healthy: got healthy result from http://127.0.0.1:2379
member f3c45714407d68f3 is healthy: got healthy result from http://127.0.0.1:2379
cluster is healthy

ECTD读写操作

基于HTTP协议的API使用起来比较简单，这里主要通过etcdctl和curl两种方式来做简单介绍。

1.下面通过给message key设置Hello值示例：

1
2
3
4
5
6
$ ./etcdctl set /message Hello
 
Hello
 
$ curl -X PUT http://127.0.0.1:2379/v2/keys/message -d value="Hello"
{"action":"set","node":{"key":"/message","value":"Hello","modifiedIndex":4,"createdIndex":4}}

2.读取message的值：

1
2
3
4
5
$ ./etcdctl  get /message
Hello
 
$ curl http://127.0.0.1:2379/v2/keys/message
{"action":"get","node":{"key":"/message","value":"Hello","modifiedIndex":9,"createdIndex":9}}

3.删除message key：

1
2
3
4
$ ./etcdctl  rm  /message
 
$ curl -X DELETE http://127.0.0.1:2379/v2/keys/message
{"action":"delete","node":{"key":"/message","modifiedIndex":10,"createdIndex":9},"prevNode":{"key":"/message","value":"Hello","modifiedIndex":9,"createdIndex":9}}

说明：因为是集群，所以message在其中一个节点创建后，在集群中的任何节点都可以查询到。

配置ETCD为启动服务

1.编辑/usr/lib/systemd/system/etcd.service，添加下面内容：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
 
[Service]
Type=notify
WorkingDirectory=/opt/etcd-v3.2.6/
# User=etcd
ExecStart=/opt/etcd-v3.2.6/etcd --config-file=/etc/etcd/conf.yml
Restart=on-failure
LimitNOFILE=65536
 
[Install]
WantedBy=multi-user.target

2.更新启动：

1
2
3
4
5
6
systemctl daemon-reload
systemctl enable etcd
systemctl start etcd
systemctl restart etcd
 
systemctl status etcd.service -l

编写测试客户端进行 kv 操作

git clone https://github.com/etcd-io/etcd.git $GOPATH/src/go.etcd.io/etcd/

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
package main

import (
	"context"
	"fmt"
	"log"
	"strconv"
	"time"

	"go.etcd.io/etcd/clientv3"
	"vincent.com/etcd/etcd/etcdserver/api/v3rpc/rpctypes"
)

func main() {

	cli, err := clientv3.New(clientv3.Config{
		Endpoints:   []string{"http://127.0.0.1:23790", "http://127.0.0.1:23791"},
		DialTimeout: 5 * time.Second,
	})
	if err != nil {
		// handle error!
		log.Fatalln(err.Error())
	}
	defer cli.Close()
	defer cli.Delete(context.Background(), "sample_key")

	for index := 0; index < 10000; index++ {
		insertKV(cli, "sample_key", "sample_value"+strconv.Itoa(index))
		if err != nil {
			break
		}
	}
	resp, err := cli.Get(context.Background(), "sample_key")
	if err != nil {
		log.Fatalln("get key error", err)
	}
	fmt.Printf("get the sample_key: %v\n", resp.Kvs)
}

func insertKV(cli *clientv3.Client, key string, value string) (err error) {
	time.Sleep(2 * time.Millisecond)
	_, err = cli.Put(context.Background(), key, value)
	// cancel()
	if err != nil {
		switch err {
		case context.Canceled:
			log.Fatalf("ctx is canceled by another routine: %v", err)

		case context.DeadlineExceeded:
			log.Fatalf("ctx is attached with a deadline is exceeded: %v", err)
		case rpctypes.ErrEmptyKey:
			log.Fatalf("client-side error: %v", err)
		default:
			log.Fatalf("bad cluster endpoints, which are not etcd servers: %v", err)
		}
	}
	return
}

集群性能检测

1
2
3
4
5
6
ETCDCTL_API=3 ./etcdctl check perf load="l" --endpoints=127.0.0.1:2379,127.0.0.1:22379,127.0.0.1:32379,127.0.0.1:42379,127.0.0.1:52379
 60 / 60 Booooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo! 100.00%1m0s
PASS: Throughput is 150 writes/s
PASS: Slowest request took 0.388842s
PASS: Stddev is 0.029037s
PASS