Docker Swarm Calico Etcd Haproxy

1.引用

参考:https://www.hujianxiong.com/archives/swarm-platform

2.安装组件

• Swarm 使用原生的集群方案
• Etcd key/value存储集群,网络信息等
• Registrator 服务注册
• Calico 网络
• Consul-Haproxy 自动代理后端服务
• Consul Registrator注册的地方也可以使用Etcd,不过有现成的Consul-Haproxy模板可以使用就不替换了,使用Etcd可以替换成confd-Haproxy

3.部署Master端

部署服务etcd,calico,consul,consul-haproxy,docker-proxy,swarm-manager,docker-daemon

1.配置ENV

ntpdate 10.20.101.251
export MASTER_IP=10.20.140.2
export DC=test02_dc 
export MASTER_HOSTNAME=docker.swarm.master01
export HAPROXYDOMAIN=test02.youmenlu.com

2.安装Etcd

yum -y install etcd
sed -i  "s/ETCD_LISTEN_CLIENT_URLS=\"http:\/\/localhost:2379\"/ETCD_LISTEN_CLIENT_URLS=\"http:\/\/0.0.0.0:2379\"/"  /etc/etcd/etcd.conf
sed  -i  "s/ETCD_ADVERTISE_CLIENT_URLS=\"http:\/\/localhost:2379\"/ETCD_ADVERTISE_CLIENT_URLS=\"http:\/\/`echo $MASTER_IP`:2379\"/" /etc/etcd/etcd.conf 
cat /etc/etcd/etcd.conf 
systemctl daemon-reload
systemctl start etcd
systemctl enable etcd

3.安装Docker-daemon

yum -y install docker-engine
sed -i "s/ExecStart=\/usr\/bin\/dockerd/ExecStart=\/usr\/bin\/dockerd -g=\/yschome\/data\/docker --insecure-registry 10.20.145.240  --registry-mirror=https:\/\/kyyfenk.mirror.acs.aliyun.com --cluster-advertise eth0:2375  --cluster-store etcd:\/\/`echo $MASTER_IP`:2379/" /usr/lib/systemd/system/docker.service
systemctl daemon-reload
systemctl start docker
systemctl enable docker

4.安装Calico

yum -y install wget 
wget -O /usr/local/bin/calicoctl http://download.ys-city.com/docker/calicoctl-1.1.0
chmod +x /usr/local/bin/calicoctl
mkdir -p /etc/calico/

cat <<EOF > /etc/calico/calicoctl.cfg
apiVersion: v1
kind: calicoApiConfig
metadata:
spec:
  datastoreType: "etcdv2"
  etcdEndpoints: "http://`echo $MASTER_IP`:2379"
EOF
cat /etc/calico/calicoctl.cfg
## calico-node:v1.1.0 官网上找这个镜像
calicoctl node run  --use-docker-networking-container-labels  --docker-networking-ifprefix=eth --node-image=10.20.145.240/library/calico-node:v1.1.0
calicoctl node status
##配置Calico内部访问规则
cat << EOF | calicoctl apply -f -
- apiVersion: v1
  kind: policy
  metadata:
    name: backend
  spec:
    order: 0
    selector: backend == 'true'
    ingress:
    - action: allow
      protocol: tcp
      source:
        selector: backend == 'true'
    - action: allow
      source:
      selector: backend == 'true'
  egress:
  - action: allow
    destination:
      selector: backend == 'true'
      
EOF
##配置外部访问规则,测试环境简单粗暴配置,生产环境需安需配置
cat << EOF | calicoctl apply -f -
- apiVersion: v1
  kind: policy
  metadata:
  name: domain
  spec:
    order: 0
    ingress:
    - action: allow
      protocol: tcp
    - action: allow
    egress:
    - action: allow
EOF

5.启动容器组件

docker run -d \
-p 8300:8300 \
-p 8301:8301 \
-p 8301:8301/udp \
-p 8302:8302 \
-p 8302:8302/udp \
-p 8400:8400 \
-p 8500:8500 \
-p 8600:53 \
-p 53:53/udp \
-v /yschome/data/consul:/data \
-h `echo $MASTER_HOSTNAME`  \
--restart=always \
--label registrator.ignored=true \
--name=consul 10.20.145.240/library/consul:0.5.2 -server -bootstrap -ui-dir=/ui -dc=`echo $DC` -advertise 10.20.141.20 -client 0.0.0.0

docker run  -d -p 80:80 --name haproxy  \
--restart always  \
-e HAPROXY_DOMAIN=`echo $HAPROXYDOMAIN` \
-e CONSUL_CONNECT=`echo $MASTER_IP`:8500 \
10.20.145.240/library/haproxy:1.1

docker run -ti -d \
-p 2375:2375 \
--hostname=`echo $MASTER_HOSTNAME` \
--restart=always \
--name devops-proxy \
--label registrator.ignored=true \
-v /var/run/docker.sock:/var/run/docker.sock \
-e PORT=2375 \
10.20.145.240/library/docker-proxy:latest
docker run -ti -d \
--restart=always -p 3375:3375 \
--label registrator.ignored=true \
--name devops-swarm-manager  10.20.145.240/library/swarm:latest  \
manage  --replication \
--addr `echo $MASTER_IP`:3375 \
--host tcp://0.0.0.0:3375  etcd://`echo $MASTER_IP`:2379

6.创建集群网络

docker network create --driver calico --ipam-driver calico-ipam   my-net

4.部署Agent端

安装组件docker-daemon,calico,docker-proxy,registrator,swarm-agent

registrator 我fork了一份https://github.com/gliderlabs/registrator代码,做了点修改

1.配置ENV

ntpdate 10.20.101.251
export MASTER_IP=10.20.140.2

2.安装Docker-daemon

yum -y install docker-engine
sed -i "s/ExecStart=\/usr\/bin\/dockerd/ExecStart=\/usr\/bin\/dockerd -g=\/yschome\/data\/docker --insecure-registry 10.20.145.240  --registry-mirror=https:\/\/kyyfenk.mirror.acs.aliyun.com --cluster-advertise eth0:2375  --cluster-store etcd:\/\/`echo $MASTER_IP`:2379/" /usr/lib/systemd/system/docker.service
cat /usr/lib/systemd/system/docker.service | grep ExecStart

systemctl daemon-reload
systemctl start docker
systemctl enable docker

3.安装Calico

yum -y install wget 
wget -O /usr/local/bin/calicoctl http://download.ys-city.com/docker/calicoctl-1.1.0
chmod +x /usr/local/bin/calicoctl
mkdir -p /etc/calico/

cat <<EOF > /etc/calico/calicoctl.cfg
apiVersion: v1
kind: calicoApiConfig
metadata:
spec:
  datastoreType: "etcdv2"
  etcdEndpoints: "http://`echo $MASTER_IP`:2379"
EOF
cat /etc/etcd/etcd.conf 
calicoctl node run  --use-docker-networking-container-labels  --docker-networking-ifprefix=eth --node-image=10.20.145.240/library/calico-node:v1.1.0

calicoctl node status
docker ps -a 

4.启动容器组件

docker run -ti -d \
-p 2375:2375 \
--hostname=`hostname` \
--label registrator.ignored=true \ 
--restart=always \
--name devops-proxy \
-v /var/run/docker.sock:/var/run/docker.sock \
-e PORT=2375 \
10.20.145.240/library/docker-proxy:latest

docker run -ti -d \
--restart=always \
--label registrator.ignored=true \
--name devops-swarm-agent \
 10.20.145.240/library/swarm:latest  \
join --addr `ifconfig eth0|grep inet|grep -v 127.0.0.1|grep -v inet6|awk '{print $2}'|tr -d 'addr:'`:2375 etcd://`echo $MASTER_IP`:2379


docker run -d \
--name=registrator \
--restart=always \
--net=host \
--volume=/var/run/docker.sock:/tmp/docker.sock \
10.20.145.240/library/registrator:v1.0 \
-internal=true consul://`echo $MASTER_IP`:8500

5.说明

集群安装完成后,默认haproxy会配置根据配置的域名做好代理,如果不希望代理的只需在启动容器时加上 --label registrator.ignored=true 即可,这里了registrator代码做了修改

容器使用calico网络后有个问题,即宿主机reboot后docker-daemon无法正常启动,由于容器启动顺序导致循环依赖,解决方法是修改依赖calico网络的容器中的restart策略为no