Use OpenVSwitch & bridge & Docker & pipework fast build complicated testing ENV-白红宇

本文将描述一下如何快速的部署一个网络较为复杂(至少应包含VLAN)的测试环境.

单机下 OVS+bridge+Docker 的架构 :

Use OpenVSwitch bridge Docker pipework fast build complicated testing ENV - 德哥@Digoal - PostgreSQL research

例子 :

添加两个网桥[root@localhost ~]# brctl addbr br0[root@localhost ~]# brctl addbr br1[root@localhost ~]# ip link set br0 up[root@localhost ~]# ip link set br1 up添加一个OVS交换机[root@localhost ~]# ovs-vsctl showf345b7e3-fcb0-4ef3-8295-36d3ef69ceef    ovs_version: "2.3.0"[root@localhost ~]# ovs-vsctl add-br ovs0[root@localhost ~]# ip link set ovs0 up将网桥加入交换机, 设置VLAN tag不一致.[root@localhost ~]# ovs-vsctl add-port ovs0 br0 tag=10[root@localhost ~]# ovs-vsctl add-port ovs0 br1 tag=11[root@localhost ~]# ovs-vsctl showf345b7e3-fcb0-4ef3-8295-36d3ef69ceef    Bridge "ovs0"        Port "br1"            tag: 11            Interface "br1"        Port "ovs0"            Interface "ovs0"                type: internal        Port "br0"            tag: 10            Interface "br0"    ovs_version: "2.3.0"创建两个容器[root@localhost ~]# docker run -t -i --rm --name=test0 --net=none centos:centos7 /bin/bash[root@localhost ~]# docker run -t -i --rm --name=test1 --net=none centos:centos7 /bin/bash使用pipework设置容器网络, 分别加入两个网桥.[root@localhost ~]# ./pipework.sh br0 test0 172.1.0.1/24[root@localhost ~]# ./pipework.sh br1 test1 172.1.0.2/24验证两个容器不通[root@71000703ed99 /]# ip addr1: lo: 
      
        mtu 65536 qdisc noqueue state UNKNOWN     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00    inet 127.0.0.1/8 scope host lo       valid_lft forever preferred_lft forever    inet6 ::1/128 scope host        valid_lft forever preferred_lft forever61: eth1: 
       
         mtu 1500 qdisc pfifo_fast state UP qlen 1000    link/ether 12:7c:dc:54:64:b9 brd ff:ff:ff:ff:ff:ff    inet 172.1.0.1/24 scope global eth1       valid_lft forever preferred_lft forever    inet6 fe80::107c:dcff:fe54:64b9/64 scope link        valid_lft forever preferred_lft forever[root@0c5a5dba1420 /]# ip addr1: lo: 
        
          mtu 65536 qdisc noqueue state UNKNOWN     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00    inet 127.0.0.1/8 scope host lo       valid_lft forever preferred_lft forever    inet6 ::1/128 scope host        valid_lft forever preferred_lft forever63: eth1: 
         
           mtu 1500 qdisc pfifo_fast state UP qlen 1000    link/ether ba:51:9e:d2:1c:a4 brd ff:ff:ff:ff:ff:ff    inet 172.1.0.2/24 scope global eth1       valid_lft forever preferred_lft forever    inet6 fe80::b851:9eff:fed2:1ca4/64 scope link        valid_lft forever preferred_lft forever[root@71000703ed99 /]# ping 172.1.0.2PING 172.1.0.2 (172.1.0.2) 56(84) bytes of data.^C--- 172.1.0.2 ping statistics ---1 packets transmitted, 0 received, 100% packet loss, time 0ms[root@localhost ~]# brctl show br0bridge name     bridge id               STP enabled     interfacesbr0             8000.1ad3bd8b93f9       no              veth1pl226106[root@localhost ~]# brctl show br1bridge name     bridge id               STP enabled     interfacesbr1             8000.fa8e634c6116       no              veth1pl226172[root@localhost ~]# ovs-vsctl showf345b7e3-fcb0-4ef3-8295-36d3ef69ceef    Bridge "ovs0"        Port "br1"            tag: 11            Interface "br1"        Port "ovs0"            Interface "ovs0"                type: internal        Port "br0"            tag: 10            Interface "br0"    ovs_version: "2.3.0"将网桥设置为同一个VLAN tag[root@localhost ~]# ovs-vsctl del-port ovs0 br1[root@localhost ~]# ovs-vsctl add-port ovs0 br1 tag=10测试两个容器可通讯.[root@71000703ed99 /]# ping 172.1.0.2PING 172.1.0.2 (172.1.0.2) 56(84) bytes of data.64 bytes from 172.1.0.2: icmp_seq=1 ttl=64 time=0.472 ms64 bytes from 172.1.0.2: icmp_seq=2 ttl=64 time=0.058 ms^C--- 172.1.0.2 ping statistics ---2 packets transmitted, 2 received, 0% packet loss, time 1000msrtt min/avg/max/mdev = 0.058/0.265/0.472/0.207 ms

如果单主机创建多个OVS交换机的话, 要让他们通讯也很简单,

例如

1. 使用peer将OVS交换机联合起来.

例如 :

[root@db-172-16-3-221 ~]# ovs-vsctl add-br ovs1[root@db-172-16-3-221 ~]# ovs-vsctl add-br ovs2[root@db-172-16-3-221 ~]# ovs-vsctl add-br ovs3[root@db-172-16-3-221 ~]# ovs-vsctl add-br ovs4[root@db-172-16-3-221 ~]# ip link set ovs1 up[root@db-172-16-3-221 ~]# ip link set ovs2 up[root@db-172-16-3-221 ~]# ip link set ovs3 up[root@db-172-16-3-221 ~]# ip link set ovs4 up[root@db-172-16-3-221 ~]# ip link add vv1 type veth peer name vvp1[root@db-172-16-3-221 ~]# ip link add vv2 type veth peer name vvp2[root@db-172-16-3-221 ~]# ip link add vv3 type veth peer name vvp3[root@db-172-16-3-221 ~]# ip link set vv1 up[root@db-172-16-3-221 ~]# ip link set vv2 up[root@db-172-16-3-221 ~]# ip link set vv3 up[root@db-172-16-3-221 ~]# ip link set vvp1 up[root@db-172-16-3-221 ~]# ip link set vvp2 up[root@db-172-16-3-221 ~]# ip link set vvp3 up[root@db-172-16-3-221 ~]# ovs-vsctl add-port ovs1 vv1[root@db-172-16-3-221 ~]# ovs-vsctl add-port ovs2 vvp1[root@db-172-16-3-221 ~]# ovs-vsctl add-port ovs2 vv2[root@db-172-16-3-221 ~]# ovs-vsctl add-port ovs3 vvp2[root@db-172-16-3-221 ~]# ovs-vsctl add-port ovs3 vv3[root@db-172-16-3-221 ~]# ovs-vsctl add-port ovs4 vvp3[root@db-172-16-3-221 ~]# brctl addbr bbr0[root@db-172-16-3-221 ~]# brctl addbr bbr1[root@db-172-16-3-221 ~]# ip link set bbr0 up[root@db-172-16-3-221 ~]# ip link set bbr1 up[root@db-172-16-3-221 ~]# ovs-vsctl add-port ovs1 bbr0 tag=100[root@db-172-16-3-221 ~]# ovs-vsctl add-port ovs4 bbr1 tag=101[root@db-172-16-3-221 ~]# ./pipework.sh bbr0 test0 172.1.0.100/24[root@db-172-16-3-221 ~]# ./pipework.sh bbr1 test1 172.1.0.101/24bash-4.2# ping 172.1.0.101PING 172.1.0.101 (172.1.0.101) 56(84) bytes of data.^C--- 172.1.0.101 ping statistics ---2 packets transmitted, 0 received, 100% packet loss, time 999ms[root@db-172-16-3-221 ~]# ovs-vsctl del-port ovs4 bbr1[root@db-172-16-3-221 ~]# ovs-vsctl add-port ovs4 bbr1 tag=100bash-4.2# ping 172.1.0.101PING 172.1.0.101 (172.1.0.101) 56(84) bytes of data.64 bytes from 172.1.0.101: icmp_seq=1 ttl=64 time=1.80 ms64 bytes from 172.1.0.101: icmp_seq=2 ttl=64 time=0.106 ms64 bytes from 172.1.0.101: icmp_seq=3 ttl=64 time=0.074 ms

2. 将所有OVS交换机加入同一个网桥.

例如 :

[root@db-172-16-3-221 ~]# ovs-vsctl del-port ovs1 vv1[root@db-172-16-3-221 ~]# ovs-vsctl del-port ovs2 vvp1[root@db-172-16-3-221 ~]# ovs-vsctl del-port ovs2 vv2[root@db-172-16-3-221 ~]# ovs-vsctl del-port ovs3 vvp2[root@db-172-16-3-221 ~]# ovs-vsctl del-port ovs3 vv3[root@db-172-16-3-221 ~]# ovs-vsctl del-port ovs4 vvp3[root@db-172-16-3-221 ~]# brctl addbr bbr2[root@db-172-16-3-221 ~]# ip link set bbr2 up[root@db-172-16-3-221 ~]# brctl addif bbr2 ovs1[root@db-172-16-3-221 ~]# brctl addif bbr2 ovs2[root@db-172-16-3-221 ~]# brctl addif bbr2 ovs3[root@db-172-16-3-221 ~]# brctl addif bbr2 ovs4bash-4.2# ping 172.1.0.101PING 172.1.0.101 (172.1.0.101) 56(84) bytes of data.64 bytes from 172.1.0.101: icmp_seq=1 ttl=64 time=0.402 ms^C--- 172.1.0.101 ping statistics ---1 packets transmitted, 1 received, 0% packet loss, time 0msrtt min/avg/max/mdev = 0.402/0.402/0.402/0.000 msbash-4.2# ping 172.1.0.100PING 172.1.0.100 (172.1.0.100) 56(84) bytes of data.64 bytes from 172.1.0.100: icmp_seq=1 ttl=64 time=0.024 ms^C--- 172.1.0.100 ping statistics ---[root@db-172-16-3-221 ~]# ovs-vsctl del-port ovs4 bbr1[root@db-172-16-3-221 ~]# ovs-vsctl add-port ovs4 bbr1 tag=101bash-4.2# ping 172.1.0.100PING 172.1.0.100 (172.1.0.100) 56(84) bytes of data.64 bytes from 172.1.0.100: icmp_seq=1 ttl=64 time=0.026 ms^C--- 172.1.0.100 ping statistics ---1 packets transmitted, 1 received, 0% packet loss, time 0msrtt min/avg/max/mdev = 0.026/0.026/0.026/0.000 msbash-4.2# ping 172.1.0.101PING 172.1.0.101 (172.1.0.101) 56(84) bytes of data.^C--- 172.1.0.101 ping statistics ---2 packets transmitted, 0 received, 100% packet loss, time 999ms

如果是多台主机的OVS需要通讯(并且多台主机在不同的广播域, 或者在同一个广播域并且与之相连的交换机接口是ACCESS口)的话, 可以使用GRE来通讯. (

在同一个广播域并且与之相连的交换机接口是TRUNK口不需要使用GRE, 直接就通了

)

同时需要确保 :

1. 主机间可以相互通讯

2. OVS交换机和相互通讯的IP所在的接口需要在一个广播域内.

例如OVS internal interface配置的IP互通.

例如ovs交换机在网桥内.

例如下图, 黑色为eth0, eth0加入OVS交换机, 同时IP配置在OVS交换机的internal interface.

Bridge "ovs4"

Port "ovs4"

Interface "ovs4"

type: internal

但是这里的gre链路最好建成星型, 防止断裂一条出现问题.

OVS会处理环路问题, 不需要STP来解决. (如果是bridge的话, 需要打开STP来解决环路问题)

例子 :

主机Aovs-vsctl add-br ovs1ovs-vsctl add-port ovs1 eth0ip link set ovs1 upip addr add 172.17.0.100/24 dev ovs1主机Bovs-vsctl add-br ovs1ovs-vsctl add-port ovs1 eth0ip link set ovs1 upip addr add 172.17.0.101/24 dev ovs1使用GRE联通主机A和主机B的ovs1在主机A执行ovs-vsctl add-port ovs1 gre0 -- set interface gre0 type=gre options:remote_ip=172.17.0.101

如果主机间相通的地址不在OVS交换机内, 那么可以使用桥来使接口和OVS相通.

例如 :

网关架构 :

[参考]