Rancher裡面運行的容器暴露的埠為什麼netstat看不到

首先拋出結論：Rancher沒有使用docker-proxy來暴露服務。

原因：如果使用docker的userland proxy，如果開10000個服務，意味著主機上要開銷出10000個埠來暴露服務，對內核來說，是一筆不小的開銷。Rancher的設計是利用iptables到netfilter暴露出來的hook來控制報文在host上的流向。

論證過程：在你的測試環境中選一台host進行如下操作：

  1. 清理IPTABLES mangle表所有chain下面的規則：iptables -t mangle -F

  2. 給iptables mangle表的5個鉤子添加日誌追蹤的行為，追蹤的報文協議是tcp，源/目標埠為3306，設置列印日誌的級別然後列印日誌

    Advertisements

iptables -t mangle -A PREROUTING -p tcp --dport 3306 -j LOG --log-prefix "M-PREROUTING:" --log-level 7

iptables -t mangle -A POSTROUTING -p tcp --dport 3306 -j LOG --log-prefix "M-POSTROUTING:" --log-level 7

iptables -t mangle -A FORWARD -p tcp --dport 3306 -j LOG --log-prefix "M-FORWARD:" --log-level 7

Advertisements

iptables -t mangle -A OUTPUT -p tcp --dport 3306 -j LOG --log-prefix "M-OUTPUT:" --log-level 7

iptables -t mangle -A INPUT -p tcp --dport 3306 -j LOG --log-prefix "M-INPUT:" --log-level 7

iptables -t mangle -A PREROUTING -p tcp --sport 3306 -j LOG --log-prefix "M-PREROUTING:" --log-level 7

iptables -t mangle -A POSTROUTING -p tcp --sport 3306 -j LOG --log-prefix "M-POSTROUTING:" --log-level 7

iptables -t mangle -A FORWARD -p tcp --sport 3306 -j LOG --log-prefix "M-FORWARD:" --log-level 7

iptables -t mangle -A OUTPUT -p tcp --sport 3306 -j LOG --log-prefix "M-OUTPUT:" --log-level 7

iptables -t mangle -A INPUT -p tcp --sport 3306 -j LOG --log-prefix "M-INPUT:" --log-level

3.在Rancher上部署mysql應用，暴露埠為3306

4.查看Rancher為服務添加的IPTABLES主要規則，為下面的報文流

  • NAT表

  • -A CATTLE_HOSTPORTS_POSTROUTING -s 10.42.158.152/32 -d 10.42.158.152/32 -p tcp -m tcp --dport 3306 -j MASQUERADE

  • -A CATTLE_OUTPUT -p tcp -m tcp --dport 3306 -m addrtype --dst-type LOCAL -j DNAT --to-destination 10.42.158.152:3306

  • -A CATTLE_PREROUTING ! -i docker0 -p tcp -m tcp --dport 3306 -j DNAT --to-destination 10.42.158.152:3306

  • -A CATTLE_PREROUTING -p tcp -m tcp --dport 3306 -m addrtype --dst-type LOCAL -j DNAT --to-destination 10.42.158.152:3306

  • FILTER表

  • -A CATTLE_FORWARD -m mark --mark 0x1068 -j ACCEPT

  • -A CATTLE_FORWARD -m mark --mark 0x4000 -j ACCEPT

  • -A CATTLE_FORWARD -d 10.42.0.0/16 -o docker0 -j ACCEPT

    5.跨主機訪問mysql服務，查看/var/log/kern.log日誌，分析報文流轉

    • Nov 17 07:07:39 cattleh2 kernel: [46619.196296] M-POSTROUTING:IN= OUT=docker0 SRC=172.168.1.204 DST=10.42.158.152 LEN=60 TOS=0x10 PREC=0x00 TTL=63 ID=63457 DF PROTO=TCP SPT=54022 DPT=3306 WINDOW=29200 RES=0x00 SYN URGP=0 MARK=0x1068

    • Nov 17 07:07:39 cattleh2 kernel: [46619.196283] M-FORWARD:IN=enp0s8 OUT=docker0 MAC=08:00:27:e7:fe:f9:08:00:27:b5:2f:92:08:00 SRC=172.168.1.204 DST=10.42.158.152 LEN=60 TOS=0x10 PREC=0x00 TTL=63 ID=63457 DF PROTO=TCP SPT=54022 DPT=3306 WINDOW=29200 RES=0x00 SYN URGP=0 MARK=0x1068

    • Nov 17 07:07:39 cattleh2 kernel: [46619.196230] M-PREROUTING:IN=enp0s8 OUT= MAC=08:00:27:e7:fe:f9:08:00:27:b5:2f:92:08:00 SRC=172.168.1.204DST=172.168.1.200 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=63457 DF PROTO=TCP SPT=54022 DPT=3306 WINDOW=29200 RES=0x00 SYN URGP=0 MARK=0x1068

    • SYN報文到達mangle表的PREROUTING chain，寫下了下面的日誌，同時在NAT表的PREROUTING chain做了DNAT。

    • SYN報文到達mangle表的FORWARD chain，寫下了下面的日誌，同時在FILTER表的FORWARD chain 被ACCEPT。

    • SYN報文到達mangle表的POSTROUTING chain，寫下了下面的日誌，然後過NAT表的POSTROUTING chain，沒有做SNAT操作，然後出協議棧，最終SYN報文到達容器，容器收到SYN報文之後，回復ACK+SYN報文，這部分流轉本文沒有深究列出。

    • Nov 17 07:07:39 cattleh2 kernel: [46619.196717] M-PREROUTING:IN=docker0 OUT= PHYSIN=vethr1368e17ce3 MAC=02:42:2f:b4:a7:5d:02:24:37:a3:1c:c0:08:00 SRC=10.42.158.152 DST=172.168.1.204 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=3306 DPT=54022 WINDOW=27000 RES=0x00 ACK SYN URGP=0

    • ACK+SYN報文到達mangle表的PREROUTING chain，寫下下面的日誌，然後過NAT表的PREROUTING chain，沒有做DNAT操作。

    • ACK+SYN報文到達mangle表的FORWARD chain，寫下了下面的日誌，同時在FILTER表的FORWARD chain 被ACCEPT。

    • Nov 17 07:07:39 cattleh2 kernel: [46619.196729] M-FORWARD:IN=docker0 OUT=enp0s8 PHYSIN=vethr1368e17ce3 MAC=02:42:2f:b4:a7:5d:02:24:37:a3:1c:c0:08:00 SRC=10.42.158.152 DST=172.168.1.204 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=3306 DPT=54022 WINDOW=27000 RES=0x00 ACK SYN URGP=0

    • ACK+SYN報文到達mangle表的POSTROUTING chain，寫下了下面的日誌，然後過NAT表的POSTROUTING chain，做SNAT的操作，出主機協議棧，ACK+SYN報文到達發起訪問的主機網卡設備。

    • Nov 17 07:07:39 cattleh2 kernel: [46619.196984] M-PREROUTING:IN=enp0s8 OUT= MAC=08:00:27:e7:fe:f9:08:00:27:b5:2f:92:08:00 SRC=172.168.1.204 DST=172.168.1.200 LEN=52 TOS=0x10 PREC=0x00 TTL=64 ID=63458 DF PROTO=TCP SPT=54022 DPT=3306 WINDOW=229 RES=0x00 ACK URGP=0 MARK=0x1068

    • Nov 17 07:07:39 cattleh2 kernel: [46619.196996] M-FORWARD:IN=enp0s8 OUT=docker0 MAC=08:00:27:e7:fe:f9:08:00:27:b5:2f:92:08:00 SRC=172.168.1.204 DST=10.42.158.152 LEN=52 TOS=0x10 PREC=0x00 TTL=63 ID=63458 DF PROTO=TCP SPT=54022 DPT=3306 WINDOW=229 RES=0x00 ACK URGP=0 MARK=0x1068

    • Nov 17 07:07:39 cattleh2 kernel: [46619.197002] M-POSTROUTING:IN= OUT=docker0 SRC=172.168.1.204 DST=10.42.158.152 LEN=52 TOS=0x10 PREC=0x00 TTL=63 ID=63458 DF PROTO=TCP SPT=54022 DPT=3306 WINDOW=229 RES=0x00 ACK URGP=0 MARK=0x1068

    • Nov 17 07:07:39 cattleh2 kernel: [46619.196735] M-POSTROUTING:IN= OUT=enp0s8 PHYSIN=vethr1368e17ce3 SRC=10.42.158.152 DST=172.168.1.204LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=3306 DPT=54022 WINDOW=27000 RES=0x00 ACK SYN URGP=0

    • ACK報文到達主機，處理方式跟第一步一樣。

    6.本機(127.0.0.1)訪問主機3306埠，和跨主機訪問流轉一致，下面是流轉mangle表打出來的日誌，可以參考上面的分析邏輯幫助理解。

    • Nov 17 07:52:36 cattleh2 kernel: [49315.906692] M-OUTPUT:IN= OUT=lo SRC=127.0.0.1 DST=127.0.0.1 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=19503 DF PROTO=TCP SPT=49508 DPT=3306 WINDOW=43690 RES=0x00 SYN URGP=0

    • Nov 17 07:52:36 cattleh2 kernel: [49315.906708] M-POSTROUTING:IN= OUT=docker0 SRC=127.0.0.1 DST=10.42.158.152 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=19503 DF PROTO=TCP SPT=49508 DPT=3306 WINDOW=43690 RES=0x00 SYN URGP=0

    • Nov 17 07:52:36 cattleh2 kernel: [49315.906776] M-PREROUTING:IN=docker0 OUT= PHYSIN=vethr1368e17ce3 MAC=02:42:2f:b4:a7:5d:02:24:37:a3:1c:c0:08:00 SRC=10.42.158.152 DST=10.42.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=3306 DPT=49508 WINDOW=27000 RES=0x00 ACK SYN URGP=0

    • Nov 17 07:52:36 cattleh2 kernel: [49315.906791] M-INPUT:IN=docker0 OUT= PHYSIN=vethr1368e17ce3 MAC=02:42:2f:b4:a7:5d:02:24:37:a3:1c:c0:08:00 SRC=10.42.158.152 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=3306 DPT=49508 WINDOW=27000 RES=0x00 ACK SYN URGP=0

    • Nov 17 07:52:36 cattleh2 kernel: [49315.906806] M-OUTPUT:IN= OUT=lo SRC=127.0.0.1 DST=127.0.0.1 LEN=52 TOS=0x10 PREC=0x00 TTL=64 ID=19504 DF PROTO=TCP SPT=49508 DPT=3306 WINDOW=342 RES=0x00 ACK URGP=0

    • Nov 17 07:52:36 cattleh2 kernel: [49315.906812] M-POSTROUTING:IN= OUT=docker0 SRC=127.0.0.1 DST=10.42.158.152 LEN=52 TOS=0x10 PREC=0x00 TTL=64 ID=19504 DF PROTO=TCP SPT=49508 DPT=3306 WINDOW=342 RES=0x00 ACK URGP=0

    • Nov 17 07:52:36 cattleh2 kernel: [49315.907322] M-PREROUTING:IN=docker0 OUT= PHYSIN=vethr1368e17ce3 MAC=02:42:2f:b4:a7:5d:02:24:37:a3:1c:c0:08:00 SRC=10.42.158.152 DST=10.42.0.1 LEN=147 TOS=0x08 PREC=0x00 TTL=64 ID=34614 DF PROTO=TCP SPT=3306 DPT=49508 WINDOW=211 RES=0x00 ACK PSH URGP=0

    • Nov 17 07:52:36 cattleh2 kernel: [49315.907331] M-INPUT:IN=docker0 OUT= PHYSIN=vethr1368e17ce3 MAC=02:42:2f:b4:a7:5d:02:24:37:a3:1c:c0:08:00 SRC=10.42.158.152 DST=127.0.0.1 LEN=147 TOS=0x08 PREC=0x00 TTL=64 ID=34614 DF PROTO=TCP SPT=3306 DPT=49508 WINDOW=211 RES=0x00 ACK PSH URGP=0

    • Nov 17 07:52:36 cattleh2 kernel: [49315.907362] M-OUTPUT:IN= OUT=lo SRC=127.0.0.1 DST=127.0.0.1 LEN=52 TOS=0x10 PREC=0x00 TTL=64 ID=19505 DF PROTO=TCP SPT=49508 DPT=3306 WINDOW=342 RES=0x00 ACK URGP=0

    • Nov 17 07:52:36 cattleh2 kernel: [49315.907367] M-POSTROUTING:IN= OUT=docker0 SRC=127.0.0.1 DST=10.42.158.152 LEN=52 TOS=0x10 PREC=0x00 TTL=64 ID=19505 DF PROTO=TCP SPT=49508 DPT=3306 WINDOW=342 RES=0x00 ACK URGP=0

    • Nov 17 07:52:37 cattleh2 kernel: [49317.290761] M-OUTPUT:IN= OUT=lo SRC=127.0.0.1 DST=127.0.0.1 LEN=54 TOS=0x10 PREC=0x00 TTL=64 ID=19506 DF PROTO=TCP SPT=49508 DPT=3306 WINDOW=342 RES=0x00 ACK PSH URGP=0

    • Nov 17 07:52:37 cattleh2 kernel: [49317.290777] M-POSTROUTING:IN= OUT=docker0 SRC=127.0.0.1 DST=10.42.158.152 LEN=54 TOS=0x10 PREC=0x00 TTL=64 ID=19506 DF PROTO=TCP SPT=49508 DPT=3306 WINDOW=342 RES=0x00 ACK PSH URGP=0

    • Nov 17 07:52:37 cattleh2 kernel: [49317.290939] M-PREROUTING:IN=docker0 OUT= PHYSIN=vethr1368e17ce3 MAC=02:42:2f:b4:a7:5d:02:24:37:a3:1c:c0:08:00 SRC=10.42.158.152 DST=10.42.0.1 LEN=52 TOS=0x08 PREC=0x00 TTL=64 ID=34615 DF PROTO=TCP SPT=3306 DPT=49508 WINDOW=211 RES=0x00 ACK URGP=0

    • Nov 17 07:52:37 cattleh2 kernel: [49317.290948] M-INPUT:IN=docker0 OUT= PHYSIN=vethr1368e17ce3 MAC=02:42:2f:b4:a7:5d:02:24:37:a3:1c:c0:08:00 SRC=10.42.158.152 DST=127.0.0.1 LEN=52 TOS=0x08 PREC=0x00 TTL=64 ID=34615 DF PROTO=TCP SPT=3306 DPT=49508 WINDOW=211 RES=0x00 ACK URGP=0

    • Nov 17 07:52:38 cattleh2 kernel: [49317.473238] M-OUTPUT:IN= OUT=lo SRC=127.0.0.1 DST=127.0.0.1 LEN=54 TOS=0x10 PREC=0x00 TTL=64 ID=19507 DF PROTO=TCP SPT=49508 DPT=3306 WINDOW=342 RES=0x00 ACK PSH URGP=0

    • Nov 17 07:52:38 cattleh2 kernel: [49317.473255] M-POSTROUTING:IN= OUT=docker0 SRC=127.0.0.1 DST=10.42.158.152 LEN=54 TOS=0x10 PREC=0x00 TTL=64 ID=19507 DF PROTO=TCP SPT=49508 DPT=3306 WINDOW=342 RES=0x00 ACK PSH URGP=0

    • Nov 17 07:52:38 cattleh2 kernel: [49317.473378] M-PREROUTING:IN=docker0 OUT= PHYSIN=vethr1368e17ce3 MAC=02:42:2f:b4:a7:5d:02:24:37:a3:1c:c0:08:00 SRC=10.42.158.152 DST=10.42.0.1 LEN=52 TOS=0x08 PREC=0x00 TTL=64 ID=34616 DF PROTO=TCP SPT=3306 DPT=49508 WINDOW=211 RES=0x00 ACK URGP=0

    • Nov 17 07:52:38 cattleh2 kernel: [49317.473386] M-INPUT:IN=docker0 OUT= PHYSIN=vethr1368e17ce3 MAC=02:42:2f:b4:a7:5d:02:24:37:a3:1c:c0:08:00 SRC=10.42.158.152 DST=127.0.0.1 LEN=52 TOS=0x08 PREC=0x00 TTL=64 ID=34616 DF PROTO=TCP SPT=3306 DPT=49508 WINDOW=211 RES=0x00 ACK URGP=0

    Advertisements

    你可能會喜歡

    Advertisements