Demystify how traffic reaches directly to pod on using alb.ingress.kubernetes.io/target-type: ip

  1. Assuming, you have an ec2 running (amazon Linux) with secondary eni with additional Ip. If you do not have an instance, you can refer.
  2. ssh login of the instance.
  3. public ec2 machine.
  4. Required whitelisting in the security group to check internet connectivity from the instance.
  5. A cup of coffee and smile :-)
# yum install nc telnet nsenter iptables-services -y
###
Create virtual interface named veth0 and veth1
###
# ip link add veth0 type veth peer name veth1###
Create namespace named vnet0 and set the veth0 with namespace. Note: after adding link set, you won't see veth0 on host machine.
###
# ip netns add vnet
# ip link set veth0 netns vnet0
###
Verify the veth1 and eth1. Make sure secondary IP should not be there on eth1. If it is there, you need to remove the IP using "ip addr del 172.31.11.204/20 dev eth1"
###
# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc pfifo_fast state UP group default qlen 1000
link/ether 0a:7a:6a:ee:bc:34 brd ff:ff:ff:ff:ff:ff
inet 172.31.3.201/20 brd 172.31.15.255 scope global dynamic eth0
valid_lft 3489sec preferred_lft 3489sec
inet6 fe80::87a:6aff:feee:bc34/64 scope link
valid_lft forever preferred_lft forever
3: veth1@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 12:fa:5d:42:9c:9c brd ff:ff:ff:ff:ff:ff link-netns vnet0
inet6 fe80::10fa:5dff:fe42:9c9c/64 scope link
valid_lft forever preferred_lft forever
5: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc pfifo_fast state UP group default qlen 1000
link/ether 0a:5d:69:b0:f9:6e brd ff:ff:ff:ff:ff:ff
inet 172.31.7.223/20 brd 172.31.15.255 scope global dynamic eth1
valid_lft 3002sec preferred_lft 3002sec
inet6 fe80::85d:69ff:feb0:f96e/64 scope link
valid_lft forever preferred_lft forever
### 
Assign IP to the interface. You can replace the IP with your secondary IP. I am using the Secondary private IPv4 of secondary interface.
###
# ip -n vnet1 addr add 172.31.11.204/32 dev veth0###
Bring up the vnet1 interface
###
# ip -n vnet1 link set eth1 up
# ip -n vnet1 link set lo up
###
confirm that the interfaces are up
###
# ip netns exec vnet0 ip addr show1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
4: veth0@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 86:b0:f8:b2:36:0c brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.31.11.204/32 scope global veth0
valid_lft forever preferred_lft forever
inet6 fe80::84b0:f8ff:feb2:360c/64 scope link
valid_lft forever preferred_lft forever
### 
Gateway IP '169.254.1.1' will be mapped with 'veth1@if4' mac address '12:fa:5d:42:9c:9c' in the ARP table as static entry
###
# ip netns exec vnet0 ip route add 169.254.1.1 dev veth0 scope link
# ip netns exec vnet0 ip route add default via 169.254.1.1 dev veth0
# ip netns exec vnet0 arp -i veth0 -s 169.254.1.1 12:fa:5d:42:9c:9c
###
verify arp entry
###
# ip netns exec vnet0 arp -a
? (169.254.1.1) at 12:fa:5d:42:9c:9c [ether] PERM on eth0
###
verify route table
###
# ip netns exec vnet0 ip route show
default via 169.254.1.1 dev eth0
169.254.1.1 dev eth0
###
Add rules for the routing table
###
# ip rule add from all to 172.31.11.204 lookup main prio 512
# ip rule add not from all to 172.31.0.0/16 lookup main prio 1025
# ip rule add from 172.31.11.204 lookup 10001 prio 1536
###
Check all the rules are in place, this is the important part where we are specifying rules to route traffic.
512: from all to 172.31.11.204 lookup main. used for traffic to the pod
1536: from 172.31.11.204 lookup 10001 used for traffic from the pod.
1025: not from all to 172.31.0.0/16 lookup main, used for internet traffic
###
# ip rule
0: from all lookup local
512: from all to 172.31.11.204 lookup main
1025: not from all to 172.31.0.0/16 lookup main
1536: from 172.31.11.204 lookup 10001
32766: from all lookup main
32767: from all lookup default
###
Add the route of veth0 IP in the main route table
###
# ip route add 172.31.11.204 dev veth1###
added the route for veth1 and IP inside vnet0. This vnet behaves like a pipe for communication veth1 -> veth0
###
# ip route show
default via 172.31.0.1 dev eth0
169.254.169.254 dev eth0
172.31.0.0/20 dev eth0 proto kernel scope link src 172.31.3.201
172.31.11.204 dev veth1 scope link
# ip route show table 10001
default via 172.31.0.1 dev eth1
172.31.0.0/20 dev eth1 proto kernel scope link src 172.31.7.223
###
Enable IP forwarding on the host machine
###
# sysctl net.ipv4.ip_forward=1
###
To enter into the namespace, type exit inside the terminal once you are inside vnet0 if you want to do anything on host machine. Else, you may always open up new terminal.
###
# nsenter --net=/var/run/netns/vnet0###
try some ping to self and IP of eth0 interface.
###
# ping 172.31.11.204
PING 172.31.11.204 (172.31.11.204) 56(84) bytes of data.
64 bytes from 172.31.11.204: icmp_seq=1 ttl=64 time=0.044 ms
64 bytes from 172.31.11.204: icmp_seq=2 ttl=64 time=0.045 ms
64 bytes from 172.31.11.204: icmp_seq=3 ttl=64 time=0.043 ms
# ping 172.31.3.201
PING 172.31.3.201 (172.31.3.201) 56(84) bytes of data.
64 bytes from 172.31.3.201: icmp_seq=1 ttl=64 time=0.044 ms
64 bytes from 172.31.3.201: icmp_seq=2 ttl=64 time=0.045 ms
64 bytes from 172.31.3.201: icmp_seq=3 ttl=64 time=0.043 ms
###
Start nc with 172.31.11.204 and port 80
###
# nc -l 172.31.11.204 80 -v###
Open a new terminal and try telneting 172.31.11.204 on port 80
###
# telnet 172.31.11.204 80
Trying 172.31.11.204...
Connected to 172.31.11.204.
Escape character is '^]'.
# nsenter --net=/var/run/netns/vnet0
# ping google.com
PING google.com (172.217.160.206) 56(84) bytes of data.
^C
--- google.com ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1015ms
:-(
###
Rules
delete reject from input.. you need to check the rule number before deleting. For me, it was 5.
###
iptables -D INPUT 5###
Create the following chains. This is what AWS does
###
iptables -N AWS-SNAT-CHAIN-1 -t nat
iptables -N AWS-SNAT-CHAIN-0 -t nat
###
Create the SNAT rule, here please note that the IP '172.31.3.201' is the primary IP of my primary interface. When I was writing this blog, I found it mandatory that wecan't put any other interface or secondary IP here as SNAT.
###
# iptables -t nat -A AWS-SNAT-CHAIN-1 -m addrtype ! --dst-type LOCAL -j SNAT --to-source 172.31.3.201 --random-fully# iptables -t nat -A AWS-SNAT-CHAIN-0 ! -d 172.31.0.0/16 -j AWS-SNAT-CHAIN-1# iptables -t nat -A POSTROUTING -j AWS-SNAT-CHAIN-0
###
List Iptable rules
###
# iptables -L
iptables -L -t nat
# ping google.comPING google.com (172.217.160.206) 56(84) bytes of data.64 bytes from bom07s16-in-f14.1e100.net (172.217.160.206): icmp_seq=1 ttl=50 time=1.66 ms
64 bytes from bom07s16-in-f14.1e100.net (172.217.160.206): icmp_seq=2 ttl=50 time=1.79 ms
64 bytes from bom07s16-in-f14.1e100.net (172.217.160.206): icmp_seq=3 ttl=50 time=1.76 ms
64 bytes from bom07s16-in-f14.1e100.net (172.217.160.206): icmp_seq=4 ttl=50 time=1.74 ms
64 bytes from bom07s16-in-f14.1e100.net (172.217.160.206): icmp_seq=5 ttl=50 time=1.75 ms
64 bytes from bom07s16-in-f14.1e100.net (172.217.160.206): icmp_seq=6 ttl=50 time=1.74 ms
64 bytes from bom07s16-in-f14.1e100.net (172.217.160.206): icmp_seq=7 ttl=50 time=1.76 ms
^C--- google.com ping statistics ---7 packets transmitted, 7 received, 0% packet loss, time 6011ms
rtt min/avg/max/mdev = 1.668/1.747/1.798/0.048 ms
# curl google.com<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store