2022-08-02
Contents
I monitor my home network connectivity by sending pings at well known sites like google.com and checking to see how many time out. For this purpose, I set up my own Debian box so I can have 24/7 monitoring. I plot the number of timeouts in Grafana. The following is an example of pings to facebook.com.
However, this Debian box is connected via Ethernet to my router and the Ethernet is the default network interface for the Debian machine. Because of that, my ping monitoring tool can only really monitor the status of the Ethernet connectivity. Monitoring wifi connectivity is also a goal of mine.
This sounds like it should be easy. Just have the program send the ping via
the wifi network interface instead of the ethernet based one. In fact the
default
ping program can do this with flags in both Windows
(-S) and Linux (-I). However, I'm using a Golang
library go-ping, which doesn't
support that.
However, even if go-ping supported changing interface, I'd be forced to run the program directly. I've been using continers with Docker and Kubernetes (k3s) so if possible I would prefer that. Furthermore, in k3s, changing to a different network interface is quite challenging because you need to switch from the default Flannel network overlay.
Luckily, Docker has more easily accessible networking options. While scouring the web, I stumbled upon this blog post, which begins with this line:
A question that crops up regularly on #docker is “How do I attach a container directly to my local network?”
That's exactly what I'm trying to do! So, I followed the instructions in the blog post. But, I encounterd a problem. I couldn't connect from inside the Docker container to the outside world using macvlan. It turns out macvlan does not work on wifi network interfaces. The reason appears to be the IEEE 802.11 standards do not allow this.
IEEE 802.11 doesn’t like multiple MAC addresses on a single client. It is likely macvlan sub-interfaces will be blocked by your wireless interface driver, AP or both.
Macvlans are not built to work on wireless interfaces. The reason is that all APs will reject frames originating from a MAC address which did not authenticate with them, while the whole point of macvlans is exactly to provide new subinterfaces with their own MAC address, different from that of the physical interface.
...
Incidentally, having an underlying wireless interface is just about the only reasonable use case for ipvlans instead of macvlans. In all other cases, just stick to macvlans.
Viewing the standards themselves requires a subscription, but I found some other sources on the internet also claiming that macvlan won't work with wifi network interfaces. With this newfound knowledge, I changed my commands to use ipvlan instead and everything mostly just worked!
My router/DHCP server assigns IPs from the range of 192.168.1.25-192.168.1.200 with a gateway at 192.168.1.1. In order to avoid overlap, I picked the range 192.168.1.208/28, which corresponds to the 16 ip addresses from 192.168.1.208-192.168.1.223. I picked 192.168.1.222 to reserve for use by my host interface. Also note that apparently, network interfaces can only have 15 characters in their name. Thus, my script was
docker network create -d ipvlan -o parent=my_wifi_adapter \
--subnet 192.168.1.0/24 \
--gateway 192.168.1.1 \
--ip-range 192.168.1.208/28 \
--aux-address 'host=192.168.1.222' \
wifi_ipvlan
ip link add wifi-shim link my_wifi_adapter type ipvlan mode l2
ip addr add 192.168.1.222/32 dev wifi-shim
ip link set wifi-shim up
ip route add 192.168.1.208/28 dev wifi-shim
The other thing that wasn't mentioned in the Lars Kellogg-Stedman blog post
was how to make the settings persistent (in Debian) through restarts as each
Linux distribution is different. I initially thought I could just run my
script through crontab (i.e. sudo crontab -e and adding a
line with @reboot path_to_my_script.sh). This turned out not to
be viable because the crontab would run before the interface was
discoverable. I tried adding some minor amounts of sleep, but it required a
lot more sleep than I wanted (>20s).
I ended up modifying my /etc/network/interfaces file to ensure persistency. I found some examples on various places online. I ended up with the following config:
#ipvlan for docker connecting only to wifi
#first line starts up automatically
auto wifi-shim
iface wifi-shim inet manual
pre-up /sbin/ip link add wifi-shim link my_wifi_adapter type ipvlan mode l2
up /sbin/ip addr add 192.168.1.222/32 dev wifi-shim
post-up /sbin/ip route add 192.168.1.208/28 dev wifi-shim
post-down /sbin/ip link del wifi-shim
I also set up docker-compose so I could ensure my ping utility would run on restarts. In order to make sure it would use the wifi_ipvlan network I set up earlier as well as a static ip address (192.168.1.208), I used settings like this:
services:
ping:
...
networks:
wifi_ipvlan:
ipv4_address: 192.168.1.208
networks:
wifi_ipvlan:
external: trueNote that apparently with higher versions of docker-compose, this may not work, but I'm using an old version.
With all this set up, I was finally able get some data for pinging through my wifi interface. The lighter color indicates the timeouts through the wifi interface. The number of timeouts seem similar, which indicates my wifi itself isn't having too many issues.
This seemed to work pretty well. I was able to connect to the internet from my Docker container through my wifi interface. I was also able to access my Docker container from other things on my local computer. Those were all things I had expected. However, I was able to access my Docker container even on a different computer on the network. I didn't expect that at all!
The part that confused me the most was how could a different computer on the same network know to send packets addressed to 192.168.1.208 to my Debian machine. Before diving in this, I hadn't really dug into the details of how things like this work. I had to figure out how a computer decides where to send packets in local networks in general. For the purposes of this blog, let's assume my computer is at 192.168.1.50 and we're trying to send a packet to 192.168.1.208.
First, my computer uses the subnet mask of my network (255.255.255.0), to determine if the target is in the local network or not. A subnet mask consists of 4 8-bit numbers and all the bits that are 0 are considered part of the same network. Since my network's subnet mask has the final 8-bit number as 0, that means all addresses that only differ in the final 8-bit number as considered part of the same network. That would mean the range (192.168.1.0-192.168.1.255). That is how my computer knows 192.168.1.208 is in the same network.
Given an IP address, my computer then has to find the MAC address associated with that IP address. When the result isn't cached, it does this using ARP. This works by having my computer broadcast to every single other device on the network with the destination of 192.168.1.255 asking for the MAC address of 192.168.1.208. Every other device will hear this broadcast, but only the device that claims to be 192.168.1.208 will respond with its MAC address.
Having set up ipvlan, when my Debian machine receives the ARP ping, it then knows to respond with the MAC address of its wifi interface. My computer then receives this and then it can send an Ethernet frame with the appropriate source and destination MAC addresses. My router upon receiving the Ethernet frame would then see the destination MAC address and uses an internal table that matches physical port to MAC address. The router is thus able to forward it to my Debian machine. That's how my main machine was also able to access the Docker container on a remote Debian machine.
Any error corrections or comments can be made by sending me a pull request.