Four megabits on a three-hundred line

We set out to put a hosted Home Assistant on the same LAN as the house. We found a tunnel measuring itself, an MTU with no legal value, a firewall judging traffic it was never written for, and three faults that could only fire at boot.

Updated 28 August 2026 networkinghomelinkengineering

GamlaBio was the first customer on HomeLink L2: extend the house LAN itself to the hosted VM, so Matter, mDNS and Thread see one broadcast domain. The starting symptom was 4.3 Mbps on a 300/300 line.

We were only looking at this tunnel because a bulb would not pair over it — part one is how we got here.

Almost none of what follows was guessable from the code. Every fault here will recur on the next L2 customer. The how-to is Set up HomeLink L2; the symptom table is HomeLink L2.

One thread runs through all of it, and it is worth saying before the detail rather than after. Not one of these faults looked like what it was. A fragmenting tunnel looked like a throttling ISP. A firewall that had failed to load looked like a dead tunnel. A twenty-byte MTU error looked like a missing network device. Every wrong turn below came from believing the symptom, and every right answer came from measuring something the symptom was not talking about.

What we wanted

A routed tunnel is enough for anything that already has an address. Broadcasts are designed never to leave the LAN they were sent on, so discovery and Matter cannot work across one. The fix is to stop the connection behaving like a road and make it behave like a cable.

That means a house box that bridges the LAN into a VXLAN inside WireGuard, and a second NIC on the guest sitting on that overlay.

Two wrong answers before the right one

Wrong answer one — "the ISP is policing UDP". nc -u measured a hard ceiling around 4 Mbps while TCP on the identical path managed 302 Mbps. That looks exactly like UDP policing, and we were ready to raise it with Telia. nc -u is a single-threaded sender: it was measuring itself. A paced sender tracked the offered rate to 64 Mbps with zero loss.

Wrong answer two — "the path drops large datagrams". Both tests had used 1200-byte datagrams; WireGuard's own run near 1500. Repeating at 1472 bytes collapsed. That looked conclusive. It was an artefact: the measurement ran through the measuring box's own L2 bridge, whose MTU had just been lowered by hand. It was measuring the change, not the path.

Lesson. Vary rate and datagram size, pace the sender, and check that the box you are measuring from is not itself the constraint.

What we found: an MTU with no legal value

On a box whose default route is the L2 bridge, three facts cannot all hold:

overlay_mtu = tunnel_mtu - 50      (VXLAN encapsulation)
bridge_mtu  = overlay_mtu          (a bridge takes its smallest port)
tunnel_mtu + 60 <= bridge_mtu      (WireGuard's packets leave via the bridge)

Substituting gives tunnel_mtu + 60 <= tunnel_mtu - 50. There is no solution. Lowering the tunnel lowers the bridge with it, so each attempt ratchets down and none ever fits. At the default MTU of 1420 every WireGuard packet was being fragmented on egress, which is where the 4.3 Mbps came from.

A bridge's MTU is a property of bridged frames. The host's own routed traffic to the tunnel endpoint physically leaves by the LAN NIC at 1500, so a route-specific MTU states that fact where the kernel can act on it. Same line, nothing clamped: 4.3 Mbps → 296 Mbps.

The overlay ceiling is 1390 (1500 − 60 − 50). A guest left at 1500 would silently black-hole large frames, because a bridge does not fragment and sends no ICMP. Ours sits at 1370.

Pinning the tunnel MTU only governs what the endpoint itself sends. A LAN device still negotiates MSS from 1500 and hands the router segments too big for the tunnel. Correcting that relies on an ICMP that consumer routers frequently drop — ping is perfect, large transfers crawl, and to a customer that is indistinguishable from slow hosting. Every router path now clamps MSS to the path: on Linux with a rule on SYN, and on Windows with a per-adapter setting that Windows is widely, and wrongly, said not to have.

What we found: a firewall judging traffic it was never written for

Bridged frames are passed through iptables and ip6tables because Docker and libvirt want that. So a tenant's L2 traffic — frames that have no business being anywhere near an IP firewall — is judged by one anyway.

Three separate faults came from this.

IPv6 was dropped while IPv4 worked. Roughly 130 blocks an hour of router advertisements, neighbour advertisements and MLD. The bridge-accept rule existed only in IPv4. The comment above it described this exact failure, written when someone hit it for IPv4, without noticing it applies twice. Why it survived: IPv4 mDNS worked perfectly, so the tunnel tested healthy and only IPv6-native things — Thread, Matter — saw nothing. "Works, then drops off" is the signature. A healthy IPv4 test proves nothing about IPv6.

A ufw reload took every tenant offline. Each VM's forward accept had been inserted at runtime into a chain that is rebuilt from file on every reload. The VMs stayed healthy and answered locally. Browsers served a cached shell over a connection going nowhere.

Writing the file at the wrong COMMIT put a filter rule into the nat table. ufw then failed to load its ruleset and left INPUT and FORWARD at policy DROP. Echo replies were arriving on the wire and being dropped before the pinging process saw them, so the tunnel looked dead while its handshake was 80 seconds old and 35 GiB had crossed it.

A fourth fault in the same area had a different cause, and it is worth separating rather than filing under the same heading.

Management depended on the guest's default route. DNAT preserves the caller's address, so a forwarded connection is answered to a public IP — which works only for as long as the guest's default route does. A customer changing their network config kills the management path, and nothing on the host looks wrong. Measured: from a directly-connected source, 200 in 7 ms; from a non-local source, timeout. Forwarded traffic is now masqueraded to the bridge address, so the guest replies to something on its own subnet and never consults a route.

Multicast snooping on a bridge whose members are a LAN and a tunnel silently black-holes IPv6 multicast toward the tunnel port. IPv4 is unaffected because link-local multicast is flooded whatever snooping says. Enabling a querier is the tidier fix and did not work. Flooding both ends did, immediately.

What we changed our minds about

The second NIC looks local. It is not. The guest's house tap is a port on the datacentre host's own bridge, so it looks as if the host could reach 192.168.1.66 locally. It does not: the route for the customer subnet points at WireGuard, so the traffic goes out to the house and comes back.

The tunnel went down twice during this work. On both occasions the management NIC was the only way in.

Moving management onto a house address was considered and rejected. Making it local would need the host to hold an address on the customer's subnet, and 192.168.1.0/24 is the most common home range there is. Three customers already use it.

Keep both NICs. Make the house one primary. Strip the gateway from management, so Home Assistant stops binding discovery to a segment that contains only our infrastructure. The gateway has to move, not just disappear — otherwise the guest has no internet, and Matter commissioning fails even with a border router the UI shows as healthy.

A chain of genuine network faults, each fix correct, none of them clearing the symptom. That shape is what invites one more speculative network change, and we nearly made it. Our working theory at this point was that the border router had stopped advertising itself entirely: a 60-second capture on the LAN showed a single mDNS packet and no _meshcop._udp at all.

That theory was wrong, and it is worth recording how it was wrong. The border-router side re-ran its own live check at the same time and got an immediate discovery event back from Home Assistant, carrying the router's real LAN address. Services do not rebroadcast continuously once cached, so a short capture window can land between two announcements and look exactly like silence. We had a mistimed test, not a regression.

What actually cleared it was one further firewall fault on this side, on top of the snooping fix. Afterwards the border router was consistently visible and — for the first time in the whole exercise — Home Assistant's integration entry for it persisted on the first attempt. The useful part of the wrong theory survived: visible in the Thread panel is not the same as Home Assistant holding the dataset. That distinction was correct and it is what prompted the re-check.

The reboot: three faults that could only fire at boot

On 26 August 2026 the house box was rebooted for the first time since the L2 work began. It came back with a working LAN and no link home. Three separate faults fired at once. Every one of them had been sitting in place for days, and none of them could fire while the interfaces stayed up.

They had to be fixed in order, because each one hid the next.

A tunnel MTU of 1260 is below IPv6's hard minimum of 1280. The kernel does not warn or refuse — it removes IPv6 from the interface entirely. wg-quick then reports "No such device" for a device that is plainly there, and deletes the interface: a 20-byte MTU error presented as the entire VPN refusing to start, IPv4 included. The value had been in the config for ten days while the tunnel ran perfectly, because nothing had recreated the interface.

Underneath that, AppArmor was refusing wg-quick permission to run awk — which our own hook used to find the LAN interface. ip was permitted and every iptables rule in the same hook ran fine; it was specifically the second binary that was refused. wg-quick unwinds on any hook failure, so it deleted the interface it had just finished building. A blocked text utility presented as a dead VPN, reported as a file permission with no mention of anything network-related. The workaround in the field was to disable AppArmor — a security control on a machine bridged into a customer's home network.

Underneath that, the boot unit refused to start because the LAN was carrying the default route, which is the normal state at boot. The guard was written for an interactive session, where moving that route cuts the connection you are typing into. At boot there is no session to cut. The unit had been "verified by reading it", and reading a unit file confirms what it says, not what systemd does with it.

The shape they share is the finding, and it is more useful than any of the three bugs. Every layer of a machine like this defers its effect to the next boot: sysctl files, unit files, netplan, wg-quick configs, AppArmor profiles. A change that has not survived a reboot has not been tested, however long it has appeared to work. The gap between cause and symptom here was ten days, and the only reason this was one morning rather than a fortnight is that all three surfaced together.

What it looked like, and what it was

Every row here cost hours, and several cost days. Read down the left column and it is a list of things that were never wrong.

The one at the bottom is the one to keep. It had been "verified" — by reading it.

Where it landed

The MTU pin, the MSS clamp, the IPv6 bridge accepts, the masquerade that decouples management from the guest's default route, a boot unit, and a watchdog that re-enslaves the LAN if netplan takes it back — verified end-to-end: NIC detached at 07:49, gateway down, timer caught it at 07:50, gateway back at 07:51. About 80 seconds, unattended.

Still unfinished, and written down rather than smoothed over: the house role of L2 has no provisioning path, the first migrated VM was moved by hand, AppArmor was still disabled on that house box when this was written, and a reboot belongs in the setup as a tested step. We had called one of those units verified. It could never have started.

Next: three of the faults above were the same fault — tenants kept apart by rules that a reload, a typo or a customer's route change could undo. That is not a bug list, it is a design. Isolation that fails closed.