Isolation that fails closed

A shared-bridge firewall was dropping real packets and was still the wrong design. We measured per-VM bridges, policy routing and VRFs on this host, and namespaces won — not because they are cheaper. Every tenant is now on its own stack, and moving the last two found four faults that doing it by hand had missed.

Updated 28 August 2026 networkingsecurityisolationengineering

The criterion was agreed before anything moved: strongest isolation on this host, not the cheapest that would do. A dedicated machine per tenant is stronger still. That is a product tier, not this change. This is the strongest design that still shares a hypervisor.

We did not set out to redesign this. It started with a bulb that would not pair (part one), which sent us into the tunnel to that house (part two) — and three of the faults we found there were the same fault wearing different clothes. Tenants were being kept apart by runtime rules that a reload, a misplaced COMMIT, or a customer changing their own default route could undo. Fixing each one leaves the design that produced them.

The customer-facing version is How your server is kept apart from others. What follows is why the other answers were not good enough, including the ones that work.

What we wanted

Every hosted VM sat on one shared bridge in one flat subnet. They were on the same layer-2 segment, and what kept them apart was a firewall rule: drop traffic from that bridge back to itself. It had dropped 2,537 packets and 455 KB, so it was live, not decorative.

The problem is what it depends on. Bridged frames only reach the firewall because a kernel switch is on; turn it off and the DROP never sees the traffic. A ufw reload rebuilt its chains from file and silently discarded every runtime-inserted rule, taking all four tenants offline — isolation held only because a second copy of the DROP happened to live in FORWARD as well. The save file is written and never replayed, so every rule there is runtime-only, surviving on the host not having rebooted.

None of these fail loudly. A tenant would not notice gaining reachability to another tenant, and neither would we. Silent loss of isolation is the one failure mode a multi-tenant host must not have.

What we measured

Per-VM bridges, still in the host's network stack. Removes layer-2 adjacency. Does not isolate at layer-3. The host holds a connected route to every /30 and has forwarding on. Demonstrated: a lookup from one test VM's address to another's is a valid forwarding path. What stops it is still a FORWARD rule. Better than today — the rule no longer depends on bridged-frame filtering — not strong.

Per-VM routing rules plus a blackhole. Fails open: lose the rule and the lookup lands in the main table, which has every tenant. That is the current HomeLink mechanism, and the same failure mode.

A VRF per VM. Measured, and it needs more than a VRF. A VRF alone does not fail closed: connected routes are confined, but a lookup miss falls through to the main table and lands on the shared bridge. An unreachable default at the VRF's own metric closes that. Granting internet egress re-opens it: a leaked default via the uplink sends everything unmatched to the WAN, other tenants included. Upstream drops private address space; isolation that relies on someone else's router is not isolation. Blackholes for private space make it correct — until someone unslaves the bridge, deletes the VRF, or omits the unreachable default, at which point the routes move to main and the join is silent.

That is still one network stack with several tables. The leak we found cannot be designed out of a VRF; it can only be configured around. Do not layer a namespace on top of a recipe you are replacing.

What we changed our minds about

If the only gain were "no main-table fallthrough", VRF would be the better trade. Namespaces make every diagnostic one command further away. Nearly every fault in Four megabits on a three-hundred line was found by running tcpdump on a host interface. Under this design those interfaces are inside the tenant's namespace, at exactly the moment someone is under pressure.

What tips it is HomeLink. Three customers already use 192.168.1.0/24. Each currently sits in its own routing table selected by the VM's source address, and that house subnet is not in the main table — so a VM with no matching rule reaches no house at all. That rule must move in lockstep with any address change or the customer silently loses their house LAN. Inside a per-VM namespace those are ordinary routes in an ordinary table. That deletes a live class of bug rather than mitigating it.

We had been reasoning from topology ("the tap is on our bridge, so the house address is local") and getting a confident wrong answer. Only the routing decides. The same week taught that twice.

The shape

One network namespace per VM. The guest's bridge, taps and HomeLink live inside it. The host's default namespace holds the WAN, the port forwards, and one veth per tenant — and will not forward between those veths.

Libvirt creates taps in the host namespace. Putting the bridge in a namespace therefore means moving the tap after qemu has opened it. Qemu keeps the file descriptor; moving the netdev does not break the guest. Each VM gets a stub bridge in the host namespace that never holds any other tenant. If the hook fails, the tap sits alone on the stub: the VM has no uplink, and it cannot reach another tenant. A shared fallback is a silent join the first time a hook errors. Do not use one.

The default namespace still has a veth into every tenant, because inbound DNAT to a shared public IP always lands there. With forwarding on, the host will route one tenant to another. That hairpin is the remaining VM-to-VM path. Close it in a table ufw cannot rebuild, policy drop, persisted by its own unit. An empty table takes tenants offline (loud) rather than joining them (silent). A host-side VRF on the veths would also close it. Do not add one. The filter does the same job without the fallthrough traps already measured, and the guest is already in a namespace.

Namespaces mean a tenant cannot reach another by routing inside its own stack. They do not mean no rule is load-bearing. Claiming otherwise would repeat the mistake this design already corrects once.

Where it landed

Done, 28 August 2026. Every tenant is in its own namespace, the production instance included, and the shared bridge has retired: no ports, no carrier, no address. Isolation is probed from inside each tenant's own stack, each stance carrying its own positive control.

Moving the last two tenants was written as a script rather than done at a prompt, and writing it down is what found the things doing it by hand had missed. Four of them, each invisible in a different way:

The third is the one worth dwelling on. The forwarded port kept working, the guest answered on the host, everything looked migrated — and the public URL timed out, because that guest's primary default route is its own house, so it was answering the outside world down the customer's broadband instead of back through us. Testing from the host was not a test.

Isolation is probed from a tenant's position, not by confirming a rule exists. The probe is the claim, and today it passes from three stances.

We say: each tenant has its own network stack, we do not forward between them, and we try to break in on a schedule. We do not say "virtually separated". We do not say a firewall is no longer involved — the nft table is involved, and that is a virtue, because it is the part that fails closed.

That is as far as this series goes. It started with one IKEA bulb that would not pair. Finding it took us into the tunnel to the house, then into the design of the host those tunnels land on. The three parts, in the order they happened:

1. A bulb that would not join
2. Four megabits on a three-hundred line
3. Isolation that fails closed