VXLAN overlay broken on VMware/NSX: hypervisor silently drops valid VXLAN packets on UDP 8472

[UMMAN2005]

Environment:

• RKE2 with Canal CNI (Flannel VXLAN + Calico)

• VMware vCloud Director with NSX-backed networking

• Nodes across multiple AZs on different L3 subnets (10.6.10.0/24 and 10.10.175.0/24)

• Flannel VXLAN: VNI 1, UDP 8472, MTU 1450

Problem:

Pod-to-pod communication fails between nodes on different subnets. The Flannel VXLAN data plane appears fully configured (FDB entries, ARP, routes all correct), but encapsulated packets never reach the destination node.

Diagnosis:

All standard checks passed:

• FDB, ARP/neighbor tables, routes: correct on all nodes

• rp_filter: loose mode (2)

• UDP 8472 listening on all nodes

• ICMP at MTU 1450 between nodes: 0% loss

• Generic UDP (socat) between nodes: works bidirectionally

Simultaneous tcpdump on sender and receiver showed VXLAN replies leave the sender's NIC but never arrive at the receiver's NIC. To isolate, we sent crafted packets from the same source to the same destination on UDP 8472:

Payload | Arrived? -- | -- Random bytes (not valid VXLAN) | Yes Valid VXLAN header, VNI=1 | No Valid VXLAN header, VNI=999 | No Valid VXLAN header, VNI=4096 | No

Root Cause:

The VMware/NSX hypervisor performs deep packet inspection on UDP 8472 and intercepts any packet containing a valid VXLAN header structure, regardless of VNI. NSX uses VXLAN for its own overlay networking, and the vSwitch VTEP consumes guest-generated VXLAN traffic before it reaches the physical network.

The issue is asymmetric across subnets -- likely depending on which ESXi host and vSwitch configuration each VM lands on.

Workarounds to evaluate:

1. Change Flannel's VXLAN backend port to a non-standard value (e.g. 4790)

2. Disable VXLAN offloading/interception on the VM portgroups at the vSphere level

3. Switch overlay to WireGuard or IPsec (avoids VXLAN entirely)

4. Switch to Geneve (UDP 6081) -- unclear if NSX also intercepts this