Back for part two of this look at how I went about configuring a single host for NestedESXi deployments. In the previous post I looks at the physical configuration of the ESXi Host networking as well as what needed to be configured in the ESXi switching to prepare the top level to correctly have the NestedESXi networking act as close to possible as a production level multi-tiered network. In part two, I’m going to lay out how I have configured my NestedESXi hosts and networking on how to handle all nested VMs as well as how to talk to the physical layer without the need for external switching or routing.

I needed a solution that was sustainable and relatively simple to maintain without spending a lot of cash, while also keeping the actual hardware required down to a minimum. At the same time, I wanted to have the NestedESXi solution to still mimic real production platforms in that the networking should not be flat, and that there should be dedicated subnets for specific purposes.

As a refresher, the diagram below represents a high level design of the what I have setup:

Nested ESXi Hosts

As you can see from the diagram each NestedESXi host has two network interfaces that are connected to the top level Distributed Switch. These are connected to the LAB-TRUNK PortGroup which is configured to accept VLAN 0-4094.

The Vyos Edge also has this PortGroup connected and is configured without a specific subnet configuration. The Edge has the interfaces configured and provides routing and firewalling between the subnets to, from and within the whole environment. The 172.17.x.x addresses shown below are the NestedESXi subnets for VM Traffic, Management and vSAN and they match back to the Private VLAN configuration as mentioned in Part 1.

vCenter and other NestedESXi management VMs get assigned the specific top level ports which are configured with the VLAN Type of Private and get assigned the Private VLAN ID as configured in part 1.

The result is that any VM deployed into the Private VLAN configured PortGroups can communicate with any VM that is Nested and part of the same VLAN configuration.

Remembering that at this point we are still in the top physical layer, but have configured all the required settings for communications to flow between the Nested and Physical.

Nested ESXi Host Networking

When looking at the Nested vCenter, we see a different world. This is the NestedESXi world. In my current setup, I have three NestedESXi hosts setup with vSAN configured for the storage (vSAN also has its own dedicated storage network). If I was going to leverage NFS or iSCSI I would create those Virtual Appliances on the physical Host and configure their networking as shown above.

The irony here is that, once inside the NestedESXi world, the networking is more straight forward and traditional. But it is what has been configured above that makes it all tick. The configuration of the nested Distributed Switch is simply a case of creating PortGroups with VLAN IDs that match what is configured on the Private VLANs. All NestedESXi traffic flows through the LAB-TRUNK PortGroups and back through each other if the traffic is local to the same subnet… otherwise it will hit the Vyos Edge and be routed through and out as required.

And for the NestedESXi VMs, the PortGroup and IP addressing matches the configured subnets on the Vyos Edge.

The last thing to mention is in regards to the Distributed Switch settings. The only change i’ve made is to change the MTU to 1600 so that in future it can carry NSX traffic. Apart from that it’s a standard configuration. Each PortGroup is configured with two Active Uplinks and the Load Balancing is Route Based on Original Virtual Port… but in my testing it hasn’t made ant much difference.

Part Two Wrap Up

So there you have it. This two part series showed how it is possible to leverage a single host to configure and run an advanced NestedESXi instance with advanced VLAN backed networking. Again, while this isn’t for everyone as I know there are a lot in the community who like to splurge on networking kit, cabling and other production like hardware, for me this setup gives me the flexibility to test anything I want in the vSphere world. As mentioned in the first post, I am all setup and ready to deploy NSX-T into this lab and also start to deploy VCD as well. I’ll cover that in future posts as well as a look at options to backup homelabs with Veeam.

If anyone has any questions on this setup, please post them in the comments below and I will try to answer them as best I can.

Happy NestedESXi Home Labbing!