There are a variety ways of configuring networking within Equinix Metal environments, and the following procedure is by no means the only option. But it achieves a few things:
- Redundancy of VM network
- Removal of public IPs
- Redundancy of iSCSI network
This configuration will end up in Layer 2 bonded mode, which means that the final state of the ESXi host will be using a Virtual Distributed Switch (VDS) with LACP enabled with both metal network interfaces in the VDS LACP bond.
Create the Virtual Distributed Switch
First, in the vSphere Client, click on the Networking screen, then right-click on the desired datacenter and choose Distributed Switch > New Distributed Switch.
Give the VDS a name:
Choose a number of uplinks (it doesn't matter since we will be creating a LAG) and enter a name for a port group (this also is up to you, initially we will need a public network so this one is named PublicNetwork).
Complete the wizard. Now click on the VDS, then the Configure tab, then LACP and click New.
Give the LAG a name or leave the default, and choose 2 ports, then Active. Leave the rest as default. Click OK.
Next, right-click on the VDS and choose Distributed Port Group > New Distributed Port Group.
We now want to create a port group for the management VLAN--the private VLAN that will be the management network for ESXi. If you plan to use vVols this should be on the management network for the FlashArray (though it does not have to be, you can create a VMK later on that network if needed).
Leave most at defaults, but choose VLAN type to VLAN and enter the VLAN ID. Also, select Customize default policies configuration.
Leave the defaults until you get to Teaming and failover. Make your LAG the active uplink and demote anything else to unused.
Complete the wizard.
For your original port group that was created during the initial VDS creation, right click on it and choose Edit Settings.
Skip to Teaming and failover and make the LAG the active uplink and demote anything else to unused. Repeat this for any other port groups you create in this VDS.
Add ESXi Host to vCenter and the VDS
Once the VDS is created you can now add hosts to the vCenter. This guide is assuming you have deployed ESXi according to the default settings and will be using the same private VLAN for ESXi management as the FlashArray management network.
In this case, there is a new host that was just provisioned:
Currently in Layer 3 mode. The first step is convert to Hybrid Unbonded mode. This will allow us to move some management to LACP and use the secondary NIC as management during the configuration via a private VLAN. So first, click on Convert to Other Network Type and choose Hybrid and then Unbonded.
Scroll down and in the Layer 2 panel, click Add New Vlan.
Add the management VLAN to eth1. Repeat and add any VM network VLAN to eth1 as well.
You do not technically have to add two VLANs, but it is advisable to have more than one VLAN assigned in order to force the VLAN tagging requirement in ESXi. If there is only a single VLAN, tagging is not required, but once a second one is added tagging is required. So generally I find it best to start tagging from the start.
Now you will have two VLANs added to eth1.
Now identify the public management IP and enter it into a browser. Login with root credentials.
Go to Networking > Virtual switches and click on Add standard virtual switch.
Name it vSwitch1 and click Add.
Next, go to Networking > VMkernel NICs and then Add VMKernel NIC.
Now, enter in a new port group name, choose vSwitch1 and enter in the management VLAN ID. Then assign an IP address from that VLAN and check Management. Click Create.
Ensure that the vmkernel port is accessible, by either pinging or logging into the ESXi Web UI via that IP.
If the IP does not come online you like either 1) didn't choose the right vSwitch, 2) Only added 1 VLAN AND you tagged the port group (no tag is needed then) 3) Added two VLANs but forgot to tag it in the port group 3) typo'ed the IP information.
Now log back into the vSphere Client.
Ensure that the above IP is added into DNS so that it is resolvable by vCenter.
Right-click on your cluster and choose Add Hosts.
Enter in the host name and credentials and complete the wizard to add the host.
Go to the Network inventory screen then right-click on the VDS and choose Add and Manage Hosts.
Choose Add Hosts and use the New hosts pop-up to add the new host.
Select vmnic0 and click Assign uplink. In the pop-up, choose lag1-0 (or whatever you called your LAG) and click OK. Click Next.
The next step of migrating VMKs is optional. If you plan on using Hybrid Bonded mode, then you should migrate the VMK ports. If you plan on moving to Layer 2 Bonded mode, you do not need to migrate the vmks as they can be deleted.
If you would like to migrate them do the following: select the affected vmkernel ports one by one and choose Assign port group. For the VMK(s) with the public/non-layer 2 IP, ensure that you add them to a port group with no VLAN ID.
Ensure that you leave the recently created VMK with the private IP untouched. Do not move it.
Click Next. Complete the wizard.
Now click on the ESXi hosts in Hosts and Clusters and go to Configure > VMkernel adapters and click Add Networking.
Choose VMkernel adapter then choose the existing network that is the port group for the private management VLAN on the VDS.
Enter in an IP on the management VLAN and a subnet mask. Finish the wizard.
Once you are ready, you can delete the non-private VMK ports from the host--once we convert the host to Layer 2 bonded. If you are planning to convert to Hybrid Bonded mode you can leave these VMKs alone.
If you are moving to Layer 2 Bonded, all that will be left is the VLAN-based VMkernel adapters.
Now log into the Equinix Console and remove the VLANs from eth1.
Once complete scroll up and click on Convert to Other Network Type.
Choose Layer 2 Bonded (alternatively you can choose Hybrid Bonded if you want/need public IPs for your hosts/VMs)
Next add back the VLANs (the management and one more so that the tags are needed).
The IP from the VDS should come online:
Login to the ESXi server directly via the new VLAN IP.
Go to Networking and click on TCP/IP Stacks and choose the default stack and click Edit Settings.
Fill out the IP stack with the information for the private VLAN.
Lastly, remove the original VMK that is not associated with the VDS. Having two VMKernel ports on the same subnet will prevent vCenter from adding the host back successfully. Go to Networking and click on VMkernel NICs, choose the original VMK and go to Actions > Remove.
Now update any DNS records to the new private IP that was assigned via the VDS.
Note that you will likely need to update the vCenter DNS record cache. If you see a "host not found" error in the step to re-add, this is why. SSH into vCenter and run "systemctl restart dnsmasq"
Go back to the vSphere Client. Right-click on the host and choose Remove from Inventory.
This step is required because the ESXi host was originally registered with a different management IP address. This process makes it "seem" like the management IP was changed outside of vCenter which breaks the vCenter management agent access to the host. Removing and re-adding is the simplest way to resolve this issue.
Now right-click on the cluster and click Add Hosts. Add the host back.
If you see the following error, after adding the host, you likely forgot to remove the original VMkernel port. Login to the host via the direct web UI and remove it.
Go to Networking and right-click on the VDS and choose Add and Manage Hosts then choose Add Hosts.
Choose the host. Click Next.
Choose vmnic0 and click Assign uplink. Choose lag1-0 (or your LAG name).
Repeat for vmnic1 but with the other LAG port.
Finish the wizard.
Now go back to Hosts and Clusters, click on the host and choose Configure then Virtual Switches. Select the standard vSwitches and delete them.
The host has now been configured.