Skip to main content
Contact Us
Register
Pure Technical Services

Equinix Metal Guide: Configuring LACP on ESXi

  1. Last updated
  2. Save as PDF

Currently viewing public documentation. Please login to access the full scope of documentation.

There are a variety ways of configuring networking within Equinix Metal environments, and the following procedure is by no means the only option. But it achieves a few things:

  • Redundancy of VM network
  • Removal of public IPs
  • Redundancy of iSCSI network

This configuration will end up in Layer 2 bonded mode, which means that the final state of the ESXi host will be using a Virtual Distributed Switch (VDS) with LACP enabled with both metal network interfaces in the VDS LACP bond.

Video Demo

Create the Virtual Distributed Switch

First, in the vSphere Client, click on the Networking screen, then right-click on the desired datacenter and choose Distributed Switch > New Distributed Switch.

clipboard_e05a3b2fc5c6da646239afdcc01c848cd.png

Give the VDS a name:

clipboard_e63b161844d3f3593bbe130b5079b4661.png

Choose a number of uplinks (it doesn't matter since we will be creating a LAG) and enter a name for a port group (this also is up to you, initially we will need a public network so this one is named PublicNetwork).

clipboard_efe8091632e42f034d859b2f730733a48.png

Complete the wizard. Now click on the VDS, then the Configure tab, then LACP and click New.

clipboard_eb67af73038f4e1f1993b610e51b2ce08.png

Give the LAG a name or leave the default, and choose 2 ports, then Active. Leave the rest as default. Click OK.

clipboard_e958aa44846c715f1cebf2d923cf9440c.png

Next, right-click on the VDS and choose Distributed Port Group > New Distributed Port Group.

clipboard_e0f5c497f446c4791a6d9e8832241bfe4.png

We now want to create a port group for the management VLAN--the private VLAN that will be the management network for ESXi. If you plan to use vVols this should be on the management network for the FlashArray (though it does not have to be, you can create a VMK later on that network if needed).

clipboard_ee1eaf2ef7a659df09cb41dc81d6c0a25.png clipboard_e893f5901f59731557f9759f75fe442da.png

Leave most at defaults, but choose VLAN type to VLAN and enter the VLAN ID. Also, select Customize default policies configuration.

clipboard_e00ea12a3f049fd3d37ac8005177ac5f8.png

Leave the defaults until you get to Teaming and failover. Make your LAG the active uplink and demote anything else to unused.

clipboard_eeda27921abd88ce3948831e4e489b16e.png

Complete the wizard.

For your original port group that was created during the initial VDS creation, right click on it and choose Edit Settings.

clipboard_e8485ba28c8b83145c6086773c9c930e7.png

Skip to Teaming and failover and make the LAG the active uplink and demote anything else to unused. Repeat this for any other port groups you create in this VDS.

clipboard_e1bf6253ccab0b7f7d14b5f6300d79d09.png

Click OK.

Add ESXi Host to vCenter and the VDS

Once the VDS is created you can now add hosts to the vCenter. This guide is assuming you have deployed ESXi according to the default settings and will be using the same private VLAN for ESXi management as the FlashArray management network.

In this case, there is a new host that was just provisioned:

clipboard_ee7871470d3f2e3dc8f9a046a70218d49.png

Currently in Layer 3 mode. The first step is convert to Hybrid Unbonded mode. This will allow us to move some management to LACP and use the secondary NIC as management during the configuration via a private VLAN. So first, click on Convert to Other Network Type and choose Hybrid and then Unbonded.

clipboard_e92cb17776c1144a4c191bc42c915855c.png

Scroll down and in the Layer 2 panel, click Add New Vlan.

clipboard_ef53f2aa6656d5e9cb586a08f1d447ebd.png

Add the management VLAN to eth1. Repeat and add any VM network VLAN to eth1 as well.

You do not technically have to add two VLANs, but it is advisable to have more than one VLAN assigned in order to force the VLAN tagging requirement in ESXi. If there is only a single VLAN, tagging is not required, but once a second one is added tagging is required. So generally I find it best to start tagging from the start.

clipboard_e2d260f655b8489c7a2880b78a01e12ad.png clipboard_e68c2e3db588078bc60533ba8a3f5fad9.png

Now you will have two VLANs added to eth1.

clipboard_e9ce42e8c3b1777b613696f97d98afe32.png

Now identify the public management IP and enter it into a browser. Login with root credentials.

clipboard_e6eebf7fb2e348baf200a345f1fba3c2d.png clipboard_e84011cf9071b7548a014d11b48cf90f9.png

Go to Networking > Virtual switches and click on Add standard virtual switch.

clipboard_ec6f2f32fd42a224bafd936bf5e107809.png

Name it vSwitch1 and click Add.

clipboard_e61a4e758594ef109925afd7e00d95992.png

Next, go to Networking > VMkernel NICs and then Add VMKernel NIC.

clipboard_ec744c43ce554a9d147e20d304ffc9d36.png

Now, enter in a new port group name, choose vSwitch1 and enter in the management VLAN ID. Then assign an IP address from that VLAN and check Management. Click Create.

clipboard_ef97f729d40e618e6e39ea450715f76da.png

Ensure that the vmkernel port is accessible, by either pinging or logging into the ESXi Web UI via that IP.

clipboard_e7149871366a90e4cce4a7cfd3641cd63.png clipboard_ed4c88c94e63765fba2183a1707a19dba.png

If the IP does not come online you like either 1) didn't choose the right vSwitch, 2) Only added 1 VLAN AND you tagged the port group (no tag is needed then) 3) Added two VLANs but forgot to tag it in the port group 3) typo'ed the IP information.

Now log back into the vSphere Client.

Ensure that the above IP is added into DNS so that it is resolvable by vCenter. 

clipboard_e2477c40be4e219456764627069771db7.png

Right-click on your cluster and choose Add Hosts.

clipboard_e95bb175ba334b2b1e050e9378cba87f0.png

Enter in the host name and credentials and complete the wizard to add the host.

clipboard_e760bc228ecb288086558521699a4be02.png

Go to the Network inventory screen then right-click on the VDS and choose Add and Manage Hosts.

clipboard_ee4878b543110c1f19d090aaaeb976d73.png

Choose Add Hosts and use the New hosts pop-up to add the new host.

clipboard_ed7a842ec68d6a750b6232b3a926ab398.png clipboard_eb4a0c79efc2df3467fe76d97ee553a04.png

Select vmnic0 and click Assign uplink. In the pop-up, choose lag1-0 (or whatever you called your LAG) and click OK. Click Next.

clipboard_e69573a8da86e3d55a3ec4b09872fc0b7.png clipboard_e50497ecec11b12a339b69fdd237e82dc.png clipboard_ead3fd12d71a487a4fa0302109df55a5b.png

The next step of migrating VMKs is optional. If you plan on using Hybrid Bonded mode, then you should migrate the VMK ports. If you plan on moving to Layer 2 Bonded mode, you do not need to migrate the vmks as they can be deleted.

If you would like to migrate them do the following: select the affected vmkernel ports one by one and choose Assign port group. For the VMK(s) with the public/non-layer 2 IP, ensure that you add them to a port group with no VLAN ID. 

clipboard_e43c4091d204e6311ca978dd4f018962e.png clipboard_ee011fcae1335d7e448fe78ce39a816f6.png

Ensure that you leave the recently created VMK with the private IP untouched. Do not move it.

clipboard_ec93c000abf57740057ad5163afa6a672.png

Click Next. Complete the wizard. 

Now click on the ESXi hosts in Hosts and Clusters and go to Configure > VMkernel adapters and click Add Networking.

clipboard_e8cb488ec7c1306e44e9198c486e91f6a.png

Choose VMkernel adapter then choose the existing network that is the port group for the private management VLAN on the VDS.

clipboard_e4f2160afb12f54d75c9bb3000a3b862e.png clipboard_ec4a6eba06cfc8a7495f486418ad96e3c.png

Select Management.

clipboard_e2149fa8da301b096ecf3918bff24e3ad.png

Enter in an IP on the management VLAN and a subnet mask. Finish the wizard.

clipboard_e10f5a501ea1b8c9952134b3454677815.png

Once you are ready, you can delete the non-private VMK ports from the host--once we convert the host to Layer 2 bonded. If you are planning to convert to Hybrid Bonded mode you can leave these VMKs alone.

clipboard_e6e00988eba0971f8d92fb2c296503b61.png clipboard_e042066b45f802c70f1f7b5aaac1c17d8.png

If you are moving to Layer 2 Bonded, all that will be left is the VLAN-based VMkernel adapters.

clipboard_eb90232aef2dc0c53fabbc928922eb913.png

Now log into the Equinix Console and remove the VLANs from eth1.

clipboard_e49074fe808562bebb1d30be20d738ed2.png

Once complete scroll up and click on Convert to Other Network Type.

clipboard_e8e065c4a2bc13262e1250c6e7b801c13.png

Choose Layer 2 Bonded (alternatively you can choose Hybrid Bonded if you want/need public IPs for your hosts/VMs)

clipboard_ec7fb2157325420b388c7635e61b9e488.png

Next add back the VLANs (the management and one more so that the tags are needed).

clipboard_ec98e2715f3b80f1f50b6710f658eb6cf.png
clipboard_e586a8f4cd143c588a688fd90a5efa3d3.png clipboard_e30244e02ed111b23493e9b531daae392.png

The IP from the VDS should come online:

clipboard_e17e0760d3244d6e8427bd6e06e2a96ac.png

Login to the ESXi server directly via the new VLAN IP.

clipboard_ea1d70e00068fa8d00fe6c83c6c87249a.png

Go to Networking and click on TCP/IP Stacks and choose the default stack and click Edit Settings.

clipboard_ee28cb02f36c949cb6e58156d793dfe23.png

Fill out the IP stack with the information for the private VLAN.

clipboard_e91bb4b244c3441db3e4fc5d1fb8e4786.png

Lastly, remove the original VMK that is not associated with the VDS. Having two VMKernel ports on the same subnet will prevent vCenter from adding the host back successfully. Go to Networking and click on VMkernel NICs, choose the original VMK and go to Actions > Remove.

clipboard_eab3bb921c69fcb1eee6c591db8c94b4b.png

Now update any DNS records to the new private IP that was assigned via the VDS.

Note that you will likely need to update the vCenter DNS record cache. If you see a "host not found" error in the step to re-add, this is why. SSH into vCenter and run "systemctl restart dnsmasq"

clipboard_e80e1b0154b9b69086d76596779676a46.png

Go back to the vSphere Client. Right-click on the host and choose Remove from Inventory.

This step is required because the ESXi host was originally registered with a different management IP address. This process makes it "seem" like the management IP was changed outside of vCenter which breaks the vCenter management agent access to the host. Removing and re-adding is the simplest way to resolve this issue.

clipboard_e7d1a02587b28fefee1b22ae90b912098.png

Now right-click on the cluster and click Add Hosts. Add the host back.

clipboard_e73f7659966eebc4256a830b299f93716.png clipboard_ebdc1707cbae731e3f7a6389a02048449.png

If you see the following error, after adding the host, you likely forgot to remove the original VMkernel port. Login to the host via the direct web UI and remove it.

clipboard_e06edb146e2586530009b8297279eff5a.png

Go to Networking and right-click on the VDS and choose Add and Manage Hosts then choose Add Hosts.

clipboard_eedddcdc8be06069a583fd40eefed0597.png clipboard_ebd403e8be1d48370a6c75eb7e1fe0a30.png

Choose the host. Click Next.

clipboard_e2e73b3cd24eb3b5e116f38fc6f74d7e1.png clipboard_e5be1693af93822bc84f7c733a1bcec22.png clipboard_e35c0ff380ee4193beb61764cea3df4e3.png

Choose vmnic0 and click Assign uplink. Choose lag1-0 (or your LAG name).

clipboard_e9ccc0363f0a0acb2fee7509d2c5d276d.png clipboard_eb277092487726c5c84bacef4c12ac45c.png

Repeat for vmnic1 but with the other LAG port.

clipboard_eeeb963ddd4b47f3e9e6783367dbf5e96.png clipboard_e6b839e2b5e05ca037ad73c2f04eee4b8.png

Click Next.

clipboard_e276f6173f5e4984d17518eee559cfe43.png

Finish the wizard.

Now go back to Hosts and Clusters, click on the host and choose Configure then Virtual Switches. Select the standard vSwitches and delete them.

clipboard_ee651dc95e58e5d30096df0b04891c0db.png
clipboard_e0824e8e106c03d7d5ebbc7e1c790de54.png

The host has now been configured.