Equinix Metal Guide: Network Design

Overview

• Layer 3 Bonded mode - In Layer 3 mode, individual network interfaces are placed in an LACP bond where all management IPs are assigned.

• Hybrid Bonded mode - Both Layer 3 and Layer 2 is supported on the bonded interface. VLANs can be attached to the bonded interface.

• Hybrid Unbonded Mode - One network interface is removed from the bond and placed in Layer 2 mode. VLANs can then be attached to this interface for Layer 2 connectivity. This preserves Layer 3 connectivity to the server via bond0, so it can be accessed via the public IP.

• Layer 2 Bonded mode - This mode converts the bonded network interface to pure Layer 2 mode. This means all access to the public Internet is lost, and the host can only be reached by the Serial Over SSH (SOS) console. In this configuration the network bond is intact, so only one network interface will be available for attaching VLANs.

• Layer 2 Unbonded mode - This mode is similar to the Layer 2 Bonded mode configuration, except the network bond is also dismantled, providing two network interfaces available for VLANs. All access to the public Internet is lost, and the host can only be reached by the Serial Over SSH (SOS) console.

Layer 3 Bonded mode

This is the default network configuration for hosts when they get deployed. A server will be assigned a routable public IP (something like 145.40.80.154), a private IP (like 10.70.19.18) and a few additional IPs (two /29 IP blocks) reserved for use with that host. For ESXi, these additional public and private IPs can be used for vmkernel ports or virtual machines. For instance if you see this for a server:

The available IPs are given as below (the exact IPs of course change, but the numerical incrementing/allocation does not vary).

By default this will configure two vmkernel ports in vSwitch0 (standard switch) with only the first physical NIC in the switch. Since this is a bonded deployment, both NICs are usable, but they cannot be both directly added to the vSwitch--instead, in order to use them both they need to be placed in an LACP configuration which in vSphere is only available via a virtual Distributed Switch (vDS). A vDS requires vCenter Server, so this is why it is not configured upon deployment.

To configure LACP, follow this KB once you have a vCenter deployed.

https://kb.vmware.com/s/article/2034277

These networks (and respective port groups) do not need to be tagged with any VLAN.

While layer 3 is the simplest mode (no VLANs etc.) it is generally not the best choice for running a persistent VMware environment as public IPs are generally not desired for ESXi management. It is recommended to migrate to layer 2 and provisioning VLANs for the ESXi network. This will be discussed in the ESXi network management KB.

These IPs are not really meant to be mobile--they are assigned via a specific subnet to that server--so virtual machines that might vMotion should not use these IPs. There best use is for static hardware--like a vmkernel adapter that does not move amongst hosts. For virtual machine networks, it is best to use private VLANs that can be shared across hosts--for this you need to convert to Hybrid mode or direct layer 2.

Hybrid Bonded mode

In a hybrid bonded mode, Equinix gives you the ability to create and assign shared VLANs to your servers while still being able to provision and use the provisioned default IPs. This is useful for servers that host virtual machines that might need to have a static/public IP address while still also hosting virtual machines that use private addressing. Just like with Layer 3 bonded mode you must leverage LACP to use both NICs--therefore requiring vCenter to be deployed and a virtual Distributed Switch to be created with LACP support enabled.

While you can use any bonded mode without a VDS, there will not be NIC-level redundancy. If there is a switching failure or a NIC failure the host/VMs will be offline that rely on that NIC. The second NIC can only be used in LACP mode--so it will sit unused in standard virtual switch environment.

Hybrid Unbonded mode

In a hybrid unbonded mode, Equinix gives you the ability to create and assign shared VLANs to your servers while still being able to provision and use the provisioned default IPs. This is useful for servers that host virtual machines that might need to have a static/public IP address while still also hosting virtual machines that use private addressing. Unlike bonded mode, there is no bond on the NICs, so both can be use independently but in this mode they offer no NIC or switch-level redundancy as they will connected to separate VLANs and networks. Adding the same VLAN to both interfaces in unbonded mode is not allowed. Therefore, this mode is not recommended for environments that have VMs that need highly-available network access.

Layer 2 Bonded mode

In a layer 2 bonded mode, Equinix gives you the ability to create and assign shared VLANs to your servers but does not provide any pre-provisioned IPs (public or private) and upon conversion to this mode any assigned IPs will be removed. This is useful for compute environments that do not have any need for public IPs.

Just like with other bonded modes you must leverage LACP to use both NICs--therefore requiring vCenter to be deployed and a virtual Distributed Switch to be created with LACP support enabled. While you can use layer 2 bonded mode without a VDS, there will not be NIC-level redundancy. If there is a switching failure or a NIC failure the host/VMs will be offline that rely on that NIC. The second NIC can only be used in LACP mode--so it will sit unused in standard virtual switch environment.

Layer 2 Unbonded mode

In a layer 2 unbonded mode, Equinix gives you the ability to create and assign shared VLANs to your servers but does not provide any pre-provisioned IPs (public or private) and upon conversion to this mode any assigned IPs will be removed. This is useful for compute environments that do not have any need for public IPs.

Unlike bonded mode, there is no bond on the NICs, so both can be use independently but in this mode they offer no NIC or switch-level redundancy as they will connected to separate VLANs and networks. Adding the same VLAN to both interfaces in unbonded mode is not allowed. Therefore, this mode is not recommended for environments that have VMs that need highly-available network access.

Changing Network Mode

To change a network mode, login to the Equinix Console and click on servers and choose your desired server.

Next, click on Network then Convert to Other Network Type.

Next choose the network mode:

And if converting to a layer 2 type mode, choose any relevant VLANs. Click Convert to... when done.