Network Architecture and Best Practices for Pure Cloud Block Store on AWS
This document is intended to help users architect and configure the network for deploying Pure Cloud Block Store. It provides an overview of the different network connectivity options as well as best practices to how to control the network security, enhancing network performance, and what tools to use to monitor the network.
Cloud Block Store Networking
Interfaces/Subnets
Cloud Block Store deploys with the following four Ethernet interfaces on each controller:
- System
- iSCSi
- Management
- Replication
During deployment, users will be asked to provide a private subnet for each interface. There are two network configuration options that can be implemented to accommodate the CBS networking requirements.
Simple Network Configuration
The simplest topology is to place all Cloud Block Store interfaces (system, iSCSI, management, replication) onto a dedicated subnet. In this scenario, create a new private subnet with /26 subnet mask (255.255.255.192) which is dedicated to only Cloud Block Store interfaces.
Best Practices
It is a recommended best practice that the Cloud Block Store Ethernet interfaces reside on a subnet that is different from the EC2 host initiators. iSCSI traffic between Cloud Block Store and EC2 host initiators can flow using route tables. In most cases, route tables between subnets are set to "local", which allows traffic to flow between subnets in the same Amazon VPC. There are two reasons for this recommendation:
- Separate subnets minimize the chances that an EC2 host initiator would accidentally use IP addresses belonging to the Cloud Block Store Ethernet interfaces.
- Cloud Block Store instances will need seven new IP addresses for a capacity upgrade. Placing Cloud Block Store Ethernet interfaces on a separate subnet from EC2 host initiators reduces the chance that the subnet runs out of IP addresses.
Ideal Network Configuration
This network topology places each Cloud Block Store interface in a separate private subnet. It offers an optimal solution because it allows traffic isolation. It also minimizes the chance of having a network overlap. The following network configuration diagram is an example of this topology.
.png?revision=1)
Mandatory Requirements
- All subnets for Cloud Block Store must be private subnets for greater security.
- If multiple private subnets are used for Cloud Block Store, they must be all in the same Availability zone.
- Ensure the private subnet for the System interface has internet access; see the Internet Access section.
The following table summarizes the requirements for each interface type:
| Interface Name | Interface | Subnet Type | Internet Access Required? |
|---|---|---|---|
| System | eth0 | Private | Y (port 443) |
| Replication | eth1 | Private | N |
| iSCSI | eth2 | Private | N |
| Management | eth3 | Private | N |
IP Addresses
Deploying a new Cloud Block Store instance requires 20 private IP addresses. For a capacity upgrade, an additional seven private IP addresses are required from the subnet where System interfaces reside. This results in a maximum of 27 IP addresses per V10 or V20 instance. It is recommended that the subnet used for the System interfaces has a network mask of 255.255.255.192 (/26). This ensures that there is enough space for capacity expansion.
Internet Access
The private subnet for the System interface must have internet access. Internet access ensures that Cloud Block Store can phone home to Pure1 providing logs, alerts, and additional cloud management services. The simplest configuration is to route traffic from the System private subnet to a NAT Gateway, which resides in the public subnet.
NAT Gateway
A NAT Gateway is the simplest way to provide outbound internet traffic for a private subnet. To configure this with CBS system interface, you just need to create NAT Gateway on a public subnet and configure a route table associated to the system subnet to it. Below you can see the network and the simplified route tables.
.png?revision=1)
Transit Gateway
By following hub-spoke network design, you can leverage Transit Gateway centralization feature to route outbound internet traffic from a VPC without an internet gateway to a VPC that contains a NAT gateway and an internet gateway. Read more about it here.
VPC Endpoint to Amazon S3 and Amazon DynamoDB
A Cloud Block Store instance copies all written data to Amazon S3 to ensure high durability. In addition, a Cloud Block Store instance stores metadata information pertaining to the Cloud Block Store configuration in DynamoDB.
It is important to ensure this traffic travels within the AWS network rather than through the public internet. The benefits of VPC Endpoints include:
- VPC Endpoints ensures that data stays within the AWS network which eliminates the dramatic egress costs for sending data to Amazon S3 and Amazon DynamoDB.
- VPC Endpoints for Amazon S3 prevents additional network hops through the public internet which can significantly improve performance.
- VPC Endpoints is a standard AWS best practice for security reasons.
It is highly advised that customers use VPC Endpoints for Amazon S3 and Amazon DynamoDB. See Appendix A on the Implementation guide to learn how to add an Amazon S3 and Amazon DynamoDB VPC Endpoint to an existing subnet.
Host Initiators Networking
Storage Connectivity
In order to attach storage volumes to EC2 host initiators, route tables associated between the Cloud Block Store subnet and the EC2 host initiators subnet must allow iSCSI traffic to flow. There are two possible cases depending on where EC2 hosts are located.
Intra VPC Connectivity
Intra VPC connectivity refers to traffic flow between subnets in the same VPC. In most cases, route tables between subnets are set to "local", which allow traffic from any subnet range inside the VPC CIDR block. See the network diagram below.
Best Practices
It is recommended to keep EC2 hosts subnets on the same Availability Zone as CBS subnet(s) to avoid data transfer egress charges.
Inter VPC Connectivity
Inter VPC connectivity can be achieved by leveraging VPC peering. VPC peering is used to establish network connection between two VPCs to allow traffic between them using private IP addresses. VPC peering doesn't allow transit communication; therefore, it is required to establish peering connecting between each VPC separately in 1:1 relationship.
There are three possible peering scenarios that fit Cloud Block Store host connectivity:
- Intra AZ peering: EC2 initiator hosts subnet are located in the same AZ as CBS iSCSi subnet.
- Inter AZ peering: EC2 initiator hosts subnet are not located in the same AZ as CBS iSCSi subnet.
- Inter Region peering: EC2 initiator hosts subnet are located in the different region.
Best Practices
- For same region VPC peering, It is recommended to keep EC2 hosts subnets on the same Availability Zone as CBS subnet(s) to avoid data transfer egress charges.
- It is not recommended to attach a volume to an EC2 instance in a different region due to the high latency.
Transit Gateway
A Transit Gateway may be utilized to establish connectivity between Cloud Block Store and EC2 hosts when they reside in different VPCs and within the same AZ. The use of a Transit Gateway to establish connectivity between a Cloud Block Store instance's VPC and EC2 host's VPC when they reside in different AZs is not supported.
Cross-Account VPC Connectivity
Cross-Account VPC connectivity can be established using the same VPC peering in the same account. Once you share VPC to another account using the account ID, the receiving account must accept the request to activate it.
You can use this document for a detailed procedure: Create a VPC peering connection with a VPC in another AWS account.
Each account has its own AZ mapping, make sure to map same AZ ID to avoid egress charges.
For more information check here Working with AZ IDs.
Initiator Performance Enhancement
Increasing the performance of an EC2 instance depends on several factors, mostly it depends on the instance type, hence the number of vCPUs that it has. Therefore, there is a baseline to the amount of traffic the instance can utilize. This section covers what are the other options can be leveraged to enhance the network performance.
Jumbo Frames
Increasing the instance network MTU size to 9001 bytes (also commonly knows as jumbo frames) improves the network efficiency by allowing bigger payloads to be sent over the network in fewer packets while maintaining the same network overhead. Although enabling jumbo frame allows for higher network throughput, it is not a straight forward decision to make and requires proper investigation into your AWS network environment and how your workload is handling I/O requests.
In general, It is not recommended to enable jumbo frames on a network interface that sends traffic over an internet gateway therefore leaves your VPC, since this might lead to packets fragmentation by an intermediate or receiver systems. The same rule applies to sending traffic over an inter-region VPC peering connection or traffic over VPN connections.
To prevent slower performance and packets dropping due to MTU size mismatching, follow the best practices below.
Pure generally recommends performing proper end-to-end tests prior to configuring jumbo frames.
Best Practices - AWS Network Topology
- Pure recommends the usage of jumbo frames for iSCSi traffic between CBS and EC2 instance located in the same VPC or intra-region VPC peering.
- Pure recommends the usage of jumbo frames for replication traffic between your CBS located in VPC that is connected to your FlashArray on-premises network over AWS Direct Connect (only applies to private virtual interface).
- it is not recommended to enable jumbo frames for traffic sent over the following: internet gateway, VPN connections, and inter-region VPC peering, since traffic is limited to 1500 bytes.
- Traffic over AWS Transit Gateway is not supported. A transit gateway supports an MTU of 8500 bytes.
Best Practices - EC2 Configuration
- Don't configure jumbo frames on elastic network interfaces (ENI) has Internet-bound traffic.
- Use an additional ENI for storage "iSCSI" traffic and configure it with jumbo frames.
- Use Path MTU Discovery technicals to validate network route between CBS and EC2 supports jumbo frames.
For Additional References, check AWS documention for Linux EC2 and Windows.
iSCSI Sessions
It is important to note that AWS has bandwidth limits on each TCP connection. A single TCP connection is only capable of 5 Gb networking in AWS. Since a single iSCSI session equates to a single TCP connection, each iSCSI session is also limited on throughput. For applications that need higher throughput, it is advised to increase the number of iSCSI sessions on the EC2 host. If you are looking to maximize throughput from a given EC2, the number of iSCSI sessions will vary depending on the size of the EC2 instance. The Windows and Linux examples above already includes steps to set these values.
The approximate guidance is to have 2 iSCSI sessions for each EC2 "xlarge" size. You can always increase the number of sessions beyond approximate guidance (up to 32 iSCSI per controller) if needed.
For example, an application running on an EC2 size of:
- c5.2xlarge would have 4 iSCSI sessions to each CBS controller.
- m5.4xlarge would have 8 iSCSI sessions to each CBS controller.
- r5.8xlarge would have 16 iSCSI sessions to each CBS controller.
- r5n.16xlarge would have 32 iSCSI sessions to each CBS controller.
The guidance under Host Management for Cloud Block Store on AWS provide approximate values and can be increased, up to 32 iSCSI sessions.
MPIO Settings
I/O should only flow to the primary controller. Ensure you followed the steps to appropriately set MPIO parameters for Windows and Linux hosts to ensure IO's are sent to the primary controller.
To confirm, run I/O on your host and run the following command on your Cloud Block Store instance:
purehost monitor --balance --interval 3
Example: Notice I/O Count should only show for the primary controller.
pureuser@CBS> purehost monitor --balance --interval 3
Name Time Initiator WWN Initiator IQN Initiator NQN Target Target WWN Failover I/O Count I/O Relative to Max
Linux2AMI 2019-08-26 10:31:32 PDT - iqn.1994-05.com.redhat:b9ddc64322ef - (primary) - - 1626 100%
iqn.1994-05.com.redhat:b9ddc64322ef (secondary) - 0 0%
Network Bandwidth Limits
Keep in mind that each Amazon EC2 instance type has network bandwidth limits. See Amazon EC2 instance types. When running performance load testing with a Cloud Block Store instance, ensure the Amazon EC2 instance that the application is running on has enough network bandwidth. For example, a Cloud Block Store V20A-R1 instance uses C5n.18xlarge instances for the controllers which has up to 100 Gb of bandwidth. Therefore the application host should use a single Amazon EC2 instance with matching network limits (ex: 1 x C5n.18xlarge), or multiple Amazon EC2 instances with networks that add up to the 100 Gb limit (ex: 4 x r5n.8xlarge). This ensures that the application is not the bottle neck.
Elastic Network Adapter (ENA)
Leveraging Elastic Network Adapter (ENA) on EC2 instances can deliver high-performance networking capabilities on supported instance types. Enhanced networking uses single root I/O virtualization (SR-IOV) which resulting in near bare metal performance.
Most of the latest AMI have ENA enabled by default. You can find the AMI list here.
If you have an old custom-built AMI, check the additional reference below to test and verify that the ENA module is installed and enabled on your instance.
For Additional References, check AWS documentation for Linux EC2 and Windows.
Security Controls
Security Groups
A Cloud Block Store instance has four Ethernet interfaces that are used for the following types of traffic: iSCSI, management, replication, and system intercommunication. Each Ethernet interface requires different types of TCP access.
- As a security best practice, create three different security group as shown in the table below with the appropriate TCP access for the replication, iSCSI, and management interfaces. Each security group will be applied during the Cloud Block Store deployment.
- The security group for the System interface is auto-created "PureSystemSecurityGroup" during the deployment.
- The three security groups must be in the same region and VPC.
| Security Group | Inbound | Outbound |
|---|---|---|
| System (eth0)* | Auto-Created | Auto-Created |
| Replication (eth1) | 8117 |
8117, 443 (if using CloudSnap) |
| iSCSI (eth2) | 3260 | |
| Mgmt (eth3) | 22, 80, 443, 8084 | 443 |
* Note: For the System interface, a fourth security group called " PureSystemSecurityGroup" is automatically created and applied as part of the Cloud Block Store deployment.
Network ACL
Besides security groups that control traffic on interface level. Network Access Control List (ACL) is an optional layer of security and can be configured at a subnet level. It uses a set of rule that are evaluated from top to down based on numbering assigned to the rule. And it can allow or block a CIDR block on a range of ports or a specific port.
It is recommended to reinforce your security layer with network ACL rules, this will prevent from accidentally making a new attached security group that is too permissive.
For Additional References, see Control traffic to subnets with Network ACLs.
Replication
When replicating from a FlashArray on-premises to a Cloud Block Store instance in an Amazon VPC, ensure that there is network connectivity between the respective sites. Likewise, replication between multiple Cloud Block Store instances requires network connectivity between the instances. More specifically, in order to replicate between a Cloud Block Store instance and a physical FlashArray (or another Cloud Block Store instance), the management and replication ports must communicate. Configure all security groups, network ACLs, and route tables to allow traffic between the two sites for the respective management and replication ports. The following table provides the port number for each interface.
| Service Type | Firewall Port |
|---|---|
| Management interfaces | 443 |
| Replication interfaces |
8117 443 (if using CloudSnap) |
For additional details on replication requirements and limits, see the Purity Replication Requirements and Interoperability Matrix.
Important consideration to put in mind before configuration the replication.
- AWS network charges may apply for replicating across Availability Zones, VPCs, and/or regions.
- Egress charges may apply for any data that sent from Cloud Block Store to FlashArray.
FlashArray to CBS
When replicating between a physical datacenter and the Amazon VPC, you can achieve network connectivity in a number of ways, including AWS Direct Connect or a Site-to-Site-VPN connection.
Replication between Cloud Block Store instances or using CloudSnap can greatly minimize customer cloud spend on ingress/egress charges since snapshots are deduped, compressed and we don’t have to send the same bits twice.
AWS Direct Connect
Direct Connect provides a private connectivity, bypassing the internet to deliver consistent and lower-latency performance. It allows you to establish a dedicated network connection to AWS. This can reduce network costs, increase bandwidth, and provide a more consistent network experience than internet-based connections.
AWS Site-to-Site VPN
Create IPsec VPN connection between from your remote datacenter to VPC over the internet. It has three main components:
- Site to Site VPN Connection: A secure connection between your on-premises equipment and your VPCs.
- Virtual Private Gateway: Gateway for the Amazon side of the Site-to-Site VPN connection. It’s used on the VPC to allow this Site to Site connection.
- Customer Gateway: Physical or software appliance that you own or manage in your on-premises network to allow this Site to Site connection.
CBS to CBS
AWS to AWS
Similar to Host Connectivity options discussed in previous section. Establishing replication connectivity between two CBS arrays can be done in various ways, but the same concept is applied. The optimal way to ensure the best network performance and regulate with the security control configured, is to use VPC Peering connections. Whether you are peering two CBSs in the same VPC or different region.
Pure recommends using VPC Peering for replication between CBS array in different VPC inter/intra-region. Put in mind when using ActiveCluster, ensure the latency between two CBS arrays are below 11 ms round trip.
AWS to Azure
There are various options to establish communication between two cloud providers, those options differ in speed, latency, reliability, service-level agreements (SLAs), complexity, and costs.
A common and easiest way is to use Site-to-Site VPN tunnels. It is certainly the fastest deployment option, yet it wont provide a high throughout and low latency comparing to AWS DirectConnect or Azure ExpressRoute.
For more guidelines on how to establish cross communication between two cloud providers check this.
Network Monitoring
VPC Flow Logs
VPC flow logs is a feature that is used to capture information about network terrific flowing in and out network interfaces in your VPC. It can Capture inbound and outbound traffic at the VPC level, subnet level, or the network interface level. Logs captured can be published to CloudWatch logs or S3 Athena for advance analysis.
For more information on how to diagnose and monitor Cloud Block Store instance or EC2 workload traffic, check this guide.