Skip to main content
Pure Technical Services

Network Architecture and Best Practices | Pure CBS on AWS

Currently viewing public documentation. Please login to access the full scope of documentation.

KP_Ext_Announcement.png

This document is intended to help users architect and configure the network for deploying Pure Cloud Block Store. It provides an overview of the different network connectivity options as well as best practices to how to control the network security, enhancing network performance, and what tools to use to monitor the network. 

Cloud Block Store Networking

Interfaces/Subnets

Cloud Block Store deploys with the following four Ethernet interfaces on each controller:

  • System 
  • iSCSi
  • Management
  • Replication

PAWS Networking - Interfaces.png

During deployment, users will be asked to provide a private subnet for each interface. There are two network configuration options that can be implemented to accommodate the CBS networking requirements. 

Simple Network Configuration

The simplest topology is to place all Cloud Block Store interfaces (system, iSCSI, management, replication) onto a dedicated subnet. In this scenario, create a new private subnet with /26 subnet mask (255.255.255.192) which is dedicated to only Cloud Block Store interfaces.

simple-network-cbs.png

Best Practices

It is a recommended best practice that the Cloud Block Store Ethernet interfaces reside on a subnet that is different from the EC2 host initiators. iSCSI traffic between Cloud Block Store and EC2 host initiators can flow using route tables. In most cases, route tables between subnets are set to "local", which allows traffic to flow between subnets in the same Amazon VPC. There are two reasons for this recommendation:

  • Separate subnets minimize the chances that an EC2 host initiator would accidentally use IP addresses belonging to the Cloud Block Store Ethernet interfaces.
  • Cloud Block Store instances will need seven new IP addresses for a capacity upgrade. Placing Cloud Block Store Ethernet interfaces on a separate subnet from EC2 host initiators reduces the chance that the subnet runs out of IP addresses.

Ideal Network Configuration

This network topology places each Cloud Block Store interface in a separate private subnet. It offers an optimal solution because it allows traffic isolation. It also minimizes the chance of having a network overlap. The following network configuration diagram is an example of this topology.

PAWS Networking - Page 3 (1).png

 

Mandatory Requirements

  • All subnets for Cloud Block Store must be private subnets for greater security.
  • If multiple private subnets are used for Cloud Block Store, they must be all in the same Availability zone.
  • Ensure the private subnet for the System interface has internet access; see the Internet Access section.

The following table summarizes the requirements for each interface type:

Interface Name Interface Subnet Type Internet Access Required?
System eth0 Private Y (port 443)
Replication eth1 Private N
iSCSI eth2 Private N
Management eth3 Private N

IP Addresses 

Deploying a new Cloud Block Store instance requires 20 private IP addresses. For a capacity upgrade, an additional seven private IP addresses are required from the subnet where System interfaces reside. This results in a maximum of 27 IP addresses per V10 or V20 instance. It is recommended that the subnet used for the System interfaces has a network mask of 255.255.255.192 (/26). This ensures that there is enough space for capacity expansion.

Internet Access

The private subnet for the System interface must have internet access. Internet access ensures that Cloud Block Store can phone home to Pure1 providing logs, alerts, and additional cloud management services. The simplest configuration is to route traffic from the System private subnet to a NAT Gateway, which resides in the public subnet.

NAT Gateway

A NAT Gateway is the simplest way to provide outbound internet traffic for a private subnet. To configure this with CBS system interface, you just need to create NAT Gateway on a public subnet and configure a route table associated to the system subnet to it. Below you can see the network and the simplified route tables.

 

 

PAWS Networking - NAT GW (1).png

Transit Gateway

By following hub-spoke network design, you can leverage Transit Gateway centralization feature to route outbound internet traffic from a VPC without an internet gateway to a VPC that contains a NAT gateway and an internet gateway. Read more about it here
 

clipboard_ec542c4e95d304e0c40fe3f8396840582.png

 

 

VPC Endpoint to Amazon S3 and Amazon DynamoDB

A Cloud Block Store instance copies all written data to Amazon S3 to ensure high durability. In addition, a Cloud Block Store instance stores metadata information pertaining to the Cloud Block Store configuration in DynamoDB. 

It is important to ensure this traffic travels within the AWS network rather than through the public internet. The benefits of VPC Endpoints include:

  • VPC Endpoints ensures that data stays within the AWS network which eliminates the dramatic egress costs for sending data to Amazon S3 and Amazon DynamoDB.
  • VPC Endpoints for Amazon S3 prevents additional network hops through the public internet which can significantly improve performance.
  • VPC Endpoints is a standard AWS best practice for security reasons.

 

PAWS Networking - VPC Endpoint.png

 

It is highly advised that customers use VPC Endpoints for Amazon S3 and Amazon DynamoDB. See Appendix A on the Implementation guide to learn how to add an Amazon S3 and Amazon DynamoDB VPC Endpoint to an existing subnet.



Host Initiators Networking

Storage Connectivity 

In order to attach storage volumes to EC2 host initiators, route tables associated between the Cloud Block Store subnet and the EC2 host initiators subnet must allow iSCSI traffic to flow. There are two possible cases depending on where EC2 hosts are located.

Intra VPC Connectivity 

Intra VPC connectivity refers to traffic flow between subnets in the same VPC. In most cases, route tables between subnets are set to "local",  which allow traffic from any subnet range inside the VPC CIDR block. See the network diagram below.

Best Practices

It is recommended to keep EC2 hosts subnets on the same Availability Zone as CBS subnet(s) to avoid data transfer egress charges.


  clipboard_ebed43d977caa5c613e02c0c58b932579.png

 

Inter VPC Connectivity 

Inter VPC connectivity can be achieved by leveraging VPC peering. VPC peering is used to establish network connection between two VPCs to allow traffic between them using private IP addresses. VPC peering doesn't allow transit communication; therefore, it is required to establish peering connecting between each VPC separately in 1:1 relationship.  

There are three possible peering scenarios that fit Cloud Block Store host connectivity:

  • Intra AZ peering: EC2 initiator hosts subnet are located in the same AZ as CBS iSCSi subnet. 
  • Inter AZ peering: EC2 initiator hosts subnet are not located in the same AZ as CBS iSCSi subnet. 
  • Inter Region peering: EC2 initiator hosts subnet are located in the different region. 

 

Best Practices

  • For same region VPC peering, It is recommended to keep EC2 hosts subnets on the same Availability Zone as CBS subnet(s) to avoid data transfer egress charges.
  • It is not recommended to attach a volume to an EC2 instance in a different region due to the high latency.

clipboard_e0d2fc3c3ab27b9689b207a16f94e6b9e.png

Transit Gateway

A Transit Gateway may be utilized to establish connectivity between Cloud Block Store and EC2 hosts when they reside in different VPCs and within the same AZ. The use of a Transit Gateway to establish connectivity between a Cloud Block Store instance's VPC and EC2 host's VPC when they reside in different AZs is not supported. 

Cross-Account VPC Connectivity 

Cross-Account VPC connectivity can be established using the same VPC peering in the same account. Once you share VPC to another account using the account ID, the receiving account must accept the request to activate it. 

You can use this document for a detailed procedure: Create a VPC peering connection with a VPC in another AWS account

Each account has its own AZ mapping, make sure to map same AZ ID to avoid egress charges. 

For more information check here Working with AZ IDs.

Initiator Performance Enhancement

Increasing the performance of an EC2 instance depends on several factors, mostly it depends on the instance type, hence the number of vCPUs that it has. Therefore, there is a baseline to the amount of traffic the instance can utilize. This section covers what are the other options can be leveraged to enhance the network performance. 

Jumbo Frames

Increasing the instance network MTU size to 9001 bytes (also commonly knows as jumbo frames) improves the network efficiency by allowing bigger payloads to be sent over the network in fewer packets while maintaining the same network overhead. Although enabling jumbo frame allows for higher network throughput, it is not a straight forward decision to make and requires proper investigation into your AWS network environment and how your workload is handling I/O requests.

In general, It is not recommended to enable jumbo frames on a network interface that sends traffic over an internet gateway therefore leaves your VPC, since this might lead to packets fragmentation by an intermediate or receiver systems. The same rule applies to sending traffic over an inter-region VPC peering connection or traffic over VPN connections. 

To prevent slower performance and packets dropping due to MTU size mismatching, follow the best practices below.

Pure generally recommends performing proper end-to-end tests prior to configuring jumbo frames. 

Best Practices - AWS Network Topology

  • Pure recommends the usage of jumbo frames for iSCSi traffic between CBS and EC2 instance located in the same VPC or intra-region VPC peering. 
  • Pure recommends the usage of jumbo frames for replication traffic between your CBS located in VPC that is connected to your FlashArray on-premises network over AWS Direct Connect (only applies to private virtual interface).
  • it is not recommended to enable jumbo frames for traffic sent over the following: internet gateway, VPN connections, and inter-region VPC peering, since traffic is limited to 1500 bytes. 
  • Traffic over AWS Transit Gateway is not supported. A transit gateway supports an MTU of 8500 bytes. 

Best Practices - EC2 Configuration

  • Don't configure jumbo frames on elastic network interfaces (ENI) has Internet-bound traffic.
  • Use an additional ENI for storage "iSCSI" traffic and configure it with jumbo frames. 
  • Use Path MTU Discovery technicals to validate network route between CBS and EC2 supports jumbo frames. 

For Additional References, check AWS documention for Linux EC2 and Windows.

iSCSI Sessions

It is important to note that AWS has bandwidth limits on each TCP connection. A single TCP connection is only capable of 5 Gb networking in AWS. Since a single iSCSI session equates to a single TCP connection, each iSCSI session is also limited on throughput. For applications that need higher throughput, it is advised to increase the number of iSCSI sessions on the EC2 host. If you are looking to maximize throughput from a given EC2, the number of iSCSI sessions will vary depending on the size of the EC2 instance. The Windows and Linux examples above already includes steps to set these values.

clipboard_e1489ec7546e197ff5c6b4ad17f8a26d3.png

The approximate guidance is to have 2 iSCSI sessions for each EC2  "xlarge" size. You can always increase the number of sessions beyond approximate guidance (up to 32 iSCSI per controller) if needed.

For example, an application running on an EC2 size of:

- c5.2xlarge would have 4 iSCSI sessions to each CBS controller.

- m5.4xlarge would have 8 iSCSI sessions to each CBS  controller.

- r5.8xlarge would have 16 iSCSI sessions to each CBS controller.

- r5n.16xlarge would have 32 iSCSI sessions to each CBS controller.

The guidance under Host Management for Cloud Block Store on AWS provide approximate values and can be increased, up to 32 iSCSI sessions.

MPIO Settings

I/O should only flow to the primary controller. Ensure you followed the steps to appropriately set MPIO parameters for Windows and Linux hosts to ensure IO's are sent to the primary controller.

To confirm, run I/O on your host and run the following command on your Cloud Block Store instance:

purehost monitor --balance --interval 3

Example: Notice I/O Count should only show for the primary controller.

pureuser@CBS> purehost monitor --balance --interval 3
Name       Time                     Initiator WWN  Initiator IQN                        Initiator NQN  Target       Target WWN  Failover  I/O Count  I/O Relative to Max
Linux2AMI  2019-08-26 10:31:32 PDT  -              iqn.1994-05.com.redhat:b9ddc64322ef  -              (primary)    -           -         1626       100%
                                                   iqn.1994-05.com.redhat:b9ddc64322ef                 (secondary)              -         0          0%
                                                  

Network Bandwidth Limits

Keep in mind that each Amazon EC2 instance type has network bandwidth limits. See Amazon EC2 instance types.  When running performance load testing with a Cloud Block Store instance, ensure the Amazon EC2 instance that the application is running on has enough network bandwidth. For example, a Cloud Block Store V20A-R1 instance uses C5n.18xlarge instances for the controllers which has up to 100 Gb of bandwidth. Therefore the application host should use a single Amazon EC2 instance with matching network limits (ex: 1 x C5n.18xlarge), or multiple Amazon EC2 instances with networks that add up to the 100 Gb limit (ex: 4 x r5n.8xlarge). This ensures that the application is not the bottle neck.

Elastic Network Adapter (ENA)

Leveraging Elastic Network Adapter (ENA) on EC2 instances can deliver high-performance networking capabilities on supported instance types. Enhanced networking uses single root I/O virtualization (SR-IOV) which resulting in near bare metal performance. 

Most of the latest AMI have ENA enabled by default. You can find the AMI list here

If you have an old custom-built AMI, check the additional reference below to test and verify that the ENA module is installed and enabled on your instance. 

For Additional References, check AWS documentation for Linux EC2 and Windows.


Security Controls

Security Groups 

A Cloud Block Store instance has four Ethernet interfaces that are used for the following types of traffic: iSCSI, management, replication, and system intercommunication. Each Ethernet interface requires different types of TCP access. 

  • As a security best practice, create three different security group as shown in the table below with the appropriate TCP access for the replication, iSCSI, and management interfaces. Each security group will be applied during the Cloud Block Store deployment.
  • The security group for the System interface is auto-created "PureSystemSecurityGroup" during the deployment.
  • The three security groups must be in the same region and VPC.
Security Group Inbound Outbound
System (eth0)* Auto-Created Auto-Created
Replication (eth1) 8117

8117, 443 (if using CloudSnap)

iSCSI (eth2) 3260  
Mgmt (eth3) 22, 80, 443, 8084 443

* Note: For the System interface, a fourth security group called " PureSystemSecurityGroup" is automatically created and applied as part of the Cloud Block Store deployment.

Network ACL

Besides security groups that control traffic on interface level. Network Access Control List (ACL) is an optional layer of security and can be configured at a subnet level. It uses a set of rule that are evaluated from top to down based on numbering assigned to the rule. And it can allow or block a CIDR block on a range of ports or a specific port.

It is recommended to reinforce your security layer with network ACL rules, this will prevent from accidentally making a new attached security group that is too permissive. 

For Additional References, see Control traffic to subnets with Network ACLs.


 

Replication 

When replicating from a FlashArray on-premises to a Cloud Block Store instance in an Amazon VPC, ensure that there is network connectivity between the respective sites. Likewise, replication between multiple Cloud Block Store instances requires network connectivity between the instances. More specifically, in order to replicate between a Cloud Block Store instance and a physical FlashArray (or another Cloud Block Store instance), the management and replication ports must communicate. Configure all security groups, network ACLs, and route tables to allow traffic between the two sites for the respective management and replication ports. The following table provides the port number for each interface.

Service Type Firewall Port
Management interfaces 443
Replication interfaces

8117

443 (if using CloudSnap)

For additional details on replication requirements and limits, see the Purity Replication Requirements and Interoperability Matrix.

Important consideration to put in mind before configuration the replication. 

  • AWS network charges may apply for replicating across Availability Zones, VPCs, and/or regions.
  • Egress charges may apply for any data that sent from Cloud Block Store to FlashArray.

FlashArray to CBS 

When replicating between a physical datacenter and the Amazon VPC, you can achieve network connectivity in a number of ways, including AWS Direct Connect or a Site-to-Site-VPN connection.

Replication between Cloud Block Store instances or using CloudSnap can greatly minimize customer cloud spend on ingress/egress charges since snapshots are deduped, compressed and we don’t have to send the same bits twice.

AWS Direct Connect

Direct Connect provides a private connectivity, bypassing the internet to deliver consistent and lower-latency performance. It allows you to establish a dedicated network connection to AWS. This can reduce network costs, increase bandwidth, and provide a more consistent network experience than internet-based connections.

AWS Site-to-Site VPN

Create IPsec VPN connection between from your remote datacenter to VPC over the internet. It has three main components:

  • Site to Site VPN Connection:  A secure connection between your on-premises equipment and your VPCs.
  • Virtual Private Gateway: Gateway for the Amazon side of the Site-to-Site VPN connection. It’s used on the VPC to allow this Site to Site connection.
  • Customer Gateway: Physical or software appliance that you own or manage in your on-premises network to allow this Site to Site connection.
     

CBS to CBS

AWS to AWS 

Similar to Host Connectivity options discussed in previous section. Establishing replication connectivity between two CBS arrays can be done in various ways, but the same concept is applied. The optimal way to ensure the best network performance and regulate with the security control configured, is to use VPC Peering connections. Whether you are peering two CBSs in the same VPC or different region. 

Pure recommends using VPC Peering for replication between CBS array in different VPC inter/intra-region. Put in mind when using ActiveCluster, ensure the latency between two CBS arrays are below 11 ms round trip. 

AWS to Azure

There are various options to establish communication between two cloud providers, those options differ in speed, latency, reliability, service-level agreements (SLAs), complexity, and costs. 

A common and easiest way is to use Site-to-Site VPN tunnels. It is certainly the fastest deployment option, yet it wont provide a high throughout and low latency comparing to AWS DirectConnect or Azure ExpressRoute. 

For more guidelines on how to establish cross communication between two cloud providers check this

Network Monitoring

VPC Flow Logs

VPC flow logs is a feature that is used to capture information about network terrific flowing in and out network interfaces in your VPC. It can Capture inbound and outbound traffic at the VPC level, subnet level, or the network interface level. Logs captured can be published to CloudWatch logs or S3 Athena for advance analysis. 

For more information on how to diagnose and monitor Cloud Block Store instance or EC2 workload traffic, check this guide.