Veritas Access 3340 Appliance for Long Term Retention of Pure Storage® FlashArray™ Snapshots

EXECUTIVE SUMMARY

Pure Storage® FlashArray™ is an all-flash block storage designed for high performance and varied workloads such as databases, enterprise applications, and virtual environments.  One of the features of FlashArray is the ability to copy volume snapshots to secondary storage for data protection, migration, cloning, and test and development operations. Access Appliance is a complementary cost-optimized storage option for offloading data volume snapshots residing on Pure Storage all-flash storage products. The Access 3340 Appliance is a low-cost, disk-based solution that is easy to manage and designed for long term retention, tape replacement, backup, and archival workloads.  

SCOPE

The purpose of this document is to provide technical details to assist in understanding Access Appliance as a solution for the preservation of FlashArray volume snapshots for long-term retention.  It describes the components of this solution, its value, sizing guidance, and some best practices. It is advised to refer to Veritas and Pure Storage product documentation for installation, configuration and administration of each of the products discussed in this white paper.  

TARGET AUDIENCE

This document is targeted for joint customers, partners, and field personnel interested in learning more about the use of Veritas Access Appliance as a secondary storage for Pure Storage FlashArray volume snapshots. It provides a technical overview of this solution, guidance in sizing, and highlights some best practices. 

FLASHARRAY

Pure Storage FlashArray is an all-flash array (100% NVMe) block storage platform optimized for mission critical (Tier 1) workloads requiring ultra-low latency and high performance. Built-in features include inline data reduction (on average, 5:1), snapshots, replication, encryption, and business continuity. In addition, it has the capability to run virtual machines and containers on the FlashArray. The FlashArray current model is the //X series which ranges from 55 TB to 3 PB of effective capacity, where effective capacity assumes high availability, RAID, and metadata overhead, GB to GiB conversion, data reduction, compression and pattern removal. For more details, refer to the FlashArray//X product web page and  datasheet. 

Integrated and native to FlashArray is the snapshot capability for efficient data protection of its volumes. FlashArray snapshots are immutable, point-in-time images of the contents of one or more volumes. For simplicity and ease of management, FlashArray offers protection groups. FlashArray protection groups provide automatic snapshot and replication scheduling as well as retention options for volume(s). 

Although the amount of storage space consumed by snapshots on FlashArray is minimized by Redirect-On-Write (redirection of all overwrites to new blocks to avoid additional writes or data copy), compression and array-wide deduplication technologies, FlashArray may not be optimal for the long-term retention of its volume snapshots. Snap to NFS and CloudSnap have been introduced with Purity version 5.2 to provide the means of protecting FlashArray volumes and their snapshots by offloading them to another media. Snap to NFS and CloudSnap allow customers to replicate protection group snapshots from FlashArray to any NFS target or AWS S3 for long term retention. This offload snapshot feature requires no additional license to use.

Snap to NFS is also highly available. During the FlashArray controller upgrade procedure or in the event of the controller failure, the offload application which runs in Purity Run environment fails over to the surviving controller and resumes operation from the point when it was interrupted.

ACCESS APPLIANCE

The Veritas Access Appliance is a turn-key storage platform designed for cost optimization and high capacity, making it well suited for long-term retention of FlashArray volume snapshots. Key features of the Access Appliance include:

• Disk-based solution
• Integrated high availability – 2 nodes configured as active/active cluster, hot spares, and redundant power modules, RAID controllers, disk power paths, and disk SAS signals. 
• Multi-protocol support – NFS, CIFS/SMB, FTP and S3.
• Scales up to 2.8 PB (2.5 PiB) of usable storage capacity
• Veritas AutoSupport to provide proactive monitoring and alerting 24x7 on the health of the appliance to reduce risk and quicker resolution.
• Symantec Data Center Security (SDCS) intrusion detection system. SDCS is a real-time monitoring and auditing software. It performs host intrusion detection, file integrity monitoring, configuration monitoring, user access tracking and monitoring, and produces logs and event reports. For more information on the Access Appliance intrusion detection system, refer to the Access Appliance Initial Configuration and Administration Guide. 
• Encryption with an external Key Management System (KMS) to create the keys for encryption.
• Replication of data on an Access Appliance to another offsite Access Appliance for disaster recovery purposes.

The Access Appliance model 3340 is comprised of two clustered nodes and one primary storage shelf and up to three additional expansion storage shelves.  The appliance can scale up to 2,800 TB of usable space as can be seen in Figure 2. Refer to the Access Appliance datasheet for more detailed specifications. 

Figure 2 - Access Appliance Rack Units

The two nodes are clustered in active/active configuration such that each node can handle I/O requests.  Storage shelves are connected to each node and configured with dynamic multipathing, so I/O can be sent to either node for performance and availability.   There are four 10 GbE uplinks (2 per node) available for client connections.  Figure 3 provides a view of these connections with 2 shelves.

Figure 3 - Access Appliance Connections Example

The redundant hardware RAID controller in the primary storage shelf configures and presents the shelves’ physical disks into disk groups (volumes) protected by a RAID 6 storage layout. With a RAID 6 configuration, data with dual parity is striped across the configured volumes.  There are 5 volumes per storage shelf with each volume containing 16 disks as pictured in Figure 4. Each data volume can remain operational despite two concurrent disk failures.

Figure 4 – Access Appliance Storage Shelves Disk Layout 

The nodes run RHEL 7.4 or later as the operating system platform and Access software version 7.3.2 or later. The Access Appliance supports multiple protocols, including NFS, CIFS/SMB, FTP, and S3.  With FlashArray, the Access Appliance is seen as an NFS target.

When using Access as an NFS target, a “cluster file system” type is created where shares on that file system can be exported and then mounted on the FlashArray.  The shares should be exported at the minimum with “rw” and sync option set on Access.  Prior to configuring the share as a target in FlashArray, permissions of the FlashArray user which has a user id and group id of 1000 would need to be set appropriately for the share exported.

For management, the appliance can be managed by the command-line shell referred to as the CLISH and/or a web-based graphical user interface (GUI) where one can provision storage pools, create file systems and provision Access as an S3, NFS, and/or SMB targets. 

SOLUTION INTEGRATION

FlashArray connects to Access Appliance as depicted in Figure 5 and as previously mentioned sends snapshots to the Access Appliance via the NFS protocol.  The hosts can connect to FlashArray using fiber channel (16 or 32 Gb), iSCSI or NVMe/RoCE (RDMA over converged Ethernet).  For Snap to NFS and CloudSnap, the FlashArray replication ports and bond are utilized, which depending on the FlashArray model are 10 or 25 GbE in active/passive mode. In this mode, only one network interface is active, and the other interface will only become active when the first network interface becomes unavailable. FlashArray can communicate with any of the four 10 GbE network interfaces on the Access Appliance side (two 10 GbE per node). For more bandwidth, the Access Appliance network interface can be bonded if desired. 

Figure 5 - Host, FlashArray and Access Appliance Solution Connection Example

SOLUTION SECURITY

At a transport level, FlashArray sends data over dedicated NFS network ports (2049, 111, 4001, and 4045) to Access. The ownership of the NFS exported share belongs to a FlashArray user with a user id and group id of 1000, hence, limiting the access of the contents to this specific user.  For enhanced security, the Access Appliance also has encryption capabilities in conjunction with an external Key Management System (KMS). The appliance encrypts the volume that the “file system” resides on. An external KMS such as IBM KMS is required to create the keys for the encryption.

DATA LAYOUT ON ACCESS APPLIANCE

The appliance contains hardware RAID 6 controllers and inherently does striping with dual parity across disks on the storage shelves for high performance and data durability. Selecting other layouts such as mirrored or erasure coding layout for data protection is not necessary. As a best practice, for backup, archival and long-term retention use cases, it is recommended to configure the Access Appliance using defaults such as clustered file system, simple layout, and block size of 8 KB. For additional performance, the layout can be configured to be stripe.  

NFS PROTOCOL

When using Access Appliance as secondary storage for FlashArray volume snapshots, it is recommended to mount the NFS share with the following options for better performance and minimize data loss.

• sync – data is flushed to disks on server instead of keeping data in the client or server caches. This option should be specified when exporting on Access to minimize data loss in case of failure.
• nfsvers=3 - it was observed that the performance when utilizing NFS version 3 is better when compared to the NFS version 4 protocol.
• rsize=1048576 and wsize=1048576 – the best performance was observed when setting the read and write size values to 1 MB. This value specifies the number of bytes that NFS utilizes to read and write files on the NFS server.  

Since multiple FlashArray can store snapshots on a single Access Appliance, it is recommended as a best practice to use a separate NFS share as the offload target for each FlashArray for segregation and ease of management.  Snapshots from different FlashArray can be seen in each FlashArray utilizing the same share.  For instance, snapshots taken from two FlashArray can be seen in both FlashArray. Additionally, those snapshots can only be removed by the FlashArray where the protection group resides or where the snapshot was created.  Furthermore, it is recommended to use the NFS share solely for FlashArray snapshots and not for other file sharing purposes.  Better performance was also observed when utilizing different file systems for each share.  

The optimum frequency to conduct “Snap to NFS” is generally dependent on size of the data, the network bandwidth, and system resources on the FlashArray. However, per Snap to NFS Overview and Administration documentation, it is recommended to offload data not more than once or twice a day.  

COMPRESSION

Compression is a good feature for better storage utilization. The Access Appliance has the capability to do compression, however, it is not recommended for use with FlashArray snapshots. The snapshots received from FlashArray are already compressed by default.

NETWORK CONNECTIVITY

The Access Appliance has two 10 GbE uplinks per node. Each physical port maps to a virtual IP. Thus, there are four virtual IP addresses.  Always present the virtual IP to clients or in this scenario FlashArray so it will automatically transition to the other node if one node fails, the physical links on one node fails, or the node becomes unreachable.

Bonding is an option on Access Appliance.  Joining or bonding multiple network interfaces on the Access appliances into a single interface improves the bandwidth and network throughput through the combined interface. Bonding is only configurable via the Access command-line interface. As a best practice, the switch that the uplinks of the Access Appliance are connected to should be configured appropriately for the link aggregation. 

For load balancing, virtual IP addresses of the nodes can be manually assigned to applications in a distributed manner. Balancing the load across the nodes and network interfaces improves overall performance especially when directing multiple FlashArray to a single Access Appliance.

MONITORING

It is important to monitor or be aware of the alerts, especially storage utilization warnings and hardware critical alerts. The AutoSupport features assists in this manner, but as a best practice, it is advisable to be pro-active instead of re-active. For instance, once the capacity reaches 70%, it might be a good time to revisit the storage utilization or plan for growth. 

EXAMPLE CAPACITY SIZING OF ACCESS APPLIANCE FOR FLASHARRAY SNAPSHOTS

The Access 3340 Appliance can scale up to 2.8 PB of usable capacity.  Although the FlashArray volume snapshots are compressed, to create portable and efficient snapshot, FlashArray has about a 5% overhead. For this simplistic sizing exercise, capacity is evaluated with an initial source data of 100 TB in size, a known data compression ratio of 2:1, retention of 30 days, and daily change rate of 5%. The sample capacity required for 30 days would be as follows:

Capacity Needed = ((Data Size + (Retention * Change Rate * Data Size) * Compression Rate) + (Data Size * 5%)

     =  ((100 TB + (30 days * 0.05 per day * 100 TB)) * 0.5 compression) + (100 TB * 0.05)

     =   125 TB + 5 TB

     =   130 TB

In addition, when old snapshots expire, the data blocks from these snapshots are not converted into free space immediately, so it conceivable that at some point in time the space consumed by the above snapshots will approach the following:

Capacity Needed = 130 TB / 70% 

= 186 TB

Therefore, capacity on Access Appliance should be calculated to include room for the above overhead.

PERFORMANCE OF ACCESS APPLIANCE FOR FLASHARRAY

Pure and Veritas conducted a joint performance study to understand the performance of sending and retrieving snapshots from FlashArray to and from the Access Appliance via the NFS protocol. Table 1 exhibits the maximum throughput results when sending, retrieving and 50%/50% send and retrieve of the volume snapshots between one FlashArray and an Access 3340 Appliance with one shelf. If using a higher model of FlashArray\, the throughput values can be higher. Based on the “sar” and “vxstat” outputs gathered during the testing, the Access Appliance system (CPU, memory, etc.), network and disk utilization were minimal.   When another FlashArray was added, performance doubled as expected since the Access Appliance system resources was not overloaded.  The overall system utilization on the Access Appliance did increase when two FlashArray were used, however, it did not fully saturate them. Thus, a single Access Appliance is capable of handling multiple FlashArray. Refer to the Appendix section for more details on this performance study.   

Table 1- Solution Performance

• Additional testing was conducted to determine the effects when modifying different parameters such as tunables, utilizing multiple FlashArray, multiple network interfaces, etc. A summary of observations includes:

• Best performance was observed when using a rsize=1048576 and wsize=1048576
• Best performance was observed when using NFS version 3 when compared to NFS version 4 protocol.
• When adding another FlashArray to send data to the Access Appliance, the throughput performance doubled. For instance, with 2 FlashArray the combined throughput observed was 457 MB/s when load was sent to 1 node and 1 network interface of the Access Appliance. 
• Utilizing a higher model of FlashArray also produced higher results. For instance, utilizing a //FA10-R2, the maximum throughput observed for “Snap to NFS” was 300 MB/s.
• Separating the volumes in different protection groups did not affect the results.
• Performance is improved when distributing the workload across the two nodes, multiple network interfaces and different file systems of the Access Appliance.  For example, with 2 FlashArray and when using 1 node, 1 network, and 1 file system on the Access Appliance the throughput was 457 MB/s whereas when using 1 node, 2 filesystems, and 2 network interfaces the combined throughput was 640 MB/s. This is an improvement of 40%. 

REFERENCES

Access 3340 Appliance

Product Documentation 

7.3.2 -  https://sort.veritas.com/documents/doc_details/AAPP/7.3.2/Appliance%203340/ProductGuides/

7.4.2 - https://sort.veritas.com/documents/doc_details/AAPP/7.4.2/Appliance%203340/ProductGuides/

         FlashArray

Snap to NFS Overview and Administration

Snap To NFS Overview and Administration

Flash Recover Snapshots

FlashArray Concepts and Features

SOLUTION DEPLOYMENT

This section describes an example of the configuration and readers are expected to refer to the Veritas Access Appliance and Pure Storage FlashArray product documentation for definitive and specific installation, administration, and configuration details.

The deployment example walks through the configuration of the Access Appliance as an NFS offload target for FlashArray as shown in Figure 10.

Figure 10 - Configuration Used in this Example Deployments

Outline of the steps involved in this example include:

• Configuration and storage provisioning of the Access Appliances as an NFS target.
• Configuration of FlashArray to utilize the Access Appliance as an NFS target. 
• Validation of configuration by conducting a send to NFS and retrieval of snapshot from the Access Appliance.

CONFIGURATION OF FLASHARRAY TO UTILIZE ACCESS AS AN NFS TARGET

Offload application needs to be installed to enable Snap to NFS and CloudSnap features. Thus, for offload application (Snap to NFS) installation contact Pure Storage support at support@purestorage.com

Step 1) Connect to the FlashArray GUI using a web browser, click on Storage on the left pane.

Step 2) Click the three menu dots (vertical ellipses) on the far right of the Offload Targets section and select Connect to NFS Offload Target.  Then, enter the name, IP address or hostname, mount point and mount options as shown below. 

Step 3) Once the connection succeeds, the NFS target details entered will appear under the Offload Target with a green dot.

Step 4) Click on Protection Groups at the top and click on a Protection Groupname (e.g. MongoDB) to get into page to specify the Target.

Step 5) In the Targets section, click on the 3 dots (vertical ellipses) on the far right and select “Add”. Select the NFS target (e.g. accessT) and click Add.

Step 6) The selected target will appear on the Targets section.  

VALIDATION OF CONFIGURATION

A snapshot and replication schedule can be defined as well, however, in this example, a manual snap and retrieval is described.  Refer to the Snap to NFS Overview and Administration on how to setup a scheduled snapshot and/or replication.

TESTING STRATEGY

The performance study conducted involved the following two scenarios:

• Snap to NFS – sending volume(s) snapshots to the Access Appliance
• Snapshot retrieval – retrieving volumes(s) from the Access Appliance

• Topology

The network topology of the test environment is shown in Figure 10.  There were two FlashArray (//M20 and //M20R2) connected to the network utilizing the 10 GbE or 25 GbE replication bonded (active/passive) ports. The four 10 GbE ports of the Access Appliance were also connected to the network. The bulk of the tests used the FlashArray //M20.  The other FlashArray //M20R2 was utilized only to observe how well the Access 3340 Appliance performs when multiple FlashArray are utilized. 

Figure 11 - Network Topology

RESULTS