Pure Cloud Block Store on Azure - Architecture and Underlying Resources
Introduction
This document is intended to help users better understand the architecture of Pure Cloud Block Store (CBS) on Azure. Insight into the design choices is provided to illustrate the meticulous planning and thought poured into the design of Pure Cloud Block Store. As well as listing the underlaying resources of different cloud native service Pure CBS is leveraging. Furthermore, the document details of the CBS instance models at each capacity point are also provided to allow users to make their own cost analysis for their various capacity requirements.
Design Decisions for CBS on Azure
Similarly to Pure Cloud Block Store on AWS, Pure Cloud Block Store on Azure was developed to provide the same unified storage capabilities and experience independent of whether users are running on AWS, Azure, or even a FlashArray on-premises. The CBS design on Azure was much more straight forward compared to AWS mainly because Azure managed disks supports "shared disks" (aka multi-attach) as well as persistent reservations. Still, Pure developers went through careful analysis and endless testing to find the right combination of CPU, memory, network bandwidth, performance, stability, region availability, and cost for the CBS controller VM. Pure developers standardized on D32s_v3 and D64s_v3 for the V10MU-R1 and V20MU-R1 models, respectively. A unique advantage does exist with CBS on Azure. The aforementioned D32s_v3 and D64s_v3 VMs have a high speed temporary storage volume directly attached. This allows for the utilization of Pure's DirectMemory™ caching software to provide an additional performance boost for workloads that have recurring read IO access to the same blocks ranges.
For the NVRAM and backend persistent storage type, Pure evaluated and tested many Azure storage types, including Azure blob storage, Premium SSD, and Ultra Disk. Ultra Disks were ultimately selected for both NVRAM and backend persistent storage due to the low latency requirements of CBS. Customer data is persisted onto the backend Ultra Disks using RAID-HA which provides high availability even in the event of dual concurrent Ultra Disk failures. A noticeable deviation from CBS on AWS is that data does not need to be mirrored to an object store. Azure managed disk provides a Annual Failure Rate (AFR) of zero, which means they will not lose any data stored on Ultra Disks. This ensures that data stored on CBS will have extremely high durability.
As Azure releases newer, faster, more cost-effective resource types, Pure will re-evaluate and integrate any new feature and benefits into the CBS product.
Architecture and Components
The below architecture diagrams shows how all the underlying Azure resources and services are connected to create a single Cloud Block Store instance. Cloud Block Stores's architecture is designed to take the best properties of each of the resources and provide an abstraction layer that gives customers the best possible storage experience.
Controllers
The Purity Operating Environment (POE) runs on the Cloud Block Store controllers, which consist of two Azure VMs. The dual controllers ensure high data accessibility and availability in the event of a single controller (Azure VM) failure. iSCSI connections can receive and transmit IO traffic through both controllers. The Cloud Block Store controllers process the data (deduplicate, compress, and encrypt) before writing to the underlying storage resources.
Read Cache
Each Cloud Block Store controller VM comes with high speed temporary storage. This temp storage accelerates read performance which can be beneficial for various datasets with recurring read workloads.
NVRAM
Cloud Block Store uses high-performance Azure Managed Disks as the NVRAM modules. All host write IOs are initially cached onto the NVRAM modules and immediately acknowledged back to the application host after it has been securely written.
Persistent Storage
- The backend persistent storage consists of 14 Azure Managed Disk. After a host IO is securely written onto the NVRAM modules, data is eventually flushed and striped across the 14 Ultra disks. For host reads, if the requested dataset does not reside in the read cache, it will be read directly from the backend Ultra disks. Azure's Ultra disk durability coupled with Cloud Block Store's RAID-HA protection provides a solution for the most mission critical applications.
CosmosDB
- CosmosDB is used to store metadata and configuration information about a Cloud Block Store instance. Customer data is not stored on CosmosDB.
KeyVault
Cloud Block Store performs in-line encryption prior to persisting data to the backend storage. To perform the encryption, it uses an internal data encryption key that is generated when the Cloud Block Store managed app is initially deployed. To increase security, the data encryption key itself is also encrypted using a customer managed key which is generated and managed by Azure Key Vault. Interaction between the Cloud Block Store instance and Key Vault only occurs during the initial deployment of a Cloud Block Store managed app or when a Cloud Block Store controllers boots up. Key Vault is never in the data path. In addition, since Cloud Block Store is a managed app, users do not have access to the customer managed key within Key Vault. Only the Cloud Block Store managed app can communicate with Key Vault to access the customer managed key.
CBS on Azure Architecture (Base or after Capacity NDU)
CBS Resource Details
This following tables can be used to provide estimates of a CBS instance at various capacity points. You can enter the components below into the Azure Pricing Calculator to get an estimated static cost of a specific CBS model and capacity point. For more accurate comparison of CBS total cost versus to native Azure storage costs, please consult with the Pure Storage account team or Pure Storage account partner to leverage the CBS TCO tool which factors in important factors like data reduction, thin provisioning, etc.
Refer Using the Azure Cost Explorer for Cloud Block Store to learn how to estimate Azure underlying infrastructure cost using Azure Price Calculator.
V20MP2-R2 on Azure Has 13 Capacity Points
Model |
Total Capacity |
Total Number of Controller VMs (Includes Temp Read Cache Disk) |
Total OS boot disks (Controller VM) |
Total Number of Azure Managed Disks for Persistent Storage
|
Total Number of disk for NVRAM
|
Internal Mgmt Load Balancer | CosmosDB | Key Vault |
V20MP2-R2 Base Deployment |
Usable 4.65 TiB (5.1 TB) Effective (4:1) 18.6 TiB (20.5 TB) |
2 x D64s_v3** Read Cache disk: 1 x 1600 GiB per Controller VM |
2 x 128 GiB disks (Premium SSD) |
14 x 512 GiB PremiumV2 SSD 5,714 IOPS per disk* 86 MB/s per disk* |
2 x 64 GiB PremiumV2 SSD 15,000 IOPS per disk* 450 MB/s per disk* |
1 |
6 tables Throughput: 800 RU/s per table |
1 |
Capacity NDU 1 |
Usable 9.45 TiB (10.4 TB) Effective (4:1) 37.8 TiB (41.6 TB) |
2 x D64s_v3** Read Cache disk: 1 x 1600 GiB per Controller VM |
2 x 128 GiB disks (Premium SSD) |
14 x 1 TiB Ultra SSD 5,714 IOPS per disk* 86 MB/s per disk* |
2 x 64 GiB Ultra SSD 15,000 IOPS per disk* 450 MB/s per disk* |
1 |
6 tables Throughput: 800 RU/s per table |
1 |
Capacity NDU 2 |
Usable 19.05 TiB (20.95 TB) Effective (4:1) 76.2 TiB (83.8 TB) |
2 x D64s_v3** Read Cache disk: 1 x 1600 GiB per Controller VM |
2 x 128 GiB disks (Premium SSD) |
14 x 2 TiB Ultra SSD 5,714 IOPS per disk* 86 MB/s per disk* |
2 x 64 GiB Ultra SSD 15,000 IOPS per disk* 450 MB/s per disk* |
1 |
6 tables Throughput: 800 RU/s per table |
1 |
Capacity NDU 3 |
Usable 28.65 TiB (31.51 TB) Effective (4:1) 114.6 TiB (126.1 TB) |
2 x D64s_v3** Read Cache disk: 1 x 1600 GiB per Controller VM |
2 x 128 GiB disks (Premium SSD) |
14 x 3 TiB Ultra SSD 5,714 IOPS per disk* 86 MB/s per disk* |
2 x 64 GiB Ultra SSD 15,000 IOPS per disk* 450 MB/s per disk* |
1 |
6 tables Throughput: 800 RU/s per table |
1 |
Capacity NDU 4 |
Usable 38.74 TiB (42.62 TB) Effective (4:1) 155 TiB (170.5 TB) |
2 x D64s_v3** Read Cache disk: 1 x 1600 GiB per Controller VM |
2 x 128 GiB disks (Premium SSD) |
14 x 4 TiB Ultra SSD 5,714 IOPS per disk* 86 MB/s per disk* |
2 x 64 GiB Ultra SSD 15,000 IOPS per disk* 450 MB/s per disk* |
1 |
6 tables Throughput: 800 RU/s per table |
1 |
Capacity NDU 5 |
Usable 49.54 TiB (54.5 TB) Effective (4:1) 198.2 TiB (218 TB) |
2 x D64s_v3** Read Cache disk: 1 x 1600 GiB per Controller VM |
2 x 128 GiB disks (Premium SSD) |
14 x 5 TiB Ultra SSD 5,714 IOPS per disk* 86 MB/s per disk* |
2 x 64 GiB Ultra SSD 15,000 IOPS per disk* 450 MB/s per disk* |
1 |
6 tables Throughput: 800 RU/s per table |
1 |
Capacity NDU 6 |
Usable 60.34 TiB (66.38 TB) Effective (4:1) 241.4 TiB (265.5 TB) |
2 x D64s_v3** Read Cache disk: 1 x 1600 GiB per Controller VM |
2 x 128 GiB disks (Premium SSD) |
14 x 6 TiB Ultra SSD 6,144 IOPS per disk* 86 MB/s per disk* |
2 x 64 GiB Ultra SSD 15,000 IOPS per disk* 450 MB/s per disk* |
1 |
6 tables Throughput: 800 RU/s per table |
1 |
Capacity NDU 7 |
Usable 71.14 TiB (78.26 TB) Effective (4:1) 284.6 TiB (313 TB) |
2 x D64s_v3** Read Cache disk: 1 x 1600 GiB per Controller VM |
2 x 128 GiB disks (Premium SSD) |
14 x 7 TiB Ultra SSD 7,168 IOPS per disk* 86 MB/s per disk* |
2 x 64 GiB Ultra SSD 15,000 IOPS per disk* 450 MB/s per disk* |
1 |
6 tables Throughput: 800 RU/s per table |
1 |
Capacity NDU 8 |
Usable 81.94 TiB (90.14 TB) Effective (4:1) 327.8 TiB (360.5 TB) |
2 x D64s_v3** Read Cache disk: 1 x 1600 GiB per Controller VM |
2 x 128 GiB disks (Premium SSD) |
14 x 8 TiB Ultra SSD 8,192 IOPS per disk* 86 MB/s per disk* |
2 x 64 GiB Ultra SSD 15,000 IOPS per disk* 450 MB/s per disk* |
1 |
6 tables Throughput: 800 RU/s per table |
1 |
Capacity NDU 9 |
Usable 92.74 TiB (102.02 TB) Effective (4:1) 371 TiB (408.1 TB) |
2 x D64s_v3** Read Cache disk: 1 x 1600 GiB per Controller VM |
2 x 128 GiB disks (Premium SSD) |
14 x 9 TiB Ultra SSD 9,216 IOPS per disk* 86 MB/s per disk* |
2 x 64 GiB Ultra SSD 15,000 IOPS per disk* 450 MB/s per disk* |
1 |
6 tables Throughput: 800 RU/s per table |
1 |
Capacity NDU 10 |
Usable 103.54 TiB (113.9 TB) Effective (4:1) 414.2 TiB (455.6 TB) |
2 x D64s_v3** Read Cache disk: 1 x 1600 GiB per Controller VM |
2 x 128 GiB disks (Premium SSD) |
14 x 10 TiB Ultra SSD 10,240 IOPS per disk* 86 MB/s per disk* |
2 x 64 GiB Ultra SSD 15,000 IOPS per disk* 450 MB/s per disk* |
1 |
6 tables Throughput: 800 RU/s per table |
1 |
Capacity NDU 11 |
Usable 114.34 TiB (125.78 TB) Effective (4:1) 457.4 TiB (503.1 TB) |
2 x D64s_v3** Read Cache disk: 1 x 1600 GiB per Controller VM |
2 x 128 GiB disks (Premium SSD) |
14 x 11 TiB Ultra SSD 11,264 IOPS per disk* 86 MB/s per disk* |
2 x 64 GiB Ultra SSD 15,000 IOPS per disk* 450 MB/s per disk* |
1 |
6 tables Throughput: 800 RU/s per table |
1 |
Capacity NDU 12 |
Usable 125.14 TiB (137.66 TB) Effective (4:1) 500.6 TiB (550.6 TB) |
2 x D64s_v3** Read Cache disk: 1 x 1600 GiB per Controller VM |
2 x 128 GiB disks (Premium SSD) |
14 x 12 TiB Ultra SSD 12,288 IOPS per disk* 86 MB/s per disk* |
2 x 64 GiB Ultra SSD 15,000 IOPS per disk* 450 MB/s per disk* |
1 |
6 tables Throughput: 800 RU/s per table |
1 |
V20MU-R1 on Azure Has 13 Capacity Points
Model |
Total Capacity |
Total Number of Controller VMs (Includes Temp Read Cache Disk) |
Total OS boot disks (Controller VM) |
Total Number of Azure Managed Disks for Persistent Storage
|
Total Number of disk for NVRAM
|
Internal Mgmt Load Balancer | CosmosDB | Key Vault |
V20MU-R1 Base Deployment |
Usable 4.65 TiB (5.1 TB) Effective (4:1) 18.6 TiB (20.5 TB) |
2 x D64s_v3** Read Cache disk: 1 x 1600 GiB per Controller VM |
2 x 128 GiB disks (Premium SSD) |
14 x 512 GiB Ultra SSD 5,714 IOPS per disk* 86 MB/s per disk* |
2 x 64 GiB Ultra SSD 15,000 IOPS per disk* 450 MB/s per disk* |
1 |
6 tables Throughput: 800 RU/s per table |
1 |
Capacity NDU 1 |
Usable 9.45 TiB (10.4 TB) Effective (4:1) 37.8 TiB (41.6 TB) |
2 x D64s_v3** Read Cache disk: 1 x 1600 GiB per Controller VM |
2 x 128 GiB disks (Premium SSD) |
14 x 1 TiB Ultra SSD 5,714 IOPS per disk* 86 MB/s per disk* |
2 x 64 GiB Ultra SSD 15,000 IOPS per disk* 450 MB/s per disk* |
1 |
6 tables Throughput: 800 RU/s per table |
1 |
Capacity NDU 2 |
Usable 19.05 TiB (20.95 TB) Effective (4:1) 76.2 TiB (83.8 TB) |
2 x D64s_v3** Read Cache disk: 1 x 1600 GiB per Controller VM |
2 x 128 GiB disks (Premium SSD) |
14 x 2 TiB Ultra SSD 5,714 IOPS per disk* 86 MB/s per disk* |
2 x 64 GiB Ultra SSD 15,000 IOPS per disk* 450 MB/s per disk* |
1 |
6 tables Throughput: 800 RU/s per table |
1 |
Capacity NDU 3 |
Usable 28.65 TiB (31.51 TB) Effective (4:1) 114.6 TiB (126.1 TB) |
2 x D64s_v3** Read Cache disk: 1 x 1600 GiB per Controller VM |
2 x 128 GiB disks (Premium SSD) |
14 x 3 TiB Ultra SSD 5,714 IOPS per disk* 86 MB/s per disk* |
2 x 64 GiB Ultra SSD 15,000 IOPS per disk* 450 MB/s per disk* |
1 |
6 tables Throughput: 800 RU/s per table |
1 |
Capacity NDU 4 |
Usable 38.74 TiB (42.62 TB) Effective (4:1) 155 TiB (170.5 TB) |
2 x D64s_v3** Read Cache disk: 1 x 1600 GiB per Controller VM |
2 x 128 GiB disks (Premium SSD) |
14 x 4 TiB Ultra SSD 5,714 IOPS per disk* 86 MB/s per disk* |
2 x 64 GiB Ultra SSD 15,000 IOPS per disk* 450 MB/s per disk* |
1 |
6 tables Throughput: 800 RU/s per table |
1 |
Capacity NDU 5 |
Usable 49.54 TiB (54.5 TB) Effective (4:1) 198.2 TiB (218 TB) |
2 x D64s_v3** Read Cache disk: 1 x 1600 GiB per Controller VM |
2 x 128 GiB disks (Premium SSD) |
14 x 5 TiB Ultra SSD 5,714 IOPS per disk* 86 MB/s per disk* |
2 x 64 GiB Ultra SSD 15,000 IOPS per disk* 450 MB/s per disk* |
1 |
6 tables Throughput: 800 RU/s per table |
1 |
Capacity NDU 6 |
Usable 60.34 TiB (66.38 TB) Effective (4:1) 241.4 TiB (265.5 TB) |
2 x D64s_v3** Read Cache disk: 1 x 1600 GiB per Controller VM |
2 x 128 GiB disks (Premium SSD) |
14 x 6 TiB Ultra SSD 6,144 IOPS per disk* 86 MB/s per disk* |
2 x 64 GiB Ultra SSD 15,000 IOPS per disk* 450 MB/s per disk* |
1 |
6 tables Throughput: 800 RU/s per table |
1 |
Capacity NDU 7 |
Usable 71.14 TiB (78.26 TB) Effective (4:1) 284.6 TiB (313 TB) |
2 x D64s_v3** Read Cache disk: 1 x 1600 GiB per Controller VM |
2 x 128 GiB disks (Premium SSD) |
14 x 7 TiB Ultra SSD 7,168 IOPS per disk* 86 MB/s per disk* |
2 x 64 GiB Ultra SSD 15,000 IOPS per disk* 450 MB/s per disk* |
1 |
6 tables Throughput: 800 RU/s per table |
1 |
Capacity NDU 8 |
Usable 81.94 TiB (90.14 TB) Effective (4:1) 327.8 TiB (360.5 TB) |
2 x D64s_v3** Read Cache disk: 1 x 1600 GiB per Controller VM |
2 x 128 GiB disks (Premium SSD) |
14 x 8 TiB Ultra SSD 8,192 IOPS per disk* 86 MB/s per disk* |
2 x 64 GiB Ultra SSD 15,000 IOPS per disk* 450 MB/s per disk* |
1 |
6 tables Throughput: 800 RU/s per table |
1 |
Capacity NDU 9 |
Usable 92.74 TiB (102.02 TB) Effective (4:1) 371 TiB (408.1 TB) |
2 x D64s_v3** Read Cache disk: 1 x 1600 GiB per Controller VM |
2 x 128 GiB disks (Premium SSD) |
14 x 9 TiB Ultra SSD 9,216 IOPS per disk* 86 MB/s per disk* |
2 x 64 GiB Ultra SSD 15,000 IOPS per disk* 450 MB/s per disk* |
1 |
6 tables Throughput: 800 RU/s per table |
1 |
Capacity NDU 10 |
Usable 103.54 TiB (113.9 TB) Effective (4:1) 414.2 TiB (455.6 TB) |
2 x D64s_v3** Read Cache disk: 1 x 1600 GiB per Controller VM |
2 x 128 GiB disks (Premium SSD) |
14 x 10 TiB Ultra SSD 10,240 IOPS per disk* 86 MB/s per disk* |
2 x 64 GiB Ultra SSD 15,000 IOPS per disk* 450 MB/s per disk* |
1 |
6 tables Throughput: 800 RU/s per table |
1 |
Capacity NDU 11 |
Usable 114.34 TiB (125.78 TB) Effective (4:1) 457.4 TiB (503.1 TB) |
2 x D64s_v3** Read Cache disk: 1 x 1600 GiB per Controller VM |
2 x 128 GiB disks (Premium SSD) |
14 x 11 TiB Ultra SSD 11,264 IOPS per disk* 86 MB/s per disk* |
2 x 64 GiB Ultra SSD 15,000 IOPS per disk* 450 MB/s per disk* |
1 |
6 tables Throughput: 800 RU/s per table |
1 |
Capacity NDU 12 |
Usable 125.14 TiB (137.66 TB) Effective (4:1) 500.6 TiB (550.6 TB) |
2 x D64s_v3** Read Cache disk: 1 x 1600 GiB per Controller VM |
2 x 128 GiB disks (Premium SSD) |
14 x 12 TiB Ultra SSD 12,288 IOPS per disk* 86 MB/s per disk* |
2 x 64 GiB Ultra SSD 15,000 IOPS per disk* 450 MB/s per disk* |
1 |
6 tables Throughput: 800 RU/s per table |
1 |
V10MU-R1 on Azure Has 3 Capacity Points
Model: |
Total Capacity |
Total Number of Controller VMs (Includes Temp Read Cache Disk) |
Total OS boot disks (Controller VM) |
Total Number of Azure Managed Disks for Persistent Storage
|
Total Number of disk for NVRAM
|
Internal Mgmt Load Balancer | CosmosDB | Key Vault |
V10MU-R1 Base Deployment |
Usable 2.25 TiB (2.47TB) Effective (4:1) 9 TiB (9.88 TB) |
2 x D32s_v3** Read Cache disk: 1 x 1600 GiB per Controller VM |
2 x 128 GiB disks (Premium SSD) |
14 x 256 GiB Ultra SSD 2,857 IOPS per disk* 55 MB/s per disk* |
2 x 64 GiB Ultra SSD 5,000 IOPS per disk* 150 MB/s per disk* |
1 |
6 tables Throughput: 800 RU/s per table |
1 |
Capacity NDU 1 |
Usable 4.65 TiB (5.1 TB) Effective (4:1) 18.6 TiB (20.5 TB) |
2 x D32s_v3** Read Cache disk: 1 x 1600 GiB per Controller VM |
2 x 128 GiB disks (Premium SSD) |
14 x 512 GiB Ultra SSD 2,857 IOPS per disk* 55 MB/s per disk* |
2 x 64 GiB Ultra SSD 5,000 IOPS per disk* 150 MB/s per disk* |
1 |
6 tables Throughput: 800 RU/s per table |
1 |
Capacity NDU 2 |
Usable 9.45 TiB (10.4 TB) Effective (4:1) 37.8 TiB (41.6 TB) |
2 x D32s_v3** Read Cache disk: 1 x 1600 GiB per Controller VM |
2 x 128 GiB disks (Premium SSD) |
14 x 1 TiB Ultra SSD 2,857 IOPS per disk* 55 MB/s per disk* |
2 x 64 GiB Ultra SSD 5,000 IOPS per disk* 150 MB/s per disk* |
1 |
6 tables Throughput: 800 RU/s per table |
1 |
* The IOPS per disk and MB/s refer to the number of provisioned IOPS and throughput for each Ultra disk. These values do not reflect the total effective IOPS available from Cloud Block Store.
** Pure strongly recommends using Reserve Virtual Machine Instances (rather than Pay-as-you-go) for the underlying controller VMs. Reserve Virtual Machines Instances provide a discount compared to pay-as-you-go pricing. Reserved Virtual Machine Instances are flexible and can easily be exchanged or returned.
CBS Architecture Comparison
CBS AWS | CBS Azure | Notes | |
Capacity (Usable) |
V10A-R1: 6.9 - 13.8 TiB V20A-R1: 27.2 - 57.5 TiB |
V10MU-R1: 2.25 - 9.45 TiB V20MU-R1: 4.65 - 125.14 TiB V20MP-R2: 4.65 - 125.14 TiB |
|
Controllers | Dual | Dual | |
NVRAM |
7 x io2 volumes |
2 x Azure Managed Disks
|
|
NVRAM mirror | Y | N* | Not needed for CBS on Azure since Azure supports an Annual Failure Rate (AFR) of zero for their managed disks. |
Backend Storage |
7 x Virtual Drives in initial base deployment (consisting of EC2 + Instance Store) 14 x Virtual Drives after capacity NDU (consisting of EC2 + Instance Store) 1 x S3 Bucket |
14 x Azure Managed Disks
|
|
Capacity NDU | CBS capacity upgrade consist of adding 7 Virtual Drives | CBS capacity upgrades involve size increases of each backend managed disk. | |
Backend RAID Protection | RAID-HA (5+2) | RAID-HA (12+2) | |
Read Cache | N | Pure DirectMemory™ Cache | CBS VM controllers on Azure have temporary disks directly attached that are used for read cache |
Database for CBS Configuration metadata | DynamoDB | CosmosDB | |
Network |
1 VPC 1-4 Subnets |
1 VNet 1-4 Subnets |
|
Service Endpoints required | Y (DynamoDB, S3) | Y (CosmosDB, KeyVault) | |
Management Virtual IP | Floating Management IP | Management Load Balancer | |
Key Management | AWS Key Management Service (KMS) | Azure Key Vault |
* Azure offers an Annual Failure Rate (AFR) of zero for their managed disks, including Ultra Disks. This means they never lose data that is stored on managed disks. Therefore Pure's confidence is high that there will be no data loss on CBS's NVRAM. However, availability SLAs on Ultra Disks are only 99.9%. To protect against NVRAM availability, NVRAM data is also mirrored onto the CBS controller VM's memory. In the event of an Ultra disk failure and NVRAM is unaccessible, data is immediately de-staged from the controller VM's memory and persisted onto the the backend storage. CBS will continue to function and serve write IO's with the remaining NVRAM. Read IO's are unaffected because NVRAM is not in the read IO data path.