Deployment Troubleshooting Guide | Pure CBS on Azure

Symptom:

CBS deployment fails. In the Azure Portal -> CBS managed application -> managed resource group (mrg) -> deployment section, the failure occurs when Azure is deploying the final step called ct0-extension and ct1-extension. If this step takes longer than 10 minutes, there's a good chance it will eventually fail.

The deployment typically takes over 45 mins until it eventually fails.

Azure failure message:

Clicking on th Error details link (as seen above), you will see:

{"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.","details":[{"code":"VMExtensionProvisioningTimeout","message":"Provisioning of VM extension cbsnoNAT-ct1-ext has timed out. Extension provisioning has taken too long to complete. The extension last reported \"Plugin enabled\".\r\n\r\nMore information on troubleshooting is available at https://aka.ms/VMExtensionCSELinuxTroubleshoot"}]}

Applies to:

All CBS deployments on Azure.

Cause:

When a CBS deployment fails at this stage, it is usually trying to validate the CBS license with Pure1. There are a few possible reasons for failures at this stage:

• There is no network connection outbound to the internet or Pure1.

Resolution:

• Ensure there is internet access to the internet and Pure1 on the CBS system subnet. This can be achieved by using on of the below options. Detailed information is provided in the CBS Deployment Guide for Azure -> Internet Access section.
  • A firewall (Azure firewall or 3rd party appliance)
  • NAT Gateway
  • CBS subnet routed through on-premises network
• Ensure firewall rules and ports are open for the CBS system subnet to reach Pure1 and pertinent Microsoft Azure public urls. 
  • See the CBS Deployment Guide for Azure -> firewall rules table.
• Ensure service endpoints to Azure Key Vault and Cosmos DB are enabled for the CBS system subnet. See the CBS Deployment Guide for Azure -> Service Endpoint section. 

Symptom:

CBS deployment fails. In the Azure Portal -> CBS managed application -> managed resource group (mrg) -> deployment section, the failure occurs when Azure is deploying the final step called ct0-extension and ct1-extension. If this step takes longer than 10 minutes, there's a good chance it will eventually fail.

This specific deployment fail typically takes about 10 minutes to fail.

Azure failure message:

Clicking on th Error details link (as seen above), you will see a very long error messaging consisting on multiple license retry messages. Eventually it will fail the license activation after about 10 mins.

Small Snippet:

{"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.","details":[{"code":"VMExtensionProvisioningError","message":"VM has reported a failure when processing extension 'CBSTest1-ct0-ext'. Error message: \"Enable failed: failed to execute command: command terminated with exit status=1\n[stdout]\nlicense key activation\nFri Jul 2 14:56:25 PDT 2021 : Retrying check for license key activation\nFri Jul 2 14:56:30 PDT 2021 : Retrying check for license key activation\nFri Jul 2 14:56:35 PDT 2021 : Retrying check for license key activation\nFri Jul 2 14:56:40 PDT 2021 : Retrying check for license key activation\nFri Jul 2 14:56:45 PDT 2021 : Retrying check for license key activation\nFri Jul 2 14:56:50 PDT 2021 : Retrying check for license key activation\nFri Jul 2 14:56:55 PDT 2021 : Retrying check for license key activation\nFri Jul 2 14:57:00 PDT 2021 : Retrying check for license key activation\nFri Jul 2 14:57:05 PDT 2021 : Retrying check for license key activation\nFri Jul 2 14:57:10 PDT 2021 : License activation failed. Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\r100 1095 100 1095 0 0 33181 0 --:--:-- --:--:-- --:--:-- 33181\n % Total % Received % Xferd Average Speed Time

Applies to:

All CBS deployments on Azure.

Cause:

When a CBS deployment fails at this stage, it is usually trying to validate the CBS license with Pure1. There are a few possible reasons for failures at this stage:

• Incorrect license was applied.
• Customer did not register their Azure Subscription ID with in their Pure1 account.

Resolution:

• Ensure the correct license is entered during the deployment steps.
• Make sure the subscription ID is registered with Pure1. See KB article on registering your Azure subscription.

Symptom:

CBS deployment fails. In the Azure Portal -> CBS managed application -> managed resource group (mrg) -> deployment section, the failure occurs when Azure is deploying the final step called ct1-extension. It usually fails after 10 min. 

Azure failure message:

Clicking on th Error details link, you will see:

{"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.","details":[{"code":"VMExtensionProvisioningError","message":"VM has reported a failure when processing extension 'PSarrayName-ct1-ext'. Error message: \"Enable failed: failed to execute command: command terminated with exit status=1\n[stdout]\nartup to finish\nWed 23 Nov 2022 10:16:24 AM EST :  Waiting for startup to finish\nWed 23 Nov 2022 10:16:29 AM EST :  Waiting for startup to finish\nWed 23 Nov 2022 10:16:34 AM EST :  ........ Waiting for startup to finish\nWed 23 Nov 2022 10:21:36 AM EST :  Startup failed\n\n[stderr]\n\"\r\n\r\nMore information on troubleshooting is available at https://aka.ms/VMExtensionCSELinuxTroubleshoot "}]}

Applies to:

All CBS deployments on Azure.

Cause:

When a CBS deployment succeed at ct0-extension and fails at ct1-extension, controller VM ct0 is trying to communicate with ct1 but it fails due to limited or no communication between both VMs. Since both controller VMs reside at the same VNet, The main reason for failures at this stage are applied Network Security Groups (NSG) on the System subnet level. Example of an NSG inbound rule that has limited port open, which will lead to a failure deployment: 

Resolution:

If NSG are used on the System subnet level, ensure that there are no rules with limited port for VNet to VNet communication. 

Symptom:

Azure failure error message:

"The requested operation cannot be performed because the database account <account value> is in the process of being created. Please wait until account creation completes and retry the operation"

Full Azure error message example:

{
    "status": "Failed",
    "error": {
        "code": "ApplianceDeploymentFailed",
        "message": "The operation to create appliance failed. Please check operations of deployment 'anandkPAZExt610V10' under resource group '/subscriptions/5f33229f-5857-48b0-a725-ee5a9266ccc7/resourceGroups/mrg-anandk-PAZ-Ext-6-1-0-V10'. Error message: 'At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.'",
        "details": [
            {
                "code": "Conflict",
                "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource operation completed with terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"Conflict\",\r\n            \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource operation completed with terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"BadRequest\\\",\\r\\n            \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"BadRequest\\\\\\\",\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"The requested operation cannot be performed because the database account 3c122546-97bb-427a-8c9e-984958e7d28d is in the process of being created. Please wait until account creation completes and retry the operation.\\\\\\\\r\\\\\\\\nActivityId: bb29810d-2f4e-476a-91e2-a1b16bef9f56, Microsoft.Azure.Documents.Common/2.11.0\\\\\\\"\\\\r\\\\n}\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
            }
        ]
    }
}

Applies to:

All CBS deployments on Azure.

Resolution:

The deployment failed within Azure's array identity creating the resources. This is a one-time error. Proceed and retry the CBS deployment.

Symptom:

CBS deployment fails. In the CBS managed application -> managed resource group -> deployment section, the failure occurs when Azure is deploying Cosmos DB. 

Azure failure error message:

"The resource provision operation did not complete within the allowed timeout period."

Full Azure failure message example:

    "eventTimestamp": "2021-03-21T04:05:11.5406379Z",
    "id": "/subscriptions/54865ec6-422e-4cd2-b7e9-73933bf1d4a6/resourcegroups/testbed-paz19-array-jenkins-purityinthecloud-root-1616292226/providers/Microsoft.DocumentDB/databaseAccounts/d9007621-42af-4b26-a4ac-f6fac6e86897/events/9266cd8f-0dc4-4ecd-a682-c6c25eb5a4df/ticks/637518963115406379",
    "level": "Error",
    "operationId": "ae1cf8b5-f2c4-474e-b6e6-0b26ecbaa424",
    "operationName": {
        "value": "Microsoft.DocumentDB/databaseAccounts/write",
        "localizedValue": "Update database account"
    },
    "resourceGroupName": "testbed-paz19-array-jenkins-purityinthecloud-root-1616292226",
    "resourceProviderName": {
        "value": "Microsoft.DocumentDB",
        "localizedValue": "Microsoft.DocumentDB"
    },
    "resourceType": {
        "value": "Microsoft.DocumentDB/databaseAccounts",
        "localizedValue": "Microsoft.DocumentDB/databaseAccounts"
    },
    "resourceId": "/subscriptions/54865ec6-422e-4cd2-b7e9-73933bf1d4a6/resourcegroups/testbed-paz19-array-jenkins-purityinthecloud-root-1616292226/providers/Microsoft.DocumentDB/databaseAccounts/d9007621-42af-4b26-a4ac-f6fac6e86897",
    "status": {
        "value": "Failed",
        "localizedValue": "Failed"
    },
    "subStatus": {
        "value": "",
        "localizedValue": ""
    },
    "submissionTimestamp": "2021-03-21T04:06:58.1848547Z",
    "subscriptionId": "54865ec6-422e-4cd2-b7e9-73933bf1d4a6",
    "tenantId": "5a3d1859-f4b7-4151-beae-773895b989fd",
    "properties": {
        "statusCode": "RequestTimeout",
        "statusMessage": "{\"error\":{\"code\":\"ResourceDeploymentFailure\",\"message\":\"The resource provision operation did not complete within the allowed timeout period.\"}}",
        "eventCategory": "Administrative",
        "entity": "/subscriptions/54865ec6-422e-4cd2-b7e9-73933bf1d4a6/resourcegroups/testbed-paz19-array-jenkins-purityinthecloud-root-1616292226/providers/Microsoft.DocumentDB/databaseAccounts/d9007621-42af-4b26-a4ac-f6fac6e86897",
        "message": "Microsoft.DocumentDB/databaseAccounts/write",
        "hierarchy": "5a3d1859-f4b7-4151-beae-773895b989fd/54865ec6-422e-4cd2-b7e9-73933bf1d4a6"
    },

Applies to:

All CBS deployments on Azure prior to March 2021.

Resolution:

This is Azure specific issue related to Cosmos DB team. Cosmos DB suffered some issue which caused customers to hit a timeout error while creating the resource on the East US region.  The Azure product team performed a hotfix and currently, the issue has been mitigated.

Symptom:

When deploying using the Azure CLI, the deployment outputs error about accepting terms and conditions.

Azure failure message:

"You have not accepted the legal terms on this subscription"

Full Azure failure message example:

New-AzResourceGroupDeployment : 10:26:28 AM - Resource Microsoft.Solutions/applications 'gsinghtest' failed with message '{
  "error": {
    "code": "ResourcePurchaseValidationFailed",
    "message": "User failed validation to purchase resources. Error message: 'You have not accepted the legal terms on this subscription: '248fd382-1957-464e-87c3-9b0a759516d9' for this plan. Before the subscription can be used, you need to accept the legal terms of the image. To read and accept legal terms, use the Azure CLI commands described at https://go.microsoft.com/fwlink/?linkid=2110637 or the PowerShell commands available at https://go.microsoft.com/fwlink/?linkid=862451. Alternatively, deploying via the Azure portal provides a UI experience for reading and accepting the legal terms. Offer details: publisher='purestorageinc1578960262525' offer = 'cbsdevelopment-preview', sku = 'pazqaplan1', Correlation Id: '5eefd35c-dfc7-4473-b747-1823288e1aa1'.'"
  }
}'

Cause:

Before deploying CBS on Azure, user needs to accept the terms and condition. This can be done in the Azure UI portal or via Azure CLI.

Resolution:

Steps to accept terms and conditions for a CBS CLI deployment provided in the CBS Deployment Guide for Azure -> Deploying Cloud Block Store via CLI section

Symptom:

Deployment failed during Cosoms DB creation. 

Azure failure message:

"Sorry, we are currently experiencing high demand in this region, and cannot fulfill your request at this time. The access to the region is currently restricted, to request region access for your subscription"

Full Azure failure message example:

{"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.","details":[{"code":"ServiceUnavailable","message":"Database account creation failed. Operation Id: 4eaeb4b5-ac80-4911-8694-fd7eedc01447, Error : Sorry, we are currently experiencing high demand in this region, and cannot fulfill your request at this time. The access to the region is currently restricted, to request region access for your subscription, please follow this link https://aka.ms/cosmosdbquota for more details on how to create a region access request.\r\nActivityId: 990b5350-399c-4475-9268-390ce751f09d, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0"}]}

Cause:

The resources in Azure Cosmos DB have default quotas/limits. However, this might be a case where CBS deployment needs more quota than the default value.

Resolution:

Please follow this link Create Support Quota Request  for how to request quota increase for Azure CosomsDB.

Symptom: 

Deployment fails even before it gets started. 

Azure failure message:

"The given managed resource group did not match that in the existing appliance definition."

Full Azure failure message example:did not match that in the existing appliance definition

{"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.","details":[{"code":"ApplianceManagedResourceGroupMismat85640347bbd1ch","message":"The given managed resource group '/subscriptions/3befd360-cc97-4c57-b412-85640347bbd1/resourceGroups/mrg-pure_storage_cloud_bl-202208' did not match that in the existing appliance definition '/subscriptions/3befd360-cc97-4c57-b412-xxxxxxxxxx/resourceGroups/mrg-pure_storage_cloud_bl-202'."}]} 

Cause: 

A previous failed deployment with the same application name did not got deleted. This causes the mismatch.

Resoution: 

The best way to resolve this issue is by redeploying with different application name. 

Symptom: 

Deployment failed during roleAssignment creation. 

Azure failure message:

"No more role assignments can be created."

Full Azure failure message example:

{"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.","details":[{"code":"RoleAssignmentLimitExceeded","message":"No more role assignments can be created."}]

Cause: 

The number of the available role assignment under the subscription has exceeded the limit. 

Resoution: 

Azure advices to try reduce the number of used role assignment, for more information follow this link Troubleshooting Azure Role Assignment Limit.

To get the number of role assignments, you can view the chart on the Access control (IAM) page in the Azure portal. 

Symptom: 

CBS can not be deployed after selecting Subscription due to the error shown on the Screenshot below. 

Azure failure message:

Cannot subscribe to this offer from Microsoft Azure Cloud Solution providers. Either choose a different subscription or contact your Microsoft Azure Cloud Solution provider for more information.

Full Azure failure message example:

Get-AzMarketplaceTerms: Cannot subscribe to this offer from Microsoft Azure Cloud Solution providers. Either choose a different subscription or contact your Microsoft Azure Cloud Solution provider for more information. Offer details:  publisher=‘purestoragemarketplaceadmin’ offer = ‘pure_storage_cloud_block_store_deployment’, name = ‘cbs_azure_6_2_6’. Correlation Id: '9f276c3c-7875-4922-b71f-1a0ad3cebfc5'.

Applies to:

CBS deployments on Azure Subscription have been provided via Azure Cloud Service Provider.

Cause: 

The Azure Cloud Service Provider (CSP) has not been authorized to resell CBS offers on the Marketplace. 

Resolution:

Pure designated Azure alliance and PM have access to the partner portal in order to authorize Azure Service Providers to resell CBS offerings. Please reach out to cbs-response-team@purestorage.com. 

Symptom:

Azure failure message:

Failed to delete managed application "<CBS app name>". Error: Deletion of resource group "<managed resource group name>" failed as resources with identifiers '<Microsoft resource ID>' could not be deleted. The provisioning state of the resource group will be rolled back. The tracking ID is '<tracking ID>'. Please check audit logs for details.

Example: 

Resolution:

1) This is usually an erroneous error. Occasionally Azure will output this error message that one of the CBS resources cannot be deleted. However, given enough time, Azure will eventually delete it and subsequently also delete the whole CBS managed application.

2) If the CBS managed application or underlying resources within the CBS managed resource group does not delete in a timely manner (usually within 10-20 mins), retry the CBS managed application deletion. If it continues to be unsuccessfully, contact Azure Support or Pure Storage Support.