Skip to main content
Contact Us
Register
Pure Technical Services

vROps: Management Pack Troubleshooting

  1. Last updated
  2. Save as PDF

Currently viewing public documentation. Please login to access the full scope of documentation.

Introduction

VMware vRealize™ Operations™, part of the VMware vRealize suite, provides a unified management experience for IT services on the VMware vCenter™ server.

The Pure Storage FlashArray Management Pack for VMware vRealize Operations Manager (vROps) helps VMware administrators gain better insight into the health of the Pure Storage arrays in a VMware context so that they are able to assess the health of the environment and identify problems and risks. The management pack provides a comprehensive view of the Pure Storage arrays, including their storage object data, capacity and performance statistics, and alerts in the vRealize monitoring environment.

The process of troubleshooting vRealize Operations Manager, and its associated management packs, can be complex and in-depth due to the many different working components throughout the environment. This article is meant to serve as a guide for troubleshooting the Pure Storage FlashArray Management Pack and what to do if you are unable to determine the underlying reason for the failures.

Terminology

Before we get started down the road of troubleshooting, it is important to ensure we are all speaking the same language when specific terminology is used throughout this document.

Component Description
Adapter Adapters are how vROps communicates with other products within the environment. For instance, when utilizing the Pure Storage Management Pack you will configure a separate adapter for each FlashArray you wish to monitor with vROps.
Analytics Nodes Analytics nodes consist of the master node and data nodes (if used).
Master Node The initial, required node in all vRealize Operations Manager deployments. All other nodes are managed by the master node. In a single-node installation, the master node manages itself, has adapters installed on it, and performs all data collection and analysis.
Management Pack A management pack is (often) a third-party solution you install to integrate with various other products within the environment. Management packs can contain dashboards, alerts definitions, reports and the adapters needed to communicate with their respective products for vROps integration.
Remote Collector Node Remote collector nodes gather data but leave the storage and processing of the data to the analytics cluster.
Replica Node To use vRealize Operations Manager high availability (HA), the cluster requires that you convert a data node into a replica of the master node.
Witness Node To use vRealize Operations Manager continuous availability (CA), the cluster requires that you have a witness node. If the network connection between the two fault domains is lost, the witness node acts as a decision maker regarding the availability of vRealize Operations Manager.

Node definitions derived from VMware documentation found here.

Log Files

Anyone who has reviewed vROps log files has seen there are a lot of different files available to review. This can be overwhelming when trying to troubleshoot an issue and often results in a generic "grep all of the logs for an error" type of behavior. While this can be helpful in some circumstances, often times it leads to false positives and down a road not associated with the actual issue. 

The below files are typically the only ones you will find yourself reviewing for a majority of the issue you encounter with the vROps Management Packs. Often times if investigation is required outside of these files Pure Storage or VMware Support will likely be involved.

File Description
collector.log

Stores information about the collector service and individual collection jobs that are executed.

This log file should be reviewed whenever there are problems with collection cycles. Keep in mind this file contains information about the collector service but also logs information from other adapters configured within vROps and can be very busy.
PureStorageAdapter*.log

Stores information about individual collection jobs that are executed for Pure Storage Adapters. 

This log file should be reviewed for any issues during the configuration or data collection with Pure Storage FlashArrays. It is important to note that for every configured FlashArray adapter there is a dedicated log file created to log evens specific to that adapter. For instance, if you had 3 FlashArray adapters configured you would expect three different log files, one for each adapter.
Viewing Logs from the UI
  1. Navigate to the Administration section.
  2. Expand Support in the left hand pane.
  3. Select Logs under the Support section.
  4. Expand the Node that you wish to review the logs on (MASTER is typically highlighted by default)
  5. Double-click the COLLECTOR folder.
  6. Under the COLLECTOR folder double click the adapters folder.
  7. Under the adapters folder double click the PureStorageAdapter folder.
  8. Double click the desired log file to review it's contents.

vrops-logs.png 

While these steps are specific to the PureStorageAdapter log files you can also review the collector logs (under the COLLECTOR folder) and other available vROps log files through the UI.

Viewing Logs from the CLI

If reviewing the log files from the UI is not something you prefer they can also be reviewed from the CLI of the vROps server. The steps to complete this are listed below.

  1. Log into the Master node (or data node) as root via SSH or Console.
  2. Navigate to the following directory to review the collector log files (along with other vROps services):
root@vrops [ ~ ]# cd /storage/vcops/log/
root@vrops [ /storage/vcops/log ]# ls collector.log
collector.log

 NOTE: You can identify all logging associated with Pure Storage FlashArray Adapters by searching for com.purestorage.vrops.adapter in the collector logs.

  1. If you need to review the "PureStorageAdapter" logs then you will navigate to the following location:
root@vrops [ ~ ]# cd /storage/vcops/log/adapters/PureStorageAdapter/
root@vrops [ /storage/vcops/log/adapters/PureStorageAdapter ]# ls
PureStorageAdapter_207565.log    PureStorageAdapter_209706.log    PureStorageAdapter_209708.log    PureStorageAdapter_234574.log    PureStorageAdapter_234925.log
PureStorageAdapter_207565.log.1  PureStorageAdapter_209706.log.1  PureStorageAdapter_209708.log.1  PureStorageAdapter_234574.log.1  PureStorageAdapter_234925.log.1

Enabling Debug Logging

There may come a time when debug logging is required to troubleshoot more complex issues. The steps below will best instruct how to go about this process.

Please do not enable debug logging unless directed to do so by Pure Storage or VMware Support. With debug logging enabled the log file threshold size will quickly be reached and may result in running out of space or not capturing the error you are trying to troubleshoot.

vRealize Operations Manager 7.5 and later have removed the ability to enable debug logging from the UI, thus SSH access is required for this process.

vROps 7.5 - vROps 8.0
  1. Log into the Master node as root via SSH or Console.
  2. Navigate to the following directory:
root@vrops [ ~ ]# cd /usr/lib/vmware-vcops/user/conf/collector/
  1. Once in the aforementioned directory open the following file using your preferred text editor: log4j.properties
  2. With the log4j.properties file opened modify the following line:
Original Line:
log4j.rootLogger=WARN, fileAppender

Change to:
log4j.rootLogger=DEBUG, fileAppender
  1. Once the applicable line has been modified save and close the file. No services are required to be restarted in order for the logging to take effect.
  2. Wait ~ 5-10 minutes for the next collection cycle to kick off and reproduce the issue you are troubleshooting.
  3. Once the issue is reproduced capture a Full Support Bundle (outlined below) and upload to Pure Storage for review.

Once the Full Support Bundle has been captured it is important to revert the log line to its original state to ensure there are no issues with logging or storage.

vROps 8.1 and Later
  1. Log into the Master node as root via SSH or Console.
  2. Navigate to the following directory:
root@vrops [ ~ ]# cd /usr/lib/vmware-vcops/user/conf/collector/
  1. Once in the aforementioned directory open the following file using your preferred text editor: log4j2.properties
  2. With the log4j2.properties file opened modify the following line:
Original Line:
rootLogger.level = WARN

Change to:
rootLogger.level = DEBUG
  1. Once the applicable line has been modified save and close the file. No services are required to be restarted in order for the logging to take effect.
  2. Wait ~ 5-10 minutes for the next collection cycle to kick off and reproduce the issue you are troubleshooting.
  3. Once the issue is reproduced capture a Full Support Bundle (outlined below) and upload to Pure Storage for review.

Once the Full Support Bundle has been captured it is important to revert the log line to its original state to ensure there are no issues with logging or storage.

Support Bundle

Similar to other VMware products there are several easy ways to capture all of the information required to troubleshoot most issues you will experience within your environment. 

There are two different types of support bundles you can capture from vROps as outlined below:

Type Description
Light Support Bundle

The Light Support Bundle option includes 24 hours of logs and basic configuration information for the vROps environment.

The size difference between a Light and Full bundle is significant thus, unless directed by Pure Storage or VMware support, a light support bundle is recommended.
Full Support Bundle

The Full Support Bundle option contains all of the available logs and configuration files.

This option should only be used when directed by Pure Storage or VMware support or when debug logging has been enabled for advanced troubleshooting purposes.
Capturing a Support Bundle in the UI
  1. Navigate to the Administration section.
  2. Expand Support in the left hand pane.
  3. Select Logs under the Support section.
  4. Click on the green "+" symbol to begin generating a new support bundle.

vrops-support-bundle.png

  1. When the Generate Support Bundles prompt pops up select the Light Support Bundle (unless otherwise requested by support).
  2. Select the all of the nodes to capture the data on (unless otherwise requested by support).
  3. Click OK to complete the process.

vrops-generate-bundle.png

  1. Wait for the Status to change from "In Progress" to "Succeeded" and download the new log bundle.
  2. Upload the log bundle to the Pure Storage FTPS by following the Uploading Files to Pure Storage KB.
Capturing a Support Bundle from the CLI
  1. Log into the Master node as root via SSH or Console.
  2. Run the generateSupportBundle.py script using this command:

Generating a Light Support Bundle: 

root@vrops [ ~ ]# python /usr/lib/vmware-vcopssuite/utilities/bin/generateSupportBundle.py -a create -f 1,2,3,4 -m

Generating a Full Support Bundle:

root@vrops [ ~ ]# python /usr/lib/vmware-vcopssuite/utilities/bin/generateSupportBundle.py -a create -f 1,2,3,4
  1. The support bundle will be placed in the /usr/lib/vmware-vcops/support/timestamp directory (where timestamp is the milliseconds since epoch time).
  2. Once the support bundle has been successfully generated capture the ".zip" files under the stored directory and upload them to the Pure Storage FTPS by following the Uploading Files to Pure Storage KB.

Below is an example of what the output looks like following the successful creation of a support bundle. Please see below the ".zip" files mentioned in step 4.

root@vrops-8 [ ~ ]# python /usr/lib/vmware-vcopssuite/utilities/bin/generateSupportBundle.py -a create -f 1,2,3,4 -m
Default Values:
timestamp: -t=1596155572992 (current time in milliseconds)

Option -m=true so only most recent logs will be bundled.
Waiting until support bundle is created

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/vmware-vcops/tomcat-enterprise/webapps/suite-api/docs/bindings/java/vcops-suiteapi-client-1.11-all.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/vmware-vcops/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/vmware-vcops/common/lib/vcops-suiteapi-client-1.11-all.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Tech support bundle is created successfully for timestamp: 1596155572992
------------Generated support bundles----------------
/usr/lib/vmware-vcops/support/1596155572992/1,2,3-light-1596155572992.zip   <------ Upload
/usr/lib/vmware-vcops/support/1596155572992/4-light-1596155572992.zip         <------ Upload
/usr/lib/vmware-vcops/support/1596155572992/summary-1596155572992
/usr/lib/vmware-vcops/support/1596155572992/log

Useful Links

The below links are additional resources available to better understand vROps functionality and troubleshooting.

Add Data Disk Space to a vRealize Operations Manager vApp Node
Safely cleaning up log files in vRealize Operations
vRealize Operations Manager Sizing Guidelines