From Wikipedia, the free encyclopedia:
In computing, syslog is a standard for message logging.
vRealize Log Insight, part of comprehensive vRealize suite of applications from VMware is a modern log management and analysis application. With its easy installation, intuitive user interface, high level of customization and integration with other VMware as well as third-party products, vRealize Log Insight presents an attractive solution for customers interested in monitoring and troubleshooting their virtualized, cloud-based environments. To deliver extendibility and simplify third-party product integration, vendors may develop Log Insight Content Packs. This document focuses on the Pure Storage Content Pack, its installation, configuration, and use cases.
Pre-sales consulting engineers, sales engineers and customers who are interested in deploying or already have deployed the Pure Storage FlashArray in VMware vSphere-based virtualized datacenters utilizing Log Insight are the intended audience for this article. No extensive Log Insight knowledge is required however, basic familiarity with FlashArray and Log Insight is necessary. For vRealize detailed documentation refer to https://docs.vmware.com/en/vRealize-Log-Insight/index.html.
VMware vRealize Log Insight provides centralized, real-time log administration for heterogeneous environments that span across physical, virtual and cloud environments. By directing the standard syslog messages to be processed by vRealize Log Insight, administrators can easily gain insight view to their environment for monitoring, analysis and troubleshooting. Custom queries and aggregations add structure to machine-generated log data making it simpler to parse and query, custom dashboards, reports, alerts, interactive searches provide additional tools for enhanced infrastructure operational efficiency. Third-party product integration is achieved by Content Packs.
VMware vRealize Log Insight allows partners to create read-only integration plugins referred to as Content Packs. Content Packs provide additional intelligence into Log Insight and are developed by partners. Generally, Content Packs include custom queries, dashboards, alerts and fields. These custom properties are created in context of the source system sending syslog messages. The important metrics, events and alerts along with field descriptions are already defined in the Content Pack therefore, significantly reducing initial Log Insight configuration.
Pure Storage Content Pack
The Pure Storage Content Pack offers a variety of dashboards, widgets, queries and alerts specifically tailored to provide means of processing FlashArray generated syslog messages and notifying IT administrators of potential as well as existing system issues. Log Insight widgets deliver easy to visualize and decode query results. Widgets are logically group into dashboards. Besides providing customized dashboards, widgets, alerts and queries, Pure Storage Content Pack also includes extracted fields. Extracting and describing fields gives structure to unstructured syslog messages allowing for meaningful message parsing and data analysis. Pure Storage Content Pack is freely available for Pure Storage customers.
- vRealize Log Insight 4.5 or higher
- Purity 4.8 or higher
The Pure Storage Content Pack can be installed directly from vRealize Log Insight Content Pack Marketplace. If the Internet access from vRealize Log Insight is not possible, the Content Pack may be downloaded from VMware Solution Solution Exchange web site at https://marketplace.vmware.com/vsx and installed manually. See Figure 1.
Once Pure Storage Content Pack is successfully installed, it can be located in the left pane of the Dashboard window under Content Pack Dashboards.
FlashArray has a built-in syslog logging facility however, the storage system must be configured to forward messages to remote syslog servers such as vRealize Log Insight. The configuration may be done using command line interface or Graphical User Interface.
Command Line Interface
To forward syslog messages to remote syslog server (vRealize Log Insight) execute purelog command on FlashArray:
purelog create --uri URI NAME
where URI is a the URI of the remote syslog server and NAME is syslog server identification used by Purity.
The URI format is:
The NAME parameter is optional. vRealize Log Insight supports UDP and TCP protocols using port 514 and 1514, 6514 for TCP SSL connections.
purelog create --uri udp://10.21.211.105:514
Syslog configuration can be tested using the following command.
Graphical User Interface
To forward syslog messages to remote syslog server (vRealize Log Insight) using Graphical User Interface navigate:
Settings ➤ System ➤ Syslog Servers ➤ Provide URI for syslog server ➤ '+' (plus sign)
Send a test message by selecting Test button.
Successful transmission needs to be verified using Log Insight. Select Interactive Analytics and in the query field enter flasharray, optionally a second text query may be added by choosing + ADD FILTER with condition contains word test. The query will return pure_event_type test with This is a test message generated by Pure Storage FlashArray text.
Result of successful transmission is shown in Figure 4.
Using Pure Storage Content Pack
The structure of Pure Storage Content Pack provides a logical workflow for FlashArray management. Each dashboard consists of multiple widgets which provide easy to view and scan means of quickly obtaining FlashArray releated data. This data is intended to assist in maintaining, monitoring and troubleshooting vSphere FlashArray-based back-end storage. Besides the quick overview delivered by each widget, vSphere administrator can easily examine detailed information about FlashArray related events reported to Log Insight.
For instance, based on Figure 5, the sn1-405-c12-13 FlashArray reported seventeen critical alerts.
By switching to Alerts dashboard, Critical Alerts By Array and Critical Alerts - Details widgets, detailed data for each critical alert can be obtained by mousing over the appropriate graphic (bar or doughnut) or Interactive Analytics, click Open in Interactive Analytics icon on corresponding widget. See Figure 6 and Figure 7.
Similar methodology can be utilized when examining additional dashboards and widgets.
The list of dashboards and widgets is shown in table 1.
|Overview||Number of Arrays Sending Messages||Number of FlashArrays sending messages to vRealize Log Insight during selected time interval.
To test whether FlashArray is properly configured send a test message using purelog test command.
|Hardware Alerts||Count of critical hardware events during selected time interval to assist in hardware troubleshooting. Alerts should be investigated immediately.|
|Power Failures||Power component hardware failures. The alerts should be investigated immediately. This widget indicates the general location (controller # or shelf #) and specific location (such as pwr1 which is Power Connection 1) Refer to the Pure Storage GUI for the physical location of the failure. The results are sorted by failed power component and FlashArray name <failed power component, array name>.|
|Total Alerts||Total number of alerts sent by FlashArray within the specified time interval. The alerts are classified as Info, Warning, Critical. Additional details are avilable in Alerts dashboard.|
|Alerts||Critical Alerts By Array||Number of Critical events grouped by FlashArray within the specified time interval. See Critical Alerts widget fin Alerts dashboard for additional details.|
|Critical Alerts - Details||Critical alerts within specified time interval grouped by array which need to be investigated immediately. Critical alert message may be seen by mousing over displayed columns.|
|Warning Alerts By Array||Number of Warning events grouped by FlashArray within the specified time interval. See Warning Alerts by widget in Alerts dashboard for additional details.|
|Warning Alerts - Details||Warning alerts within specified time interval grouped by array which need to be investigated as soon as possible. Warning alert message may be seen by mousing over displayed columns.|
|Reminder Alerts||Reminder Alerts within the specified time interval. Reminder alerts are periodically generated by array to alert about the unresolved issues after the initial alert had been already sent.|
|Information Alerts||Information alerts within specified time interval grouped by array. Information alert message may be seen by mousing over displayed columns.|
|Volumes||Volume Operations||Volume management events such as creation, destruction, expansion, shrinking (truncating), recovering, connecting, disconnecting and snapping. The results are grouped by array. For additional details examine widgets in this (Volumes) dashboard.|
|Volume Create||Volume create management events grouped by array.|
|Volume Resize||Volume resize management events grouped by array.|
|Volume Destroy||Volume destroy management events grouped by array.|
|Volume Eradicate||Volume eradicate management events grouped by array. Once the volume is destroyed and may be manually restored or automatically eradicated after 24 hours from the deletion time.|
|Capacity Alerts||Array Capacity Alert are generated when the following conditions are met:
Information: 80 % of capacity
Warning: 90% of capacity
Critical: 100% capacity
|VVols||VVol Create||VVol create management events grouped by array. VVols are automatically created with virtual machine and when virtual machine is powered on. Adding new disks to the virtual machine may also result in creation of subsequent VVols.|
|VVol Disconnect||VVol disconnect events grouped by array. VVold disconnect events may be associated with the virtual machine deletion, virtual disk removal and resizing as well as vMotion.|
|VVol Destroy||VVol Destroy management events may be associated with the virtual machine deletion, virtual disk removal, Storage vMotion.|
|VVol Eradicate||VVol eradicate management events grouped by array. Once the VVol (volume) is destroyed and may be manually restored or automatically eradicated after 24 hours from the deletion time.|
|VVol-Based VIrtual Machines||The number of VVol-based virtual machines created within the specified time interval.|
|Replication||Protection Group Operations||Protection Group management events such as creation, destruction, recovery, renaming, snapping. The results are grouped by array.|
|Delayed Protection Groups||This widget displays Protection Groups that did not meet remote replication schedule at the configured interval. Investigate replication network, replication schedule or lack of free capacity.|
|Single Volume Snap And Copy Operations||Single volume local clone or snap operations during the defined time interval.|
|Protection Group Snap and Copy Operations||Protection group snap and copy operations during the defined time interval.|
|Protection Group Queries||Suggested queries for snapshot or remote replication messages.|
|Recent Protection Group Membership Changes||List of objects (volume,host, host,host group, target FlashArray) that have been added or removed from Protection Group.|
|ActiveCluster||Alerts||Number of ActiveCluster alerts (Information, Warning, Critical) grouped by array within the defined time interval.|
|Mediator Offline||Mediator is responsible for failover and split-brain prevention. Mediator offline need to be investigated immediately. The Mediator Offline alerts can be generated by FlashArray's inability to reach the Internet. Check your Internet connectivity.|
|Local Pod Offline||ActiveCluster Local Pod offline alert. The offline Pod is frozen and no longer writable. When remote array is reachable and a pod is synchronized, the pod will automatically unfreeze and its volumes will become writable.|
|Remote Pod Offline||The Pod remains writable and configurable locally. Data and configuration re-synchronization when communication is restored with the remote array. This alert indicates that this array lost contact with its replication peer.|
|Pod Operations||ActiveCluster POD operations such as add, clone, create, destroy, recover,eradicate, rename.|
|Pod Resynchronizing||ActiveCluster Pod resyncing alerts. During re-synchronization, the data inside the Pod is frozen on the remote FlashArray and is read-only.|
|Pod Resynchronization Complete||ActiveCluster Pod resyncing alerts. Pod has been resynchronized.|
|Auditing||User Operations||Number of user operations performed on FlashArray. Arrays with comparatively large number of user operations may indicate unauthorized access.|
|Invalid Logins By FlashArray And Access Method||Number of invalid logins from end-users or applications sorted by access method (REST, GUI or CLI). Large increases in this number may represent a pattern of unauthorized access attempts or more likely scripts/applications with outdated credentials. High numbers of REST failures indicates typically a script or application.|
|Login Access Method||Distribution of access methods (REST, GUI or CLI) by end user or application logins to the configured FlashArrays. Only initial logins are captured.|
|Host And Host Group Operations||Host and host group configuration changes. Operations like creation, deletion, host add, volume add/removal are reported here. The results are sorted by host group name and FlashArray name.|
|IPs Accessing Array||IP addresses from which FlashArrays have been accessed. This data may be used to determine which hosts connect to FlashArray.|
|User Activity||User audit activity on the FlashArray. All non-read-only operations are shown in quantity for that specific time. This data may be used to examine CLI commands executed on the array.|
Extracted fields describe syslog message fields simplifying the process of searching, identifying and understanding the content of FlashArray generated events. Log Insight provides a comprehensive set of indicators to describe a field which are called regular expressions. Generally, FlashArray messages follow a certain pattern which can be searched using regular expressions. The Pure Storage Content Pack provides forty extracted fields. Details of each described field including regular expression can be viewed from Log Insight Content Packs pane. See Figure 8.
Extracted fields are also displayed in Interactive Analytics where Log Insight automatically highlights associated part of FlashArray syslog message when mousing over the extracted field. See Figure 9.
The following fields are extracted and included in Pure Storage Content Pack:
|pure_access_ip||IP address of the host accessing FlashArray.|
|pure_access_mode||FlashArray access method (CLI, GUI, RestAPI) utilized to manage array.|
|pure_admin_operation||Administrative commands invoking FlashArray configuration changes (not involving hosts and volumes)|
|pure_alert_action||Suggested action to be taken based on condition reported.|
|pure_alert_code||Numeric value associated with an alert. The alert codes may be viewed at https://support.purestorage.com|
|pure_alert_detail||Additional details, when available associated with an alert.|
|pure_alert_message||Message describing alert.|
|pure_alert_severity||Alert severity: Info, Reminder, Warning, Critical. Alerts with critical severity require immediate attention.|
|pure_audit_user||Name of the user accessing or name of the user used to access FlashArray.|
|pure_cli_command||Command executed on FlashArray.|
|pure_controller_name||FlashArray controller: ct0 or ct1.|
|pure_controller_sn||Serial number of the FlashArray controller.|
|pure_delayed_pgroup||Protection group which experienced delayed replication.|
|pure_event_type||FlashArray event type: audit, alert, test.|
|pure_failed_hardware||Failed hardware component such as controller, SSD, power supply.|
|pure_hgroup_name||Host group name.|
|pure_hgroup_operation||Operation executed against the indicated host group.|
|pure_host_name||Name of the altered host in the indicated operation.|
|pure_host_operation||Host operation such as create, connect, disconnect, delete, setattr, remove.|
|pure_hostvol_name||Volume or volume group which has been connected or disconnected from host.|
|pure_object_setattr||FlashArray or FlashArray's object which had its attribute modified.|
|pure_percent_full||Percentage of consumed FlashArray usable capacity. Reported when 80% (Info), 90% (Warning), 100% (Critical) of storage capacity is exceeded.|
|pure_pgroup_name||Protection group name.|
|pure_pgroup_objectname||Protection group object or list of objects managed such as volume, host, host group, target FlashArray.|
|pure_pgroup_operation||Protection group operation such as create, destroy, enable, disable, destroy, eradicate, recover, rename, snamp, schedule, setattr.|
|pure_pod_name||ActiveCluster Pod name.|
|pure_pod_operation||ActiveCluster Pod operation such as create , add, clone, destroy, eradicate, recover, remove, rename, setattr.|
|pure_purity_version||Version of Purity on FlashArray.|
|pure_replicate_frequency||Protection group replication frequency.|
|pure_snapshot_frequency||Protection group snapshot frequency.|
|pure_user_name||User name executing operation on FlashArray.|
|pure_vgroup_operation||Volume group operation such as create, destroy, eradicate, rename, recover.|
|pure_vmvol_name||Name on the FlashArray for VVol-based virtual machine and corresponding volume group.|
|pure_volume_name||Volume name or volume_group/volume_name|
|pure_volume_operation||Volume operation such as create, copy, destroy, disconnect, eradicate, move, recover, remove, rename, setattr, snap.|
|pure_vvol_name||Volume name for VVol-based virtual machine volume.|
|pure_vvol_operation||VVol (volume) operation such as connect, snap, destroy, eradicate.|
|pure_vvolvg_name||Volume group name.|
The Pure Storage Content Pack includes seven alerts. Each alert is based on a specific query and when query requirements are met, vRealize Log Insight sends an email or webhook notifications and generates events in vRealize Operations Manager. When the Content Pack is installed by default, all alerts are disabled. Alerts are managed and can be enabled in Interactive Analytics window by selecting Alerts (bell) icon. See Figure 10.
The following alerts are included in Pure Storage Content Pack:
|FlashArray: Delayed Protection Group||FlashArray Protection Group has been delayed in replication.||Check the replication network (congestion/bandwidth) and rate of change on replicated volumes.|
|FlasArray: Usable Capacity Limit||FlashArray has reached or exceeded 80% of usable capacity.|
|FlashArray: ActiveCluster Alert||Active Cluster Alert.||Examine additional widgets in ActiveCluster dashboard to determine the cause of alerts.|
|FlashArray: ActiveCluster Local Pod is offline||ActiveCluster local Pod has gone offline.||Verify volume status, examine critical and warning FlashArray alerts prior to Pod going offline.|
|FlashArray: ActiveCluster Local Pod is offline||ActiveCluster local Pod has gone offline.||Verify volume status, examine critical and warning FlashArray alerts prior to Pod going offline.|
|FlashArray: ActiveCluster Pods resynchronizing||ActiveCluster Pod or Pods are resynchronizing.||Examine FlashArray logs for any potential issues which could lead to Pods becoming not synchronized.|
|FlashArray: Critical Alert||FlashArray has generated a critical alert.||Examine Alerts dashboard for additional details.|
The Pure Storage FlashArray Content Pack for VMware vRealize Log Insight is an essential management tool for all Pure Storage and Log Insight users. While Log Insight works with the FlashArray without the Content Pack, it is highly encouraged that users deploy and leverage the Content Pack to simplify configuration and accelerate the time required to more effectively use Log Insight.