Best Practices for Splunk on Pure Storage

Bucket Size

Splunk has predefined sizes for the bucket that can be configured under the maxDataSize parameter in indexes.conf as

maxDataSize = <positive integer> | auto | auto_high_volume

Default is “auto” at 750MB whereas auto_high_volume is 10GB on 64-bit systems and 1GB on 32-bit systems.

The general recommendation by Splunk for a high volume environment is to set the bucket size to auto_high_volume but for Splunk SmartStore indexes, the specific recommendation is to use “auto” (750MB) or lower. This is to avoid timeouts when downloading big sized buckets from the remote object store back to the cache.

Recommended setting:

maxDataSize = auto

TSIDX Reduction

SmartStore doesn’t support TSIDX reduction. Do not set the parameter enableTsidxReduction to “true” for SmartStore indexes.

Recommended setting:

enableTsidxReduction: false

Bloom Filters

Bloom filters play a key role with SmartStore in reducing the download of tsidx data from the remote object store to the cache. Do not set the parameter createBloomfilter to “false.”

Recommended setting:

createBloomfilter: true

Versioning

FlashBlade supports versioning which is recommended by SmartStore to protect against any accidental deletion.  Splunk data is generally deleted when it surpasses the configured data retention period.  Setting this parameter to false on S3 storage like FlashBlade that supports versioning allows Splunk to put a delete marker on the objects rather than physically deleting them which makes it possible to protect against the accidental deletion.  If this parameter is set to true, which is the default setting, all versions of the data are deleted permanently by Splunk SmartStore when it ages out and cannot be recovered.

Recommended setting:

remote.s3.supports_versioning = false 

If protection against any accidental deletion is required, it is imperative that the versioning setting is enabled at the FlashBlade bucket level upon creation as the default is no versioning.  If accidental deletion protection is not required, the versioning at the FlashBlade bucket level can be left at default (none).  The following picture shows how to enable the versioning of a bucket through FB GUI.

In case if the Purity//FB version (below 3.0) doesn’t support the online enablement of the version, use the following AWS CLI command to enable the bucket versioning. 

aws s3api put-bucket-versioning --bucket <bucket-name> --versioning-configuration Status=Enabled

Space Reclamation

As the parameter remote.s3.supports_versioning is set to false and if the versioning is enabled at the FlashBlade bucket level, the data is not physically removed when data ages out.  Hence it is recommended to set a lifecycle policy at the FlashBlade S3 bucket level to physically remove the deleted data and reclaim the space. 

Note, if the versioning at FlashBlade bucket level is not enabled but remote.s3.supports_versioning is set to false, any object deletes will physically remove the object.

Starting Purity//FB 3.1 release, the lifecycle policy can be set through the GUI.  

To set the policy, select the account under the Object Store and click on the bucket.  It should bring up the page which should have the Lifecycle Rules option.

Click the + symbol on the right against the Lifecycle Rules and specify a rule name and enter your desired days to keep the previous versions before they are removed physically.  In the example below, we have created a rule named rule1 with 3 days to keep the previous versions.  After 3 days, the previous versions of the objects are removed.  Please choose the days for "Keep Previous Version For" based on your requirements.  The minimum you can configure is 1 day.

Following is a sample python code that can be used to set the lifecycle policy of a given bucket in a FlashBlade.  This code will remove all noncurrent versions (or previous versions) of the objects (deleted or overwritten objects), say after 3 days.  Please update the value for NoncurrentDays as per your requirement.  

import boto3
s3 = boto3.resource(service_name='s3', use_ssl=False,
      aws_access_key_id='<access_key>',
      aws_secret_access_key='secret_key',
      endpoint_url='http://<FB data-vip>')

s3.meta.client.put_bucket_lifecycle_configuration (
  Bucket='<bucket-name>',
  LifecycleConfiguration={
    'Rules': [
      { 'ID' : 'rule1',
        'Filter' : {},
        'Status' : 'Enabled',
        'NoncurrentVersionExpiration': { 'NoncurrentDays': 3 },
      } 
     ] 
    } 
   )

Multi-part upload/download

FlashBlade supports multipart upload and download and the default setting of 128MB should be good enough and recommended not to modify unless the new value has been proven to improve throughput.

List Object Version

FlashBlade supports objects listing version V2 which is much more performant than V1.  To improve performance when Splunk is dealing with objects, V2 is highly recommended.

Recommended setting:

remote.s3.list_objects_version = v2

URL Version

The parameter remote.s3.url_version is used for parsing the endpoint and communicating with the remote storage. The parameter allows options v1 or v2. 

In v1, the bucket is the first element of the path like mydomain.com/bucket/remaining/path.

In v2, the bucket is the outermost element of the subdomain like bucket.mydomain.com/remaining/path.

While FlashBlade can support the use of either version, we have noticed using v2 with Splunk results in inadvertent effects like objects not getting deleted or Splunk command line with rfs not working. Hence the recommendation is to not set the parameter which defaults to v1.

eviction_policy

Splunk recommends not to change the default eviction policy of lru which evicts the buckets that are least recently used.  

max_cache_size

Specify the maximum size for the disk partition that hosts the cache in megabytes.  This setting is applied at an indexer level and not the maximum cache size across the cluster.  When the occupied space of the cache exceeds the max_cache_size, or falls below the sum of minFreeSpace and eviction_padding, the cache manager will start to evict the data.

hotlist_recency_secs

Splunk SmartStore eviction policy generally favors the least recently searched buckets meaning the cache manager will keep the buckets that are searched recently and evict the buckets that are least recently searched even if the bucket was recently created. 

If most of your searches are on the recently ingested data, it makes more sense to protect this data from being evicted using the hotlist_recency_secs parameter.  This parameter sets the cache retention period based on the bucket’s age (aka recency) of the warm buckets in the cache and helps to protect the recent buckets over other buckets.  This setting overrides the eviction policy.

The recency or the bucket age is determined by the interval between the bucket’s latest time and the current time.  As the name implies, the setting is in seconds and the default is 86400 seconds or 1 day.  The CacheManager will not evict the buckets until they reach this configured setting unless all other buckets have already been evicted.

Setting can be at an index level or at the global level within the indexes.conf file but the recommendation is to set this parameter at an index level to favor protecting data in critical indexes over non-critical indexes.

For optimal functionality of cache eviction, set this parameter in consideration with the max_cache_size settings.  Do not set a value for hotlist_recency_secs that would require cache size beyond the max_cache_size value could provide as this can impact the cache eviction functionality. 

For example, if the daily ingest adds 100GB of new buckets daily, a cache size of 500GB can only hold five days of recent data, and hence any hotlist_recency_secs over 5 days would impact the cache eviction to work optimally.   Alternatively, if your search is always within the last 30 days and limited to the data ingested within the last 30 days, set hotlist_recency_secs to 2592000 seconds or 30 days and make sure the max_cache_size can hold 30 or more days of daily ingest data.

Recommended setting:

Please set the hotlist_recency_secs parameter at the index level for critical indexes in indexes.conf to protect the data in the cache from eviction based on the required age and in alignment with the max_cache_size settings.  

hotlist_bloom_filter_recency_hours

Similar to hotlist_recency_secs, the hotlist_bloom_filter_recency_hours parameter protects the metadata files like bloomfilter from eviction.  The use of bloom filters during searches avoids the need to download larger bucket objects like the rawdata journal file or the time series index files (tsidx) from the remote object storage.

The default setting is 360 hours or 15 days.  With this setting, the cache manager will defer eviction of smaller files like bloomfilter until the interval between the bucket’s latest time and the current time exceeds this setting. If the searches are limited to the recently ingested data within say last n days, set this parameter for all the critical indexes to the hour that corresponds to n days.  If the search is limited to the last 30 days, set this parameter to 720.

Recommended setting:

Please set the hotlist_bloom_filter_recency_hours parameter at the index level for critical indexes in indexes.conf to protect the data smaller metadata files in the cache from eviction based on the required age.