Skip to main content
Contact Us
Register
Pure Technical Services

How to setup Splunk Multisite SmartStore with FlashBlade

  1. Last updated
  2. Save as PDF

Currently viewing public documentation. Please login to access the full scope of documentation.

This document will cover the object store setup on FlashBlade (FB) for Splunk SmartStore in a multisite environment.  At this time, Splunk Multisite SmartStore deployment on S3 compliant object store like FlashBlade is limited to two sites, with each site hosted in an on-premises data center.  In addition, Pure FlashBlade currently can only support object replication between two FlashBlades.  Hence this limits the Splunk Multisite SmartStore on FlashBlade to two sites.

Prerequisite:

  1. Minimum Purity//FB version required to support Splunk Multisite SmartStore: 3.3.3.

  2. Purity//FB tunables required for bi-directional object replication as well as to enable the "multi-site-writable" bucket option.  The tunables can only be set up by the Pure Storage support.  Please work with the support to set up the tunables before proceeding to the Object Store Replication configuration.

    1. To enable the bi-directional object replication, a Purity//FB tunable, objstore.repl_cycle_check.enabled, should be set to FALSE on both the FlashBlades.

    2. To enable the “multi-site-writable” bucket option, a Purity//FB tunable, objstore.enable_multi_site_writable_buckets, should be set to TRUE on both the FlashBlades.
       

Configuring FlashBlade Object Store Replication

Object Replication set up between two FlashBlades involves the following sequence of activities.

  1. Establishing FlashBlade Array connections

  2. Creating an object account, a user and an access key on the first FB

  3. Creating similar object account, user and import the key on the second FB

  4. Setting up remote credentials on both FlashBlade arrays

  5. Creating a “multi-site-writable” S3 bucket with the same name on both FlashBlades

  6. Configuring the Object Replica link with the above information on both FlashBlades

 

1. Establish FlashBlade Array Connections

The very first step for setting up asynchronous replication between two FlashBlades is to connect the arrays.  This requires a TCP/IP connection (replication network on both FlashBlades), a connection key and the Management IP address of the remote FlashBlade array.  

  1. The connection key can be obtained from the remote/secondary FlashBlade array by running the following CLI command.

pureuser@sn1-fb-a04-20> purearray create --connection-key
Connection Key                          Created                  Expires
T-00a86978-9df4-4ff2-ac31-1de71cfb4589  2020-03-02 17:50:36 PST  2020-03-02 19:50:36 PST

  1. On the source FlashBlade, go to the Array tab under the Storage view and click the + button in the FlashBlade Array Connections to configure the remote FlashBlade.

  1. Provide the Management IP address of the remote FlashBlade, the connection key and click Connect.  

clipboard_e194cae30028599ade36c140a35ae7e3d.png

         The FlashBlade Array Connections should show the connection status and would have auto-discovered the replication address.

The remote FlashBlade GUI also should show the status of the connection and the replication address.  The Management address will be empty on the remote FlashBlade to show that the connection was established from the source FlashBlade.  

The empty Management address is relevant when the FlashBlade arrays have to be disconnected as it can only be disconnected from the source array.  Trying to disconnect from the second/target FlashBlade will throw the following error message.

 

2. Create an Object account, a user and access key on first FB

The hierarchy to host an object in FlashBlade requires an object account, a user with a connection key and an S3 bucket. 

The following steps should be performed on the first FlashBlade.

  1. On the FlashBlade, go to the Object Store tab under the Storage view and click the + (add) button against the Accounts section.


     

  2. Provide an account name and click the Create button.

    clipboard_e32cf54c27588738e774dd1423645cfa4.png
     

  3. Under the Storage > Object Store > Accounts section, click the account that was created in the previous step.  This should show the Users section.  Click the + (add) button to create a new user.


     

  4. Provide a username and click the Create button.  


     

  5. Select the relevant access policies based on your security requirements.  For convenience, select the "pure:policy/full-access" policy and click Add.
     At the minimum, for Splunk Multisite to work, you need the following policies object-delete, object-list, object-read, object-write.

    The version-delete policy is not required as the Splunk indexes.conf parameter remote.s3.supports_versioning = false is a requirement for Splunk Multisite so the objects can be deleted by placing the delete marker rather than deleting by a version.  Deleting by the version will not propagate the deletes to the replicated FlashBlade and you might see object differences between two FlashBlades and hence the parameterremote.s3.supports_versioning should always be set to False.  


  6.  You have the option of creating a new key or importing an existing key.  Go ahead and select “Create a new key” and click the Create button.


     

  7.  Make a note of the access key and secret key.  Optionally use the CSV or JSON button to download them as a file in their respective format.  This access key and secret key will be imported on to the second FlashBlade.

    clipboard_e158f32810abd4b1419e4eca827998d64.png
     

Note: Save or download the key information before closing the popup.  The secret access key can only be accessed during the key creation confirmation and cannot be viewed or accessed after the confirmation popup is closed.


3. Create object account, user and import the key on the second FB

The following steps should be performed on the second FlashBlade.  The steps are similar to the one done on the first FB except we will be importing the key created from the first FB onto the second FB.

  1. On the second FlashBlade, go to the Object Store tab under the Storage view and click the + (add) button against the Accounts section.


     

  2. Provide an account name and click the Create button.

    clipboard_e05cbcb962f487edac8f4f7a38cbc31bb.png
     

  3. Under the Storage > Object Store > Accounts section, click the account that was created in the previous step.  This should show the Users section.  Click the + (add) button to create a new user.


     

  4. Provide a username and click the Create button.  


     

  5. Select the relevant access policies based on your security requirements.  For convenience, select the "pure:policy/full-access" policy and click Add.
     At the minimum, for Splunk Multisite to work, you need the following policies object-delete, object-list, object-read, object-write.

  6.  At this point, select “Import an existing key” option instead of creating a new key. It should show fields to enter the access key and secret key.  Enter the access key and secret key from the first FlashBlade here and click the Import button.



 

 

4. Setup Remote Credentials on both FlashBlades

  1. Go to the Object Replica Links tab under the Protection view on the first FlashBlade GUI. Click the + (add) button against the Remote Credentials for Object Replication.


     

  2. Select the second FlashBlade from the drop-down list, provide a meaningful name and enter the access key and the secret access key.  Create remote credentials by clicking the Create button.

    clipboard_e19cc6ddc2e28eedf8732cf9b7d0bf1cf.png
     

  3. Repeat the above on the second FlashBlade as well and click the Create button.

    clipboard_e859dcfc547a134a85992303dff2bbc38.png
     

 

5. Create an S3 bucket with the same name on both FlashBlades

  1. Go to the Object Store tab under the Storage view on the first FlashBlade.  Click on the object store account (analytics in this example) under which the S3 bucket has to be created. Click the + (add)  button against the Buckets view.


     

  2. Provide a name for the S3 bucket and select the Bucket Type as “multi-site-writable” and click the Create button.


     

  3.  Object replication requires versioning at the target bucket level.  Enable the versioning at the bucket level, by clicking the ellipsis against the bucket and selecting the Enable versioning option

    clipboard_e94b4ab842e30854c9d92fe46214912dd.png
     

  4. Confirm the dialog to enable the version on the S3 bucket by pressing the Enable button.

    clipboard_e8c71673c88221a2bd126a77533e2e118.png
     

  5.  Repeat the same on the second FlashBlade as well.  Versioning at the S3 bucket level is required on both FlashBlades to setup the bi-directional object replica link.

 

6. Configure Object Replica Link on both FlashBlades

  1. To set up the Object replica link for the S3 bucket created in the previous step, click the + button against the Object Replica Links on the source FlashBlade by navigating to the Object Replica Links tab under the Protection section.


     

  2. Select the local bucket from the source FlashBlade that will be the source of the object replication.  Select the Remote FlashBlade and enter the remote bucket (in this case, it will be the same bucket name).  Select the remote credential we just created from the previous steps and click the Create button to establish the uni-directional replication from the first FlashBlade to the second FlashBlade.


     

  3.  The status column should show the current status of the object replica link along with the recovery point timeline.


     

  4.  Repeat the above steps on the second FlashBlade as well to set up the reverse directional replication.  This is required in case of the first FlashBlade failure where Splunk indexers will be switched over to the second FlashBlade until the first source FlashBlade comes back online.


 

Configure Splunk level settings 

For Splunk Multisite SmartStore requirements and pre-requisites please check Splunk’s documentation.

Please see the “Splunk SmartStore Architecture” section within the Best Practices for Splunk on Pure Storage and configure the remote storage sections within the indexes.conf to point to the FlashBlade. 

1. Configure Replication Timeout

The parameter remote_storage_upload_timeout under [clustering] stanza on the server.conf for all the indexers across the site should be set to a time (in seconds) higher than the maximum replication lag time between the two FlashBlades.  The recommended settings is 600 (10 minutes).  This setting should be updated to a higher value if you notice the replication lag goes beyond 600 seconds.