Skip to main content
Pure Technical Services

Splunk not starting with Cold tier on NFS

Currently viewing public documentation. Please login to access the full scope of documentation.

KP_Ext_Announcement.png

Symptoms

Splunk Enterprise software would not start and displays the message 

# $SPLUNK_HOME/bin/splunk start
 
Splunk> The IT Search Engine.
 
Checking prerequisites...
       Checking http port [8000]: open
       Checking mgmt port [8089]: open
       Checking appserver port [127.0.0.1:8065]: open
       Checking kvstore port [8191]: open
       Checking configuration... Done.
       Checking critical directories...   Done
       Checking indexes...
coldPath='/cold/splunk/_introspection/colddb' of index=_introspection on unusable filesystem.
Validating databases (splunkd validatedb) failed with code '1'.  If you cannot resolve the issue(s) above after consulting documentation, please file a case online at http://www.splunk.com/page/submit_issue

Performing locktest or locktool utilities would work fine but Splunk still fails to start.

Applies to

  • Splunk Enterprise version between 8.1.0 and 8.1.2 with cold tier pointing to network filesystem (NFS) from FlashBlade mounted on the Splunk indexers.
  • Both NFS v3 and NFS v4.1

Cause

This is a known Splunk issue introduced in Splunk 8.1.0.  Splunk versions prior to 8.1.0 work fine.

Date filed Issue number Description
2021-01-21 SPL-199608, SPL-200661 Indexer is not starting up because of locktest failure after upgrade to 8.1.0

Workaround:
To startup the indexer, the workaround done was to set OPTIMISTIC_ABOUT_FILE_LOCKING=1 in $SPLUNK_HOME/etc/splunk-launch.conf to bypass locktest for Splunk to startup. Refer to: https://docs.splunk.com/Documentation/Splunk/8.1.0/Troubleshooting/FSLockingIssues

or upgrade to 8.1.3+

Resolution

Upgrade Splunk to 8.1.3 that has the filesystem locking issues fixed.

Date resolved Issue number Description
2021-03-02 SPL-199608, SPL-200661 Indexer is not starting up because of locktest failure after upgrade to 8.1.0