Skip to main content
Pure Technical Services

Splunk not starting with Cold tier on NFS

Currently viewing public documentation. Please login to access the full scope of documentation.

Symptoms

Splunk Enterprise software would not start and displays the message 

# $SPLUNK_HOME/bin/splunk start
 
Splunk> The IT Search Engine.
 
Checking prerequisites...
       Checking http port [8000]: open
       Checking mgmt port [8089]: open
       Checking appserver port [127.0.0.1:8065]: open
       Checking kvstore port [8191]: open
       Checking configuration... Done.
       Checking critical directories...   Done
       Checking indexes...
coldPath='/cold/splunk/_introspection/colddb' of index=_introspection on unusable filesystem.
Validating databases (splunkd validatedb) failed with code '1'.  If you cannot resolve the issue(s) above after consulting documentation, please file a case online at http://www.splunk.com/page/submit_issue

Performing locktest or locktool utilities would work fine but Splunk still fails to start.

Applies to

  • Splunk Enterprise version between 8.1.0 and 8.1.2 with cold tier pointing to network filesystem (NFS) from FlashBlade mounted on the Splunk indexers.
  • Both NFS v3 and NFS v4.1

Cause

This is a known Splunk issue introduced in Splunk 8.1.0.  Splunk versions prior to 8.1.0 work fine.

Date filed Issue number Description
2021-01-21 SPL-199608, SPL-200661 Indexer is not starting up because of locktest failure after upgrade to 8.1.0

Workaround:
To startup the indexer, the workaround done was to set OPTIMISTIC_ABOUT_FILE_LOCKING=1 in $SPLUNK_HOME/etc/splunk-launch.conf to bypass locktest for Splunk to startup. Refer to: https://docs.splunk.com/Documentation/Splunk/8.1.0/Troubleshooting/FSLockingIssues

or upgrade to 8.1.3+

Resolution

Upgrade Splunk to 8.1.3 that has the filesystem locking issues fixed.

Date resolved Issue number Description
2021-03-02 SPL-199608, SPL-200661 Indexer is not starting up because of locktest failure after upgrade to 8.1.0