Splunk Enterprise software would not start and displays the message
# $SPLUNK_HOME/bin/splunk start Splunk> The IT Search Engine. Checking prerequisites... Checking http port : open Checking mgmt port : open Checking appserver port [127.0.0.1:8065]: open Checking kvstore port : open Checking configuration... Done. Checking critical directories... Done Checking indexes... coldPath='/cold/splunk/_introspection/colddb' of index=_introspection on unusable filesystem. Validating databases (splunkd validatedb) failed with code '1'. If you cannot resolve the issue(s) above after consulting documentation, please file a case online at http://www.splunk.com/page/submit_issue
Performing locktest or locktool utilities would work fine but Splunk still fails to start.
- Splunk Enterprise version between 8.1.0 and 8.1.2 with cold tier pointing to network filesystem (NFS) from FlashBlade mounted on the Splunk indexers.
- Both NFS v3 and NFS v4.1
This is a known Splunk issue introduced in Splunk 8.1.0. Splunk versions prior to 8.1.0 work fine.
|Date filed||Issue number||Description|
|2021-01-21||SPL-199608, SPL-200661||Indexer is not starting up because of locktest failure after upgrade to 8.1.0
To startup the indexer, the workaround done was to set OPTIMISTIC_ABOUT_FILE_LOCKING=1 in $SPLUNK_HOME/etc/splunk-launch.conf to bypass locktest for Splunk to startup. Refer to: https://docs.splunk.com/Documentation/Splunk/8.1.0/Troubleshooting/FSLockingIssues
or upgrade to 8.1.3+
Upgrade Splunk to 8.1.3 that has the filesystem locking issues fixed.
|Date resolved||Issue number||Description|
|2021-03-02||SPL-199608, SPL-200661||Indexer is not starting up because of locktest failure after upgrade to 8.1.0|