Splunk not starting with Cold tier on NFS
Symptoms
Splunk Enterprise software would not start and displays the message
# $SPLUNK_HOME/bin/splunk start Splunk> The IT Search Engine. Checking prerequisites... Checking http port [8000]: open Checking mgmt port [8089]: open Checking appserver port [127.0.0.1:8065]: open Checking kvstore port [8191]: open Checking configuration... Done. Checking critical directories... Done Checking indexes... coldPath='/cold/splunk/_introspection/colddb' of index=_introspection on unusable filesystem. Validating databases (splunkd validatedb) failed with code '1'. If you cannot resolve the issue(s) above after consulting documentation, please file a case online at http://www.splunk.com/page/submit_issue
Performing locktest or locktool utilities would work fine but Splunk still fails to start.
Applies to
- Splunk Enterprise version between 8.1.0 and 8.1.2 with cold tier pointing to network filesystem (NFS) from FlashBlade mounted on the Splunk indexers.
- Both NFS v3 and NFS v4.1
Cause
This is a known Splunk issue introduced in Splunk 8.1.0. Splunk versions prior to 8.1.0 work fine.
Date filed | Issue number | Description |
---|---|---|
2021-01-21 | SPL-199608, SPL-200661 | Indexer is not starting up because of locktest failure after upgrade to 8.1.0 Workaround: To startup the indexer, the workaround done was to set OPTIMISTIC_ABOUT_FILE_LOCKING=1 in $SPLUNK_HOME/etc/splunk-launch.conf to bypass locktest for Splunk to startup. Refer to: https://docs.splunk.com/Documentation/Splunk/8.1.0/Troubleshooting/FSLockingIssues or upgrade to 8.1.3+ |
Resolution
Upgrade Splunk to 8.1.3 that has the filesystem locking issues fixed.
Date resolved | Issue number | Description |
---|---|---|
2021-03-02 | SPL-199608, SPL-200661 | Indexer is not starting up because of locktest failure after upgrade to 8.1.0 |