Splunk not starting with Cold tier on NFS
Symptoms
Splunk Enterprise software would not start and displays the message
# $SPLUNK_HOME/bin/splunk start
Splunk> The IT Search Engine.
Checking prerequisites...
Checking http port [8000]: open
Checking mgmt port [8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [8191]: open
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...
coldPath='/cold/splunk/_introspection/colddb' of index=_introspection on unusable filesystem.
Validating databases (splunkd validatedb) failed with code '1'. If you cannot resolve the issue(s) above after consulting documentation, please file a case online at http://www.splunk.com/page/submit_issue
Performing locktest or locktool utilities would work fine but Splunk still fails to start.
Applies to
- Splunk Enterprise version between 8.1.0 and 8.1.2 with cold tier pointing to network filesystem (NFS) from FlashBlade mounted on the Splunk indexers.
- Both NFS v3 and NFS v4.1
Cause
This is a known Splunk issue introduced in Splunk 8.1.0. Splunk versions prior to 8.1.0 work fine.
| Date filed | Issue number | Description |
|---|---|---|
| 2021-01-21 | SPL-199608, SPL-200661 | Indexer is not starting up because of locktest failure after upgrade to 8.1.0 Workaround: To startup the indexer, the workaround done was to set OPTIMISTIC_ABOUT_FILE_LOCKING=1 in $SPLUNK_HOME/etc/splunk-launch.conf to bypass locktest for Splunk to startup. Refer to: https://docs.splunk.com/Documentation/Splunk/8.1.0/Troubleshooting/FSLockingIssues or upgrade to 8.1.3+ |
Resolution
Upgrade Splunk to 8.1.3 that has the filesystem locking issues fixed.
| Date resolved | Issue number | Description |
|---|---|---|
| 2021-03-02 | SPL-199608, SPL-200661 | Indexer is not starting up because of locktest failure after upgrade to 8.1.0 |