How to Troubleshoot Splunk Indexer Blocked Queues
Author: Grace Dolby
Date: 04/10/2022
Within the Somerford Support Desk we see many common themes among our customers and the cases that they raise. Recently, we have seen many being caused by blocked queues on Splunk Indexers and Forwarders. This can manifest itself in many ways, such as no data coming into the Splunk indexers and users noting there are no results to their searches, or slowness in data being ingested.
To start with, if you are experiencing issues with your queues, either due to seeing slowness or no data being indexed, or you are seeing blocked queues within the Monitoring Console, we need to isolate where the queues are blocked. We can do this by reviewing the Monitoring Console -> Performance -> Indexing Performance: Instance.
For us to see which is the first queue that is causing an issue, review the order of queues in the pipeline:
For more information on which queue is blocked, you can add the below to your limits.conf:
[default]
Regex_cpu_profiling = true
And restart Splunk.
This will allow you to identify CPU usage by queue and can be seen in the Monitoring Console -> Performance -> Indexing Performance : Advanced
You can then review if a specific source or sourcetype is blocking the rogue queue, and limit the monitoring console performance views to this source or sourcetype. Often, it may be due to regex replacement, which can be seen within this view and within the metrics.log, or due to a throttling of ingestion due to too many events per second.
Once you review which source or sourcetype is the issue, and where in the queue it is becoming blocked, you can use the Troubleshooting blocked queues guide to implement fixes and use the Monitoring Console to review how these are impacting the queues. You may need to re architecture your data inputs to prevent this, such as utilising Heavy Forwarders to take some of the load off of your Indexers.
If any of our customers or potential customers are experiencing issues with blocked queues, or are having indexer performance issues – please do not hesitate to contact our support team and we will be happy to discuss this with you.
