How to Filter Events From Two Sources with One Common Field
Author: Grace Dolby
Release Date: 26/02/2024
Introduction
Often the power of Splunk can really be shown by correlating data that previously would have been very disparate or difficult to compare together. By leveraging Splunk’s flexible and verbose search language, you can ask any question of the data that you want, which includes searches similar to that of a “join” in SQL.
If you have a common field in two data inputs, you may think it's impossible to compare them in a single search, especially if the field has alternative field names in each data set. However, utilising the OR operator or by leveraging field aliases can make this much simpler.
Using OR
This method is the easiest if your field name is the same, but can also be used in instances where the field is named differently. The key is searching two indexes at the same time with an OR command between, to reduce the performance impact of this search you may also want to include the field name and result wanted (or a * for every value) across the two indexes. Let's look at the example below (please note Somerford Company Emails have been removed):
In the above screenshot we have a search which is using the OR operator to search two separate indexes, although this has performance impacts this is still a very useful way to correlate across fields in different datasets. Our DLP software has a field called user, and the same is included in our CRM monitoring as Owner, as you can see from the above search by using the OR operator we are pulling these two disparate fields into one dataset, we could then do analytics for example number of CRM changes compared to DLP records.
Utilising Field Aliases
Of course, the field is still named something different in each log, (Owner and user), if we want to normalise the field names to be the same you can leverage a field alias. A field alias can be used to normalise the field names of a sourcetype of data, in this example I have created an alias of the field currently called Owner as user, which is the same in our DLP logs.
Note: If you tick the box for overwriting field values the existing fields will no longer be present, we encourage you to only do this once you are sure the field is named as you wish and you will no longer need the original field. Field alias’s will by default be private to you so share this in app or globally if other users will use this.
Now when I search the two datasets together, I can now search for user data by the one field name:
Field Extraction and Preparation
If your fields are not currently extracted correctly to be able to leverage the OR operator or utilise field names, you may need to do some field extraction up front via the UI extractor or writing your own regular expressions. If you would like to learn more about field naming, alias’s or the building blocks of searches we would love to welcome you to our next Advanced Searching and Reporting workshop date!
More Resources like this one:
Case Management for Splunk Enterprise Security Explained—Bite-Size Webinar Series (Part 1)
Virtual Workshops & Webinars - Somerford Events
