112

Splunk Lookups

In this section we will learn about the Lookups in the Splunk platform. We will also learn what is lookup in Splunk, types of lookups, lookup table files, search command and lookup, operations in lookups.

Lookups

Lookups expand our event data by adding variations of the field-value from the search tables. Splunk software uses lookups to match combinations of a field value in our event data with combinations of a field value in external lookup tables. If Splunk software finds these combinations of a field value in our search table, Splunk software adds the corresponding combinations of field value from the table to the table.

Types of Lookups

There are four types of lookups:

• CSV lookups
• External lookups
• KV Store lookups
• Geospatial lookups

In Splunk Web, we can create lookups through the Settings search pages.

We can customize the lookups by editing the configuration files, whether we have Splunk Enterprise or Splunk Light, and we have access to the Splunk deployment configuration files.

Lookup table files

The files that contain a table of views are known as lookup table file. A standard lookup takes fields out of this table and adds them to our events when matching the respective fields in our events.

All kinds of lookups use lookup tables, but only two types of lookups require us to upload a lookup table file. They are CSV lookups and geospatial lookups. Multiple lookup definitions can make use of a single lookup table file.

For, e.g., say we have a file that provides the definitions of http status fields in the CSV lookup table. If we have events that include http status = 503, we can have a lookup that finds the 503 value for the http status field in the lookup table column and takes the corresponding status description value out of that lookup table.

The lookup then adds status description = Service Non-available, Server Error to any http status = 503 events.

Lookup definitions

The definition of a lookup provides a name for the lookup and a path to find the table. The definitions of lookup can include additional settings such as matching rules, or restrictions on the fields that match the lookup in our Splunk platform. One lookup table can have definitions of multiple lookups.

All types of lookups require a definition of the lookup. But this has to be done after we have created the lookup definition.

Automatic lookups

To apply automatic lookups to all searches at search time, use automatic lookups. After creating an automated search for a description of a search, we do not need to invoke it manually with the look-up button.

Search commands and lookups

After defining and sharing our lookups with apps, we can interact with them via commands that are used for searching:

• lookup: It is used to add fields to the events in the results of the search in the lookup.
• inputlookup: It is used to search the contents of a lookup table.
• outputlookup: It is used to write fields in search results to a static lookup table file or KV store collection that we specify. We cannot use the outputlookupcommand with external lookups.

Lookups and the search-time operations sequence

Search-time operation order

Lookups are sixth in position in the search-time operation order and are processed before event types after calculated fields.

Restrictions

The Splunk software processes lookups in ASCII sort order that belong to a particular host, source, or source type.

Lookup configurations can reference fields by field extractions, field aliases, and calculated fields added to events. They cannot reference the types and tags of the events.

Define a CSV lookup in Splunk Web

The lookups that are file-based and match the field values in the static table represented by a CSV file from our events to field values. The output corresponding field values to our events from the table. They are often called static lookups.

Lookups on CSV are best for small data sets. The general workflow to create a CSV lookup in Splunk Web is to upload a file, share the file for the lookup table, and then create the lookup definition from the file for the lookup table. CSV inline search table files, and inline search descriptions using CSV files, are both types of datasets. See Types and Use Dataset.

Restrictions in CSV file

In the lookups, there are a few restrictions that we can use in the CSV lookup.

• The table should have at least 2 columns in the CSV format. In which, one column represents a field with a set of values in our events. This must include the values that belong to a field. The column need not have the same name as the area for the case. Any column may have equal value in multiple instances, which is a multivalued field in the Splunk lookup.
• In lookups, CSV files cannot have “r” line endings.
• Also, in the Splunk lookups, the CSV files must not have header rows that exceed 4096 characters.

Upload the lookup table file

We must upload the file to our Splunk platform for the use of a lookup table file.

Prerequisites:

• The table must be in .csv or.gz format.
• Our job should have the capability to upload lookup files. We can’t upload lookup table files into Splunk Site without it.

Steps:

1. To go to the Lookups manager tab, select Settings > Lookups.
2. Click on the Add new files next to the Lookup table in the Actions column.
3. Select from the list an App for Destination. The table file for the lookup will be saved in the place where the application resides in the system. Such as: $SPLUNK HOME / etc / users///lookups/.
4. Click on the Select Password to scan for an upload of the CSV file. The Splunk software will save our CSV file in $SPLUNK HOME / etc / system / lookups/ or in $SPLUNK HOME / etc/lookups/ if it belongs to a particular application.
5. Enter the filename for the destination. This is the name it will have on the Splunk server for the lookup table file. If we have a gzipped CSV file uploaded, enter a filename ending in .gz. When uploading a plaintext CSV file, use a filename that ends in .csv.
6. Click on the Save button.

Share a table lookup file with apps:

After uploading the lookup file, tell the Splunk software which applications are allowed to use this file. The default app for this is Launcher. Follow the steps given below to share a table lookup file with apps.

1. Click on the Settings option > Lookups.
2. Now, from the Lookup manager option, click on the Lookup table files.
3. Now, click on the Permissionsin the Sharing column of the lookup we want to share.
4. Now in the Permissions dialog box, under Object should appear in, select All appsto share globally. If we want the lookup to be specific to this app, select This app only. We can also keep our lookup private by selecting Keep private.
5. Click Save.

Create a CSV lookup definition

We must use the lookup table file to create a lookup definition.

Prerequisites

We must share the lookup table file so that Splunk can see it, to create the lookup definition.

1. Select Settings > Lookups.
2. Click Lookup definitions.
3. Click New.
4. Select a Destination app from the list.
   Our file of the lookup table is saved in the place where the application resides.
5. For an instance: $SPLUNK HOME / etc / users/<username>/<app name>/lookups/
6. The lookup definition must be given a unique name.
7. Now click on the File-based option as the lookup Type.
8. Now select the Lookup file from the list. The file extension must be.csv for a CSV lookup,
9. If the CSV file contains time fields, then select the Configure time-based lookup checkbox to make the CSV lookup time-bounded. (Optional Step)
10. Click Save.

Our lookup is defined as a CSV file-based search and appears in the search definitions list.

Share the lookup definition with apps

After creating the definition for the lookup, specify which applications we want to use the definition in.

1. Click on the Settings option > Lookups.
2. Next, click on the Lookup definitions option.
3. In the Lookup definitions list, now click Permissionsin the Sharing column of the lookup definition we want to share.
4. In the permissions dialog box, pick All apps to share globally under Object will appear in. If we only want this app to be specific to the lookup, just select This app. Selecting Keep private will also keep our lookup private.
5. Click Save.

We can use this lookup field to add information to our events from the lookup table file. In a search string, we can use the field lookup with the searchup button. We can automatically set the field lookup to run for information about how to create an automatic lookup.

Handle large CSV lookup tables

Lookup tables are built on a search head and updated. The search head replicates to other search heads a new or modified lookup table or indexers for certain tasks.

• A replication bundle of knowledge. A related bundle of knowledge is also distributed to the indexers when a search head distributes the searches to indexers. The knowledge bundle contains objects of knowledge, such as search tables, which the indexers need to conduct their searches.
• Replication of configuration (search head-clusters). For search head clusters, all other search heads in the cluster are automatically repeated with runtime modifications made on one search head. If a user creates or updates a search table on a cluster search head, that search head replicates the updated table onto the other search heads.
• The updated version of the search table must be replicated by the search head or indexers, or both, depending on the situation. By default, the search head sends any part of the table changes to the whole table each time.

Make the lookup automatic

We can set the lookup to run automatically instead of using the lookup command in our search when we want to apply a field lookup for our events.

---