Splunk Removing Index data
In this section, we are going to learn about How to remove index and index data, delete event from the search, who can delete, how can we delete, clusters of delete operations and indexers etc.
Remove indexes and indexed data
We may delete indexed data from the indexer, or even whole indexes. Those are the key possibilities:
- Delete events from searches which follow.
- Delete all data from a single index or from more.
- Remove or deactivate the entire index.
- Delete older data, based on the pension policy.
Note: Removing data is irreversible. If we want to get our data back once we’ve removed data using any of the techniques described in this section, we must re-index the relevant data sources.
Delete events from searches afterwards
The delete command is provided by the Splunk search language to delete event data from subsequent searches.
The Delete command is only available with indexed events. We can’t do it with index metrics.
We cannot execute the delete command to search in real-time. If we attempt to use delete during a real-time scan, an error will appear on Splunk Enterprise.
Note: The delete command in Splunk only deletes the events from searches that follow. The data is left in the index itself.
Who can delete?
The delete command can only be executed by a user having the capability “delete by keyword.” The Splunk Enterprise ships with a special function, “can delete,” which has that (and no other) capability. By nature, the admin function does not have the capability. When we intend to delete index data, it is recommended that we create a special user that we log in to.
How to delete
First, run a search that returns the events which we want to delete. Make sure that the search returns only certain events that we want to delete. After that, pip the search results to the delete button.
For example, if we want to delete the events that we have indexed from a source /fflanda / incoming / cheese.log so that they will no longer appear in searches, do:
1. Disable or delete the source to prevent it from being indexed any more.
2. Check inside our index for events from that source:
3. Look at the results to confirm that the data we want to delete is this.
4. Once we have checked that these are the data that we want to remove, pip the quest to delete:
Note: Replace the forward slashes (/) with backslashes () while running Splunk under Windows.
Piping a search to the delete command labels all the events returned by that search so that it is not retrieved by subsequent searches. When searching, no user (even with admin permissions) will be able to view that data.
Note: Delete piping won’t reclaim disk space. Really, the data is not removed from the index; searches are simply invisible.
The delete command will not change the event metadata, and any searches for metadata will still contain the events, but they are not searchable. The key All indexed data dashboard will still display event counts for the sources, hosts, or sourcetypes that have been removed.
Clusters of delete operations and indexers
The effect of a delete operation gets quickly propagated across all bucket copies in the cluster, typically within a few seconds or minutes. It depends on the cluster load and amount of data and buckets affected by the delete operation. During this propagation interval, a search can return results that have already been deleted.
Additionally, if a peer who had primary copies of the bucket at the time of the delete procedure goes down before the outcomes are propagated, some of the deletes are lost. In that scenario, after reassigning the primary copies from the downed peer, we will re-run the process.
Delete any or more data from one index
To permanently delete indexed data from disk, use the CLI Clean button. The command deletes the data in one or more indexes entirely, depending on whether we have an justification for < index name >. We usually run clean before re-indexing all of our details.
Note: On clustered indexes, the clean command does not work.
How to use the clean command
Here are the key ways to use the Command Clean:
- To access the Clean Help page, type:
- To delete the data permanently from all indexes, type the command:
- And to permanently remove data from a single index, type:
where <index_name> is the name which is given to the targeted index.
Important: Before running the Clean order, we must stop the indexer.
Note: Running the clean command in pre-5.0 versions of Splunk Enterprise caused the indexer to reset the next bucket ID value to 0 for the index. This is no longer the case from version 5.0 on. So, after running the Clean command, if the new bucket ID is 3, the next bucket ID will be 4, not 0.
Examples
The following example takes data off all indexes:
It eliminates data from the inner index and causes Splunk to skip the prompt for confirmation:
Remove an index entirely in Splunk platform
We can use Splunk Web or the CLI to delete an index entirely (and not just the data stored in it) from a non-clustered indexer. We can also edit the indexes.conf directly.
Check through all inputs.conf files on our indexer and any forwarders that send data to the indexer before deleting an index, and make sure that none of the stanzas guide data to the index that we want to remove. For instance, if we want to delete a nogood index, make sure that the following attribute / value pair does not appear in any of our input stanzas: index=nogood. Upon deletion of the index, the indexer will discard any data that is still sent to that index.
Navigate to Settings > Indexes to delete an index in Splunk Internet, and click Delete to the right of the index we want to disable. The intervention eliminates the data directories of the index, and eliminates the stanza of the index from indexes.conf
To delete an index via CLI, execute the order Splunk delete index:
The command removes the data directories of the index, and also removes the stanza of the index from indexes.conf.
When the indexer runs, we can also run splunk delete index. We do not need to reboot the indexer after the completion of the instruction.
The process of deletion of indexes is normally quick but the period depends on several factors:
- The amount of data that will be deleted.
- If we are actually doing heavy writes on the same disk to other indexes.
- We ‘re deleting a significant number of small.tsidx files in the index.
We may also delete an index directly by editing indexes.conf, and removing the stanza of the index. Restart the indexer, then delete directories of the index.
To delete an index from an indexer cluster, we need to edit indexes.conf, and extract the stanza of the index. We can’t use either Splunk Page, or CLI. We edit the file on the master node first, as with all such changes on an indexer cluster, and then apply the changes to the peer nodes.
Configure the peer indexes in an indexer cluster, remove the index directories from each peer node once we have applied the changes to indexes.conf, and the peer nodes have restarted.
Disable an index without removing it
Once an index is deactivated, the indexer will no longer accept data targeted to it. However, disabling an index does not delete index data, and this process is reversible.
In Splunk Web, we can also deactivate an index. To do this, navigate to Settings > Indexes and click Disable, we can see the index that we want to deactivate is to the right of the window. To re-enable the index, press the index button on the right.
We can also deactivate an index with the splunk disable index for CLI command:
splunk enable index command to re-enable that index.
Remove older data based on retirement policy.
The bucket rolls to the “frozen” state when a bucket in an index hits a specified age or when the index expands to a specified size, at which point the indexer removes it from the index. The indexer will transfer it to an archive just before removing the bin, depending on how we configure our retirement policy.