Splunk Series: Feeding Data to Splunk

Splunk Index Time Process

Data ingestion for Splunk is broken down into 3 different phases

  • Input Phase – Data is handled at the source and is usually done by a forwarder
  • Parsing Phase – Handled by the indexer or Heavy Forwarders, the data is broken into events
  • Indexing Phase – The license meter runs as data is been written to disk, this happens before the compression of the data. After this process runs the data cannot be changed

What data can Splunk ingest?

There are many supported files for ingestion by Splunk, here is a quick list

  • Files and Directories
  • Network Data
  • Scripts Output
  • Windows Logs
  • HTTP events

Metadata Settings

When indexing a data source, Splunk assigns metadata values to the source, applies defaults to visualize and organize de data. Visualization and defaults can be changed at the time of the input or later in the process

  • Source
  • Host
  • Sourcetype
  • Index

Sourcetype is what Splunk uses to determine how the data will be presented to Splunk and in further searches

Uploading data to Splunk

Splunk has a way to help ingest data by using APPs that are used for common types of data ingestion, like in the image below
UntitledImage

Also, there are 3 other different ways to upload/present data to Splunk

  • Upload – Local log files Local structured files (e.g. CSV)
  • Monitor – Provides one time or continuous monitoring of files and directories
  • Forward – This one is the main source of input in production environments, collected after the forwarder plugin is installed in the source server

UntitledImage

Source Types

Splunk has default settings for many different types of data

UntitledImage

What is next?

Basic Search

About the Author:

Andres Sarmiento, CCIE # 53520 (Collaboration)
With more than 15 years of experience, Andres is specialized in Unified Communications and Collaboration technologies. Consulted for several companies in South Florida, also Financial Institutions on behalf of Cisco Systems. Andres has been involved in high-profile implementations including Cisco technologies; such as Data Center, UC & Collaboration, Contact Center Express, Routing & Switching, Security and Hosted IPT Service provider infrastructures.

You can follow Andres using Twitter, LinkedIn or Facebook

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.