You are on page 1of 125

Using Splunk 5.

Document usage guidelines


Should be used only for enrolled students
Not meant to be a self-paced document
Not for distribution

Listen to your data.

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Course goals
Using Splunk Web, run searches and save results
Create and use knowledge objects, such as:
- Saved searches
- Custom field extractions
- Tags
- Event types
- Views (Dashboards)

Create reports
Find out where and how to get help
Listen to your data.

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Course outline
1. Starting Searches
2. Saving Results and Searches
3. Using Fields
4. Creating and Using Tags and Event Types
5. Creating Alerts
6. Creating Reports and Dashboards

Listen to your data.

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Section 1:
Start Searching

Listen to your data.

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Section objectives
Describe Splunk and its uses
Describe the Search app
Run basic searches
Identify the contents of search results
Use the output of a search to refine your search
Control a search job
Set the time range of a search

Listen to your data.

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

One Splunk. Many uses.

Listen to your data.

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

What are Apps?


Apps allow different workspaces,
tailored to a specific use case or user
role to share a single Splunk instance
- You are using an app at all times
- Your Splunk administrator may assign a

default app for you

This class focuses on the Splunk


Search app
- It is the default app for the Student labs

Additional apps can be added to your


Splunk instance from Splunkbase
Listen to your data.

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Describing knowledge objects


Splunk knowledge gives you different ways to
interpret, classify, enrich, and normalize your event
Create knowledge objects to capture knowledge
and add value to your data
- Can be reused and shared

Knowledge objects include:


- Saved searches
- Tags
- Event types
- Views (dashboards)
- And more
Listen to your data.

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Users and roles


Splunk users are assigned roles
- Roles determine capabilities and data

access

admin

Out of the box, there are 3 roles:


- Admin
- Power

power

- User

This class utilizes the Power role


Splunk administrators can create other
roles
Listen to your data.

10

user
Using Splunk 5.0
Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

User settings
To display and edit your user
settings, click your name in the
main menu
- The Time zone setting allows you to

view search results in your own


time zone

Listen to your data.

11

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Describing the Search app


Provides a default interface for searching and analyzing data
Enables you to add knowledge, build reports, and create alerts
and dashboards
Provides use across many areas of IT including application
management, operations management, security, and compliance

Listen to your data.

12

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Search app Summary view


current app

app navigation
current view

search bar
time range
picker

global stats

start
search

data sources

Listen to your data.

13

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Sources, sourcetypes, hosts


Source
- name of the file, stream, or

other input

Sourcetype
- specific data type or data

format

Host
- hostname, IP address, or

name of network host from


which the events originated
Listen to your data.

14

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Events
Searches return events
In Splunk, an event is a single
piece of data, such as a record
in a log file or other data input
Splunk breaks up input data
into individual events and
gives each a timestamp, host,
source, and sourcetype

Listen to your data.

15

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Everything is searchable
* wildcard supported
Search terms are case
insensitive
Booleans AND, OR, NOT

fail*
fail* nfs

error OR 404

- Must be uppercase
- Implied AND between terms
- Use () for complex searches

Quote phrases

Listen to your data.

error OR failed OR (sourcetype=access_*(500 OR 503))


"login failure"

16

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Search assistant
Quick reference for Splunk search language that updates as you type
- Includes links to documentation
- Shows matching searches, matching terms, and examples

updates as you type


shows examples and
help

Matching search terms


and data values
toggle off / on
Listen to your data.

17

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Search results
Matching results are
returned immediately
- Displayed in reverse

chronological order
- Matching search terms
are highlighted

Listen to your data.

18

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

search mode
time range picker

timeline
paginator
Fields sidebar

event data
timestamp

selected fields

search terms highlighted


19

Search mode
Three modes
1. Smart [default]
2. Fast Performance over completeness
3. Verbose Completeness over performance
You learn more about search mode in
Section 3: Using Fields

Listen to your data.

20

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Navigating search results


Mouse over search results
- Keywords and parts of

keywords are highlighted

To add a term to the


search, click it
- AND is implied
- To remove, click again

To exclude a term from a


search, alt+click it
- Adds NOT [term] to search
Listen to your data.

21

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Selecting search time range


First time you log into
Splunk, by default, it
searches All time
- Can consume a great deal

of resources
- Ideal for looking at long
term patterns, such as,
advanced persistent threat

To narrow your search,


use the time range
picker
Listen to your data.

22

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Viewing results within a time range


Timeline shows distribution of events over time
- Mouse over for details
return

drill down

Listen to your data.

23

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Modifying time range with timeline


To select a specific
time range, click a bar
in the timeline and
drag across a series
of bars
This action does not
re-run the search, it
simply filters the
current search results

Listen to your data.

24

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Identifying other time line controls


Hide
- Hides or shows the timeline

Zoom out
- Expands the time focus &

re-runs the search

Zoom to selection
- Narrows the time focus &

re-runs the search

Deselect
- If in a drill down, returns to the

original time frame


- Otherwise, grayed out
Listen to your data.

25

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Selecting a specific time


For specific date or
relative time ranges,
use custom time

Listen to your data.

26

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Describing real-time searches


Standard searches
present a static snapshot
of event information
Real-time searches
- Constantly update the

timeline with live data


- Use raw data streams as
new matching events are
scanned

Listen to your data.

27

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Selecting real-time search ranges


From time range picker
1.
2.

Select Real-time
Select a time window

Events returned match


your search within your
selected time window
- Value updates constantly

as new matching events


are scanned

Listen to your data.

28

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Search actions
Every search is a job
Available actions are:
Send to background
Pause [toggles to resume]
Finalize
Cancel

Listen to your data.

29

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Lab 1
Log into the Search app
Perform a search and remove unwanted events from results
Change the search time range
Use the flash timeline
Drill-down into results

Listen to your data.

30

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Section 2: Saving
Results and Searches

Listen to your data.

31

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Section objectives
Export search results
Save and share search results
Save searches
Schedule searches

Listen to your data.

32

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Exporting search results


Export results to a file using the Export link
- Select a file format and max number of results to export
- Results are saved in your Download folder

Listen to your data.

33

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Saving results
1. From the Save menu,
select Save results
2. From the jobs manager,
manage the results

Listen to your data.

34

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Sharing results
Can save and share your results with other Splunk users
Generates a link you can copy and paste anywhere
Accessible in the Jobs Manager

Listen to your data.

35

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Print or PDF search results

Use the print icon to print or


PDF search results
- Formats the results for printing
- Invokes browsers native print

functionality

Listen to your data.

36

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Saving search criteria


From the Save menu
1.
2.
3.

Select Save search


Name the search
Keep search private or share it

Listen to your data.

37

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Saving a search set permissions


All knowledge objects you create in Splunk
have associated permissions
- To share knowledge objects, must have at least

Power role privileges

Keep search private [default]


- Only the creator can access and run the search

Share as read-only to all users of current app


- All Search app users can access and run the search
- More specific permissions can be set using Manager

Listen to your data.

38

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Saving a search finish


To save the search, click Finish
The confirmation box includes a link to
Manager > Searches and Reports
where you can modify the search

Listen to your data.

39

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Running a saved search


Run saved searches from the Searches and Reports menu
- Lists all searches you have permission to run
- There are three built-in sub-menus, Error, Inputs, and Admin
- Searches whose name one of these words are automatically added to the

appropriate sub-menu
- A green dot indicates a private search

Listen to your data.

40

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Managing saved searches


Can edit the search, permissions, enable/disable, run, clone, move, or
delete

Listen to your data.

41

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Using scheduled searches


Scheduled searches are useful for:
- Monthly, weekly, daily executive/managerial roll up reports
- Dashboard performance
- Automatically sending reports via email
- Adding reports to RSS feeds

Listen to your data.

42

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Creating a scheduled search


First create your search
Select Scheduled search from the Create menu

Listen to your data.

43

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Scheduling a search define schedule


Name the search
From the Schedule menu, select
how often the search should run
Define the time range of the search
- Can select the schedule to be relative

to the time range of the search


- Example: a daily schedule could have
a search time range of -24h@h or
-1d@d

Listen to your data.

44

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Scheduling a search define actions


Send email
- Enter addresses, separated by a comma
- Keep the default subject or edit the $name$

variable automatically places the search name in


the subject line
- Include results inline, PDF, or in .csv format
- Once search is scheduled, can add RSS feed

Run a script
- Enter the name of the script file
- All scripts must be in Splunks bin/script directory
- Administrators have access to this location
Listen to your data.

45

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Scheduling a search set permissions and save


As with any knowledge object you create, you can choose to keep
private or share as read-only to all users of the current app
Clicking Finish saves the scheduled search

Listen to your data.

46

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Editing a saved search set time range


Time range is automatically populated
with time setting of the search you ran
- Use abbreviations for time range syntax:

s=seconds m=minutes h=hours d=days w=week


mon=months y=year
-
-

@ symbol "snaps" to time unit you specify


Example: Current time when the search
starts is 09:37:12
-5m looks back to 09:32:12
-5m@m looks back to 09:32:00
-30m@h looks back to 09:00:00
Listen to your data.

47

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Editing a saved search set time range


Time range is automatically populated
with time setting of the search you ran
- Use abbreviations for time range syntax:

s=seconds m=minutes h=hours d=days w=week


mon=months y=year
-
-

@ symbol "snaps" to time unit you specify


Example: Current time when the search
starts is 09:37:12
-5m looks back to 09:32:12
-5m@m looks back to 09:32:00
-5m@h looks back to 09:00:00
Listen to your data.

48

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Lab 2
Save and share search results
Create, edit, and run a saved search

Listen to your data.

49

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Section 3:
Using Fields

Listen to your data.

50

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Section objectives
Understand fields
Use fields in searches
Use the fields sidebar

Listen to your data.

51

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

What are fields?


Fields are searchable key/value pairs in your event data
- Example: host=www1, status=503

All fields have names and can be searched with those names
- Example: Separating an http status code of 404 from Atlantas area code

There are 2 types of fields


default fields

data-specific fields

Listen to your data.

52

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Identifying default fields


Added to every event by Splunk during indexing
Can be configured by Splunk administrators
Most commonly used: source, sourcetype, and host
source

Listen to your data.

sourcetype

53

host

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Identifying data-specific fields


Data-specific field values come from your data
- Sometimes indicated by obvious key=value pairs

- Sometimes not

- For more information, please see:

http://docs.splunk.com/Documentation/Splunk/latest/Data/Listofpretrainedsourcetypes
Listen to your data.

54

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Fields sidebar
selected fields

For the current search, shows


- Selected fields

interesting fields

- Interesting fields

(#) indicates number


of unique values

- Link to view all fields

Fields returned are those Splunk


recognized from your search results
Interesting fields are fields that show up in at
least 50% of events
Total fields varies depending on your search
mode
Listen to your data.

55

view more fields


Using Splunk 5.0
Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Field discovery
A field is a name/value pair extracted by Splunk
At search time, Splunk automatically extracts
- Fields used in the search
- Default fields, such as _time, _raw, host, source, and sourcetype

Field discovery also extracts other fields in the event data not directly
related to the search
The fields that display in the fields sidebar depends on the search
mode that you select

Listen to your data.

56

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Fast mode
Emphasizes performance, returning only essential
and required data
Field discovery is OFF
- Returns data on default fields and fields required to fulfill

your search

Reporting
Fields sidebar
Timeline
Chart/visualization
Listen to your data.

57

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Verbose mode
Emphasizes completeness, returning all field and
event data it possibly can
Field discovery is ON
- Splunk returns all of the fields it can

Reporting
Fields sidebar
Timeline
Chart/visualization

Listen to your data.

58

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Smart mode (default)


Designed to give you the best results for your search
Combination of Fast and Verbose modes
Field discovery is ON [Verbose]
- Splunk returns all of the fields it can

Reporting [Fast]
Fields sidebar
Timeline
Chart/visualization

Listen to your data.

59

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Extracted fields are related to results


Most fields and their values are
extracted from the events
themselves

Fast mode

Smart mode

When you run an event search,


the total number of fields
extracted depends on your
search mode

Listen to your data.

60

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Describing selected fields


Selected fields and their values are listed under every event that
include the fields
By default, host, source, and sourcetype are selected
Can dynamically add to selected fields

field added to selected list


field added to event display

Listen to your data.

61

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Adding fields to selected fields


Search by name or %

Listen to your data.

62

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

More ways to use the fields sidebar


remove events from
results that dont have
the field

create charts

click a value to add to a


search
Listen to your data.

ALT + click a value to


remove from a search
63

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Using fields in searches


Efficient way to pinpoint searches and refine results
Consider:
vs.

vs.
- Field names ARE case sensitive, field values are NOT

- Example: Splunk extracts a field in linux_secure data named user


- These two searches return results

Listen to your data.

this one does not

64

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Using fields in searches (contd)


For IP fields, Splunk is subnet/CIDR aware
is more efficient than

Use wildcards to match a range of field values

Use comparison operators for numeric field values

Listen to your data.

65

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Lab 3
Use fields to refine your search
Use fields to examine search results

Listen to your data.

66

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Section 4:
Creating & Using Tags
and Event Types
Listen to your data.

67

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Section objectives
Describe tags
Create tags and use tags in a search
Describe event types and their uses
Create, tag, and use event types in a search

Listen to your data.

68

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Describing tags
Tags allow you to search for events with related field values
- Can assign one or more tags to any field/value combination
- Example:

Server names arent always very helpful!

- Sometimes they contain ambiguous information


- Tag field/value pairs with meaningful terms

Listen to your data.

69

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Using tags
Search with tags the same way you search with fields
Tags are case sensitive, in this example, tag=dmz returns no events

Listen to your data.

70

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Managing tags
Use Splunk Manager to enable/disable, copy, delete, and edit tags
youve created

Listen to your data.

71

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

What is an event type?


An Event type:
- Is a method of categorizing events based on a search
- Is a knowledge object created by users
- Is useful for institutional knowledge capture and sharing
- Can be tagged to group similar types of events

Listen to your data.

72

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Event type scenario


You want to monitor potential security issues, one of which is failed
password events
These events fall into two categories
- Failed password for invalid users
- Failed password for valid users

The first group may need further investigation


The second group is potentially more dangerous
- To gain entry to a system, attackers need a valid user name and a password
- With a valid user name, they are half way in and need only discover the

password
Listen to your data.

73

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Event type scenario (contd)


To differentiate these 2 events, save event types
- less_risky - password

- risky - password

Listen to your data.

fail* "for invalid user"

fail* NOT "for invalid user"

74

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Creating an event type


Run a search and verify all results meet your event type criteria
Create the event type
- Name should not contain spaces

Listen to your data.

75

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Using event types


In our scenario, there are two event types
Displays in the fields sidebar and can be added as a selected field
Search for
eventtype=*risky
As events come in, Splunk
evaluates them and applies
appropriate event type
Using the fields sidebar, you
can easily view numbers
and percentages
Listen to your data.

76

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Tagging an event type


Use tags to groups similar
event types
In our scenario, we were
monitoring security issues,
of which one was failed
passwords
By tagging this and other
security event types, you
can view all
these security events

Listen to your data.

77

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Managing event types


From Manager, you can edit the underlying search, enable/disable,
clone, move, or delete event types

Listen to your data.

78

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Setting permissions for event types


As with all knowledge objects, you can set permissions

Listen to your data.

79

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Lab 4
Add tags to hosts
Use those tags in a search
Create and use event types

Listen to your data.

80

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Section 5:
Creating Alerts

Listen to your data.

81

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Section objectives
Describe alerts
Create alerts
- Run the underlying search
- Set the schedule, conditions, and actions

View fired alerts

Listen to your data.

82

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Alerting overview
Splunk alerts are based on searches that can run either
- On a regular scheduled interval
- In real-time

Alerts are triggered when the results of the search meet a specific
condition that you define
Based on your needs, alerts can
- Send emails
- Trigger scripts
- Write to RSS feeds

Listen to your data.

83

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Alerts real-time / monitored / scheduled


Real-time alerts
- Always trigger immediately for every returned result

Real-time monitored alerts


- Monitor a real-time window
- Can trigger immediately or you can define conditions

Scheduled alerts
- Run a search on a regular interval that you define
- Triggers based on conditions that you define

Listen to your data.

84

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Creating an alert defining the search and name


Run a search
From the Create menu,
select Alert
Name the alert

Listen to your data.

85

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Selecting alert schedule real-time


Schedule determines how often Splunk searches for the alert-worthy
events
- Trigger in real-time whenever a result matches

A real-time search continuously runs


Every time a matching result is returned the alert triggers

Listen to your data.

86

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Setting alert schedule set a schedule or rt window


Run on a schedule once every...

run on schedule

- Can run the search every hour, day, week,

month, or on a cron schedule

Monitor in real-time over a rolling


window of
- Runs in real time over a period you specify

Both types allow for Trigger if


conditions

Listen to your data.

87

monitor in real-time

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Setting alert schedule set conditions


Trigger if conditions allow you to
capture a larger data set, then apply
more stringent criteria to results before
executing the alert
Example:
- Search looks for failed logins for the root

user
- Trigger if more than 5 results are returned
within the selected schedule timeframe

Listen to your data.

88

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Defining alert actions enable actions


Send Email sends an email with results to
recipients that you define
Run a script runs a script that can perform
some other action
Show triggered alerts in Alert manager
adds an item to the Alert manager that
indicates severity and links to search results
- Choose an appropriate severity for the alert

Listen to your data.

89

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Execute actions on All Results


All Results executes alert actions once for all
matching events within the scheduled time
and conditions
Select a Throttling option
- After executing the alert actions once, dont execute

again for a specified time range


- Example: within a 1 minute real-time window, the
conditions may be met 70 times, but only execute
alert actions once

Listen to your data.

90

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Execute Actions on Each result


Each result executes the alert actions once for
each result that matches the conditions
- Selecting the Throttling option can suppress the

actions for results that have the same field value, within
a specified time range

Certain situations can cause a flood of alerts, when really


you only want one

- Example:

only execute the actions once per host for


the next 30 seconds

70 results are returned in a 1 minute window


50 results include host=sf013 and 20 include host=sf027
Actions would execute 4 times once for each of the 2
hosts, every 30 seconds
Listen to your data.

91

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Setting permissions
As with any knowledge object you create, you can keep the alert
private or share as read-only to other users
To save the alert, click Finish

Listen to your data.

92

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Viewing triggered alerts


To display the Alerts manager,
use Alerts menu item
To view the events that
triggered the alert, click View
results
To edit the alert settings, click
Edit search
To remove an alert from the list,
mark the checkbox and click
Delete
- Alert is not deleted, just
removed from list
Listen to your data.

93

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Edit alerts
To display and edit the alert settings, click Edit
search in the alert manager

Listen to your data.

94

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Lab 5
Create an alert

Listen to your data.

95

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Section 6:
Creating Reports &
Dashboards
Listen to your data.

96

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Section objectives
Create reports and charts
Create dashboards
Edit dashboards to add panels

Listen to your data.

97

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Reports and Charts


Create compelling visualizations and show trends
Allow you to drill down to see the events
Can be saved and shared
Can be added to dashboards

Listen to your data.

98

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Reporting with Splunk


Reporting turns Splunk data into operational intelligence
- Search and Investigation
- Proactive Monitoring
- Operational Visibility
- Real-time Business Insight

Listen to your data.

99

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Methods to create reports


Three main methods to create reports in Splunk
1.
2.
3.

From the Fields sidebar


From the Results Chart options in the events viewer
In the Report Builder

This course mainly focuses on creating reports from the Fields sidebar
and Results Chart options
- Searching and Reporting with Splunk course discussed how to create reports

using commands
- Note: Report Builder is not discussed in the Searching and Reporting class

Listen to your data.

100

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Quick and easy reporting from the sidebar


From the fields sidebar, select a field and a chart definition
- Splunk automatically "pipes" to the required search commands

These commands are covered in the


Searching and Reporting course

Listen to your data.

101

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Search mode and reporting

Listen to your data.

102

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Reporting on top values


For non-numeric fields, you can
choose to report on top values
by time or top values overall

Top values by time

- Top values by time displays a

trend over time (timechart) of the


top values of the selected field

| timechart
Top values overall

- Top values overall displays the top

values of the selected field by


number of occurrences

| top
Listen to your data.

103

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Reporting on numeric fields


For fields with numeric values,
you can create time-based
statistical reports on field values
- Average value over time
- Max value over time
- Minimum value over time

Average value over time

- Top values by time


- Top values overall
104

Listen to your data.

104

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Formatting reports
Click the Formatting options link to
display formatting tools
From the chart type menu, you
can select from several types
You can also enter a title for the
chart, adjust the legend
placement, and more
- The Searching and Reporting course

covers these options in detail

Listen to your data.

105

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Saving and sharing reports


You can save and share your
reports in the same way as you
would a search results
For one-off reports, you can
select Save & share results
from the Save menu
More efficient to save the search
and have the ability to run as
needed
Listen to your data.

106

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Splunk dashboards
Dashboards are collections of
searches and reports
A great way to group together
related reports and events
Easy to create and edit

Listen to your data.

107

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Adding a report to a dashboard


After running a search or
creating a report, select
Dashboard panel from
the Create menu
- Panels are individual reports or

items on a dashboard

Name the search

Listen to your data.

108

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Selecting dashboard and setting permissions


Create a new dashboard or add
the report to an existing
dashboard
If creating a new dashboard,
name it and set permissions

Listen to your data.

109

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Defining the panel


Name the panel
- Defaults to search name

Select the visualization


Choose a schedule
- Run the search when the dashboard loads

Splunk runs the search when the dashboard is


loaded

- Run scheduled search

Data loads from most recent run of the scheduled


search
Faster loading, less "fresh" data

Listen to your data.

110

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Saving and viewing the dashboard

Listen to your data.

111

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Editing the dashboard


To edit a panel, you must first toggle the dashboard Edit button to On

Listen to your data.

112

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Adding a new panel


Click +New Panel
Name the panel
Select a Saved search from the menu or use
the Inline search string to create a new search
Click Save
When creating a panel from the dashboard
view, by default, it will be either a data table
or an event listing

Listen to your data.

113

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Editing a panel
From the Edit menu, you can edit the search and/or the visualization

Listen to your data.

114

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Editing a panel (contd)


You can select a new
visualization and edit the title
Depending on the type of
visualization you choose, a
number of advanced options are
available
- These options are discussed in

detail in the Searching and


Reporting course

Listen to your data.

115

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Arranging and deleting dashboard panels


Click and drag panel items to
reorder
- Dashboards can contain up to three

panels per column

To remove a panel, from the


panels Edit menu, select Delete
- Does not delete the underlying

saved search

Listen to your data.

116

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Lab 6
Create reports
Add reports to a dashboard
Edit dashboard panels

Listen to your data.

117

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Wrap up
Search
- By keywords and booleans
- By time
- By fields
- All of the above

Refine searches
- Click to add/remove terms

Save search results and searches


Use tags and event types
Create alerts
Make reports and charts from your
searches
Create dashboards

- Use time line, time modifiers, fields

Listen to your data.

118

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Support programs
Community
- Splunkbase

Answers: answers.splunk.com
Post specific questions and get them answered by Splunk community experts.

- Splunk

Docs: docs.splunk.com
These are constantly updated. Be sure to select the version of Splunk you are using.

- Wiki:

wiki.splunk.com
A community space where you can share what you know with other Splunk users.

- IRC

Channel: #splunk on the EFNet IRC server Many well-informed Splunk users hang out here.

Global Support
Support for critical issues, a dedicated resource to manage your account 24 x 7 x 365.
- Email: support@splunk.com
- Web: http://www.splunk.com/index.php/submit_issue

Enterprise Support
Access you customer support team by phone and manage your cases online 24 x 7
(depending on support contract).
Listen to your data.

119

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

.conf2013: The 4th Annual Splunk WWUC


Las Vegas: Sept 30-Oct 3, 2013
- The Cosmopolitan

1500+ IT & Business Professionals


8 Tracks across 8 soluQon areas
3 days of content, 100+ sessions
2 days of Splunk University
30+ Customer speakers
25+ Apps in Splunkbase Labs
20+ Technology Partners
hTp://www.splunk.com/goto/conf
Listen to your data.

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Thank You

Please fill out the class survey


http://www.surveymonkey.com/s/splunkclasses

Listen to your data.

121

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Appendix:
Using IFX

Listen to your data.

122

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

How to make fields


If Splunk is not extracting a field, do not despair!
You can do one of these:
- Use the Interactive field extractor (IFX)
- Use the rex command (advanced topic)
- Ask your friendly Splunk administrator to add

But dont go crazy too many extractions will slow things down

Listen to your data.

123

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Interactive field extractor


1. From the field menu,
select, Extract fields

2. Add examples of the fields


values; here the example shows
IP addresses taken from the
returned data

3. Generate

Listen to your data.

124

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

Generating fields
Will tell you if the field
is already extracted

Can edit the expression,


test, and save

To improve accuracy,
edit sample extractions

Listen to your data.

125

Using Splunk 5.0


Copyright 2013 Splunk, Inc. All rights reserved | 3 March 2013

You might also like