You are on page 1of 10

Version 1.

Accelerate 2016
Partner Program

AccelOps

Using Combine threat


feeds to extend
AccelOps Threat
Intelligence
Transform the threat feeds into a format we can
import into our CMDB

COMBINE TO CSV AND ACCELOPS IMPORT


(c) AccelOps 2016
Partner Training Services

April 2016

Contents
Page

Introduction

Installing Combine on your AccelOps Supervisor

Configuring Combine

Importing the accelops_combine.csv into AccelOps through the GUI

Other Guides Available

This guide is based upon testing with the following software versions
AccelOps v4.6.1 (1025)
Operation/Results may be different for later versions, check with your local SE.

Note: AccelOps does not provide any support for the third party or
modified scripts and this information is provided for reference only.
The same concepts can be applied to importing other CSV files into the
AccelOps CMDB.

COMBINE TO CSV AND ACCELOPS IMPORT

Introduction
AccelOps is aware of the changing threat landscape and the ways in which the security
community fights off attacks. While we do our best to integrate with the latest commercial
threat intelligence solutions and feeds, we also provide you with the option of adding other
intelligence feeds by allowing you to import CSV files containing all the hosts from these
feeds. Importing through our API is also supported but not covered in this guide.
While there are many sources of open source intelligence, we find that Combine
(https://github.com/mlsecproject/combine) , which is a project from the Machine Learning
Project (http://www.mlsecproject.org) does an excellent job of aggregating the most valuable
ones.
The Combine script mentioned in this guide aggregates many open source feeds into a
single CSV file. This can provide you with thousands of ip addresses and additional
information that can automatically be correlated with all of your Network and DNS logs, in
real-time, for no additional cost.
This guide will show you how to import these feeds into AccelOps by using the
Combine scripts and a custom python script that will transform the combine CSV file into a
format we can import directly through the GUI, into the Blocked IP section of the CMDB.

COMBINE TO CSV AND ACCELOPS IMPORT

Installing Combine on your AccelOps Supervisor


We need to prepare our Supervisor to download a few python modules needed by the
combine script. Login to your supervisor and execute the following commands:
cd /etc/yum.repos.d.orig/
cp epel* ../yum.repos.d/
yum install python-devel python-virtualenv
wget https://github.com/mlsecproject/combine/archive/master.zip
mkdir /opt/scripts
cp master.zip /opt/scripts
cd /opt/scripts/
unzip master.zip
cd /opt/scripts/combine-master/
pip install grequests bs4 feedparser unicodecsv pygeoip netaddr
sortedcontainers
Figure 1. Python dependencies

COMBINE TO CSV AND ACCELOPS IMPORT

Configuring Combine
Combine depends on the combine.cfg configuration file. You should create edit /opt/
scripts/combine-master/combine.cfg and copy/paste the information below, you may fill in
the dnsdb_api, crits_username and crits_campaign attributes if you have that information. If
not, leave it as it is.
[Reaper]
inbound_urls = inbound_urls.txt
outbound_urls = outbound_urls.txt
[Winnower]
dnsdb_server = https://api.dnsdb.info/
dnsdb_api = YOUR_API_KEY_HERE
enrich_dns = 0
enrich_ip = 1
[Baler]
tiq_directory = tiq_test
winnow = 1
crits_url = http://crits_url:crits_port/api/v1/
crits_username = CRITS_USERNAME
crits_api_key = CRITS_API_KEY
crits_campaign = combine
crits_maxThreads = 10
Figure 2. combine.cfg

Next, we can run combine with the following command inside the /opt/scripts/
combine-master/ directory to obtain the enriched.csv file that contains all the threat
intelligence:
./combine.py -t CSV -e

COMBINE TO CSV AND ACCELOPS IMPORT

The resulting enriched.csv file does not match the format that AccelOps expects to
import as a Blocked IP entry into its CMDB. We need to respect the CSV formats in order to
leverage all the correlation rules that are running in memory.

CSV Formats supported by AccelOps


CMDB Watchlist

Blocked Domains

CSV Format

Name, IP, Reverse Lookup, Malware Type, Confidence, Severity,


ASN, Org, Country,Description, Date Found(MM/DD/
YYYY),Last Seen(MM/DD/YYYY)

Blocked IPs

Name, Low IP, High IP, Malware Type, Confidence, Severity, ASN,
Org, Country ,Description,Data Found(MM/DD/YYYY),Last
Seen(MM/DD/YYYY)

Blocked Processes
Malware Hash

Process Name,Description
Botnet Name, Algorithm, Has Code, Controller IP, Malware Type,
Confidence, Severity, Asn, Org, Country, Description, Data
Found(MM/DD/YYYY), Last Seen(MM/DD/YYYY)

Default Password

Vendor, Model, Access Protocol, User Name, Password

Anonymity

IP, Port, Malware Type, Confidence, Severity, Asn, Org, Country,

Network

Description, Data Found(MM/DD/YYYY), Last Seen(MM/DD/


YYYY)

User Agents

User Agent (regular expression)

Table 1. CSV Formats supported by Accelops

COMBINE TO CSV AND ACCELOPS IMPORT

We want to import the CSV file into the Blocked IPs list and to simplify this process we
can use the following python script to transform the data.
Save the script below as accelops_combine.py inside the /opt/scripts/combinemaster/ directory.
Make sure you keep the indentation as it is below and if you get any errors running the
script, ensure you do not have any special characters due to the copy/paste:
#!/usr/bin/env python
import csv
import datetime
import sys

f = open('enriched.csv', 'r') #open file


filename = open("accelops_combine.csv",'w') #write the output to accelops_combine.csv
sys.stdout = filename

try:
r = csv.reader(f) #init csv reader
r.next() #jump over the CSV headers as we don't need them
for row in r:
print "Combine-"+row[0]+","+row[0]+","+row[0]+","+row[3]+",,,"+row[6]+","+str(row[7]).replace(',','')
+","+row[8]+","+row[2]+","+datetime.datetime.strptime(row[5],"%Y-%m-%d").strftime('%m/%d/%Y')

finally:
f.close() #cleanup
Figure 3. accelops_combine.py script

You can then run the script by making the script executable (chmod +x
accelops_combine_py), executing ./accelops_combine.py and obtaining the
accelops_combine.csv file.

COMBINE TO CSV AND ACCELOPS IMPORT

Importing the accelops_combine.csv into AccelOps


through the GUI
First of all, we need to pull the accelops_combine.csv file from the supervisor to the
laptop/workstation we use to access AccelOps.
In OS X its as simple as opening up Terminal and executing the following command
(replace 192.168.0.148 with your supervisor ip address):
scp root@192.168.0.148:/opt/scripts/combine-master/accelops_combine.csv

Now that we have the accelops_combine.csv file in our possession, we have to login
into AccelOps, go to the CMDB tab and under the Blocked IPs section, select Update.

Figure 4. Uploading CSV file into AccelOps CMDB

COMBINE TO CSV AND ACCELOPS IMPORT

Browse, find the accelops_combine.csv


file you downloaded and click the
Upload button. Wait while the file is
uploaded and processed by AccelOps.

Figure 5. Upload popup window

You should see the following window


when it completes.
If the file is very large it may cause the
upload to timeout. If that happens, split

Figure 6. Uploaded succeeded message

the accelops_combine.csv file into


smaller ones until it succeeds.

COMBINE TO CSV AND ACCELOPS IMPORT

Other Guides Available


Other guides available in the Accelerate 2016 Partner Program include:

Identity and Location Guide

Available Soon

Windows Agent Deployment

Available Soon

MSP Edition
Quick Start Guide

Enterprise Edition
Quick Start Guide

Using a Microsoft CA
Certificate

Available Soon

Deployment Architectures

Please check with your local SE or Sales Representative for an up to date list.

COMBINE TO CSV AND ACCELOPS IMPORT

You might also like