8/23/2015

Apache Flume Tutorial : Send data to another server and write that data to a file This And That

Menu

This And That

FLUME TUTORIAL

Apache Flume Tutorial : Send data to

another server and write that data to a file
BY PARTH SAVANI MAY 22, 2015 0 COMMENTS

http://thisandthat.io/blog/flume-part3/

1/9

8/23/2015

Apache Flume Tutorial : Send data to another server and write that data to a file This And That

Objective
http://thisandthat.io/blog/flume-part3/

2/9

8/23/2015

Apache Flume Tutorial : Send data to another server and write that data to a file This And That

Setup configuration on Server 1

Setup configuration on Server 2
Run the example
Output

Objective
To cat a text file on server 1 and pass the data via flume to server 2 and write the data to a file on server 2

What we will do: We will cat a file and pass that data to flume on server1 and that data will be passed to server2s
flume and written to a file on server2

http://thisandthat.io/blog/flume-part3/

3/9

8/23/2015

Apache Flume Tutorial : Send data to another server and write that data to a file This And That

Setup configuration on Server 1

SERVER 1
1) Go to the conf directory of your flume installation.
cd apache-flume/conf/

2) Write configuration file

flume_send.conf

and put it in your conf directory

# configure the agent

agent.sources = r1
agent.channels = k1
agent.sinks = c1
# using memory channel to hold upto 1000 events
agent.channels.k1.type = memory
agent.channels.k1.capacity = 1000
agent.channels.k1.transactionCapacity = 100
# connect source, channel, sink
agent.sources.r1.channels = k1
agent.sinks.c1.channel = k1

http://thisandthat.io/blog/flume-part3/

4/9

8/23/2015

Apache Flume Tutorial : Send data to another server and write that data to a file This And That

# cat the file

agent.sources.r1.type = exec
agent.sources.r1.command = cat file.txt
# connect to another box using AVRO and send the data
agent.sinks.c1.type = avro
agent.sinks.c1.hostname = 10.37.158.144
#NOTE: use your server 2s ip address here
agent.sinks.c1.port = 4545
#NOTE: This port should be open on Server 2

Save the above configuration in a file. Lets call it

flume_send.conf

and put it under conf directory

I am using port 4545. You will have to make sure that port is open on Server 2

Setup configuration on Server 2

SERVER 2
1) Go to the conf directory of your flume installation.
cd apache-flume/conf/

2) Write configuration file

http://thisandthat.io/blog/flume-part3/

flume_write.conf

and put it in your conf directory on Server2

5/9

8/23/2015

Apache Flume Tutorial : Send data to another server and write that data to a file This And That

# THIS ONE WRITES TO A FILE

# configure the agent
agent.sources = r1
agent.channels = k1
agent.sinks = c1
# using memory channel to hold upto 1000 events
agent.channels.k1.type = memory
agent.channels.k1.capacity = 1000
agent.channels.k1.transactionCapacity = 100
# connect source, channel, sink
agent.sources.r1.channels = k1
agent.sinks.c1.channel = k1
# here source is listening at the specified port using AVRO for data
agent.sources.r1.type = avro
agent.sources.r1.bind = 0.0.0.0
agent.sources.r1.port = 4545
# this is whats different.
# We use file_roll and write file at specified directory.
agent.sinks.c1.type = file_roll
agent.sinks.c1.sink.directory = /home/ec2-user/parth
#Note: Change this path to your server path

Save the above configuration in a file. Lets call it

http://thisandthat.io/blog/flume-part3/

flume_write.conf

and put it under conf directory

6/9

8/23/2015

Apache Flume Tutorial : Send data to another server and write that data to a file This And That

Run the example

3) Before running the Example, Lets create a file on
ON Server1
cd apache-flume/
echo "test test test" > file.txt

4) Lets run flume on both servers now

ON SERVER 2
cd apache-flume/
bin/flume-ng agent --conf conf --conf-file conf/flume_write.conf --name agent -Dflume.root.logger=INFO,console

ON SERVER 1
cd apache-flume/

http://thisandthat.io/blog/flume-part3/

7/9

8/23/2015

Apache Flume Tutorial : Send data to another server and write that data to a file This And That

bin/flume-ng agent --conf conf --conf-file conf/flume_send.conf --name agent -Dflume.root.logger=INFO,console

Output
When flume runs, it should WRITE the contents of file.txt on Server 2 in the directory path you provided in the conf
file.
0 Comments

thisandthat.io

Share

Recommend

Login

Sort by Best

Start the discussion

Be the first to comment.

Subscribe

Add Disqus to your site

Privacy

comments powered by Disqus

http://thisandthat.io/blog/flume-part3/

8/9

8/23/2015

Apache Flume Tutorial : Send data to another server and write that data to a file This And That

Previous

Next

2015 Parth Savani. Powered by Jekyll using the So Simple Theme.

http://thisandthat.io/blog/flume-part3/

9/9