Flume简单使用

netcat-memory-logger 模式配置(监听某个端口并且通过内存缓冲后输出为日志)

  1. 安装目录conf文件夹新建一个配置文件(便于管理)

# vim netcat-memory-logger.conf

写入如下内容

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# example.conf: A single-node Flume configuration

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444

# Describe the sink
a1.sinks.k1.type = logger

# Use a channel which buffers events in memory
a1.channels.c1.type = memory

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

  1. 启动
    启动命令为
    1
    2
    3
    4
    5
    flume-ng agent \
    --name a1 \
    --conf $FLUME_HOME/conf \
    --conf-file $FLUME_HOME/conf/netcat-memory-logger.conf \
    -Dflume.root.logger=INFO,console

启动后该窗口不要关闭

  1. 测试
    使用telnet命令
    telnet localhost 44444
    然后输入消息 如果前面的窗口可以接受到就是成功了

exec-memory-logger 模式配置(监听某个文件并且通过内存缓冲后输出为日志)

  1. 安装目录conf文件夹新建一个配置文件(便于管理)
    # vim exec-memory-logger.conf
    写入如下内容

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    #Agent选型:exec source + memory channel +logger sink
    # Name the components on this agent
    a1.sources = r1
    a1.sinks = k1
    a1.channels = c1

    # Describe/configure the source
    a1.sources.r1.type = exec
    a1.sources.r1.command = tail -F /Users/duanjiaxing/data/flume-test.log
    a1.sources.r1.shell = /bin/sh -c

    # Describe the sink
    a1.sinks.k1.type = logger

    # Use a channel which buffers events in memory
    a1.channels.c1.type = memory

    # Bind the source and sink to the channel
    a1.sources.r1.channels = c1
    a1.sinks.k1.channel = c1
  2. 启动
    启动命令为

    1
    2
    3
    4
    5
    flume-ng agent \
    --name a1 \
    --conf $FLUME_HOME/conf \
    --conf-file $FLUME_HOME/conf/exec-memory-logger.conf \
    -Dflume.root.logger=INFO,console

启动后该窗口不要关闭

  1. 测试
    使用telnet命令
    echo hello >> flume-test.log
    然后输入消息 如果前面的窗口接受到 hello 就是成功了

exec-memory-avro + avro-memory-logger 模式配置(两个Flume对接,将一台机器上的Flume的输出作为另一个Flume的输入)

  1. 安装目录conf文件夹新建两个配置文件(便于管理)
    # vim exec-memory-avro.conf
    # vim avro-memory-logger.conf

exec-memory-avro.conf写入如下内容

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# Name the components on this agent
exec-memory-avro.sources = exec-source
exec-memory-avro.sinks = avro-sink
exec-memory-avro.channels = memory-channel


# Describe/configure the source
exec-memory-avro.sources.exec-source.type = exec
exec-memory-avro.sources.exec-source.command = tail -F /Users/duanjiaxing/data/flume-test.log
exec-memory-avro.sources.exec-source.shell = /bin/sh -c


# Describe the sink
exec-memory-avro.sinks.avro-sink.type = avro
exec-memory-avro.sinks.avro-sink.hostname = localhost
exec-memory-avro.sinks.avro-sink.port = 44444


# Use a channel which buffers events in memory
exec-memory-avro.channels.memory-channel.type = memory


# Bind the source and sink to the channel
exec-memory-avro.sources.exec-source.channels = memory-channel
exec-memory-avro.sinks.avro-sink.channel = memory-channel

avro-memory-logger.conf 写入如下内容

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# Name the components on this agent
exec-memory-avro.sources = exec-source
exec-memory-avro.sinks = avro-sink
exec-memory-avro.channels = memory-channel

# Describe/configure the source
exec-memory-avro.sources.exec-source.type = exec
exec-memory-avro.sources.exec-source.command = tail -F /Users/duanjiaxing/data/flume-test.log
exec-memory-avro.sources.exec-source.shell = /bin/sh -c

# Describe the sink
exec-memory-avro.sinks.avro-sink.type = avro
exec-memory-avro.sinks.avro-sink.hostname = localhost
exec-memory-avro.sinks.avro-sink.port = 44444

# Use a channel which buffers events in memory
exec-memory-avro.channels.memory-channel.type = memory

# Bind the source and sink to the channel
exec-memory-avro.sources.exec-source.channels = memory-channel
exec-memory-avro.sinks.avro-sink.channel = memory-channel
  1. 启动

    1. 先启动avro-memory-logger
      1
      2
      3
      4
      5
      flume-ng agent \
      --name avro-memory-logger \
      --conf $FLUME_HOME/conf \
      --conf-file $FLUME_HOME/conf/avro-memory-logger.conf \
      -Dflume.root.logger=INFO,console
  2. 再启动exec-memory-avro.conf

    1
    2
    3
    4
    5
    flume-ng agent \
    --name exec-memory-avro \
    --conf $FLUME_HOME/conf \
    --conf-file $FLUME_HOME/conf/exec-memory-avro.conf \
    -Dflume.root.logger=INFO,console
  3. 测试
    使用telnet命令
    echo hello223 >> flume-test.log
    然后输入消息 如果avro-memory-logger窗口接受到 hello223 就是成功了

-------------End Of This ArticleThank You For Reading-------------