Flume監聽檔案目錄sink至hdfs配置

一：flume介紹

flume是乙個分布式、可靠、和高可用的海量日誌聚合的系統，支援在系統中定製各類資料傳送方，用於收集資料；同時，flume提供對資料進行簡單處理，並寫到各種資料接受方（可定製）的能力。，flume架構分為三個部分源-source，接收器-sink，通道-channel。

二：配置檔案

此配置檔案source為乙個目錄，注意，該目錄下的檔案應為唯讀，不可寫，且檔名不能相同，採用的channels為file，sink為hdfs，此處往hdfs寫的策略是當時間達到3600s或者檔案大小達到128m。

agent1.sources = spooldirsource

agent1.channels = filechannel

agent1.sinks = hdfssink

agent1.sources.spooldirsource.type=spooldir

agent1.sources.spooldirsource.spooldir=/data/lwq/new_log

agent1.sources.spooldirsource.channels=filechannel

agent1.sinks.hdfssink.type=hdfs

agent1.sinks.hdfssink.hdfs.path=hdfs://dev228:8020/raw/lwq/%y-%m-%d

agent1.sinks.hdfssink.hdfs.fileprefix=lwq

agent1.sinks.sink1.hdfs.round = true

# number of seconds to wait before rolling current file (0 = never roll based on time interval)

agent1.sinks.hdfssink.hdfs.rollinterval = 3600

# file size to trigger roll, in bytes (0: never roll based on file size)

agent1.sinks.hdfssink.hdfs.rollsize = 128000000

agent1.sinks.hdfssink.hdfs.rollcount = 0

agent1.sinks.hdfssink.hdfs.batchsize = 1000

#rounded down to the highest multiple of this (in the unit configured using hdfs.roundunit), less than current time.

agent1.sinks.hdfssink.hdfs.roundvalue = 1

agent1.sinks.hdfssink.hdfs.roundunit = minute

agent1.sinks.hdfssink.hdfs.uselocaltimestamp = true

agent1.sinks.hdfssink.channel=filechannel

agent1.sinks.hdfssink.hdfs.filetype = datastream

agent1.channels.filechannel.type = file

agent1.channels.filechannel.checkpointdir=/usr/share/apache-flume-1.5.0-bin/checkpoint

agent1.channels.filechannel.datadirs=/usr/share/apache-flume-1.5.0-bin/datadir

複製**

三：啟動命令

$/bin/flume-ng agent --conf ./conf/ -f conf/flume-site.xml -dflume.root.logger=debug,console -n agent1 > log.log 2>&1 &

一 flume介紹 flume是乙個分布式可靠和高可用的海量日誌聚合的系統，支援在系統中定製各類資料傳送方，用於收集資料同時，flume提供對資料進行簡單處理，並寫到各種資料接受方可定製的能力。flume架構分為三個部分源 source，接收器 sink，通道 channel。二配置檔...