flume 將資料匯入hive
1:建立.conf檔案
#cd /usr/tmp
#mkdir flumesrc
#vim netcat_to_hive_user.conf
a1.sources=r1
a1.sinks=s1
a1.channels=c1
a1.sources.r1.type=netcat
a1.sources.r1.bind=master
a1.sources.r1.port=44444
a1.sinks.s1.type=hive
a1.sinks.s1.channel = c1
a1.sinks.s1.hive.metastore = thrift://master:9083
a1.sinks.s1.hive.database = bd18
a1.sinks.s1.hive.table = flume_user
a1.sinks.s1.uselocaltimestamp = false
a1.sinks.s1.serializer = delimited
a1.sinks.s1.serializer.delimiter = "\t"
a1.sinks.s1.serializer.serdeseparator = '\t'
a1.sinks.s1.serializer.fieldnames=user_id,user_name,age
a1.channels.c1.type=memory
a1.channels.c1.capacity=1000
2:執行.conf檔案
#flume-ng agent -c conf -f netcat_to_hive_user.conf --name a1 -dflume.root.logger=info,console
3:將hive的四個jar包複製到flume安裝目錄的lib資料夾下
4:啟動hive的metastore
#hive --service metastore
5:進入hive,建立資料庫和表
hive> create database bd18;
hive> create table flume_user(
> user_id int,
> user_name string,
> age int
> )
> clustered by(user_id) into 2 buckets
> stored as orc
> tblproperties("transactional"='true');
hive> select * from flume_user;
failed: semanticexception [error 10265]: this command is not allowed on an acid table bd18.flume_user with a non-acid transaction manager. failed command: select * from flume_user
hive> set hive.support.concurrency=true；
hive> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.dbtxnmanager;
hive> select * from flume_user;
oktime taken: 5.809 seconds
6:操作flume,向hive輸入資料
[root@master ~]# telnet master 44444
trying 192.168.2.100...
connected to master.
escape character is '^]'.
1 jim 18ok
注：中間用tab鍵隔開
7:查詢hive表中的內容
042 將資料匯入hive，將資料從hive匯出
一 將資料匯入hive 六種方式 1.從本地匯入 load data local inpath file path into table tbname 用於一般的場景。2.從hdfs上匯入資料 load data inpath hafd file path into table tbname 應用場...
04 flume將資料匯入到Hbase
我們已經了解了使用flume監聽埠以及檔案內容，本節將展示使用flume匯入資料到hbase。進到hbase啟動目錄 cd root hbase 1.2.6 bin 啟動hbase服務 start hbase.sh 登入到hbase客戶端 hbase shell 在hbase命令列建立乙個表 hba...
將excel中的資料匯入hive
步驟二，將該txt檔案匯入linux指定目錄中 步驟三，轉換編碼格式，在指定目錄下執行如下命令 piconv f gb2312 t utf 8 companycode.txt c.txt 步驟四，根據文件中的列，建立表，建表語句如下 hive create table companycode ccn...