Hive中將查詢結果匯出到指定分隔符的檔案中

2021-07-03 12:36:33 字數 4594 閱讀 7901

在hive0.11.0版本中新引進了乙個新的特性,當使用者將hive查詢結果輸出到檔案中時,使用者可以指定列的分割符,而在之前的版本是不能指定列之間的分隔符。

在hive0.11.0之前版本如下使用,無法指定分隔符,預設為\x01:

hive (hive)>insertoverwrite local directory '/home/hadoop/export_hive' select * from a;

query id =hadoop_20150627174342_64852f3a-56ed-48d5-a545-fc28f109be74

total jobs = 1

launching job 1 out of 1

number of reduce tasks is set to 0since there's no reduce operator

kill command =/home/hadoop/hadoop-2.6.0/bin/hadoop job -kill job_1435392961740_0025

2015-06-27 17:43:54,395 stage-1 map =0%,  reduce = 0%

2015-06-27 17:44:07,615 stage-1 map =100%,  reduce = 0%, cumulative cpu 4.82sec

mapreduce total cumulative cpu time: 4seconds 820 msec

ended job = job_1435392961740_0025

copying data to local directory /home/hadoop/export_hive

copying data to local directory/home/hadoop/export_hive

mapreduce jobs launched:

stage-stage-1: map: 1   cumulative cpu: 4.82 sec   hdfs read: 2416833 hdfs write: 1188743success

total mapreduce cpu time spent: 4seconds 820 msec

oka.key     a.value

time taken: 26.475 seconds

(目錄不存在時,會自動建立)

檢視生成的檔案內容:

[hadoop@gpmaster export_hive]$head-n 10 /home/hadoop/export_hive/000000_0 | cat -a

2610^aaaanz$

32196^aaaaownz$

78606^aaaayxfz$

3804^aaaaz$

30102^aaabewez$

21744^aaabukz$

39666^aaabz$

1632^aaabz$

82464^aaabz$

88320^aaaccaz$

我使用cat -a引數,將檔案中每行的結尾$符號和分隔符^a(即是\x01)列印了出來。

接下來,我們使用hive0.11.0版本新引進的新特性,指定輸出結果列之間的分隔符:

hive (hive)>insertoverwrite local directory '/home/hadoop/export_hive' row format delimitedfields terminated by '*' select * from a;

query id =hadoop_20150627180045_fced1513-8f1b-44a8-8e88-3cd678552aa5

total jobs = 1

launching job 1 out of 1

number of reduce tasks is set to 0since there's no reduce operator

kill command =/home/hadoop/hadoop-2.6.0/bin/hadoop job -kill job_1435392961740_0028

2015-06-27 18:00:57,354 stage-1 map =0%,  reduce = 0%

2015-06-27 18:01:10,567 stage-1 map =100%,  reduce = 0%, cumulative cpu 4.68sec

mapreduce total cumulative cpu time: 4seconds 680 msec

ended job = job_1435392961740_0028

copying data to local directory/home/hadoop/export_hive

copying data to local directory/home/hadoop/export_hive

mapreduce jobs launched:

stage-stage-1: map: 1   cumulative cpu: 4.68 sec   hdfs read: 2417042 hdfs write: 1188743success

total mapreduce cpu time spent: 4seconds 680 msec

oka.key     a.value

time taken: 26.607 seconds

檢視指定分隔符為*後,匯出的資料如下:

[hadoop@gpmaster export_hive]$head -n10 /home/hadoop/export_hive/000000_0

2610*aaanz

32196*aaaownz

78606*aaayxfz

3804*aaaz

30102*aabewez

21744*aabukz

39666*aabz

1632*aabz

82464*aabz

88320*aaccaz

可以看到列的分隔符的確是我們指定的*號分隔符。

如果是復合型別,比如struct,map型別等也可以指定對應的分隔符:

以下我們做個例項操作實際操作以下:

(1)  建立復合型別的表

hive (hive)>create table userinfo(id int,name string,job_listarray,perf map,info struct);

(2)  構造資料(使用預設分隔符構造)

1^a小明^ait工程師^b教師^a10086^c正常^b10010^c不正常^a北京市^b130

2^a小花^a保姆^b**^a10086^c正常^b10010^c正常^a南京市^b130

注釋:\001使用^a代替,\002使用^b,\003使用^c代替

造資料在使用vi編輯器裡面,用ctrl+v然後再ctrl+a可以輸入這個控制符\001。按順序,\002的輸入方式為ctrl+v,ctrl+b,依次類推。

(3)  匯入資料

hive (hive)>load data local inpath'/home/hadoop/hivetestdata/userinfo.txt' overwrite into table userinfo;

(4)  查詢資料

hive (hive)> select * from userinfo;

okuserinfo.id   userinfo.name   userinfo.job_list userinfo.perf     userinfo.info

1    小明     ["it工程師","教師"]   

2    小花     ["保姆","**"] 

time taken: 0.088 seconds, fetched: 2 row(s)

(5)  匯出資料

我們指定的分隔符為:

列分隔符為 \t

map keys分隔符為:(冒號)

collection items分隔符為:,(逗號)

執行匯出命令:

hive (hive)>insert overwrite localdirectory '/home/hadoop/export_hive'

> row formatdelimited fields terminated by '\t'

> collectionitems terminated by ','

> map keysterminated by ':'

>select * from userinfo;

檢視匯出的資料為:

[hadoop@gpmaster export_hive]$ cat 000000_0

1    小明     it工程師,教師         10086:正常,10010:不正常     北京市,130

2    小花     保姆,**    10086:正常,10010:正常   南京市,130

mysql查詢結果匯出到檔案

方法一 直接執行命令 mysql select count 1 fromtable into outfile tmp test.xls query ok,31 rows affected 0.00 sec 在目錄 tmp 下會產生檔案test.xls 遇到的問題 mysql select count...

mysql查詢結果匯出到檔案

環境 mysql 進入mysql shell mysql u username password password database database然後輸入匯出查詢結果命令 select from table1 into outfile data test1.txt 報錯error 1045 28...

MySQL查詢結果匯出到Excel

mysql uroot p use databa xx select user,host,password from user into outfile tmp data.xls select b.name,a.title,a.answer from plugin faq question a jo...