使用hivexmlserde解析xml檔案

hive xml serde是乙個基於hive serde(序列化/反序列化)框架的xml處理庫。它依賴於apache mahout專案中的xmlinputformat，根據特定的開始和結束標記將輸入檔案分解成xml片段。 xml serde的本質其實是使用xpath處理器查詢xml片段來填充hive表。

準備資料：

首先將jar包新增到hive中，將jar新增到hive中有很多種方式，這裡我們採用新增臨時jar的方法

add jar /home/hadoop/hive_jar/hivexmlserde- 1.0.5

.3.jar;

注意：如果在hive中整合了spark sql請用spark sql完成本操作

建表語句：

create
table ebay_listing(seller_name string, 
seller_rating bigint
, bidder_name string, 
location string, bid_history map
, item_info map
)row format serde 'com.ibm.spss.hive.serde2.xml.xmlserde'
with serdeproperties (
"column.xpath.seller_name"
="/listing/seller_info/seller_name/text()"
,"column.xpath.seller_rating"
="/listing/seller_info/seller_rating/text()"
,"column.xpath.bidder_name"
="/listing/auction_info/high_bidder/bidder_name/text()"
,"column.xpath.location"
="/listing/auction_info/location/text()"
,"column.xpath.bid_history"
="/listing/bid_history/*"
,"column.xpath.item_info"
="/listing/item_info/*"
)stored as
inputformat 'com.ibm.spss.hive.serde2.xml.xmlinputformat'
outputformat 'org.apache.hadoop.hive.ql.io.ignorekeytextoutputformat'
tblproperties (
"xmlinput.start"=""
,"xmlinput.end"=""
);

載入資料：

load data local inpath "/home/hadoop/hive_data/ebay.xml" into

table ebay_listing;

接下來進行測試：

select seller_name, bidder_name, location, bid_history[
"highest_bid_amount"
], item_info[
"cpu"
]from ebay_listing

結果：

seller_name bidder_name location bid_history[highest_bid_amount] item_info[cpu] cubsfantony [email protected] usa/chicago $620.00 pentium iii 933 system ct-inc [email protected] usa/los angeles $680.00 intel pentium iii 800eb-mhz coppermine cpu ct-inc [email protected] usa/los angeles $1 ,025.00 intel pentium iii 933eb-mhz coppermine cpu bestbuys4systems wizbang4 allentown, pa 18109 $610.00 genuine intel pentium iii 1000mhz processor [email protected] [email protected] los angeles, ca $535.00 intel pentium iii 800mhz time taken: 2.197 seconds, fetched 5

row(s)

成功將xml檔案解析為 relational table !

使用hivexmlserde解析xml檔案

Berkeley DB使用全解

crontab使用祥解

Python使用ElementTree解析XML

使用hivexmlserde解析xml檔案

Berkeley DB使用全解

crontab使用祥解

Python使用ElementTree解析XML

相關推薦