hive udf 寫入到redis的Set 集合中

1，因業務需求，需要把hive的資料寫入到reids的set集合中，看網上都是寫入到hash的實現，故自己寫了乙個寫入hash的實現

2,需要的pom.xml檔案

org.apache.hadoop

hadoop-client
$log4j
log4j
org.slf4j
slf4j-log4j12
provided
org.apache.hadoop
hadoop-common
$provided
org.apache.hive
hive-exec
$provided
redis.clients
jedis
2.9.0
junit
junit
4.12
test

import com.crgt.redis.jedisutil;

import org.apache.hadoop.hive.ql.exec.description;
import org.apache.hadoop.hive.ql.exec.udfargumentexception;
import org.apache.hadoop.hive.ql.metadata.hiveexception;
import org.apache.hadoop.hive.ql.udf.generic.genericudf;
import org.apache.hadoop.hive.serde2.objectinspector.constantobjectinspector;
import org.apache.hadoop.hive.serde2.objectinspector.objectinspector;
import org.apache.hadoop.hive.serde2.objectinspector.objectinspector.category;
import org.apache.hadoop.hive.serde2.objectinspector.primitiveobjectinspector;
import org.apache.hadoop.hive.serde2.objectinspector.primitive.primitiveobjectinspecto***ctory;
import org.apache.hadoop.hive.serde2.objectinspector.primitive.stringobjectinspector;
import org.apache.hadoop.io.intwritable;
import redis.clients.jedis.hostandport;
/** * 
* 入參校驗，可以簡單點
*/@description(name = "redis_batch_sadd", value = "_func_(host_and_port,keyfield, value(string)) - return ret ")
public class redisbatchsetudf extends genericudf 
jedisutil.getjedis(hostandport).sadd(datakey,data);
return new intwritable(1);
} catch (exception e) 
}@override
public string getdisplaystring(string arg0) 
@override
public objectinspector initialize(objectinspector arg0) throws udfargumentexception 
//第乙個引數校驗
if (arg0[0].getcategory() == category.primitive
&& ((primitiveobjectinspector) arg0[0]).getprimitivecategory() == primitiveobjectinspector.primitivecategory.string) 
constantobjectinspector redishost_and_port = (constantobjectinspector) arg0[0];
string host_and_port = redishost_and_port.getwritableconstantvalue().tostring().split(":");
hostandport = new hostandport(host_and_port[0], integer.parseint(host_and_port[1]));
}// 1. 檢查是否接收到正確的引數型別
objectinspector key = arg0[1];
objectinspector value = arg0[2];
if (!(key instanceof stringobjectinspector) ) 
if ( !(value instanceof stringobjectinspector) ) 
this.keyelementoi = (stringobjectinspector) key;
this.valueelementoi = (stringobjectinspector) value;
return primitiveobjectinspecto***ctory.writableintobjectinspector;
}}

public class jedisutil 

} catch (exception e)  finally 
}}));
}return jedis;
}}

日誌寫入到檔案 多檔案寫入

在上篇文章的基礎上，修改配置 log4j.rootlogger info,fout 說明 rootlogger是可以多樣式定義的，如log4j.rootlogger info,myout,fout。myout,fout是自定義樣式。這樣就會在c盤下建立a.log檔案。有這樣一種需求，不同模組需要各自...

Redis批量寫入資料

生產中的有些場景,我們經常需要大批量的往redis中寫入資料,如果我們採用單條迴圈寫入的話,不僅效率低下,而且可能會出現頻繁的建立和銷毀redis連線,這些都是很不合理的.對此,我們可以採用jedis的父類中的pipelined 方法獲取管道,它可以實現一次性傳送多條命令並一次性返回結果,這樣就大量...

將資料寫入到redis快取 取出 刪除 訊息佇列

1 寫入快取 1獲取redis操作介面 var redisclient redismanager.getclient 2將新增的資料儲存到redis answer redis裡面儲存的名稱，類似於一張表一樣，新增成功的資料 model.tojson 需要儲存的資料 model.tojson的資料 會...