19/10/16 11:22:06 error yarnclusterscheduler: lost executor 28 on **********: container marked as failed: container_********** on host: **********. exit status: 137. diagnostics: container killed on request. exit code is 137 killed by external signal
19/10/16 11:32:59 error yarnclusterscheduler: lost executor 38 on 100.76.80.197: container marked as failed: container_********** on host: **********. exit status: 137. diagnostics: container killed on request. exit code is 137 killed by external signal
19/10/16 11:40:27 error yarnclusterscheduler: lost executor 39 on **********: container marked as failed: container_1567762627991_1638740_01_000343 on host: **********. exit status: 137. diagnostics: container killed on request. exit code is 137 killed by external signal
19/10/16 11:49:29 error yarnclusterscheduler: lost executor 40 on **********: container marked as failed: container********** on host: **********. exit status: 137. diagnostics: container killed on request. exit code is 137 killed by external signal
19/10/16 11:49:29 error tasksetmanager: task 51 in stage 4.0 failed 4 times; aborting job
driver stack trace:
org.apache.spark.sparkexception: job aborted due to stage failure: task 51 in stage 4.0 failed 4 times, most recent failure: lost task 51.3 in stage 4.0 (tid 160, **********, executor 40): executor lost failure (executor 40 exited caused by one of the running tasks) reason: container marked as failed: container_********** on host: 100.76.26.136. exit status: 137. diagnostics: container killed on request. exit code is 137 killed by external signal
幾個關鍵的spark資源配置如下：
由報錯資訊可以看出，yarn丟失了executor，極有可能還是因為executor被關閉了，所以還是要檢查一下自己的driver-memory和executor-memory是不是夠大。
使用spark 2.0的scala api，使用jedis客戶端api，dependency如下：
redis.clients
jedis
2.9.0
jar
sampledata.repartition(500).foreachpartition(
rows => )
pipe.sync()
})
解決方法是在foreachpartition之前，repartition整個dateset，確保每個分割槽的資料不要太大。推薦控制在1k~20k左右。如上，將sampledata分為500個分割槽，每個分割槽10000條，那麼sampledata的總數為500萬左右。但是，如果資料總量太大，單個分割槽過小，會導致分割槽數過大，這樣需要提高driver的記憶體，否則會導致driver記憶體溢位。
。觀察在不同數量executor下，併發寫入redis的qps，直到qps達到乙個可以接受的範圍。
spark寫sql語句範例
建立乙個sparksession物件 val sparksession sparksession.builder createdf2 master local getorcreate val rdd sparksession.sparkcontext.textfile c data person.t...
使用Python寫spark 示例
python寫spark我認為唯一的理由就是 你要做資料探勘，ai相關的工作。因為很多做數挖的他們的基礎語言都是python，他們如果重新學scala比較耗時，而且，python他的強大類庫是他的優勢，很多演算法庫只有python有。python的安裝 解壓python包，在環境變數裡面配上bin的...
Spark（六） Spark計算模型
整個spark框架都是基於rdd運算元來進行計算的。what is rdd？resilient distributed dataset rdd 分布式彈性資料集，是spark上的乙個核心抽象 表示用於平行計算的，不可修改的，對資料集合進行分片的資料結構 簡單地，可以將rdd看成是spark平台上的通...