CM5 x配置spark錯誤解決

2022-10-11 05:51:11 字數 4426 閱讀 9869

通過cloudera manager 5.x新增spark服務,在建立服務過程中,發現spark服務建立失敗,可以通過控制台錯誤輸出看到如下日誌資訊:

+ perl -pi -e 's#}#/etc/spark/conf.cloudera.spark_on_yarn/yarn-conf#g' /opt/cm-5.9.2/run/cloudera-scm-agent/process/ccdeploy_spark-conf_etcsparkconf.cloudera.spark_on_yarn_1615663591259519890/spark-conf/yarn-conf/yarn-site.xml

++ get_default_fs /opt/cm-5.9.2/run/cloudera-scm-agent/process/ccdeploy_spark-conf_etcsparkconf.cloudera.spark_on_yarn_1615663591259519890/spark-conf/yarn-conf

++ get_hadoop_conf /opt/cm-5.9.2/run/cloudera-scm-agent/process/ccdeploy_spark-conf_etcsparkconf.cloudera.spark_on_yarn_1615663591259519890/spark-conf/yarn-conf fs.defaultfs

++ local conf=/opt/cm-5.9.2/run/cloudera-scm-agent/process/ccdeploy_spark-conf_etcsparkconf.cloudera.spark_on_yarn_1615663591259519890/spark-conf/yarn-conf

++ local key=fs.defaultfs

++ '[' 1 == 1 ']'

++ /opt/cloudera/parcels/cdh-5.9.2-1.cdh5.9.2.p0.3/lib/hadoop/../../bin/hdfs --config /opt/cm-5.9.2/run/cloudera-scm-agent/process/ccdeploy_spark-conf_etcsparkconf.cloudera.spark_on_yarn_1615663591259519890/spark-conf/yarn-conf getconf -confkey fs.defaultfs

exception in thread "main" j**a.lang.unsupportedclassversionerror: org/apache/hadoop/hdfs/tools/getconf : unsupported major.minor version 51.0

at j**a.lang.classloader.defineclass1(native method)

at j**a.lang.classloader.defineclass(classloader.j**a:643)

at j**a.security.secureclassloader.defineclass(secureclassloader.j**a:142)

at j**a.net.urlclassloader.defineclass(urlclassloader.j**a:277)

at j**a.net.urlclassloader.access$000(urlclassloader.j**a:73)

at j**a.net.urlclassloader$1.run(urlclassloader.j**a:212)

at j**a.security.accesscontroller.doprivileged(native method)

at j**a.net.urlclassloader.findclass(urlclassloader.j**a:205)

at j**a.lang.classloader.loadclass(classloader.j**a:323)

at j**a.lang.classloader.loadclass(classloader.j**a:268)

could not find the main class: org.apache.hadoop.hdfs.tools.getconf. program will exit.

+ default_fs=

根據輸出日誌資訊大致可以判斷這是因為jdk版本導致的新增spark服務失敗。因為這是我全權安裝的環境,所以印象中jdk版本是滿足cm5安裝要求的,我這裡使用的是jdk1.7.0_67,如下:

# j**a -version

j**a version "1.7.0_67"

j**a(tm) se runtime environment (build 1.7.0_67-b01)

j**a hotspot(tm) 64-bit server vm (build 24.65-b04, mixed mode)

you h**e new mail in /var/spool/mail/root

針對於目前j**a版本並沒有問題,滿足當前安裝的cm5版本要求。所以判斷可能是因為我是通過tar包方式安裝的j**a的原因,正常通過rpm包安裝應該沒有這個問題。下面驗證自己的猜測:

這裡使用alternatives命令,alternatives命令通常用來管理伺服器上的相同軟體多版本問題。

--檢視伺服器j**a版本,發現jdk1.7.0_67沒有再伺服器管理之下:

[root@db01 ~]# alternatives --config j**a

there are 2 programs which provide 'j**a'.

selection    command

1           /usr/lib/jvm/jre-1.5.0-gcj/bin/j**a

*+ 2           /usr/lib/jvm/jre-1.6.0-openjdk.x86_64/bin/j**a

enter to keep the current selection[+], or type selection number:

--將jdk1.7.0_67新增到伺服器管理中:

[root@db01 ~]# alternatives --install /usr/bin/j**a j**a /opt/j**a/jdk1.7.0_67/bin/j**a 3

--再次檢視伺服器j**a版本資訊,並且調整優先順序最高的為jdk1.7.0_67:

[root@db01 ~]# alternatives --config j**a

there are 3 programs which provide 'j**a'.

selection    command

1           /usr/lib/jvm/jre-1.5.0-gcj/bin/j**a

*+ 2           /usr/lib/jvm/jre-1.6.0-openjdk.x86_64/bin/j**a

3           /opt/j**a/jdk1.7.0_67/bin/j**a

enter to keep the current selection[+], or type selection number: 3

[root@db01 ~]# alternatives --config j**a

there are 3 programs which provide 'j**a'.

selection    command

1           /usr/lib/jvm/jre-1.5.0-gcj/bin/j**a

*  2           /usr/lib/jvm/jre-1.6.0-openjdk.x86_64/bin/j**a

+ 3           /opt/j**a/jdk1.7.0_67/bin/j**a

enter to keep the current selection[+], or type selection number:

調整j**a版本資訊後,再次新增spark服務,成功。

或者解除安裝原生的j**a版本,如:

# rpm -e j**a-1.5.0-gcj-1.5.0.0-29.1.el6.x86_64 j**a-1.6.0-openjdk-1.6.0.0-1.66.1.13.0.el6.x86_64 tzdata-j**a-2013g-1.el6.noarch j**a_cup-0.10k-5.el6.x86_64 j**a-1.6.0-openjdk-devel-1.6.0.0-1.66.1.13.0.el6.x86_64 gcc-j**a-4.4.7-4.el6.x86_64 --nodeps

如果以上方法不可用,採用以下方法,直接指定環境變數:

find / -type f -name "*cc.sh"

定位到/opt/program/cm-5.9.0/lib64/cmf/service/client/deploy-cc.sh

直接在上面加上

j**a_home=/opt/j**a

export j**a_home=/opt/j**a

0x8007007E 錯誤解決

message system.io.filenotfoundexception the specified module could not be found.exception from hresult 0x8007007e 這個錯誤表示程式用到了某個dll檔案,但本地windows系統並沒有註冊...

Oracle tns配置的6種常見錯誤解決問題

經過長時間學習oracle,你可能會遇到oracle tns配置問題,這裡將介紹oracle tns配置問題的解決方法。最近忙著安裝oracle資料庫,本來挺簡單的,可老是出現問題,最後自己在網上整理了一些tns錯誤解決方法,希望對初學者有益。常見問題 1 ora 12541 tns 沒有 顯而易見...

MySql免安裝版 配置 3543錯誤解決

1.解壓mysql免安裝版 2.環境變數 path 裡面新增mysql路徑 例子 f install soft mysql 8.0.15 winx64 bin 3.配置my.ini mysqld basedir f install soft mysql 8.0.15 winx64 datadir f...