Hadoop安裝與集群配置

環境準備

節點teacher1

teacher2

teacher3

hdfs

namenode，datanode

datanode

datanode，secondarynamenode

yarn

nodemanager

resourcemanager,nodemanager

nodemanager

192.168.52.133 teacher1

192.168.52.135 teacher2
192.168.52.136 teacher3

192.168.52.133 teacher1

192.168.52.135 teacher2
192.168.52.136 teacher3

192.168.52.133 teacher1

192.168.52.135 teacher2
192.168.52.136 teacher3

ssh-keygen -t  rsa

ssh-copy-id -i ~/.ssh/id_rsa.pub root@teacher1 
ssh-copy-id -i ~/.ssh/id_rsa.pub root@teacher2 
ssh-copy-id -i ~/.ssh/id_rsa.pub root@teacher3

hadoop安裝

解壓在teacher1解壓，解壓路徑/opt/hadoop環境配置

分別配置三颱主機的hadoop環境

export hadoop_home=/opt/hadoop/hadoop-2.7.2

export path=$path:$hadoop_home/bin

hadoop配置檔案在解壓路徑的 etc下/opt/hadoop/hadoop-2.7.2/etc/hadoop

fs.defaultfs

hdfs://teacher1:9000
hadoop.tmp.dir
/opt/hadoop/hadoop-2.7.2/data/tmp

dfs.replication

3 輔助namenode工作
!-- 指定yarn的resourcemanager的位址 -->
yarn.resourcemanager.hostname
teacher2
yarn.nodemanager.resource.memory-mb
3072

<

!-- 指定mr執行在yarn上 -->
mapreduce.framework.name
yarn

teacher1

teacher2
teacher3

scp -r hadoop-2.7.2 teacher1:/opt/hadoop/

scp -r hadoop-2.7.2 teacher2:/opt/hadoop/

hadoop-daemon.sh  start|stop  namenode|datanode|secondarynamenode

yarn-daemon.sh start|stop resourcemanager|nodemanager

各個模組分開啟動/停止(配置ssh是前提)常用

(1)整體啟動/停止hdfs(在namenode節點啟動) start-dfs.sh

stop-dfs.sh
(2)整體啟動/停止yarn (在resourcemanager節點啟動)
start-yarn.sh
stop-yarn.sh

web端檢視secondarynamenode

這裡我配置的host是在虛擬機器配置的，宿主機直接ip訪問了，當然也可以配置和虛擬機器一樣的hosts，通過host訪問。

Hadoop集群配置

hadoop的配置檔案分為兩類 1.唯讀型別的預設檔案 src core core default.xml src hdfs hfds default.xml src mapred mapred default.xml conf mapred queues.xml 2.定位設定 conf core ...

hadoop集群配置

建立目錄 修改 home hadoop hadoop 2.7.1 etc hadoop下的配置檔案 把配置好的hadoop傳送到從節點 如果沒有目錄，請新建，沒有jdk，請提前裝，路徑和master保持一致 scp r home hadoop hadoop 2.7.1 root centos02 h...

Hadoop 集群配置

重要目錄介紹 2 etc目錄 hadoop的配置檔案目錄，存放hadoop的配置檔案 3 lib目錄 存放hadoop的本地庫 對資料進行壓縮解壓縮功能 5 share目錄 存放hadoop的依賴jar包 文件 和官方案例 hadoop執行模式包括 本地模式 偽分布式模式以及完全分布式模式。本地模式...