Spark On Yarn（HDFS HA）详细配置

时间：2014-02-23 15:23:12 阅读：507 评论：0 收藏：0 [点我收藏+]

######################################################################################################

一、服务器分布及相关说明

######################################################################################################

1、服务器角色

服务角色/服务器	172.18.35.29（namenode1）	172.18.35.30（namenode2）	172.18.34.232（datanode1）	172.18.24.136（datanode2）
NameNode	YES	YES	NO	NO
DataNode	NO	NO	YES	YES
JournalNode	YES	YES	YES	NO
ZooKeeper	YES	YES	YES	NO
ZKFC	YES	YES	NO	NO

2、Hadoop（HDFS HA）总体架构

######################################################################################################

二、基础环境部署

######################################################################################################

1、JDK安装

http://download.oracle.com/otn-pub/java/jdk/7u45-b18/jdk-7u45-linux-x64.tar.gz

# tar xvzf jdk-7u45-linux-x64.tar.gz -C/usr/local

# cd /usr/local

# ln -s jdk1.7.0_45 jdk

# vim /etc/profile

exportJAVA_HOME=/usr/local/jdk

exportCLASS_PATH=$JAVA_HOME/lib:$JAVA_HOME/jre/lib

exportPATH=$PATH:$JAVA_HOME/bin

# source /etc/profile

=========================================================================================

2、Scala安装

http://www.scala-lang.org/files/archive/scala-2.10.3.tgz

# tar xvzf scala-2.10.3.tgz -C/usr/local

# cd /usr/local

# ln -s scala-2.10.3 scala

# vim /etc/profile

exportSCALA_HOME=/usr/local/scala

exportPATH=$PATH:$SCALA_HOME/bin

# source /etc/profile

=========================================================================================

3、SSH免密码登录

可参考文章：

http://blog.csdn.net/codepeak/article/details/14447627

......

=========================================================================================

4、主机名设置

# vim /etc/hosts

172.18.35.29 namenode1

172.18.35.30 namenode2

172.18.34.232 datanode1

172.18.24.136 datanode2

######################################################################################################

三、ZooKeeper集群部署

######################################################################################################

1、ZooKeeper安装

http://apache.dataguru.cn/zookeeper/stable/zookeeper-3.4.5.tar.gz

# tar xvzf zookeeper-3.4.5.tar.gz -C/usr/local

# cd /usr/local

# ln -s zookeeper-3.4.5 zookeeper

# vim /etc/profile

exportZOO_HOME=/usr/local/zookeeper

exportZOO_LOG_DIR=/data/hadoop/zookeeper/logs

exportPATH=$PATH:$ZOO_HOME/bin

# source /etc/profile

=========================================================================================

2、ZooKeeper配置与启动

# mkdir -p/data/hadoop/zookeeper/{data,logs}

# vim /usr/local/zookeeper/conf/zoo.cfg

tickTime=2000

initLimit=10

syncLimit=5

dataDir=/data/hadoop/zookeeper/data

clientPort=2181

server.1=172.18.35.29:2888:3888

server.2=172.18.35.30:2888:3888

server.3=172.18.34.232:2888:3888

在172.18.35.29上执行：

echo 1 > /data/hadoop/zookeeper/data/myid

在172.18.35.30 上执行：

echo 2 > /data/hadoop/zookeeper/data/myid

在172.18.34.232 上执行：

echo 3 > /data/hadoop/zookeeper/data/myid

## 启动ZooKeeper集群

# cd /usr/local/zookeeper

# bin/zkServer.shstart

# ./bin/zkCli.sh -server localhost:2181

测试zookeeper集群是否建立成功，如无报错表示集群创建成功

# bin/zkServer.sh status

######################################################################################################

四、Hadoop（HDFS HA）集群部署

######################################################################################################

1、hadoop环境安装

Hadoop的源码编译部分可以参考：

http://sofar.blog.51cto.com/353572/1352713

# tar xvzf hadoop-2.2.0.tgz -C/usr/local

# cd /usr/local

# ln -s hadoop-2.2.0 hadoop

# vim /etc/profile

exportHADOOP_HOME=/usr/local/hadoop

exportHADOOP_PID_DIR=/data/hadoop/pids

exportHADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

exportHADOOP_OPTS="$HADOOP_OPTS-Djava.library.path=$HADOOP_HOME/lib/native"

exportHADOOP_MAPRED_HOME=$HADOOP_HOME

exportHADOOP_COMMON_HOME=$HADOOP_HOME

exportHADOOP_HDFS_HOME=$HADOOP_HOME

exportYARN_HOME=$HADOOP_HOME

exportHADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

exportHDFS_CONF_DIR=$HADOOP_HOME/etc/hadoop

exportYARN_CONF_DIR=$HADOOP_HOME/etc/hadoop

exportJAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native

exportPATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

# mkdir -p /data/hadoop/{pids,storage}

# mkdir -p/data/hadoop/storage/{hdfs,tmp,journal}

# mkdir -p/data/hadoop/storage/tmp/nodemanager/{local,remote,logs}

# mkdir -p/data/hadoop/storage/hdfs/{name,data}

=========================================================================================

2、core.site.xml配置

# vim/usr/local/hadoop/etc/hadoop/core-site.xml

<?xmlversion="1.0" encoding="UTF-8"?>

<?xml-stylesheettype="text/xsl" href=\‘#\‘" /span>

<name>fs.defaultFS</name>

<value>hdfs://appcluster</value>

</property>

<name>io.file.buffer.size</name>

</property>

<name>hadoop.tmp.dir</name>

<value>file:/data/hadoop/storage/tmp</value>

</property>

<name>ha.zookeeper.quorum</name>

</property>

<name>ha.zookeeper.session-timeout.ms</name>

</property>

<name>fs.trash.interval</name>

</property>

<name>hadoop.http.staticuser.use</name>

</property>

<name>hadoop.proxyuser.hadoop.hosts</name>

</property>

<name>hadoop.proxyuser.hadoop.groups</name>

</property>

<name>hadoop.native.lib</name>

</property>

</configuration>

=========================================================================================

3、hdfs-site.xml配置

# vim/usr/local/hadoop/etc/hadoop/hdfs-site.xml

<?xmlversion="1.0" encoding="UTF-8"?>

<?xml-stylesheettype="text/xsl" href=\‘#\‘" /span>

<name>dfs.namenode.name.dir</name>

<value>file:/data/hadoop/storage/hdfs/name</value>

</property>

<name>dfs.datanode.data.dir</name>

<value>file:/data/hadoop/storage/hdfs/data</value>

</property>

<name>dfs.replication</name>

</property>

<name>dfs.blocksize</name>

</property>

<name>dfs.datanode.du.reserved</name>

</property>

<name>dfs.webhdfs.enabled</name>

</property>

<name>dfs.permissions</name>

<value>false</value>

</property>

<name>dfs.permissions.enabled</name>

<value>false</value>

</property>

<name>dfs.nameservices</name>

<value>appcluster</value>

</property>

<name>dfs.ha.namenodes.appcluster</name>

</property>

<name>dfs.namenode.rpc-address.appcluster.nn1</name>

<value>namenode1:8020</value>

</property>

<name>dfs.namenode.rpc-address.appcluster.nn2</name>

<value>namenode2:8020</value>

</property>

<name>dfs.namenode.servicerpc-address.appcluster.nn1</name>

<value>namenode1:53310</value>

</property>

<name>dfs.namenode.servicerpc-address.appcluster.nn2</name>

<value>namenode2:53310</value>

</property>

<name>dfs.namenode.http-address.appcluster.nn1</name>

<value>namenode1:8080</value>

</property>

<name>dfs.namenode.http-address.appcluster.nn2</name>

<value>namenode2:8080</value>

</property>

<name>dfs.datanode.http.address</name>

</property>

<name>dfs.namenode.shared.edits.dir</name>

<value>qjournal://namenode1:8485;namenode2:8485;datanode1:8485/appcluster</value>

</property>

<name>dfs.client.failover.proxy.provider.appcluster</name>

<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>

</property>

<name>dfs.ha.fencing.methods</name>

<value>sshfence(root:36000)</value>

</property>

<name>dfs.ha.fencing.ssh.private-key-files</name>

</property>

<name>dfs.ha.fencing.ssh.connect-timeout</name>

</property>

<name>dfs.journalnode.edits.dir</name>

<value>/data/hadoop/storage/hdfs/journal</value>

</property>

<name>dfs.ha.automatic-failover.enabled</name>

</property>

<name>ha.failover-controller.cli-check.rpc-timeout.ms</name>

</property>

<name>ipc.client.connect.timeout</name>

</property>

<name>dfs.image.transfer.bandwidthPerSec</name>

</property>

<name>dfs.namenode.accesstime.precision</name>

</property>

<name>dfs.datanode.max.transfer.threads</name>

</property>

</configuration>

=========================================================================================

4、mapred-site.xml配置

# vim/usr/local/hadoop/etc/hadoop/mapred-site.xml

<?xmlversion="1.0"?>

<?xml-stylesheettype="text/xsl" href=\‘#\‘" /span>

<name>mapreduce.framework.name</name>

</property>

<name>mapreduce.jobhistory.address</name>

<value>namenode1:10020</value>

</property>

<name>mapreduce.jobhistory.webapp.address</name>

<value>namenode1:19888</value>

</property>

</configuration>

=========================================================================================

5、yarn-site.xml配置

# vim/usr/local/hadoop/etc/hadoop/yarn-site.xml

<?xmlversion="1.0"?>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

<value>org.apache.hadoop.mapred.ShuffleHandler</value>

</property>

<name>yarn.resourcemanager.scheduler.address</name>

<value>namenode1:8030</value>

</property>

<name>yarn.resourcemanager.resource-tracker.address</name>

<value>namenode1:8031</value>

</property>

<name>yarn.resourcemanager.address</name>

<value>namenode1:8032</value>

</property>

<name>yarn.resourcemanager.admin.address</name>

<value>namenode1:8033</value>

</property>

<name>yarn.nodemanager.address</name>

<value>namenode1:8034</value>

</property>

<name>yarn.nodemanager.webapp.address</name>

<value>namenode1:80</value>

</property>

<name>yarn.resourcemanager.webapp.address</name>

<value>namenode1:80</value>

</property>

<name>yarn.nodemanager.local-dirs</name>

<value>${hadoop.tmp.dir}/nodemanager/local</value>

</property>

<name>yarn.nodemanager.remote-app-log-dir</name>

<value>${hadoop.tmp.dir}/nodemanager/remote</value>

</property>

<name>yarn.nodemanager.log-dirs</name>

<value>${hadoop.tmp.dir}/nodemanager/logs</value>

</property>

<name>yarn.nodemanager.log.retain-seconds</name>

</property>

<name>yarn.nodemanager.resource.cpu-vcores</name>

</property>

<name>yarn.nodemanager.resource.memory-mb</name>

</property>

<name>yarn.scheduler.minimum-allocation-mb</name>

</property>

<name>yarn.scheduler.maximum-allocation-mb</name>

</property>

<name>yarn.scheduler.minimum-allocation-vcores</name>

</property>

<name>yarn.scheduler.maximum-allocation-vcores</name>

</property>

</configuration>

【注意：上面红色标注的部分，需要根据服务器的硬件配置进行修改】

=========================================================================================

6、配置hadoop-env.sh、mapred-env.sh、yarn-env.sh【在开头添加】

文件路径：

/usr/local/hadoop/etc/hadoop/hadoop-env.sh

/usr/local/hadoop/etc/hadoop/mapred-env.sh

/usr/local/hadoop/etc/hadoop/yarn-env.sh

添加内容：

exportJAVA_HOME=/usr/local/jdk

exportCLASS_PATH=$JAVA_HOME/lib:$JAVA_HOME/jre/lib

export HADOOP_HOME=/usr/local/hadoop

exportHADOOP_PID_DIR=/data/hadoop/pids

exportHADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

exportHADOOP_OPTS="$HADOOP_OPTS-Djava.library.path=$HADOOP_HOME/lib/native"

exportHADOOP_PREFIX=$HADOOP_HOME

export HADOOP_MAPRED_HOME=$HADOOP_HOME

exportHADOOP_COMMON_HOME=$HADOOP_HOME

exportHADOOP_HDFS_HOME=$HADOOP_HOME

exportYARN_HOME=$HADOOP_HOME

exportHADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

exportHDFS_CONF_DIR=$HADOOP_HOME/etc/hadoop

exportYARN_CONF_DIR=$HADOOP_HOME/etc/hadoop

exportJAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native

exportPATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

=========================================================================================

7、数据节点配置

# vim /usr/local/hadoop/etc/hadoop/slaves

datanode1

datanode2

=========================================================================================

8、集群启动

(1)、在namenode1上执行，创建命名空间

# hdfs zkfc -formatZK

----------------------------------------------------------------------------------------------------------------------------------------------

(2)、在对应的节点上启动日志程序journalnode

# cd /usr/local/hadoop &&./sbin/hadoop-daemon.sh start journalnode

----------------------------------------------------------------------------------------------------------------------------------------------

(3)、格式化主NameNode节点（namenode1）

# hdfs namenode -format

----------------------------------------------------------------------------------------------------------------------------------------------

(4)、启动主NameNode节点

# cd /usr/local/hadoop &&sbin/hadoop-daemon.sh start namenode

----------------------------------------------------------------------------------------------------------------------------------------------

(5)、格式备NameNode节点（namenode2）

# hdfs namenode -bootstrapStandby

----------------------------------------------------------------------------------------------------------------------------------------------

(6)、启动备NameNode节点（namenode2）

# cd /usr/local/hadoop &&sbin/hadoop-daemon.sh start namenode

----------------------------------------------------------------------------------------------------------------------------------------------

(7)、在两个NameNode节点（namenode1、namenode2）上执行

# cd /usr/local/hadoop && sbin/hadoop-daemon.shstart zkfc

----------------------------------------------------------------------------------------------------------------------------------------------

(8)、启动所有的DataNode节点（datanode1、datanode2）

# cd /usr/local/hadoop &&sbin/hadoop-daemon.sh start datanode

----------------------------------------------------------------------------------------------------------------------------------------------

(9)、启动Yarn（namenode1）

# cd /usr/local/hadoop &&sbin/start-yarn.sh

主NameNode节点上的信息：

DataNode节点上的信息：

----------------------------------------------------------------------------------------------------------------------------------------------

(10)、测试Yarn是否可用

# hdfs dfs -put/usr/local/hadoop/etc/hadoop/yarn-site.xml /tmp

# hadoop jar/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jarwordcount /tmp/yarn-site.xml /mytest

-----------------------------------------------------------------------------------------------------------------------------------------------

(11)、HDFS的HA功能测试

切换前的状态：

# kill -9 11466

# cd /usr/local/hadoop &&sbin/hadoop-daemon.sh start namenode

切换后的状态：

-----------------------------------------------------------------------------------------------------------------------------------------------

(12)、后续维护

HDFS的关闭与启动：

# cd /usr/local/hadoop &&sbin/stop-dfs.sh

# cd /usr/local/hadoop &&sbin/start-dfs.sh

YARN的关闭与启动：

# cd /usr/local/hadoop &&sbin/stop-yarn.sh

# cd /usr/local/hadoop &&sbin/start-yarn.sh

【注意】

需要在NameNode节点上执行。

######################################################################################################

五、Spark集群部署

######################################################################################################

1、Spark安装与配置

Spark的源码编译请参考：

http://sofar.blog.51cto.com/353572/1358457

# tar xvzf spark-0.9.0-incubating.tgz -C/usr/local

# cd /usr/local

# ln -s spark-0.9.0-incubating spark

# vim /etc/profile

export SPARK_HOME=/usr/local/spark

export PATH=$PATH:$SPARK_HOME/bin

# source /etc/profile

# cd /usr/local/spark/conf

# mkdir -p /data/spark/tmp

----------------------------------------------------------------------------------------------------------------------------------------------

# vim spark-env.sh

export JAVA_HOME=/usr/local/jdk

export SCALA_HOME=/usr/local/scala

export HADOOP_HOME=/usr/local/hadoop

SPARK_LOCAL_DIR="/data/spark/tmp"

SPARK_JAVA_OPTS="-Dspark.storage.blockManagerHeartBeatMs=60000-Dspark.local.dir=$SPARK_LOCAL_DIR -XX:+PrintGCDetails -XX:+PrintGCTi

meStamps -Xloggc:$SPARK_HOME/logs/gc.log -XX:+UseConcMarkSweepGC-XX:+UseCMSCompactAtFullCollection -XX:CMSInitiatingOccupancyFracti

on=60"

----------------------------------------------------------------------------------------------------------------------------------------------

# vim slaves

datanode1

datanode2

# cd /usr/local/spark &&sbin/start-all.sh

=========================================================================================

2、相关测试

(1)、本地模式

# bin/run-exampleorg.apache.spark.examples.SparkPi local

----------------------------------------------------------------------------------------------------------------------------------------------

(2)、普通集群模式

# bin/run-exampleorg.apache.spark.examples.SparkPi spark://namenode1:7077

# bin/run-exampleorg.apache.spark.examples.SparkLR spark://namenode1:7077

# bin/run-exampleorg.apache.spark.examples.SparkKMeans spark://namenode1:7077file:/usr/local/spark/data/kmeans_data.txt 2 1

----------------------------------------------------------------------------------------------------------------------------------------------

(3)、结合HDFS的集群模式

# hadoop fs -put README.md .

# MASTER=spark://namenode1:7077bin/spark-shell

scala> val file =sc.textFile("hdfs://namenode1:9000/user/root/README.md")

scala> val count = file.flatMap(line=> line.split(" ")).map(word => (word, 1)).reduceByKey(_+_)

scala> count.collect()

scala> :quit

----------------------------------------------------------------------------------------------------------------------------------------------

(4)、基于YARN模式

#SPARK_JAR=assembly/target/scala-2.10/spark-assembly_2.10-0.9.0-incubating-hadoop2.2.0.jar\

bin/spark-classorg.apache.spark.deploy.yarn.Client \

--jarexamples/target/scala-2.10/spark-examples_2.10-assembly-0.9.0-incubating.jar \

--classorg.apache.spark.examples.SparkPi \

--args yarn-standalone \

--num-workers 3 \

--master-memory 4g \

--worker-memory 2g \

--worker-cores 1

----------------------------------------------------------------------------------------------------------------------------------------------

(5)、最终的目录结构及相关配置

目录结构：

/etc/profile中的配置：

本文出自 “人生理想在于坚持不懈” 博客，请务必保留此出处http://sofar.blog.51cto.com/353572/1362240

Spark On Yarn（HDFS HA）详细配置

原文：http://sofar.blog.51cto.com/353572/1362240

踩

(0)

评论一句话评论（0）

分享档案

更多>

2021年09月23日 (328)
2021年09月24日 (313)
2021年09月17日 (191)
2021年09月15日 (369)
2021年09月16日 (411)
2021年09月13日 (439)
2021年09月11日 (398)
2021年09月12日 (393)
2021年09月10日 (160)
2021年09月08日 (222)