本文主要介绍Shark Hive Spark Hadoop2 单机运行环境的搭建。先简单对各个系统进行简单的介绍。
各个系统的版本：
| 名称 | 下载地址 |
| Spark 0.8.1 |
http://d3kbcqa49mib13.cloudfront.net/spark-0.8.1-incubating-bin-cdh4.tgz |
| Shark 0.8.1 |
https://github.com/amplab/shark/releases/download/v0.8.1/shark-0.8.1-bin-cdh4.tgz |
| Hive 0.9.0 | https://github.com/amplab/shark/releases/download/v0.8.1/hive-0.9.0-bin.tgz |
| Hadoop2-CDH4.3 | http://archive.cloudera.com/cdh4/cdh/4/hadoop-2.0.0-cdh4.3.0.tar.gz |
| Scala 0.9.3 | http://www.scala-lang.org/files/archive/scala-2.9.3.tgz |
Scala: 下载之后配置到环境变量中，此处略
hadoop2：
#core-site.xml
<configuration> <property> <name>fs.defaultFS</name> <value>hdfs://localhost:8020</value> </property> <property> <name>dfs.replication</name> <value>1</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/usr/yourlocal</value> </property> </configuration>
#hadoop-env.sh
....
export JAVA_HOME=$JAVA_HOME #本机的jdk路径
....
spark:
$cd $YOUR_SPARK_HOME/
$vim conf/spark-env.sh
修改spark-env.sh
export SCALA_HOME=$YOUR_SCALA_HOME
export JAVA_HOME=$YOUR_JAVA_HOME
export SPARK_MASTER_IP=$YOUR_MASTER_HOST
export SPARK_MASTER_PORT=$YOUR_MASTER_PORT
export SPARK_WORKER_CORES=1
export SPARK_WORKER_MEMORY=1g
export SPARK_WORKER_INSTANCES=1
export SPARK_JAVA_OPTS="-verbose:gc -XX:-PrintGCDetails -XX:+PrintGCTimeStamps"
export HADOOP_HOME=$YOUR_HADOOP_HOME
Hive:
如果使用mysql保存hive的元数据相关信息，需要拷贝mysql-connector-java-3.1.13-bin.jar 到$HIVE_HOME/lib 目录下。
#cd $YOUR_HIVE_HOME
#cp conf/hive-env.sh.template conf/hive-env.sh
#cp conf/hive-default.xml.template conf/hive-site.xml
#vim conf/hive-env.sh
修改hive-env.xml
....
HADOOP_HOME=$YOUR_HADOOP2_HOME
....
修改hive-site.xml
.....
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://localhost:3306/SHARK_DATABASE?createDatabaseIfNotExist=true</value>
<description>JDBC connect string for a JDBC
metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
<description>Driver
class name for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>shark</value>
<description>username to use against
metastore database</description>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>shark</value>
<description>password to use against
metastore database</description>
</property>
......
Shark:
#cd $YOUR_SHARK_HOME
#cp conf/shark-env.sh.template conf/shark-env.sh
#vim conf/shark-env.sh
修改shark-env.sh
......
export SCALA_HOME=$YOUR_SCALA_HOME export HIVE_HOME=$YOUR_HIVE_HOME export HADOOP_HOME=$YOUR_HADOOP_HOME export SPARK_HOME=$YOUR_SPARK_HOME export MASTER=spark://localhost:8888
......
#cd $YOUR_HADOOP_HOME #bin/hadoop namenode -format #sbin/start-alll.sh #cd $YOUR_SPARK_HOME #bin/start-all.sh
#cd $YOUR_SHARK_HOME
#bin/shark
Starting the Shark Command Line Client
......
......
shark>show databases;
shark>create database SHARK_DB;
shark>use SHARK_DB;
shark>create table tbl_test(ID STRING);
搭建完毕。
----DONE----
知识源于网络 转载请注明出处http://www.cnblogs.com/nb591/p/3644388.html
Shark Hive Spark Hadoop2 环境搭建,布布扣,bubuko.com
原文：http://www.cnblogs.com/nb591/p/3644388.html