首页 > 其他 > 详细

spark操作hbase

时间:2019-09-29 20:20:38      阅读:131      评论:0      收藏:0      [点我收藏+]

hbase的操作命令 https://www.cnblogs.com/lzh-boy/p/8966826.html

环境配置略

需要注意:在Spark 2.0版本上缺少相关把hbase的数据转换python可读取的jar包

code:查看表数据

from pyspark.sql import SparkSession
import os
os.environ[‘PYSPARK_PYTHON‘]=‘/opt/anaconda2/bin/python‘
sc = SparkSession.builder.master("local").appName("hbase").getOrCreate()
host = ‘localhost‘
table = ‘student‘
conf = {"hbase.zookeeper.quorum": host, "hbase.mapreduce.inputtable": table}
keyConv = "org.apache.spark.examples.pythonconverters.ImmutableBytesWritableToStringConverter"
valueConv = "org.apache.spark.examples.pythonconverters.HBaseResultToStringConverter"
hbase_rdd = sc.sparkContext.newAPIHadoopRDD("org.apache.hadoop.hbase.mapreduce.TableInputFormat","org.apache.hadoop.hbase.io.ImmutableBytesWritable","org.apache.hadoop.hbase.client.Result",keyConverter=keyConv,valueConverter=valueConv,conf=conf)
count = hbase_rdd.count()
hbase_rdd.cache()
output = hbase_rdd.collect()
for (k, v) in output:
print (k, v)

spark操作hbase

原文:https://www.cnblogs.com/xiennnnn/p/11609290.html

(0)
(0)
   
举报
评论 一句话评论(0
关于我们 - 联系我们 - 留言反馈 - 联系我们:wmxa8@hotmail.com
© 2014 bubuko.com 版权所有
打开技术之扣,分享程序人生!