首页 > Web开发 > 详细

通过 微软 pai-fs 上传数据到HDFS (Microsoft OpenPAI)

时间:2019-04-16 15:06:29      阅读:248      评论:0      收藏:0      [点我收藏+]

 准备环境

(个人使用记录,方便下次使用查阅~~)

首先保证PAI是登陆状态:

技术分享图片

 

进入GitHub项目所在地址:

https://github.com/Microsoft/pai/

然后切换分支到 技术分享图片

技术分享图片

具体 使用文档再这里:

技术分享图片

https://github.com/Microsoft/pai/tree/pai-0.5.y/pai-fs

 

将代码clone下来后执行:

`python pai-fs.py --config host=xxx.xxx.234.79 port=50070 user=root`(配置Server)

上传数据到HDFS

进到pai-fs.py目录下(也可以直接绝对路径),然后通过 :

python pai-fs.py + 参数 +路径

进行数据上传

技术分享图片

 

具体使用方法:

(Github README.md)

example use:
  pai-fs --config host=10.0.3.9 port=50070 user=root         -- store hdfs config
  pai-fs -ls hdfs://                                         -- list the contents of a root HDFS directory 
  pai-fs -ls hdfs:// --host 10.0.3.9                         -- list the contents of a root HDFS directory with host specified
  pai-fs -ls hdfs:// --host 10.0.3.9 --port 50070 --user root    -- list the contents of a root HDFS directory with host, port and user specified
  pai-fs -ls -r hdfs://                                      -- list the contents of a root HDFS directory, recursively 
  pai-fs -mkdir hdfs://mydir/mysubdir/mysubdir2              -- makes mysubdir2 and all directories along the way 
  pai-fs -rm hdfs://mydir/mysubdir/myfile                    -- removes myfile from mysubdir 
  pai-fs -rm hdfs://mydir/mysubdir                           -- removes mysubdir and all files and directories in it 
  pai-fs -cp c:\mylocalfile hdfs://mydir/myremotedir         -- copy mylocalfile into myremotedir 
  pai-fs -cp -r c:\mylocaldir hdfs://mydir/myremotedir       -- copy mylocaldir into myremotedir, recursively 
  pai-fs -cp -r c:\mylocaldir\* hdfs://mydir/myremotedir     -- copy mylocaldirs contents into myremotedir, recursively 
  pai-fs -cp c:\mylocaldir\\a hdfs://mydir/myremotedir/b     -- copy file a from mylocaldir to myremotedir and rename to b 
  pai-fs -cp -r hdfs://mydir/myremotedir c:\mylocaldir       -- copy myremotedir into mylocaldir, recursively 
  pai-fs -cp -r hdfs://mydir/myremotedir/* c:\mylocaldir     -- copy myremotedirs contents into mylocaldir, recursively 
exit code:
  0   -- Success 
  1   -- An exception happened during the operation including bad connection 
  2   -- PAI_VC environment variable not set to valid VC or insufficient/invalid command line argument(s) 
  3   -- Path not found 
  4   -- Unauthorized access 
  5   -- Path not empty 
  6   -- Check failed after operation 
  100 -- Failed to copy too many times 
  101 -- Failed to concat chunks into file 

使用演示

执行命令后,开始上传

技术分享图片

 

通过 微软 pai-fs 上传数据到HDFS (Microsoft OpenPAI)

原文:https://www.cnblogs.com/bigtreei/p/10717069.html

(0)
(0)
   
举报
评论 一句话评论(0
关于我们 - 联系我们 - 留言反馈 - 联系我们:wmxa8@hotmail.com
© 2014 bubuko.com 版权所有
打开技术之扣,分享程序人生!