先来个对比图看一下, 左边图是普通 rsync 目录拷贝, 右边图是借助 parallel 工具并发起了多个 rsync
centos6.5安装 parallel
#!/bin/bash # Install parallel on CentOS 6. # Assumes you are root. Prefix w/ sudo if not. cd /etc/yum.repos.d/ #wget http://download.opensuse.org/repositories/home:tange/CentOS_CentOS-5/home:tange.repo wget http://download.opensuse.org/repositories/home:/tange/CentOS_CentOS-6/home:tange.repo yum install parallel
开始 rsync 拷贝文件 [从/data/ 拷贝到 /hadoop/]
[10 ~]$ df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup-lv_root 50G 6.1G 41G 13% / tmpfs 32G 12K 32G 1% /dev/shm /dev/sda1 485M 39M 421M 9% /boot /dev/mapper/VolGroup-lv_home 5.5G 140M 5.1G 3% /home /dev/mapper/data-apps 12T 5.2T 6.7T 44% /data /dev/mapper/bigdata-hadoop 12T 3.5T 8.3T 30% /hadoop
如果用普通的 rsync, 再加上很多小文件,一晚上只传了1T 多点,使用 parallel 工具 1小时大概传了500G
cd /data find . -type f | parallel ‘mkdir -p /hadoop/{//}; rsync -a {} /hadoop/{}‘
原文：http://www.cnblogs.com/txwsqk/p/7570162.html