python dask 搭建分布式集群

1.conda安装：conda install dask distributed-cconda-forge

2.pip 安装：pip install dask distributed --upgrade

3.source安装：

git clone https://github.com/dask/distributed.git

cd distributed 

python  setup.py install

dask-scheduler

distributed.scheduler - INFO - -----------------------------------------------

distributed.scheduler - INFO - Clear task state

distributed.scheduler - INFO -  Scheduler at:  tcp://192.168.1.42:8786

distributed.scheduler - INFO -                                                      :8787

distributed.scheduler - INFO - Local Directory: C:\Users\User\AppData\Local\Temp\scheduler-gd9uk980

distributed.scheduler - INFO - -----------------------------------------------

dask-worker 192.168.1.42:8786

工作节点启动成功后，此时主节点会显示多出信息：

distributed.scheduler - INFO - Register tcp://192.168.1.184:45772

distributed.scheduler - INFO - Starting worker compute stream, tcp://192.168.1.184:45772

distributed.core - INFO - Starting established connection

distributed.scheduler - INFO - Register tcp://192.168.1.183:43405

distributed.scheduler - INFO - Starting worker compute stream, tcp://192.168.1.183:43405

distributed.core - INFO - Starting established connection

distributed.scheduler - INFO - Register tcp://192.168.1.188:38095

distributed.scheduler - INFO - Starting worker compute stream, tcp://192.168.1.188:38095

distributed.core - INFO - Starting established connection

"""分布式dask"""

import time

from dask.distributed import Client

client = Client(‘192.168.1.42:8786‘ ,asynchronous=True)

ts = time.time()

A = client.map(square, range(10000))

B = client.map(neg, A)

total = client.submit(sum, B)

print(total.result())

print(‘cost time :%s‘%(time.time()-ts))

cost time :3.793848991394043