使用multiprocessing.Pool实现并发执行_pool = multiprocessing.pool(processes=7) #并发sql数

整理自博友文章https://www.cnblogs.com/kaituorensheng/p/4445418.html，以下代码都在python3.6上测试（win10）

类：multiprocessing.Pool(processes)

• 非阻塞

例子：

#coding: utf-8
import multiprocessing
import time
 
def func(msg):
    print("msg:", msg)
    time.sleep(3)
    print("end")
 
if __name__ == "__main__":
    start = time.time()
    pool = multiprocessing.Pool(processes = 3)
    for i in range(4):
        msg = "hello %d" %(i)
        pool.apply_async(func, (msg, ))   #维持执行的进程总数为processes，当一个进程执行完毕后会添加新的进程进去
 
    print("Mark~ Mark~ Mark~~~~~~~~~~~~~~~~~~~~~~")
    pool.close()
    pool.join()   #调用join之前，先调用close函数，否则会出错。执行完close后不会有新的进程加入到pool,join函数等待所有子进程结束
    print("Sub-process(es) done.")
    end = time.time()
    print(f"time: {end-start}")

输出结果：

Mark~ Mark~ Mark~~~~~~~~~~~~~~~~~~~~~~
msg: hello 0
msg: hello 1
msg: hello 2
end
msg: hello 3
end
end
end
Sub-process(es) done.
time: 6.277066230773926

分析：pool为类实例

apply_async(func[, args=()[, kwds={}[, callback=None]]])方法加入函数func，以及函数参数args，功能是把函数执行进程加入到进程池中，并且不会阻塞主进程，这一点在输出结果中可以看出来，因为Mark~ Mark~ Mark~~~~~~~~~~~~~~~~~~~~~~在func函数的输出之前输出；

close()方法关闭进程池（pool），使其不在接受新的任务；

join()方法将主进程阻塞等待子进程的退出，join方法必须在close或terminate之后使用。

• 阻塞

例子：（与上面的例子只差在apply上面）

#coding: utf-8
import multiprocessing
import time
 
def func(msg):
    print("msg:", msg)
    time.sleep(3)
    print("end")
 
if __name__ == "__main__":
    start = time.time()
    pool = multiprocessing.Pool(processes = 3)
    for i in range(4):
        msg = "hello %d" %(i)
        pool.apply(func, (msg, ))   #维持执行的进程总数为processes，当一个进程执行完毕后会添加新的进程进去
 
    print("Mark~ Mark~ Mark~~~~~~~~~~~~~~~~~~~~~~")
    pool.close()
    pool.join()   #调用join之前，先调用close函数，否则会出错。执行完close后不会有新的进程加入到pool,join函数等待所有子进程结束
    print("Sub-process(es) done.")
    end = time.time()
    print(f"time: {end-start}")

输出结果：

msg: hello 0
end
msg: hello 1
end
msg: hello 2
end
msg: hello 3
end
Mark~ Mark~ Mark~~~~~~~~~~~~~~~~~~~~~~
Sub-process(es) done.
time: 12.32795238494873

• 捕捉函数返回结果

例子：（使用get()得到函数返回的结果）

import multiprocessing
import time
 
def func(msg):
    print("msg:", msg)
    time.sleep(3)
    print("end")
    return("done " + msg)
 
if __name__ == "__main__":
    pool = multiprocessing.Pool(processes=4)
    result = []
    for i in range(5):
        msg = "hello %d" %(i)
        result.append(pool.apply_async(func, (msg, )))
    pool.close()
    pool.join()
    for res in result:
        print(":::", res.get())
    print("Sub-process(es) done.")

输出结果：msg: hello 0
msg: hello 1
msg: hello 2
msg: hello 3
end
msg: hello 4
end
end
end
end
::: done hello 0
::: done hello 1
::: done hello 2
::: done hello 3
::: done hello 4
Sub-process(es) done.