Python多进程（process）（一）进程和进程池_python 进程池

Python实用教程_spiritx的博客-CSDN博客

在Python中因为有GIL的原因，线程能提供的并发效果并不理想，Python在多核上如果要释放并发的性能，更多的依靠多进程，我们会比线程更深入的去学习多进程。

进程（Process），顾名思义，就是进行中的程序。有一句话说得好：程序是一个没有生命的实体，只有处理器赋予程序生命时，它才能成为一个活动的实体。进程是资源分配的最小单元，也就是说每个进程都有其单独的内存空间。

Unix/Linux系统通过fork系统调用创建一个进程，但是在Windows中并没有fork调用。但是别担心，Python中内置的multiprocessing模块是跨平台的，我们可以通过对multiprocess模块中的Process类进行实例化创建一个进程对象，如：

import os
from multiprocessing import Process
 
def run_a_sub_proc(name):
    print(f'子进程：{name}（{os.getpid()}）开始...')
 
if __name__ == '__main__':
    print(f'主进程（{os.getpid()}）开始...')
    # 通过对Process类进行实例化创建一个子进程
    p = Process(target=run_a_sub_proc, args=('测试进程', ))
    p.start()
    p.join()
 
'
运行
运行

Process对象

定义：

class multiprocessing.Process(group=None, target=None, name=None, args=(), kwargs={}, *, daemon=None)

进程对象表示在单独进程中运行的活动。

Process 类拥有和 threading.Thread 等价的大部分方法。

• 应始终使用关键字参数调用构造函数。
• group 应该始终是 None ，它仅用于兼容 threading.Thread 。
• target 是由 run() 方法调用的可调用对象，可以是一个函数，也可以是一个可调用的函数对象，默认为 None ，None意味着什么都没有被调用。
• name 是进程名称。
• args 是目标调用的参数元组，元组中的每个元素对应一个函数的参数。
• kwargs 是目标调用的关键字参数字典。
• daemon 将进程 daemon 标志设置为 True 或 False 。如果是 None （默认值），则该标志将从创建的进程继承。

在默认情况下，不会将任何参数传递给 target。 args 参数默认值为 ()，可被用来指定要传递给 target 的参数列表或元组。

属性和方法

注意 start() 、 join() 、 is_alive() 、 terminate() 和 exitcode 方法只能由创建进程对象的进程调用。

子进程的创建

与线程一样，子进程也有两种创建方式，一种是直接使用process对象来拉起一个对象，另外一种是使用process的子类。

通过Process创建子进程

和使用 thread 类创建子线程的方式非常类似，使用 Process 类创建实例化对象，其本质是调用该类的构造方法创建新进程。Process 类的构造方法格式如下：

def __init__(self,group=None,target=None,name=None,args=(),kwargs={})

其中，各个参数的含义为：

• group：该参数未进行实现，不需要传参；
• target：为新建进程指定执行任务，也就是指定一个函数或者一个可执行对象；
• name：为新建进程设置名称；
• args：为 target 参数指定的参数传递非关键字参数；
• kwargs：为 target 参数指定的参数传递关键字参数。

import os, time
from multiprocessing import Process
 
URLS=['https://blog.csdn.net/spiritx/article/details/132783171',
      'https://blog.csdn.net/spiritx/article/details/132782806',
      'https://blog.csdn.net/spiritx/article/details/132778558',
      'https://blog.csdn.net/spiritx/article/details/132773698']
 
def downloadUrl(url, nsecs):
    time.sleep(1)
    print(f'进程ID={os.getpid()}, {__name__=} download: {url}')
 
 
# p1 = Process(target=downloadUrl, args=(URLS[0], 3)) #RuntimeError
# p1.start()
# downloadUrl(URLS[1], 4)
# p1.join()
 
if __name__ == '__main__':
    p1 = Process(target=downloadUrl, args=(URLS[0], 3))
    p2 = Process(target=downloadUrl, args=(URLS[1], 4))
    p1.start()
    p2.start()
    downloadUrl(URLS[3], 4)
    p1.join()
    p2.join()
 
‘’'
进程ID=64482, __name__='__main__' download: https://blog.csdn.net/spiritx/article/details/132773698
进程ID=64484, __name__='__mp_main__' download: https://blog.csdn.net/spiritx/article/details/132783171
进程ID=64485, __name__='__mp_main__' download: https://blog.csdn.net/spiritx/article/details/132782806
‘''

注意：因为进程和子进程复用同一套代码，一定要写if __name__ == '__main__':这段代码，否则会出现RuntimeError的异常。

从输出来看，子进程和父进程的__name__并不是一个，父进程是__main__，而子进程是__mp_main__

通过Process子类创建子进程

和使用 thread 子类创建线程的方式类似，除了直接使用 Process 类创建进程，还可以通过创建 Process 的子类来创建进程。
需要注意的是，在创建 Process 的子类时，需在子类重写 run() 方法。实际上，该方法所起到的作用，就如同第一种创建方式中 target 参数执行的函数。
另外，通过 Process 子类创建进程，和使用 Process 类一样，先创建该类的实例对象，然后调用 start() 方法启动该进程。下面程序演示如何通过 Process 子类创建一个进程。

import os, time
import multiprocessing
 
URLS=['https://blog.csdn.net/spiritx/article/details/132783171',
      'https://blog.csdn.net/spiritx/article/details/132782806',
      'https://blog.csdn.net/spiritx/article/details/132778558',
      'https://blog.csdn.net/spiritx/article/details/132773698']
 
 
def downloadUrl(url='', secs=1):
    time.sleep(secs)
    print(f'进程ID={os.getpid()}, {__name__=} download: {url}')
 
 
class MyChildProcess(multiprocessing.Process):
    def __init__(self, url, secs):
        #super.__init__(self) #TypeError: descriptor '__init__' requires a 'super' object but received a 'MyChildProcess'
        multiprocessing.Process.__init__(self)
        self.url = url
        self.secs = secs
 
    def run(self):
        downloadUrl(self.url, self.secs)
 
 
 
if __name__ == '__main__':
    p1 = MyChildProcess(URLS[0], 3)
    p2 = MyChildProcess(URLS[1], 4)
    p1.start()
    p2.start()
    downloadUrl(URLS[3], 4)
    p1.join()
    p2.join()
 
‘’'
进程ID=65688, __name__='__mp_main__' download: https://blog.csdn.net/spiritx/article/details/132783171
进程ID=65686, __name__='__main__' download: https://blog.csdn.net/spiritx/article/details/132773698
进程ID=65689, __name__='__mp_main__' download: https://blog.csdn.net/spiritx/article/details/132782806
‘''

有几个地方要特别注意：

• 注意一定要重写__init__()和run()方法，如果不重写__init__()方法，参数要和Process一致。
• 在子类的__init__()方法中一定要调用Process.__init__()方法，写super.__init__()方法会报TypeError的异常。

进程池

当需要创建的子进程数量不多时，可以直接利用multiprocessing中的Process动态成生多个进程，但如果是上百甚至上千个目标，手动的去创建进程的工作量巨大，此时就可以用到multiprocessing模块提供的Pool方法。

class multiprocessing.Pool([processes[, initializer[, initargs[, maxtasksperchild[, context]]]]])

• processes — 进程池中进程数量，如果为 None，则使用 os.cpu_count() 返回的值
• initializer — 如果该参数不为 None，则所有进程池中的进程启动时都会先执行 initializer(*initargs)
• maxtasksperchild — 如果该参数不为 None，则进程在执行 maxtasksperchild 次任务后会被自动销毁、重启
• context — 用于指定进程池中进程运行的上下文

初始化Pool时，可以指定一个最大进程数，当有新的请求提交到Pool中时，如果池还没有满，那么就会创建一个新的进程用来执行该请求；但如果池中的进程数已经达到指定的最大值，那么该请求就会等待，直到池中有进程他的任务结束，新的任务会交给这个空闲进程去处理。

import os, time
import multiprocessing
import random
 
URLS=['https://blog.csdn.net/spiritx/article/details/132783171',
      'https://blog.csdn.net/spiritx/article/details/132782806',
      'https://blog.csdn.net/spiritx/article/details/132778558',
      'https://blog.csdn.net/spiritx/article/details/132773698',
      'https://blog.csdn.net/spiritx/article/details/5',
      'https://blog.csdn.net/spiritx/article/details/6',
      'https://blog.csdn.net/spiritx/article/details/7',
      'https://blog.csdn.net/spiritx/article/details/8',
      'https://blog.csdn.net/spiritx/article/details/9']
 
INDEXS=[1,2,3,4,5,6,7,8,9]
PARAMS=list(zip(URLS, INDEXS)) #通过zip()方法将多个参数压缩成list对象
 
def downloadUrl(url, index):
    #url, index = param  # 对多个参数进行解包
    print(f'进程ID={os.getpid()}, {__name__=} start download: {url=},{index=}')
    time.sleep(random.randint(1,3))
    print(f'进程ID={os.getpid()}, {__name__=} end download: {url=},{index=}')
    return index
 
 
if __name__ == '__main__':
    pool = multiprocessing.Pool(4)
    results = [pool.apply_async(downloadUrl, param) for param in PARAMS]
    pool.close()
    pool.join()
 
    for result in results:
        print(f'{result.get()=}')
 
    
 
‘’'
进程ID=75676, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/132783171',index=1
进程ID=75677, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/132782806',index=2
进程ID=75675, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/132778558',index=3
进程ID=75678, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/132773698',index=4
进程ID=75677, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/132782806',index=2
进程ID=75677, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/5',index=5
进程ID=75675, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/132778558',index=3
进程ID=75675, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/6',index=6
进程ID=75678, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/132773698',index=4
进程ID=75678, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/7',index=7
进程ID=75676, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/132783171',index=1
进程ID=75676, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/8',index=8
进程ID=75677, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/5',index=5
进程ID=75677, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/9',index=9
进程ID=75676, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/8',index=8
进程ID=75675, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/6',index=6
进程ID=75678, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/7',index=7
进程ID=75677, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/9',index=9
result.get()=1
result.get()=2
result.get()=3
result.get()=4
result.get()=5
result.get()=6
result.get()=7
result.get()=8
result.get()=9
‘''

从上面的结果可以看出，75677进程在处理任务结束后，立即就得到了一个新的任务进行处理。

属性和方法

进程池提交任务的方法

进程池有多种方式可以提交任务，这里举例说明几种常用的方法。

一般方法apply()和apply_async()

apply(func [，args [，kwds ] ] )

使用参数args和关键字参数kwds调用func。它会阻塞，直到结果准备就绪。鉴于此块，更适合并行执行工作。此外，func 仅在池中的一个工作程序中执行。

import os
from multiprocessing import Pool
import time
def foo(p):
       print(f'进程ID{os.getpid()},{p=}')
       time.sleep(0.1)
 
if __name__=="__main__":
    pool = Pool(processes=3)
    for i  in range(10):
        pool.apply(foo, args=(i,))
    print('子进程提交完成')
    pool.close()
    pool.join()
 
‘’'
进程ID69407,p=0
进程ID69406,p=1
进程ID69409,p=2
进程ID69407,p=3
进程ID69406,p=4
进程ID69409,p=5
进程ID69407,p=6
进程ID69406,p=7
进程ID69409,p=8
进程ID69407,p=9
子进程提交完成
‘''

import os
from multiprocessing import Pool
import time
def foo(p):
       print(f'进程ID{os.getpid()} 开始....{p=}')
       time.sleep(0.1)
       print(f'进程ID{os.getpid()} 结束.{p=}')
 
if __name__=="__main__":
    pool = Pool(processes=3)
    for i  in range(10):
        pool.apply(foo, args=(i,))   #维持执行的进程总数为10，当一个进程执行完后启动一个新进程.
        print(f'进程{i}启动...')
 
    print('子进程提交完成')
    pool.close()
    pool.join()
 
‘’'
进程ID69541 开始....p=0
进程ID69541 结束.p=0
进程0启动...
进程ID69542 开始....p=1
进程ID69542 结束.p=1
进程1启动...
进程ID69543 开始....p=2
进程ID69543 结束.p=2
进程2启动...
进程ID69541 开始....p=3
进程ID69541 结束.p=3
进程3启动...
进程ID69542 开始....p=4
进程ID69542 结束.p=4
进程4启动...
进程ID69543 开始....p=5
进程ID69543 结束.p=5
进程5启动...
进程ID69541 开始....p=6
进程ID69541 结束.p=6
进程6启动...
进程ID69542 开始....p=7
进程ID69542 结束.p=7
进程7启动...
进程ID69543 开始....p=8
进程ID69543 结束.p=8
进程8启动...
进程ID69541 开始....p=9
进程ID69541 结束.p=9
进程9启动...
子进程提交完成
‘''

从上面的例子可以看到，apply()是顺序启动进程的，在被启动的进程执行完后，再启动下一个进程，实际是没有任何并发执行。

apply_async(func[, args[, kwds[, callback[, error_callback]]]])

异步进程池（非阻塞）,返回结果对象的方法的变体。如果指定了回调，则它应该是可调用的，它接受单个参数。当结果变为就绪时，将对其应用回调，即除非调用失败，在这种情况下将应用error_callback。如果指定了error_callback，那么它应该是一个可调用的，它接受一个参数。如果目标函数失败，则使用异常实例调用error_callback。回调应立即完成，否则处理结果的线程将被阻止。

import os
from multiprocessing import Pool
import time
def foo(p):
       print(f'进程ID{os.getpid()} 开始....{p=}')
       time.sleep(0.1)
       print(f'进程ID{os.getpid()} 结束.{p=}')
 
if __name__=="__main__":
    pool = Pool(processes=3)
    for i  in range(10):
        pool.apply_async(foo, args=(i,))   
        print(f'进程{i}启动...')
 
    print('子进程提交完成')
    pool.close()
    pool.join()
 
’’’
进程0启动...
进程1启动...
进程2启动...
进程3启动...
进程4启动...
进程5启动...
进程6启动...
进程7启动...
进程8启动...
进程9启动...
子进程提交完成
进程ID69599 开始....p=0
进程ID69598 开始....p=1
进程ID69600 开始....p=2
进程ID69599 结束.p=0
进程ID69599 开始....p=3
进程ID69598 结束.p=1
进程ID69598 开始....p=4
进程ID69600 结束.p=2
进程ID69600 开始....p=5
进程ID69599 结束.p=3
进程ID69599 开始....p=6
进程ID69598 结束.p=4
进程ID69598 开始....p=7
进程ID69600 结束.p=5
进程ID69600 开始....p=8
进程ID69599 结束.p=6
进程ID69599 开始....p=9
进程ID69598 结束.p=7
进程ID69600 结束.p=8
进程ID69599 结束.p=9
‘’‘

从上面的例子可以看到，apply_async()是异步启动的，子进程还未完全启动，apply_async()就已经返回。

AsyncResult对象

AsyncResult是Pool.apply_async() 和 Pool.map_async() 返回的对象所属的类。

主要方法有

import os, time
import multiprocessing
 
URLS=['https://blog.csdn.net/spiritx/article/details/132783171',
      'https://blog.csdn.net/spiritx/article/details/132782806',
      'https://blog.csdn.net/spiritx/article/details/132778558',
      'https://blog.csdn.net/spiritx/article/details/132773698']
 
def downloadUrl(url, nsecs):
    time.sleep(1)
    print(f'进程ID={os.getpid()}, {__name__=} download: {url}')
    return os.getpid()
 
 
if __name__ == '__main__':
    pool = multiprocessing.Pool(4)
    results = [pool.apply_async(downloadUrl, (url, 3)) for url in URLS]
    pool.close()
    pool.join()
 
    for result in results:
        print(f'{result.get()=}')
 
‘’'
进程ID=72390, __name__='__mp_main__' download: https://blog.csdn.net/spiritx/article/details/132783171
进程ID=72391, __name__='__mp_main__' download: https://blog.csdn.net/spiritx/article/details/132778558
进程ID=72392, __name__='__mp_main__' download: https://blog.csdn.net/spiritx/article/details/132782806
进程ID=72393, __name__='__mp_main__' download: https://blog.csdn.net/spiritx/article/details/132773698
result.get()=72390
result.get()=72392
result.get()=72391
result.get()=72393
‘''

多个参数的可调用对象

apply()和apply_async()支持多参数的可调用对象，但是在传入参数的时候，需要将多个参数封装为tuple进行传入，apply()和apply_async()会自动解包传给可调用对象。

如下：

import os, time
import multiprocessing
import random
 
URLS=['https://blog.csdn.net/spiritx/article/details/132783171',
      'https://blog.csdn.net/spiritx/article/details/132782806',
      'https://blog.csdn.net/spiritx/article/details/132778558',
      'https://blog.csdn.net/spiritx/article/details/132773698']
 
INDEXS=[1,2,3,4,5,6,7,8,9]
PARAMS=list(zip(URLS, INDEXS)) #通过zip()方法将多个参数压缩成list对象
 
def downloadUrl(url, index): #使用*param也是可以的，需要在函数里解包
    #url, index = param  # 对多个参数进行解包
    print(f'进程ID={os.getpid()}, {__name__=} start download: {url=},{index=}')
    time.sleep(random.randint(1,3))
    print(f'进程ID={os.getpid()}, {__name__=} end download: {url=},{index=}')
    return index
 
 
if __name__ == '__main__':
    pool = multiprocessing.Pool(4)
    results = [pool.apply_async(downloadUrl, param) for param in PARAMS]
    pool.close()
    pool.join()
 
    for result in results:
        print(f'{result.get()=}')
 
 
 
’’’
进程ID=72786, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/132783171',index=1
进程ID=72785, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/132782806',index=2
进程ID=72784, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/132778558',index=3
进程ID=72787, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/132773698',index=4
进程ID=72787, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/132773698',index=4
进程ID=72786, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/132783171',index=1
进程ID=72785, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/132782806',index=2
进程ID=72784, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/132778558',index=3
result.get()=1
result.get()=2
result.get()=3
result.get()=4
‘''

注意：apply_async的可调用对象接收的是多个参数。

map()和map_async()

内置 map() 函数的并行版本 。 它会保持阻塞直到获得结果。

map(func, iterable[, chunksize])

这个方法会将可迭代对象分割为许多块，然后提交给进程池。可以将 chunksize 设置为一个正整数从而（近似）指定每个块的大小可以。

注意，map()和map_async()的可调用对象不支持多个参数，需要将多参数包装为tuple，或者使用starmap()和starmap_async()。

import os, time
import multiprocessing
import random
 
URLS=['https://blog.csdn.net/spiritx/article/details/1',
      'https://blog.csdn.net/spiritx/article/details/2',
      'https://blog.csdn.net/spiritx/article/details/3',
      'https://blog.csdn.net/spiritx/article/details/4',
      'https://blog.csdn.net/spiritx/article/details/5',
      'https://blog.csdn.net/spiritx/article/details/6',
      'https://blog.csdn.net/spiritx/article/details/7',
      'https://blog.csdn.net/spiritx/article/details/8',
      'https://blog.csdn.net/spiritx/article/details/9']
 
INDEXS=[1,2,3,4,5,6,7,8,9]
PARAMS=list(zip(URLS, INDEXS)) #通过zip()方法将多个参数压缩成list对象
 
def downloadUrl(param): #不能接受多个参数
    url,index = param #对多个参数进行解包
    print(f'进程ID={os.getpid()}, {__name__=} start download: {url=},{index=}')
    time.sleep(random.randint(1,3))
    print(f'进程ID={os.getpid()}, {__name__=} end download: {url=},{index=}')
    return index
 
 
if __name__ == '__main__':
    pool = multiprocessing.Pool(4)
    pool.map(downloadUrl, PARAMS)
    print('pool.map submit ok.')
    pool.close()
    pool.join()
 
 
 
‘’'
进程ID=72929, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/1',index=1
进程ID=72930, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/2',index=2
进程ID=72932, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/3',index=3
进程ID=72931, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/4',index=4
进程ID=72929, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/1',index=1
进程ID=72929, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/5',index=5
进程ID=72930, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/2',index=2
进程ID=72930, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/6',index=6
进程ID=72930, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/6',index=6
进程ID=72929, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/5',index=5
进程ID=72930, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/7',index=7
进程ID=72929, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/8',index=8
进程ID=72932, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/3',index=3
进程ID=72932, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/9',index=9
进程ID=72931, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/4',index=4
进程ID=72929, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/8',index=8
进程ID=72930, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/7',index=7
进程ID=72932, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/9',index=9
pool.map submit ok.
‘''

从上面的输出可以看出，pool.map submit ok是在最后输出的，start download是按顺序输出的，但end download没有按顺序输出，说明：

• pool.map是按顺序启动的子进程
• 子进程是并行执行的
• 主进程在子进程执行完后，才会执行map之后的代码。

map_async(func, iterable[, chunksize[, callback[, error_callback]]])

map_async()是map() 方法的一个变种，返回一个 AsyncResult 对象。
如果指定了 callback , 它必须是一个接受单个参数的可调用对象。当执行成功时， callback 会被用于处理执行后的返回结果，否则，调用 error_callback 。
如果指定了 error_callback , 它必须是一个接受单个参数的可调用对象。当目标函数执行失败时， 会将抛出的异常对象作为参数传递给 error_callback 执行。
回调函数应该立即执行完成，否则会阻塞负责处理结果的进程。

import os, time
import multiprocessing
import random
 
URLS = ['https://blog.csdn.net/spiritx/article/details/1',
        'https://blog.csdn.net/spiritx/article/details/2',
        'https://blog.csdn.net/spiritx/article/details/3',
        'https://blog.csdn.net/spiritx/article/details/4',
        'https://blog.csdn.net/spiritx/article/details/5',
        'https://blog.csdn.net/spiritx/article/details/6',
        'https://blog.csdn.net/spiritx/article/details/7',
        'https://blog.csdn.net/spiritx/article/details/8',
        'https://blog.csdn.net/spiritx/article/details/9']
 
INDEXS = [1, 2, 3, 4, 5, 6, 7, 8, 9]
PARAMS = list(zip(URLS, INDEXS))  # 通过zip()方法将多个参数压缩成list对象
 
 
def downloadUrl(param):  # 只支持一个参数
    url, index = param  # 对多个参数进行解包
    print(f'进程ID={os.getpid()}, {__name__=} start download: {url=},{index=}')
    time.sleep(random.randint(1, 3))
    print(f'进程ID={os.getpid()}, {__name__=} end download: {url=},{index=}')
    return index
 
 
if __name__ == '__main__':
    pool = multiprocessing.Pool(4)
    pool.map_async(downloadUrl, PARAMS)
    print('pool.map submit ok.')
    pool.close()
    pool.join()
 
 
‘’'
pool.map submit ok.
进程ID=73087, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/1',index=1
进程ID=73088, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/2',index=2
进程ID=73089, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/3',index=3
进程ID=73090, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/4',index=4
进程ID=73087, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/1',index=1
进程ID=73087, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/5',index=5
进程ID=73088, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/2',index=2
进程ID=73088, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/6',index=6
进程ID=73089, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/3',index=3
进程ID=73089, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/7',index=7
进程ID=73087, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/5',index=5
进程ID=73087, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/8',index=8
进程ID=73090, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/4',index=4
进程ID=73090, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/9',index=9
进程ID=73087, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/8',index=8
进程ID=73088, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/6',index=6
进程ID=73089, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/7',index=7
进程ID=73090, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/9',index=9
‘''

从上面的例子可以看出，pool.map submit打印在最前面，start download是顺序启动的：

• map_async在启动子进程时，不等子进程执行完，立即就执行主进程后面的语句
• 子进程是按序启动的，并且都是并发的
• 子进程与主进程也是并发的

imap()和imap_unordered()

imap(func, iterable[, chunksize])

map() 的延迟执行版本。
chunksize 参数的作用和 map() 方法的一样。对于很长的迭代器，给 chunksize 设置一个很大的值会比默认值 1 极大 地加快执行速度。
同样，如果 chunksize 是 1 , 那么 imap() 方法所返回的迭代器的 next() 方法拥有一个可选的 timeout 参数： 如果无法在 timeout 秒内执行得到结果，则 next(timeout) 会抛出 multiprocessing.TimeoutError 异常。

所谓延迟执行版本，就是并不是所有的子进程都执行完了，才能获得结果，可以在部分进程执行完后，就拿到结果。

下面是map()与imap()拿结果的对比

map版本：

import os, time
import multiprocessing
import random
 
URLS = ['https://blog.csdn.net/spiritx/article/details/1',
        'https://blog.csdn.net/spiritx/article/details/2',
        'https://blog.csdn.net/spiritx/article/details/3',
        'https://blog.csdn.net/spiritx/article/details/4',
        'https://blog.csdn.net/spiritx/article/details/5',
        'https://blog.csdn.net/spiritx/article/details/6',
        'https://blog.csdn.net/spiritx/article/details/7',
        'https://blog.csdn.net/spiritx/article/details/8',
        'https://blog.csdn.net/spiritx/article/details/9']
 
INDEXS = [1, 2, 3, 4, 5, 6, 7, 8, 9]
PARAMS = list(zip(URLS, INDEXS))  # 通过zip()方法将多个参数压缩成list对象
 
 
def downloadUrl(param):  # 只支持一个参数
    url, index = param  # 对多个参数进行解包
    print(f'进程ID={os.getpid()}, {__name__=} start download: {url=},{index=}')
    #time.sleep(random.randint(1, 3))
    time.sleep(9 - index)
    print(f'进程ID={os.getpid()}, {__name__=} end download: {url=},{index=}')
    return 'result:', index
 
 
if __name__ == '__main__':
    pool = multiprocessing.Pool(4)
    its = pool.map(downloadUrl, PARAMS)
    print('pool.map submit ok.')
    for it in its:
        print(it)
    pool.close()
    pool.join()
 
‘’’
进程ID=73691, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/1',index=1
进程ID=73693, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/2',index=2
进程ID=73690, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/3',index=3
进程ID=73692, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/4',index=4
进程ID=73692, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/4',index=4
进程ID=73692, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/5',index=5
进程ID=73690, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/3',index=3
进程ID=73690, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/6',index=6
进程ID=73693, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/2',index=2
进程ID=73693, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/7',index=7
进程ID=73691, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/1',index=1
进程ID=73691, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/8',index=8
进程ID=73691, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/8',index=8
进程ID=73691, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/9',index=9
进程ID=73691, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/9',index=9
进程ID=73693, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/7',index=7
进程ID=73690, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/6',index=6
进程ID=73692, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/5',index=5
pool.map submit ok.
('result:', 1)
('result:', 2)
('result:', 3)
('result:', 4)
('result:', 5)
('result:', 6)
('result:', 7)
('result:', 8)
('result:', 9)
’‘’

map()版本的返回值都是要在所有进程都执行完后才能获得。

imap()版本：

import os, time
import multiprocessing
import random
 
URLS = ['https://blog.csdn.net/spiritx/article/details/1',
        'https://blog.csdn.net/spiritx/article/details/2',
        'https://blog.csdn.net/spiritx/article/details/3',
        'https://blog.csdn.net/spiritx/article/details/4',
        'https://blog.csdn.net/spiritx/article/details/5',
        'https://blog.csdn.net/spiritx/article/details/6',
        'https://blog.csdn.net/spiritx/article/details/7',
        'https://blog.csdn.net/spiritx/article/details/8',
        'https://blog.csdn.net/spiritx/article/details/9']
 
INDEXS = [1, 2, 3, 4, 5, 6, 7, 8, 9]
PARAMS = list(zip(URLS, INDEXS))  # 通过zip()方法将多个参数压缩成list对象
 
 
def downloadUrl(param):  # 只支持一个参数
    url, index = param  # 对多个参数进行解包
    print(f'进程ID={os.getpid()}, {__name__=} start download: {url=},{index=}')
    #time.sleep(random.randint(1, 3))
    time.sleep(9 - index)
    print(f'进程ID={os.getpid()}, {__name__=} end download: {url=},{index=}')
    return 'result:', index
 
 
if __name__ == '__main__':
    pool = multiprocessing.Pool(4)
    its = pool.imap(downloadUrl, PARAMS)
    print('pool.map submit ok.')
    for it in its:
        print(it)
    pool.close()
    pool.join()
 
‘’'
pool.map submit ok.
进程ID=73772, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/1',index=1
进程ID=73773, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/2',index=2
进程ID=73776, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/3',index=3
进程ID=73775, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/4',index=4
进程ID=73775, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/4',index=4
进程ID=73775, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/5',index=5
进程ID=73776, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/3',index=3
进程ID=73776, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/6',index=6
进程ID=73773, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/2',index=2
进程ID=73773, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/7',index=7
进程ID=73772, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/1',index=1
进程ID=73772, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/8',index=8
('result:', 1)
('result:', 2)
('result:', 3)
('result:', 4)
进程ID=73772, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/8',index=8
进程ID=73772, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/9',index=9
进程ID=73772, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/9',index=9
进程ID=73773, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/7',index=7
进程ID=73776, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/6',index=6
进程ID=73775, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/5',index=5
('result:', 5)
('result:', 6)
('result:', 7)
('result:', 8)
('result:', 9)
‘''

pool.map submit在首行就打印出来，说明imap的子进程与父进程也是并行执行的。

结果数据打印夹杂在子进程输出的结果打印中，是在一项任务执行完后打印出来。但仍可以看出来，结果数据是按任务的进入顺序打印的。（为了演示，我们实际的任务执行时间是，越早提交的任务运行的时间越长。）

imap_unordered(func, iterable[, chunksize])

和 imap() 相同，只不过通过迭代器返回的结果是任意的。（当进程池中只有一个工作进程的时候，返回结果的顺序才能认为是"有序"的）

import os, time
import multiprocessing
import random
 
URLS = ['https://blog.csdn.net/spiritx/article/details/1',
        'https://blog.csdn.net/spiritx/article/details/2',
        'https://blog.csdn.net/spiritx/article/details/3',
        'https://blog.csdn.net/spiritx/article/details/4',
        'https://blog.csdn.net/spiritx/article/details/5',
        'https://blog.csdn.net/spiritx/article/details/6',
        'https://blog.csdn.net/spiritx/article/details/7',
        'https://blog.csdn.net/spiritx/article/details/8',
        'https://blog.csdn.net/spiritx/article/details/9']
 
INDEXS = [1, 2, 3, 4, 5, 6, 7, 8, 9]
PARAMS = list(zip(URLS, INDEXS))  # 通过zip()方法将多个参数压缩成list对象
 
 
def downloadUrl(param):  # 只支持一个参数
    url, index = param  # 对多个参数进行解包
    print(f'进程ID={os.getpid()}, {__name__=} start download: {url=},{index=}')
    #time.sleep(random.randint(1, 3))
    time.sleep(9 - index)
    print(f'进程ID={os.getpid()}, {__name__=} end download: {url=},{index=}')
    return 'result:', index
 
 
if __name__ == '__main__':
    pool = multiprocessing.Pool(4)
    its = pool.imap_unordered(downloadUrl, PARAMS)
    print('pool.map submit ok.')
    for it in its:
        print(it)
    pool.close()
    pool.join()
 
‘’’
pool.map submit ok.
进程ID=73862, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/1',index=1
进程ID=73864, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/2',index=2
进程ID=73863, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/3',index=3
进程ID=73865, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/4',index=4
进程ID=73865, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/4',index=4
进程ID=73865, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/5',index=5
('result:', 4)
进程ID=73863, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/3',index=3
进程ID=73863, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/6',index=6
('result:', 3)
进程ID=73864, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/2',index=2
进程ID=73864, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/7',index=7
('result:', 2)
进程ID=73862, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/1',index=1
进程ID=73862, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/8',index=8
('result:', 1)
进程ID=73862, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/8',index=8
进程ID=73862, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/9',index=9
进程ID=73862, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/9',index=9
('result:', 8)
('result:', 9)
进程ID=73864, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/7',index=7
('result:', 7)
进程ID=73863, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/6',index=6
('result:', 6)
进程ID=73865, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/5',index=5
('result:', 5)
’‘’

imap_unordered()与imap()几乎一样，只是结果数据是谁先完成谁就能先获得。

starmap()和starmap_async()

map()和map_async()不能支持多参数的可调用对象，starmap()和starmap_async()看上去就像是打的补丁，让他们可以支持多参数的可调用对象。

starmap(func, iterable[, chunksize])
和 map() 类似，不过 iterable 中的每一项会被解包再作为函数参数。
比如可迭代对象 [(1,2), (3, 4)] 会转化为等价于 [func(1,2), func(3,4)] 的调用。

import os, time
import multiprocessing
import random
 
URLS=['https://blog.csdn.net/spiritx/article/details/1',
      'https://blog.csdn.net/spiritx/article/details/2',
      'https://blog.csdn.net/spiritx/article/details/3',
      'https://blog.csdn.net/spiritx/article/details/4',
      'https://blog.csdn.net/spiritx/article/details/5',
      'https://blog.csdn.net/spiritx/article/details/6',
      'https://blog.csdn.net/spiritx/article/details/7',
      'https://blog.csdn.net/spiritx/article/details/8',
      'https://blog.csdn.net/spiritx/article/details/9']
 
INDEXS=[1,2,3,4,5,6,7,8,9]
PARAMS=list(zip(URLS, INDEXS)) #通过zip()方法将多个参数压缩成list对象
 
def downloadUrl(url, index): #可以支持多个参数
    #url,index = param #对多个参数进行解包
    print(f'进程ID={os.getpid()}, {__name__=} start download: {url=},{index=}')
    time.sleep(random.randint(1,3))
    print(f'进程ID={os.getpid()}, {__name__=} end download: {url=},{index=}')
    return index
 
 
if __name__ == '__main__':
    pool = multiprocessing.Pool(4)
    pool.starmap(downloadUrl, PARAMS)
    print('pool.map submit ok.')
    pool.close()
    pool.join()
 
‘’'
进程ID=73960, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/1',index=1
进程ID=73962, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/2',index=2
进程ID=73961, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/3',index=3
进程ID=73963, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/4',index=4
进程ID=73960, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/1',index=1
进程ID=73960, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/5',index=5
进程ID=73962, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/2',index=2
进程ID=73962, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/6',index=6
进程ID=73961, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/3',index=3
进程ID=73961, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/7',index=7
进程ID=73960, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/5',index=5
进程ID=73960, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/8',index=8
进程ID=73963, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/4',index=4
进程ID=73963, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/9',index=9
进程ID=73962, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/6',index=6
进程ID=73961, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/7',index=7
进程ID=73963, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/9',index=9
进程ID=73960, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/8',index=8
pool.map submit ok.
‘''

starmap_async(func, iterable[, chunksize[, callback[, error_callback]]])
相当于 starmap() 与 map_async() 的结合，迭代 iterable 的每一项，解包作为 func 的参数并执行，返回用于获取结果的对象。

import os, time
import multiprocessing
import random
 
URLS = ['https://blog.csdn.net/spiritx/article/details/1',
        'https://blog.csdn.net/spiritx/article/details/2',
        'https://blog.csdn.net/spiritx/article/details/3',
        'https://blog.csdn.net/spiritx/article/details/4',
        'https://blog.csdn.net/spiritx/article/details/5',
        'https://blog.csdn.net/spiritx/article/details/6',
        'https://blog.csdn.net/spiritx/article/details/7',
        'https://blog.csdn.net/spiritx/article/details/8',
        'https://blog.csdn.net/spiritx/article/details/9']
 
INDEXS = [1, 2, 3, 4, 5, 6, 7, 8, 9]
PARAMS = list(zip(URLS, INDEXS))  # 通过zip()方法将多个参数压缩成list对象
 
 
def downloadUrl(url, index):  # 只支持一个参数
    #url, index = param  # 对多个参数进行解包
    print(f'进程ID={os.getpid()}, {__name__=} start download: {url=},{index=}')
    time.sleep(random.randint(1, 3))
    print(f'进程ID={os.getpid()}, {__name__=} end download: {url=},{index=}')
    return index
 
 
if __name__ == '__main__':
    pool = multiprocessing.Pool(4)
    pool.starmap_async(downloadUrl, PARAMS)
    print('pool.map submit ok.')
    pool.close()
    pool.join()
 
‘’'
pool.map submit ok.
进程ID=73991, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/1',index=1
进程ID=73990, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/2',index=2
进程ID=73993, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/3',index=3
进程ID=73992, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/4',index=4
进程ID=73993, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/3',index=3
进程ID=73993, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/5',index=5
进程ID=73992, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/4',index=4
进程ID=73992, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/6',index=6
进程ID=73991, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/1',index=1
进程ID=73991, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/7',index=7
进程ID=73990, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/2',index=2
进程ID=73990, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/8',index=8
进程ID=73993, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/5',index=5
进程ID=73993, __name__='__mp_main__' start download: url='https://blog.csdn.net/spiritx/article/details/9',index=9
进程ID=73991, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/7',index=7
进程ID=73992, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/6',index=6
进程ID=73990, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/8',index=8
进程ID=73993, __name__='__mp_main__' end download: url='https://blog.csdn.net/spiritx/article/details/9',index=9
‘''