multiprocessing.dummy python使用笔记_multiprocessing.dummy处理数组

代码

pool.map

不管chunksize多大，输出的list永远和a的list 顺序一样，只是中间处理的先后顺序不同

from multiprocessing.dummy import Pool
import os 
import time 
import threading

def task(i):
    print("{} begin".format(threading.current_thread().name))
    print(i)#1-5(1-5输出的顺便随机) 6-10 11-15 16-20 (每次输出的随机) 每次5个线程进行处理 但是先后顺序没有关系，不影响输出的顺序
    time.sleep(5)
    return i
    # print("{} end".format(threading.current_thread().name))

if __name__ == "__main__":
    a=range(1,21)
    thread_count=5
    P=Pool(thread_count)
    #map  每次从iter a(1-20)调用5个元素分别给5个task 进行并行处理，所以会进行4(20/5)次 
    #map  每次得到的结果都会等4次5个task都处理完，同时将最后的结果按照传入的顺序转为list输出
    listres=P.map(task,a,chunksize=1)
    print(listres)#得到结果list，最后一次输出
   # P.map(task,a,chunksize=2)
   #1-20 每次从iter a调用5个元素（隔chunksize个元素）分别给5个task 五个task 进行并行处理，
   # 最后还是输出[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

终端输出结果

Thread-1 begin
Thread-2 begin
Thread-3 begin
Thread-5 begin
2
3
5
1
Thread-4 begin
4
Thread-5 begin
Thread-4 begin
7
6
Thread-2 begin
8
Thread-3 begin
9
Thread-1 begin
10
Thread-4 begin
11
Thread-5 begin
Thread-3 begin
12
13
Thread-2 begin
14
Thread-1 begin
15
Thread-3 begin
16
Thread-4 begin
17
Thread-5 begin
18
Thread-1 begin
19
Thread-2 begin
20
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

pool.imap

imap作用同map 但imap是一个存放所有结果的迭代器 需要在主进程中主动使用next来驱动子进程的调用，
可以不用等5个线程都处理完才返回，这里每次处理完一个task函数将得到一个迭代器结果
代码

from multiprocessing.dummy import Pool
import os 
import time 
import threading

def task(i):
    print("{} begin".format(threading.current_thread().name))
    print(i)#1-5(1-5输出的顺便随机) 6-10 11-15 16-20 (每次输出的随机) 每次5个线程进行处理 但是先后顺序没有关系，不影响输出的顺序
    time.sleep(1)
    return i
    # print("{} end".format(threading.current_thread().name))

if __name__ == "__main__":
    a=range(1,21)
    thread_count=5
    P=Pool(thread_count)
    for i,res in enumerate(P.imap(task,a,chunksize=1)):
        print("res:",res)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

输出：最终res还是按照1-20的顺序进行输出，chunksize=2时输出相同，此时res:2需要等第2次5个task处理时才会输出

Thread-1 begin
Thread-2 begin
1
Thread-5 begin
5
Thread-4 begin
4
Thread-3 begin
3
2
Thread-1 begin
6
res: 1
Thread-3 begin
7
res: 2
Thread-5 begin
Thread-2 begin
Thread-4 begin
9
8
res: 3
res: 4
10
res: 5
Thread-1 begin
11
res: 6
Thread-4 begin
12
Thread-3 begin
Thread-5 begin
Thread-2 begin
13
res: 7
15
14
res: 8
res: 9
res: 10
Thread-1 begin
16
res: 11
Thread-4 begin
17
res: 12
Thread-3 begin
res: 13
18
Thread-2 begin
Thread-5 begin
19
res: 14
20
res: 15
res: 16
res: 17
res: 18
res: 19
res: 20
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60

pool.imap_unordered

终端输出：可以不是按照a的顺序，每次处理完一个线程函数task，就会返回一个结果

Thread-1 begin input=1
Thread-2 begin input=3
Thread-4 begin input=7
Thread-5 begin input=9
Thread-3 begin input=5
Thread-2 begin input=4
Thread-1 begin input=2
Thread-5 begin input=10
Thread-3 begin input=6
Thread-4 begin input=8
Thread-3 begin input=11
res= 5
Thread-2 begin input=13
Thread-1 begin input=15
res= 6
Thread-5 begin input=17
Thread-4 begin input=19
res= 3
res= 4
res= 1
res= 2
res= 9
res= 10
res= 7
res= 8
Thread-3 begin input=12
Thread-4 begin input=20
Thread-5 begin input=18
Thread-2 begin input=14
Thread-1 begin input=16
res= 11
res= 12
res= 13
res= 14
res= 17
res= 18
res= 19
res= 20
res= 15
res= 16

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41