Python多线程（threading）、多进程（multiprocessing）适用范围，运算速度对比_python多进程同时修改矩阵

Python多线程（threading）、多进程（multiprocessing）适用范围，运算速度对比

• IO密集型，用多线程，如：文件读写、网络数据传输（下载图片、视频）
• 计算密集型，用多进程，如：大量的数据计算（累加计算）

以下例子是读取创建好的txt文件，并提取数据输出累加结果

1. read.txt 为 no:0 – no:9999999
2. 关于多进程，进程需运行在 if __name__ == "__main__" 中，否者将报错，这是由于windows系统默认设置spawn，linux直接运行即可，linux默认设置fork，虽然mac支持fork和spawn，但直接运行默认也会报错

from multiprocessing import Pool, cpu_count
import os
import time
import threading

start_time = time.time()


def task(row_list):
    num_list = [int(row.split(':')[-1]) for row in row_list]
    result = sum(num_list)
    print(result)
    print("------------son result---------------")


# 多进程处理 大约7s
# def run():
#     p = Pool(cpu_count())
#     file = open("read.txt", "r", encoding = 'utf-8')
#     file.readline()
#     row_list = []
#     for line in file:
#         row_list.append(line.strip())
#         if len(row_list) == 1000:
#             p.apply_async(task, args = (row_list,))
#             row_list = []
#     if row_list:
#         p.apply_async(task, args = (row_list,))
#     p.close()
#     p.join()
#     file.close()


# 多线程处理 大约9s
def run():
    file = open('read.txt', 'r', encoding = 'utf-8')
    file.readline()
    row_list = []
    for line in file:
        row_list.append(line.strip())
        if len(row_list) == 1000:
            t = threading.Thread(target = task, args = (row_list,))
            t.start()
            row_list = []
    if row_list:
        t = threading.Thread(target = task, args = (row_list,))
        t.start()
    file.close()


if __name__ == "__main__":
    print(f'主进程id：{os.getpid()}')
    print("------------------start-------------------")
    run()
    print(time.time() - start_time)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55

运行速度对比如下：