python 多线程并发 写文件,python中的多线程进程使用队列写入文件以检查工作是否完成...

from multiprocessing.dummy import Pool as ThreadPool

import multiprocessing as mp

def func(a):

pthData = "C:/temp/temp.txt"

with open(pthData, 'r') as file:

done = file.read().splitlines()

if a in done:

return 'done'

q.put(a)

return a

def listener(q):

pthData = "C:/temp/temp.txt"

m = q.get()

with open(pthData, 'a') as the_file:

the_file.write( m + '\n')

#he_file.write(str(m) + '\n')

a = ['a', 'b', 'c', 'd', 'a', 'b']

# Make the Pool of workers

pool = ThreadPool(4)

#must use Manager queue here, or will not work

manager = mp.Manager()

q = manager.Queue()

#put listener to work first

watcher = pool.apply_async(listener, (q,))

pool.starmap(func, a, q)

## TypeError: '<=' not supported between instances of 'AutoProxy[Queue]' and 'int'

pool.starmap(func, a)

## Runs but only writes 'a' to temp file

pool.starmap(func, (a, q))

## func() takes 1 positional argument but 6 were given

pool.apply_async(func, (a, q))

## freezes on pool.join

# Close the pool and wait for the work to finish

pool.close()

pool.join()

Why is the apply_async freezing on the pool.join()? I tried putting it into a if name == 'main' but it had the same result.

How do I properly call func passing 1 argument (a) and the queue (q)?

解决方案How do I properly call func passing 1 argument (a) and the queue (q)?

This at-least does not freeze:

Ensure temp.txt exists before execution.

Add a q parameter to func.

def func(a,q):

print(f'func({a})')

...

Use apply_async in a list comprehension.

if __name__ == '__main__':

# Make the Pool of workers

with ThreadPool(4) as pool:

q = queue.Queue()

#put listener to work first

watcher = pool.apply_async(listener, (q,))

results = [pool.apply_async(func, (item, q)) for item in a]

# just check stuff

for result in results:

result.wait()

print(result, result.successful(),result.get())

pool.close()

pool.join()

You will need to work out some other problems like listener running once then stopping.

Many other ways to do this, I used apply_async because it was one of the options in your question.

I like using concurrent.futures myself.

You may benefit from reading through the search results using variations of python threading producer consumer site:stackoverflow.com