Python异步请求实践：使用grequests实现多线程请求

安装与gevent一起工作的^{} module(requests不是为异步而设计的)：pip install grequests

然后将代码更改为以下内容：import grequests

class Test:

def __init__(self):

self.urls = [

'http://www.example.com',

'http://www.google.com',

'http://www.yahoo.com',

'http://www.stackoverflow.com/',

'http://www.reddit.com/'

]

def exception(self, request, exception):

print "Problem: {}: {}".format(request.url, exception)

def async(self):

results = grequests.map((grequests.get(u) for u in self.urls), exception_handler=self.exception, size=5)

print results

test = Test()

test.async()Blocking Or Non-Blocking?

With the default Transport Adapter in place, Requests does not provide any kind of non-blocking IO. The Response.content property will block until the entire response has been downloaded. If you require more granularity, the streaming features of the library (see Streaming Requests) allow you to retrieve smaller quantities of the response at a time. However, these calls will still block.

If you are concerned about the use of blocking IO, there are lots of projects out there that combine Requests with one of Python's asynchronicity frameworks. Two excellent examples are grequests and requests-futures.

使用此方法可以显著提高10个url的性能：0.877s与原来方法的3.852s。