爬虫中遇到中文乱码解决方法_爬中文乱码

1.requests

设置响应的编码

response.encoding = response.apparent_encoding
1

2.scrapy

中间件中添加process_response代码

from scrapy.http import HtmlResponse
class RandomUserAgentMiddleware(object):
    def process_request(self, request, spider):
        ua = random.choice(USER_AGENT_LIST)
        request.headers.setdefault('User-Agent', ua)

    def process_response(self, request, response, spider):
        response = HtmlResponse(
            url=response.url,
            body=response.body,
            encoding='GB2312'
        )
        return response
1
2
3
4
5
6
7
8
9
10
11
12
13

GB2312不行的话，可以改成utf-8之类的