devbox
你好赵伟
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

Python爬虫项目实战-Scrapy+Charles+MongoDB+Redis实现分布式京东全网信息爬取2020最新版_python charles 实现爬虫

作者：你好赵伟 | 2024-04-17 00:19:59

赞

踩

python charles 实现爬虫

目录

 

一、基础知识

二、开发环境及项目结构

三、结果展示

四、实战源码

4.1 数据模型-items.py

4.2 存储操作（以MongoDB为存储数据库）-pipelines.py

4.3 项目配置-settings.py

4.4 中间件配置-middlewares.py

4.5 分类信息抓取-jd_category.py

4.6 抓取商品详细信息-jd_product.py

4.7 分布式操作的实现-add_category_to_redis.py

五、项目使用方法

一、基础知识及相关安装包

下面是一些博客链接

HTTP基础知识理解：Python网络爬虫-原理及基础知识

Charles抓包：Windows环境下配合Charles进行抓包

JS渲染：如何结合Splash对JS进行渲染

代理池的构建：多线程结合Flask构建免费高匿代理池

MongoDB基础教程：MongoDB基础教程

Redis基础教程：Redis基础教程

scrapy基础项目开发：暂无，后续会做一个小节介绍

二、开发环境及项目结构

软件环境：Pycharm2019 + MongoDB + Redis3.20 + MongoDB+ WIndows10

三、结果展示

四、实战源码

4.1 数据模型-items.py

  1. # -*- coding: utf-8 -*-
  2.  
  3. # Define here the models for your scraped items
  4. #
  5. # See documentation in:
  6. # https://doc.scrapy.org/en/latest/topics/items.html
  7.  
  8. import scrapy
  9.  
  10.  
  11. class MallSpiderItem(scrapy.Item):
  12.     # define the fields for your item here like:
  13.     # name = scrapy.Field()
  14.     pass
  15.  
  16. class Category(scrapy.Item):
  17.     '''
  18.     定义类型数据模型,明确抓取的字段
  19.     '''
  20.     # 大、中、小商品对应的名称及url
  21.     b_category_name = scrapy.Field()
  22.     b_category_url = scrapy.Field()
  23.     m_category_name = scrapy.Field()
  24.     m_category_url = scrapy.Field()
  25.     s_category_name = scrapy.Field()
  26.     s_category_url = scrapy.Field()
  27.  
  28. class Product(scrapy.Item):
  29.     '''
  30.     商品信息数据模型
  31.     '''
  32.     product_category = scrapy.Field() # 商品类别
  33.     product_category_id = scrapy.Field() # 类别ID
  34.     product_sku_id = scrapy.Field() # 商品ID
  35.     product_name = scrapy.Field() # 商品名称
  36.     product_img_url = scrapy.Field() # 商品图片URL
  37.     product_book_info = scrapy.Field() # 图书信息，作者，出版社
  38.     product_option = scrapy.Field() # 商品选项
  39.     product_shop = scrapy.Field() # 商品店铺
  40.     product_comments = scrapy.Field() # 商品评论数量
  41.     product_ad = scrapy.Field() # 商品促销
  42.     product_price = scrapy.Field() # 商品价格
  43.

4.2 存储操作（以MongoDB为存储数据库）-pipelines.py

  1. # -*- coding: utf-8 -*-
  2.  
  3. # Define your item pipelines here
  4. #
  5. # Don't forget to add your pipeline to the ITEM_PIPELINES setting
  6. # See: https://doc.scrapy.org/en/latest/topics/item-pipeline.html
  7. from mall_spider.spiders.jd_category import JdCategorySpider
  8. from pymongo import MongoClient
  9. from mall_spider.settings import MONGODB_URL
  10. from mall_spider.spiders.jd_product import JdProductSpider
  11.  
  12.  
  13. class CategoryPipeline(object):
  14.     '''存储操作'''
  15.     def open_spider(self,spider):
  16.         # 当爬虫启动时执行，执行一次
  17.         if isinstance(spider,JdCategorySpider):
  18.             # 连接MongoDB数据库，获取要操作的集合
  19.             self.client = MongoClient(MONGODB_URL)
  20.             self.collection = self.client['jd']['category']
  21.  
  22.  
  23.  
  24.     def process_item(self, item, spider):
  25.         if isinstance(spider,JdCategorySpider):
  26.             # 插入数据，转换成字典
  27.             self.collection.insert_one(dict(item))
  28.         return item
  29.  
  30.  
  31.     def close_spider(self,spider):
  32.         # 关闭连接
  33.         if isinstance(spider, JdCategorySpider):
  34.             self.client.close()
  35.  
  36.  
  37. class ProductPipeline(object):
  38.     '''存储操作'''
  39.     def open_spider(self, spider):
  40.         # 当爬虫启动时执行，执行一次
  41.         if isinstance(spider, JdProductSpider):
  42.             # 连接MongoDB数据库，获取要操作的集合
  43.             self.client = MongoClient(MONGODB_URL)
  44.             self.collection = self.client['jd']['product']
  45.  
  46.     def process_item(self, item, spider):
  47.         if isinstance(spider, JdProductSpider):
  48.             # 插入数据，转换成字典
  49.             self.collection.insert_one(dict(item))
  50.         return item
  51.  
  52.     def close_spider(self, spider):
  53.         # 关闭连接
  54.         if isinstance(spider, JdProductSpider):
  55.             self.client.close()
  56.

4.3 项目配置-settings.py

  1. # -*- coding: utf-8 -*-
  2.  
  3. # Scrapy settings for mall_spider project
  4. #
  5. # For simplicity, this file contains only settings considered important or
  6. # commonly used. You can find more settings consulting the documentation:
  7. #
  8. #     https://doc.scrapy.org/en/latest/topics/settings.html
  9. #     https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
  10. #     https://doc.scrapy.org/en/latest/topics/spider-middleware.html
  11.  
  12. BOT_NAME = 'mall_spider'
  13.  
  14. SPIDER_MODULES = ['mall_spider.spiders']
  15. NEWSPIDER_MODULE = 'mall_spider.spiders'
  16.  
  17.  
  18. # Crawl responsibly by identifying yourself (and your website) on the user-agent
  19. #USER_AGENT = 'mall_spider (+http://www.yourdomain.com)'
  20.  
  21. # Obey robots.txt rules
  22. ROBOTSTXT_OBEY = True
  23.  
  24. # Configure maximum concurrent requests performed by Scrapy (default: 16)
  25. #CONCURRENT_REQUESTS = 32
  26.  
  27. # Configure a delay for requests for the same website (default: 0)
  28. # See https://doc.scrapy.org/en/latest/topics/settings.html#download-delay
  29. # See also autothrottle settings and docs
  30. #DOWNLOAD_DELAY = 3
  31. # The download delay setting will honor only one of:
  32. #CONCURRENT_REQUESTS_PER_DOMAIN = 16
  33. #CONCURRENT_REQUESTS_PER_IP = 16
  34.  
  35. # Disable cookies (enabled by default)
  36. #COOKIES_ENABLED = False
  37.  
  38. # Disable Telnet Console (enabled by default)
  39. #TELNETCONSOLE_ENABLED = False
  40.  
  41. # Override the default request headers:
  42. #DEFAULT_REQUEST_HEADERS = {
  43. #   'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
  44. #   'Accept-Language': 'en',
  45. #}
  46.  
  47. # Enable or disable spider middlewares
  48. # See https://doc.scrapy.org/en/latest/topics/spider-middleware.html
  49. #SPIDER_MIDDLEWARES = {
  50. #    'mall_spider.middlewares.MallSpiderSpiderMiddleware': 543,
  51. #}
  52.  
  53. # Enable or disable downloader middlewares
  54. # See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
  55. # 下载中间件进行反爬，随机头和随机IP,最好使用付费的
  56. # DOWNLOADER_MIDDLEWARES = {
  57. #    'mall_spider.middlewares.MallSpiderDownloaderMiddleware': 543,
  58. #    'mall_spider.middlewares.RandomUserAgent':301,
  59. # }
  60.  
  61. # Enable or disable extensions
  62. # See https://doc.scrapy.org/en/latest/topics/extensions.html
  63. #EXTENSIONS = {
  64. #    'scrapy.extensions.telnet.TelnetConsole': None,
  65. #}
  66.  
  67. # Configure item pipelines
  68. # See https://doc.scrapy.org/en/latest/topics/item-pipeline.html
  69. ITEM_PIPELINES = {
  70. 	# 数字越小越先执行
  71.    'mall_spider.pipelines.CategoryPipeline': 300,
  72.    'mall_spider.pipelines.ProductPipeline':301,
  73. }
  74.  
  75. # Enable and configure the AutoThrottle extension (disabled by default)
  76. # See https://doc.scrapy.org/en/latest/topics/autothrottle.html
  77. #AUTOTHROTTLE_ENABLED = True
  78. # The initial download delay
  79. #AUTOTHROTTLE_START_DELAY = 5
  80. # The maximum download delay to be set in case of high latencies
  81. #AUTOTHROTTLE_MAX_DELAY = 60
  82. # The average number of requests Scrapy should be sending in parallel to
  83. # each remote server
  84. #AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
  85. # Enable showing throttling stats for every response received:
  86. #AUTOTHROTTLE_DEBUG = False
  87.  
  88. # Enable and configure HTTP caching (disabled by default)
  89. # See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
  90. #HTTPCACHE_ENABLED = True
  91. #HTTPCACHE_EXPIRATION_SECS = 0
  92. #HTTPCACHE_DIR = 'httpcache'
  93. #HTTPCACHE_IGNORE_HTTP_CODES = []
  94. #HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'
  95.  
  96. # 配置MongoDB的URL
  97. MONGODB_URL = 'mongodb://127.0.0.1:27017'
  98.  
  99. # 在settings文件中配置scrapy_redis
  100. # REDIS数据链接
  101. REDIS_URL = 'redis://127.0.0.1:6379/0'
  102.  
  103. # 去重容器类: 用于把已爬指纹存储到基于Redis的set集合中
  104. DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter"
  105. # 调度器: 用于把待爬请求存储到基于Redis的队列
  106. SCHEDULER = "scrapy_redis.scheduler.Scheduler"
  107. # 是不进行调度持久化:
  108. # 如果是True, 当程序结束的时候, 会保留Redis中已爬指纹和待爬的请求
  109. # 如果是False, 当程序结束的时候, 会清空Redis中已爬指纹和待爬的请求
  110. SCHEDULER_PERSIST = True
  111.

4.4 中间件配置-middlewares.py

注释：这里以随机请求头和代理IP池返回的随机IP构成请求，但是由于京东对免费的IP池进行了反爬，所以我在settings中并没有开启middlewares的配置，仅作参考，若需要抓取万级量度的数据，可以选购付费IP池

  1. # -*- coding: utf-8 -*-
  2.  
  3. # Define here the models for your spider middleware
  4. #
  5. # See documentation in:
  6. # https://doc.scrapy.org/en/latest/topics/spider-middleware.html
  7. import random
  8. import requests
  9. import re
  10. from scrapy.downloadermiddlewares.retry import RetryMiddleware
  11. from twisted.internet import defer
  12. from twisted.internet.error import TimeoutError, DNSLookupError, \
  13.         ConnectionRefusedError, ConnectionDone, ConnectError, \
  14.         ConnectionLost, TCPTimedOutError
  15. from twisted.web.client import ResponseFailed
  16. from scrapy.core.downloader.handlers.http11 import TunnelError
  17.  
  18. # 实现代理IP中间件
  19. class MallSpiderDownloaderMiddleware(object):
  20.     EXCEPTIONS_TO_RETRY = (defer.TimeoutError, TimeoutError, DNSLookupError,
  21.                            ConnectionRefusedError, ConnectionDone, ConnectError,
  22.                            ConnectionLost, TCPTimedOutError, ResponseFailed,
  23.                            IOError, TunnelError)
  24.     def process_request(self, request, spider):
  25.         # 从代理池随机获取一个代理IP,协议和域名
  26.         response = requests.get('http://localhost:6888/random?protocal=https&domain=jd.com')
  27.         request.meta['proxy'] = response.content.decode()
  28.  
  29.     def process_exception(self, request, exception, spider):
  30.         if isinstance(exception, self.EXCEPTIONS_TO_RETRY):
  31.             # 当请求出现异常的时候, 代理池哪些代理IP在本域名下是不可以用的
  32.             url = 'http://localhost:6868/disable_domain'
  33.             proxy = request.meta['proxy']
  34.             ip = re.findall('https?://(.+?):\d+', proxy)[0]
  35.             params = {
  36.                 'ip': ip,
  37.                 'domain': 'jd.com'
  38.             }
  39.             # 发送请求,  告诉代理池这个代理IP在本域名下是不可以用的
  40.             requests.get(url, params=params)
  41.  
  42.  
  43.  
  44.  
  45.  
  46.  
  47.  
  48. #1. 准备User-Agent列表
  49. # 准备请求头
  50. USER_AGENTS = [
  51.     "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; AcooBrowser; .NET CLR 1.1.4322; .NET CLR 2.0.50727)",
  52.     "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Acoo Browser; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)",
  53.     "Mozilla/4.0 (compatible; MSIE 7.0; AOL 9.5; AOLBuild 4337.35; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)",
  54.     "Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)",
  55.     "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 2.0.50727; Media Center PC 6.0)",
  56.     "Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 1.0.3705; .NET CLR 1.1.4322)",
  57.     "Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 5.2; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.2; .NET CLR 3.0.04506.30)",
  58.     "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN) AppleWebKit/523.15 (KHTML, like Gecko, Safari/419.3) Arora/0.3 (Change: 287 c9dfb30)",
  59.     "Mozilla/5.0 (X11; U; Linux; en-US) AppleWebKit/527+ (KHTML, like Gecko, Safari/419.3) Arora/0.6",
  60.     "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.2pre) Gecko/20070215 K-Ninja/2.1.1",
  61.     "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9) Gecko/20080705 Firefox/3.0 Kapiko/3.0",
  62.     "Mozilla/5.0 (X11; Linux i686; U;) Gecko/20070322 Kazehakase/0.4.5",
  63.     "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko Fedora/1.9.0.8-1.fc10 Kazehakase/0.5.6",
  64.     "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11",
  65.     "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/535.20 (KHTML, like Gecko) Chrome/19.0.1036.7 Safari/535.20",
  66.     "Opera/9.80 (Macintosh; Intel Mac OS X 10.6.8; U; fr) Presto/2.9.168 Version/11.52",
  67.     "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.11 TaoBrowser/2.0 Safari/536.11",
  68.     "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.71 Safari/537.1 LBBROWSER",
  69.     "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; LBBROWSER)",
  70.     "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 732; .NET4.0C; .NET4.0E; LBBROWSER)",
  71.     "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.84 Safari/535.11 LBBROWSER",
  72.     "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)",
  73.     "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; QQBrowser/7.0.3698.400)",
  74.     "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 732; .NET4.0C; .NET4.0E)",
  75.     "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; SV1; QQDownload 732; .NET4.0C; .NET4.0E; 360SE)",
  76.     "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 732; .NET4.0C; .NET4.0E)",
  77.     "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)",
  78.     "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1",
  79.     "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1",
  80.     "Mozilla/5.0 (iPad; U; CPU OS 4_2_1 like Mac OS X; zh-cn) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8C148 Safari/6533.18.5",
  81.     "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:2.0b13pre) Gecko/20110307 Firefox/4.0b13pre",
  82.     "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:16.0) Gecko/20100101 Firefox/16.0",
  83.     "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11",
  84.     "Mozilla/5.0 (X11; U; Linux x86_64; zh-CN; rv:1.9.2.10) Gecko/20100922 Ubuntu/10.10 (maverick) Firefox/3.6.10"
  85. ]
  86.  
  87. # 实现RandomUSerAgent类,UserAgent的中间件
  88. class RandomUserAgent(object):
  89.     def process_request(self, request, spider):
  90.         # 如果请求是https://cdnware.m.jd.com开头，就是iPhone的user-agent
  91.         if request.url.startswith('https://cdnware.m.jd.com'):
  92.             request.headers['user-agent'] = 'JD4iPhone/164880 (iPhone; iOS 12.1.2; Scale/2.00)'
  93.         else:
  94.             request.headers['user-agent'] =random.choice(USER_AGENTS)
  95.  
  96.  
  97.  
  98.

4.5 分类信息抓取-jd_category.py

注：抓取主页面的所有分类信息

  1. # -*- coding: utf-8 -*-
  2. import scrapy
  3. import json
  4. from mall_spider.items import Category
  5.  
  6. class JdCategorySpider(scrapy.Spider):
  7.     name = 'jd_category'
  8.     allowed_domains = ['3.cn']
  9.     start_urls = ['https://dc.3.cn/category/get'] # 修改起始url
  10.  
  11.     def parse(self, response):
  12.         # print(response.body.decode('GBK')) # 指定编码方式，防止乱码
  13.         result = json.loads(response.body.decode('GBK'))
  14.         datas = result['data']
  15.         # 遍历数据列表
  16.         for data in datas:
  17.             # 利用数据模型进行存储
  18.             item = Category()
  19.             b_category = data['s'][0]
  20.             # 大分类信息
  21.             b_category_info = b_category['n']
  22.             # print("大分类：{}".format(b_category_info))
  23.             item['b_category_name'], item['b_category_url'] = self.get_actegory_name_url(b_category_info)
  24.             # 中分类信息列表
  25.             m_category_s = b_category['s']
  26.             # 遍历中分类信息列表
  27.             for m_category in m_category_s:
  28.                 # 中分类信息
  29.                 m_category_info = m_category['n']
  30.                 item['m_category_name'], item['m_category_url'] = self.get_actegory_name_url(m_category_info)
  31.                 # print("中分类：{}".format(m_category_info))
  32.                 # 获取小分类数据列表
  33.                 s_category_s = m_category['s']
  34.                 for s_category in s_category_s:
  35.                     s_category_info = s_category['n']
  36.                     # print("小分类：{}".format(s_category_info))
  37.                     item['s_category_name'], item['s_category_url'] = self.get_actegory_name_url(s_category_info)
  38.                     # print(item)
  39.                     # 把数据交给引擎
  40.                     yield item
  41.  
  42.  
  43.     def get_actegory_name_url(self,category_info):
  44.         '''
  45.         根据分类的信息，提取名称和url
  46.         :param category_info: 分类信息
  47.         :return: 分类名称和url
  48.         '''
  49.         # 商品分类有三种数据格式，部分url需要进行拼接
  50.         # mice.jd.com;1713-9278;4938-12420-12423
  51.         category = category_info.split('|')
  52.         category_url = category[0] # 分类URL
  53.         category_name = category[1] # 分类名称
  54.         # 处理第一类url
  55.         if category_url.count('jd.com') == 1:
  56.             # url进行补全即可
  57.             category_url = 'https://'+category_url
  58.         elif category_url.count('-')==1:
  59.             # 处理第二类url
  60.             category_url = 'https://channel.jd.com/{}.html'.format(category_url)
  61.         else:
  62.             # 处理第三类，把url中-替换为,
  63.             category_url = category_url.replace('-',',')
  64.             category_url = 'https://list.jd.com/list.html?cat={}'.format(category_url)
  65.         return category_name, category_url
  66.  
  67.  
  68.  
  69.  
  70.  
  71.

4.6 抓取商品详细信息-jd_product.py

注：这里主要是借用Networks面板进行分析，对网络中出现的json数据接口进行分析得出的，并没有使用selenium对页面进行操作

  1. # -*- coding: utf-8 -*-
  2. import time
  3.  
  4. import scrapy
  5. import json
  6. import pickle
  7. from mall_spider.items import Product
  8. from jsonpath import jsonpath
  9. from scrapy_redis.spiders import RedisSpider
  10. '''
  11. 分布式爬虫
  12. 1.修改爬虫类，继承RedisSpider
  14. '''
  15. class JdProductSpider(RedisSpider):
  16.     name = 'jd_product'
  17.     allowed_domains = ['jd.com','3.cn']
  18.     # start_urls = ['http://jd.com/']
  19.  
  20.     # 2. 用于指定起始url列表，在redis数据库中的key
  21.     redis_key = 'jd_product:category'
  22.  
  23.     # def start_requests(self):
  24.     #     '''重写start_requests方法，根据分类信息构建列表页的请求'''
  25.     #     category = {
  26.     #         "b_category_name":"家用电器",
  27.     #         "b_category_url":"https://jiadian.jd.com",
  28.     #         "m_category_name": "电视",
  29.     #         "m_category_url": "https://list.jd.com/list.html?cat=737,794,798",
  30.     #         "s_category_name": "超薄电视",
  31.     #         "s_category_url": "https://list.jd.com/list.html?cat=737,794,798&ev=4155_76344&sort=sort_rank_asc&trans=1&JL=2_1_0#J_crumbsBar"
  32.     #     }
  33.     #     # 根据小分类的url构建列表页面请求
  34.     #     yield scrapy.Request(category['s_category_url'],callback=self.parse,meta={'category':category})
  35.  
  36.     # 3. 重写make_request_from_data
  37.     def make_request_from_data(self, data):
  38.         '''
  39.         根据redis读取的分类信息的二进制数据，构建请求
  40.         :param data: 分类信息的二进制数据
  41.         :return: 根据小分类URl构建的请求对象
  42.         '''
  43.         # 把分类的二进制数据转换为字典
  44.         category = pickle.loads(data)
  45.         # 根据小分类的url构建列表页面请求
  46.         # 注意这里必须使用return返回请求，yield会变成生成器
  47.         return scrapy.Request(category['s_category_url'],callback=self.parse,meta={'category':category})
  48.  
  49.  
  50.  
  51.  
  52.     def parse(self, response):
  53.         category = response.meta['category']
  54.         # print(category)
  55.         # 解析列表页，提取商品的skuid
  56.         sku_ids = response.xpath('//div[contains(@class,"gl-i-wrap")]//@data-sku').extract()
  57.         for sku_id in sku_ids:
  58.             # 创建一个Product,用于保存商品数据
  59.             item = Product()
  60.             # product_category = scrapy.Field()  # 商品类别
  61.             # product_sku_id = scrapy.Field()  # 商品ID
  62.             # 设置商品类型
  63.             item['product_category'] = category
  64.             item['product_sku_id'] =sku_id
  65.             # 构建商品基本信息请求，通过Charles进行识别获得
  66.             # item传入无需深拷贝
  67.             product_base_url = 'https://cdnware.m.jd.com/c1/skuDetail/apple/7.3.0/{}.json'.format(sku_id) # 但是只能看前三十个
  68.             yield scrapy.Request(product_base_url,callback=self.parse_product_base,meta={'item':item})
  69.         # 获取下一页的url,这里源代码不能使用了，已经进行修改
  70.         # 最大页面数,下面未提取出来，所以换在上面提取,这里不知道为什么失效了，所以只抓取每个分类的前20页,也就是i为40
  71.         max_len = int(response.xpath('//*[@id="J_topPage"]/span/i/text()').extract_first())
  72.         # print("hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh",max_len)
  73.         i = 1
  74.         s = 0
  75.         # for i in range(1,max_len+1):
  76.         if i < 20:
  77.             # 进行url拼接，这里的url只为偶数，但发现为偶数时候，不显示后面30个商品，这里是用&s参数来区分的，s每次加60(s代表页指标不动) p代表上页下页
  78.             # 如果i为偶数，则代表不换页，s不变
  79.             if i %2 !=0:
  80.                 s += 30
  81.             next_url = category['s_category_url'].split('#')[0]+ "&page={}&s={}".format(i,s)
  82.             # print(next_url)
  83.             yield scrapy.Request(next_url,callback=self.parse,meta={'category':category})
  84.             i += 1
  85.  
  86.  
  87.  
  88.  
  89.     def parse_product_base(self,response):
  90.         # 取出传递来的数据
  91.         item = response.meta['item']
  92.         # print(item)
  93.         # print(response.text)
  94.         # 把json字符串转化为字典
  95.         result = json.loads(response.text)
  96.         # 提取数据
  97.         # product_name = scrapy.Field()  # 商品名称
  98.         # product_img_url = scrapy.Field()  # 商品图片URL
  99.         # product_book_info = scrapy.Field()  # 图书信息，作者，出版社
  100.         # product_option = scrapy.Field()  # 商品选项
  101.         # product_shop = scrapy.Field()  # 商品店铺
  102.         # product_comments = scrapy.Field()  # 商品评论数量
  103.         # product_ad = scrapy.Field()  # 商品促销
  104.         # product_price = scrapy.Field()  # 商品价格
  105.         item['product_name'] = result['wareInfo']['basicInfo']['name']
  106.         item['product_img_url'] = result['wareInfo']['basicInfo']['wareImage'][0]['small']
  107.         item['product_book_info'] = result['wareInfo']['basicInfo']['bookInfo']
  108.         color_size = jsonpath(result,'$..colorSize')
  109.         if color_size:
  110.             # 注意colorsize值是个列表，jsonpath返回列表
  111.             color_size = color_size[0]
  112.             product_option = {}
  113.             for option in color_size:
  114.                 title = option['title']
  115.                 value = jsonpath(option,'$..text') # 这是一个路径表达式
  116.                 product_option[title] = value
  117.             item['product_option'] = product_option
  118.         shop = jsonpath(result,'$..shop')
  119.         if shop:
  120.             shop = shop[0]
  121.             if shop:
  122.                 # 无id则为京东自营
  123.                 item['product_shop'] = {
  124.                     'shop_id':shop['shopId'],
  125.                     'shop_name': shop['name'],
  126.                     'shop_score': shop['score']
  127.                 }
  128.             else:
  129.                 item['product_shop'] = {
  130.                     'shop_name': '京东自营'
  131.                 }
  132.  
  133.         item['product_category_id'] = result['wareInfo']['basicInfo']['category'].replace(';',',')
  134.         # 准备促销信息的URL,搜索promotion可以获得相关api,5051427
  135.         ad_url = 'https://cd.jd.com/promotion/v2?skuId={}&area=12_919_922_0&cat={}'.format(item['product_sku_id'],item['product_category_id'])
  136.         # print(item)
  137.         # 构建促销请求
  138.         yield scrapy.Request(ad_url,callback=self.parse_product_ad,meta={'item':item})
  139.  
  140.  
  141.     def parse_product_ad(self,response):
  142.         item = response.meta['item']
  143.         print(item)
  144.         # print(response.body.decode['GBK'])
  145.         # 转化为字典
  146.         result = json.loads(response.body)
  147.         item['product_ad'] = jsonpath(result,'$..ad')[0] if jsonpath(result,'$..ad') else ''
  148.         # 构建评价信息请求,comments刷新页面可见
  149.         comments_url = 'https://club.jd.com/comment/productCommentSummaries.action?referenceIds={}'.format(item['product_sku_id'])
  150.         yield scrapy.Request(comments_url,callback=self.parse_product_comments,meta={'item':item})
  151.         print(item)
  152.  
  153.     def parse_product_comments(self,response):
  154.         item = response.meta['item']
  155.         # print(item)
  156.         # print(response.text)
  157.         result = json.loads(response.text)
  158.         item['product_comments'] = {
  159.             'CommentCount': jsonpath(result,'$..CommentCount')[0],
  160.             'GoodCount': jsonpath(result,'$..GoodCount')[0],
  161.             'PoorCount': jsonpath(result,'$..PoorCount')[0],
  162.             'GoodRate': jsonpath(result,'$..GoodRate')[0]
  163.         }
  164.         # print(item)
  165.         # 构造价格请求,但这里scrapy会过滤非jd.com的域名，所以前面需要加上
  166.         price_url = 'https://p.3.cn/prices/mgets?skuIds=J_{}'.format(item['product_sku_id'])
  167.         yield scrapy.Request(price_url,callback=self.parse_product_price,meta={'item':item})
  168.  
  169.  
  170.     def parse_product_price(self,response):
  171.         item = response.meta['item']
  172.         # print(response.text)
  173.         item['product_price'] = json.loads(response.text)[0]['p']
  174.         # print(item)
  175.  
  176.         # 把商品数据交给引擎
  177.         yield item

4.7 分布式操作的实现-add_category_to_redis.py

  1. #!/usr/bin/env python
  2.  
  3. # -*- encoding: utf-8 -*-
  4.  
  5. '''
  6. @Author  :   {Jack Zhao}
  8. @Time    :   2020/5/14 12:26
  10. @Contact :   {zc_dlmu@163.com}
  12. @Desc    :  链接mongodb,链接redis,将mongodb中数据添加到redis_key,关闭mongodb
  13. '''
  14.  
  15. from pymongo import MongoClient
  16. from redis import StrictRedis
  17. from mall_spider.settings import MONGODB_URL,REDIS_URL
  18. from mall_spider.spiders.jd_product import JdProductSpider
  19. import pickle
  20.  
  21. def add_category_to_redis():
  22. 	mongo =MongoClient(MONGODB_URL)
  23. 	redis =StrictRedis.from_url(REDIS_URL)
  24. 	# 获取信息
  25. 	collection = mongo['jd']['category']
  26. 	cursor = collection.find()
  27. 	for category in cursor:
  28. 		# 序列化字典数据
  29. 		data = pickle.dumps(category)
  30. 		# 添加到redis
  31. 		redis.lpush(JdProductSpider.redis_key,data)
  32.  
  33. 	mongo.close()
  34.  
  35. if __name__ == '__main__':
  36. 	# 调用方法测试
  37. 	add_category_to_redis()
  38.  
  39.

五、项目使用方法

  1. # WIn+R 输入services.msc
  2. # 先保证已经开启MongoDB,Redis
  3. cd ../mall_spider
  4. scrapy crawl jd_category # 先抓目录，再进行下面的操作
  5. scrapy crawl jd_product # 这里暂时是使用本机的IP进行爬取，不要过分爬取，防止京东封你家IP

欢迎关注公众号，DataDesigner，让我们一起白话机器学习。

 

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/你好赵伟/article/detail/437195
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号