devbox
AllinToyou
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

scrapy获取读书网书籍信息保存MongoDB、Redis、MySQL数据库和本地表格，并用ImagePipeline下载封面图_数据库中书籍的封面图在查询中显示

作者：AllinToyou | 2024-04-16 07:33:05

赞

踩

数据库中书籍的封面图在查询中显示

1、创建Scrapy项目

scrapy startproject Dushu

2.进入项目目录，使用命令genspider创建Spider

scrapy genspider dushu dushu.com

3、定义要抓取的数据（处理items.py文件）

  1. import scrapy
  2.  
  3. class DushuItem(scrapy.Item):
  4.     # 书籍ID
  5.     book_id = scrapy.Field()
  6.     # 书的链接地址
  7.     book_url = scrapy.Field()
  8.     # 书名
  9.     book_name = scrapy.Field()
  10.     # 作者
  11.     book_author = scrapy.Field()
  12.     # 简介
  13.     book_info = scrapy.Field()
  14.     # 封面图
  15.     cover_img_url = scrapy.Field()
  16.     # 价格
  17.     book_price = scrapy.Field()
  18.     # 标签
  19.     book_tag = scrapy.Field()
  20.     # ISBN，有空获取不到值的情况
  21.     book_isbn = scrapy.Field()
  22.     # 一级分类名
  23.     firstTitle= scrapy.Field()
  24.     # 二级分类名
  25.     secondTitle = scrapy.Field()
  26.     # 三级分类名
  27.     threeTitle = scrapy.Field()
  28.     # 四级分类名
  29.     fourTitle = scrapy.Field()

4、编写提取item数据的Spider（在spiders文件夹下：dushu.py）

  1. # -*- coding: utf-8 -*-
  2. # 通过一级分类页面，获取所有当前分类下的书籍，获取书籍信息后，通过面包屑导航栏来确定该书籍的四级分类
  3. import scrapy
  4. from ..items import DushuItem
  5.  
  6. class DushuSpider(scrapy.Spider):
  7.     name = 'dushu'
  8.     allowed_domains = ['dushu.com']
  9.     start_urls = ['https://www.dushu.com/book/']
  10.     base_url = "https://www.dushu.com"
  11.  
  12.     def parse(self, response):
  13.         # 获取所有需要爬取的一级分类链接地址
  14.         firstUrls = response.xpath('//div/div[@class="row"]/div/div/dl/dt/a/@href').extract()
  15.         for url in firstUrls:
  16.             yield scrapy.Request(url=self.base_url+url,callback=self.parse_second)
  17.  
  18.     def parse_second(self,response):
  19.         # 一级分类的下一页
  20.         next_pages = response.xpath('//div/div[@class="pages"]/a[not(@class="disabled")]/@href').extract()
  21.         for page in next_pages:
  22.             yield scrapy.Request(url=self.base_url+page,callback=self.parse_book)
  23.  
  24.     def parse_book(self, response):
  25.         title = response.xpath('//div/div[@class="row"]/div/div/dl[@class="active"]/dt/text()').get()
  26.         next_url = response.url
  27.         next_url_page = next_url.split('_')[-1].split('.')[0]
  28.         print("准备处理[%s]第%s页" % (title, next_url_page))
  29.         # 书籍url
  30.         all_books_url = response.xpath('//div[@class="bookslist"]/ul/li/div/h3/a/@href').extract()
  31.         for url in all_books_url:
  32.             yield scrapy.Request(url=self.base_url+url,callback=self.book_detail)
  33.  
  34.     def book_detail(self,response):
  35.         item = DushuItem()
  36.         # 书名
  37.         book_name = response.xpath('//div/div/div[@class="book-title"]/h1/text()').get() or "该项为空"
  38.         # with open(book_name+".html","w+",encoding="utf-8")as f:
  39.         #     f.write(response.text)
  40.         # 获取书籍面包屑
  41.         navbar = response.xpath('//div[@class="crumbs"]/a[position()>2]/text()').extract()
  42.         length = len(navbar)
  43.         if length == 2:
  44.             print("该书籍一级分类:%s"%book_name)
  45.             item['firstTitle'] = navbar[0]
  46.             item['secondTitle'] = "-"
  47.             item['threeTitle'] = "-"
  48.             item['fourTitle'] = "-"
  49.         elif length == 3:
  50.             print("该书籍二级分类:%s"%book_name)
  51.             item['firstTitle'] = navbar[0]
  52.             item['secondTitle'] = navbar[1]
  53.             item['threeTitle'] = "-"
  54.             item['fourTitle'] = "-"
  55.         elif length == 4:
  56.             print("该书籍三级分类%s"%book_name)
  57.             item['firstTitle'] = navbar[0]
  58.             item['secondTitle'] = navbar[1]
  59.             item['threeTitle'] = navbar[2]
  60.             item['fourTitle'] = "-"
  61.         elif length == 5:
  62.             print("该书籍四级分类:%s"%book_name)
  63.             item['firstTitle'] = navbar[0]
  64.             item['secondTitle'] = navbar[1]
  65.             item['threeTitle'] = navbar[2]
  66.             item['fourTitle'] = navbar[3]
  67.         else:
  68.             print("该书籍分类异常:%s"%book_name)
  69.             item['firstTitle'] = "-"
  70.             item['secondTitle'] = "-"
  71.             item['threeTitle'] = "-"
  72.             item['fourTitle'] = "-"
  73.  
  74.         # 作者
  75.         book_author = response.xpath('//div/div/div[@class="book-details"]/div/table//tr[1]/td[2]/text()').get()or "该项为空"
  76.         # 标签
  77.         book_tag = response.xpath('//div/div/div[@class="book-details"]/div/table//tr[4]/td[2]/text()').get()or "该项为空"
  78.         # ISBN
  79.         book_isbn = response.xpath('//div/div/div[@class="book-details"]/table//tr[1]/td[2]/text()').get() or "该项为空"
  80.         # 价格
  81.         book_price = response.xpath('//div/div/div[@class="book-details"]/div/p/span/text()').get()or "该项为空"
  82.         # 简介
  83.         book_info = response.xpath('//div/div/div[@class="book-summary"][1]/div/div/text()').get()or "该项为空"
  84.         # 封面图
  85.         cover_img_url = response.xpath('//div/div/div[@class="book-pic"]/div/img/@src').get()or "该项为空"
  86.         # 书籍详细页地址
  87.         book_url = response.url
  88.         # 书籍id
  89.         book_id = book_url.split("/")[-2]
  90.  
  91.         item['book_id'] = book_id
  92.         item['book_name'] = book_name
  93.         item['book_author'] = book_author
  94.         item['book_tag'] = book_tag
  95.         item['book_isbn'] = book_isbn
  96.         item['book_price'] = book_price[1:]
  97.         item['book_info'] = book_info.strip()
  98.         item['cover_img_url'] = "暂无封面图" if "n200.png" in cover_img_url else cover_img_url
  99.         item['book_url'] = book_url
  100.         yield item

5.处理pipelines管道文件保存数据，可将结果保存到文件中（pipelines.py）

  1. # -*- coding: utf-8 -*-
  2.  
  3. import os
  4. import time
  5. import json
  6. import scrapy
  7. import pymysql
  8. import pymongo
  9. import redis
  10. from openpyxl import Workbook
  11. from scrapy.pipelines.images import ImagesPipeline
  12. from scrapy.utils.project import get_project_settings
  13. from scrapy import Item
  14. from scrapy.exceptions import DropItem
  15.  
  16. class MyEncoder(json.JSONEncoder):
  17.     def default(self, o):
  18.         if isinstance(o, bytes):
  19.             return str(o, encoding='utf-8')
  20.         return json.JSONEncoder.default(self, o)
  21.  
  22. class ImagePipeline(ImagesPipeline):
  23.     # 获取存放图片文件夹名字
  24.     IMAGES_STORE = get_project_settings().get('IMAGES_STORE')
  25.     # 图片名字不能含以下特殊符号
  26.     char_list = ['*', '|', ':', '?', '/', '<', '>', '"', '\\']
  27.     def get_media_requests(self, item, info):
  28.         # 下载封面图的时候一定要Referer书的url地址，否则403
  29.         headers = {"User-Agent": "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:56.0) Gecko/20100101 Firefox/56.0",
  30.                    "Referer": item['book_url']}
  31.         # 无封面图的不下载
  32.         cover_img_url = item['cover_img_url']
  33.         if cover_img_url.startswith('http'):
  34.             yield scrapy.Request(cover_img_url,headers=headers)
  35.  
  36.     def item_completed(self, results, item, info):
  37.         book_name = item['book_name']
  38.         if (not os.path.exists(self.IMAGES_STORE)):
  39.             os.makedirs(self.IMAGES_STORE)
  40.         # results:[(True, {'url': 'https://a.dushu.com/img/n200.png', 'path': 'full/783a2.jpg', 'checksum': '2792e5'})]
  41.         if results:
  42.             print("封面图下载：%s"% book_name)
  43.             image_path = [x['path'] for ok, x in results if ok]
  44.             for i in self.char_list:
  45.                 if i in book_name:
  46.                     print("'%s'包含特殊符号'%s'已转义！" % (book_name, i))
  47.                     book_name = book_name.replace(i, "_")
  48.             os.rename(self.IMAGES_STORE + '/' + image_path[0], self.IMAGES_STORE + '/' + book_name + ".jpg")
  49.  
  50.         else:
  51.             print("无封面图，无需下载：%s"% book_name)
  52.         return item
  53.  
  54.     def close_spider(self, spider):
  55.         print("图片下载完成!")
  56.         # full最后如果是空文件夹删掉
  57.         path = self.IMAGES_STORE + '/full'
  58.         if not os.listdir(path):
  59.             os.rmdir(path)
  60.         if not os.listdir(self.IMAGES_STORE):
  61.             os.rmdir(path)
  62.  
  63. class XlsxPipeline(object):
  64.     def __init__(self):
  65.         self.wb = Workbook()
  66.         self.ws = self.wb.active
  67.         self.ws.title = "dushu网图书信息表"
  68.         self.ws.append(['book_id','一级分类','二级分类','三级分类',	'四级分类',	'书名',
  69.                         '作者',	'标签', 'ISBN',	'价格(元)',	'简介',
  70.                         '封面图', '书的链接地址'])
  71.  
  72.     def process_item(self, item, spider):
  73.         text = [item['book_id'],item['firstTitle'],item['secondTitle'],item['threeTitle'],item['fourTitle'],item['book_name'],
  74.                 item['book_author'],item['book_tag'],item['book_isbn'],item['book_price'],item['book_info'],
  75.                 item['cover_img_url'],item['book_url']]
  76.         self.ws.append(text)
  77.         return item
  78.  
  79.     def close_spider(self,spider):
  80.         time_file = time.strftime("%Y-%m-%d",time.localtime())
  81.         self.wb.save(spider.name + time_file + ".xlsx")
  82.         print("表格数据处理完毕，谢谢使用！")
  83.  
  84. class MysqlPipeline():
  85.     @classmethod
  86.     def from_crawler(cls,crawler):
  87.         cls.MYSQL_HOST = crawler.settings.get('MYSQL_HOST')
  88.         cls.MYSQL_PORT = crawler.settings.get('MYSQL_PORT')
  89.         cls.MYSQL_USER = crawler.settings.get('MYSQL_USER')
  90.         cls.MYSQL_PASSWD = crawler.settings.get('MYSQL_PASSWD')
  91.         cls.MYSQL_DBNAME = crawler.settings.get('MYSQL_DBNAME')
  92.         cls.MYSQL_CHARSET = crawler.settings.get('MYSQL_CHARSET')
  93.         return cls()
  94.     def open_spider(self,spider):
  95.         self.db = pymysql.connect(host=self.MYSQL_HOST,port=self.MYSQL_PORT,user=self.MYSQL_USER,
  96.                                   passwd=self.MYSQL_PASSWD,db=self.MYSQL_DBNAME,charset=self.MYSQL_CHARSET)
  97.         self.cursor = self.db.cursor()
  98.     def process_item(self, item, spider):
  99.         try:
  100.             sql = 'CREATE TABLE IF NOT EXISTS dushu(book_id BIGINT PRIMARY KEY NOT NULL ,firstTitle VARCHAR(15),' \
  101.                   'secondTitle VARCHAR(20),threeTitle VARCHAR(20),fourTitle VARCHAR(20),book_name VARCHAR(200) NOT NULL ,' \
  102.                   'book_author VARCHAR(200),book_tag VARCHAR(100),book_isbn VARCHAR(50),book_price VARCHAR(20),book_info TEXT,' \
  103.                   'cover_img_url VARCHAR(200),book_url VARCHAR(200))ENGINE=InnoDB DEFAULT CHARSET="utf8mb4";'
  104.             self.cursor.execute(sql)
  105.         except:
  106.             pass
  107.         try:
  108.             self.cursor.execute("SELECT book_id FROM dushu  WHERE book_id = %s;" ,item['book_id'])
  109.             switch = self.cursor.fetchone()
  110.             keys,values = zip(*item.items())
  111.             if switch:
  112.                 sql = """INSERT INTO dushu({})VALUES ({})ON DUPLICATE KEY UPDATE {};""".format(
  113.                     ','.join(keys),
  114.                     ','.join(['%s']*len(values)),
  115.                     ','.join(['{}=%s'.format(k)for k in keys])
  116.                 )
  117.                 self.cursor.execute(sql, values * 2)
  118.             else:
  119.                 sql = """INSERT INTO dushu({})VALUES ({});""".format(
  120.                     ','.join(keys),
  121.                     ','.join(['%s'] * len(values))
  122.                 )
  123.                 self.cursor.execute(sql,values)
  124.             self.db.commit()
  125.             return item
  126.         except Exception as e:
  127.             print("出错了:",e)
  128.             self.db.rollback()
  129.  
  130.     def close_spider(self,spider):
  131.         print("mysql数据库处理完毕")
  132.         self.cursor.close()
  133.         self.db.close()
  134.  
  135. class MongoPipeline():
  136.     @classmethod
  137.     def from_crawler(cls, crawler):
  138.         cls.MONGO_HOST = crawler.settings.get('MONGO_HOST')
  139.         cls.MONGO_PORT = crawler.settings.get('MONGO_PORT')
  140.         cls.MONGO_DB = crawler.settings.get('MONGO_DB')
  141.         return cls()
  142.  
  143.     def open_spider(self,spider):
  144.         self.client = pymongo.MongoClient(self.MONGO_HOST,self.MONGO_PORT)
  145.         # 记录插入了几条数据
  146.         self.num = 0
  147.  
  148.     def process_item(self,item,spider):
  149.         try:
  150.             self.db = self.client[self.MONGO_DB]
  151.             self.book = self.db[spider.name]
  152.             # 先获取数据库中有没有数据，如果有要去判断该值是否已经存在
  153.             count = self.book.find().count()
  154.             # 集合对象的insert方法需传入一个字典对象(不能传入Item对象)
  155.             data = dict(item) if isinstance(item,Item)else item
  156.             if count == 0:
  157.                 print("MongoDB数据库无数据，直接插入数据！")
  158.                 self.book.insert(data)
  159.                 self.num += 1
  160.             else:
  161.                 book_name = item['book_name']
  162.                 count = self.book.find({'book_name':book_name}).count()
  163.                 if count == 0:
  164.                     print("%s:添加数据库中..."%book_name)
  165.                     self.book.insert(data)
  166.                     self.num += 1
  167.                 else:
  168.                     print("%s:该数据已存在无需添加！"%book_name)
  169.             return item
  170.         except Exception as e:
  171.             print("MongoDB数据库出错：",e)
  172.  
  173.     def close_spider(self,spider):
  174.         print("已完成,本次共保存到MongoDB数据库:%s条数据!" % self.num)
  175.         self.client.close()
  176.  
  177. class RedisPipeline():
  178.     @classmethod
  179.     def from_crawler(cls,crawler):
  180.         # cls.REDIS_HOST = crawler.settings.get('REDIS_HOST')
  181.         cls.REDIS_HOST = get_project_settings().get('REDIS_HOST')
  182.         cls.REDIS_PORT = crawler.settings.get('REDIS_PORT')
  183.         cls.REDIS_DBNAME = crawler.settings.get('REDIS_DBNAME')
  184.         cls.REDIS_decode_responses = crawler.settings.get('REDIS_decode_responses')
  185.         return cls()
  186.     def open_spider(self,spider):
  187.         try:
  188.             self.redis_client = redis.StrictRedis(self.REDIS_HOST,self.REDIS_PORT,self.REDIS_DBNAME,
  189.                                                   decode_responses=self.REDIS_decode_responses)
  190.         except Exception as e:
  191.             print("redis数据库出错:",e)
  192.  
  193.     def process_item(self,item,spider):
  194.         if self.redis_client.sadd('dushu:items',item['book_name']):
  195.             return item
  196.         raise DropItem
  197.  
  198.     def close_spider(self,spider):
  199.         print("redis处理完成！")

6.配置settings文件（settings.py)

  1.  
  2. LOG_FILE = "dushu.log"
  3. LOG_LEVEL = "DEBUG"
  4. LOG_STDOUT = True
  5.  
  6. IMAGES_STORE = './images'
  7.  
  8. # 配置MYSQL
  9. MYSQL_HOST = "localhost"
  10. MYSQL_PORT = 3306
  11. MYSQL_USER = "root"
  12. MYSQL_PASSWD = "123456"
  13. MYSQL_DBNAME = "python5"
  14. MYSQL_CHARSET = "utf8mb4"
  15.  
  16. # 配置mongodb
  17. MONGO_HOST = 'localhost'
  18. MONGO_PORT = 27017
  19. MONGO_DB = 'py4'
  20.  
  21. # 配置redis
  22. REDIS_HOST = 'localhost'
  23. REDIS_PORT = 6379
  24. REDIS_DBNAME = 4
  25. REDIS_decode_responses = True
  26.  
  27. ROBOTSTXT_OBEY = False
  28. DOWNLOAD_DELAY = 3
  29. DEFAULT_REQUEST_HEADERS = {
  30.     'User-Agent': 'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0);',
  31.   # 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
  32.   # 'Accept-Language': 'en',
  33. }
  34. ITEM_PIPELINES = {
  35.    'Dushu.pipelines.XlsxPipeline': 301,
  36.    'Dushu.pipelines.ImagePipeline': 3,
  37.     'Dushu.pipelines.RedisPipeline': 300,
  38.    'Dushu.pipelines.MysqlPipeline': 302,
  39.    'Dushu.pipelines.MongoPipeline': 303,
  40. }

7.记得提前打开mysq/redis/MongoDBl数据库，并且建立好相应的表

  1. CREATE TABLE IF NOT EXISTS dushu(book_id BIGINT PRIMARY KEY NOT NULL ,
  2. book_name VARCHAR(200) NOT NULL ,book_author VARCHAR(200),
  3. book_tag VARCHAR(100),book_isbn VARCHAR(50),
  4. book_price VARCHAR(20),book_info TEXT,cover_img_url VARCHAR(200),book_url VARCHAR(200))ENGINE=InnoDB DEFAULT CHARSET="utf8mb4";

8.以上设置完毕，进行爬取：执行项目命令crawl，启动Spider：

scrapy crawl dushu

 

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/AllinToyou/article/detail/433043
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号