devbox
Cpp五条
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

《大众点评爬虫程序实战：爬取店铺展示信息》_大众点评 csdn

作者：Cpp五条 | 2024-04-17 20:27:48

赞

踩

大众点评 csdn

 一、引言 

  •  在日常生活中，我们经常会遇到一个问题：不知道吃什么。尤其是在陌生的城市或附近的地方，面对众多的餐馆选择，很难做出决策。随着互联网的发展，大众点评等餐饮评价平台应运而生，为我们提供了海量的餐馆信息和用户评价。然而，即使在这样的平台上，面对庞大的数据量，我们仍然很难迅速找到最适合我们口味的美食店

二、爬取目标

  • 采集以下数据：店名，美食类型，地点，评分，评价人数以及人均消费

三、准备工作

  • 版本：python版本3.x及以上
  • 需要用到的包：requests，selenium，re，bs4，tqdm，subprocess，time，random，bag（自写包，可私聊获取）

  • json文件：完整json文件过大，这里只截取了部分数据进行展示

    1. # city.json
    2. {
    3.     "郑州": "https://www.dianping.com/zhengzhou",
    4.     "珠海": "https://www.dianping.com/zhuhai",
    5.     "张家口": "https://www.dianping.com/zhangjiakou"
    6. }
    7.  
    8.  
    9. # menu.json
    10. {
    11.     "美食": "https://www.dianping.com/{}/ch10",
    12.     "丽人": "https//:www.dianping.com/{}/beauty",
    13.     "周边游": "https//:www.dianping.com/{}/ch35",
    14. }
    15. """menu.json这个文件通过后面代码自动生成，生成格式如上所示"""
    16.  
    17. # cookies.json
    18. [{}]
    19. """这里涉及到隐私问题就没把cookies展示出来，
    20. 下面会一步步带领大家如何自动获得可用的cookies
    21. 并且保存到本地需要时直接调用"""

四、爬虫实现

  1. 使用selenium获取登录后的cookies

    1. @echo off
    2. cd "C:\Program Files\Google\Chrome\Application"
    3. start chrome.exe --remote-debugging-port=9222 --user-data-dir="C:\selenium\AutomationProfile"
    1. #!/usr/bin/env python3
    2. # coding:utf-8
    3. import subprocess
    4. import bag
    5. import time
    6. import random
    7.  
    8. # batch_file_content = r'''
    9. # @echo off
    10. # cd "C:\Program Files\Google\Chrome\Application"
    11. # start chrome.exe --remote-debugging-port=9222 --user-data-dir="C:\selenium\AutomationProfile"
    12. # '''
    13. #
    14. # with open('run_chrome.bat', 'w') as f:
    15. #     f.write(batch_file_content)
    16.  
    17. subprocess.Popen('run_chrome.bat', shell=True)
    18.  
    19. web = bag.Bag.web_debug()
    20.  
    21. web.get(r'https://www.dianping.com/')
    22. time.sleep(random.randint(5, 10))
    23. cookie = web.get_cookies()
    24.  
    25. web.close()
    26.  
    27. bag.Bag.save_json(cookie, r'./cookies.json')

    • 新建一个文本文件，将第一个代码复制过去并修改后缀为.bat文件，至于为什么要这样做呢，主要是因为这样做了后可以用subprocess来控制程序

    • 运行下面的代码一个可用的cookies便会自动生成

  2. 选择需要爬取的类型并生成menu.json文件

    1. #!/usr/bin/env python3
    2. # coding:utf-8
    3. import bag
    4. from bs4 import BeautifulSoup
    5. import re
    6.  
    7. session = bag.session.create_session()
    8.  
    9. for cookie in bag.Bag.read_json(r'./cookies.json'):
    10.     session.cookies.set(cookie['name'], cookie['value'])
    11.  
    12.  
    13. # 输入需要爬取的城市名称
    14. def choose_city():
    15.     js_data = bag.Bag.read_json('./city.json')
    16.     choose = input('输入城市名：')
    17.     judge = js_data.get(choose)  # 判断输入的城市是否存在
    18.  
    19.     # pattern = re.compile(r'<a.*?data-click-title="first".*?href="(.*?)".*?>(.*?)</a>', re.S)
    20.     pattern = re.compile(r'<a.*?href="(.*?)".*?>(.*?)</a>', re.S)
    21.  
    22.     dic = {}
    23.  
    24.     if judge:
    25.         resp = session.get(judge)
    26.         html = BeautifulSoup(resp.text, 'lxml')
    27.         soup = html.findAll('span', class_='span-container')
    28.         for info in soup:
    29.             data = re.findall(pattern, str(info))
    30.             mid: list = data[0][0].split('/')
    31.             mid[-2] = '{}'
    32.             dic[data[0][1]] = 'https:' + ''.join(mid)
    33.     else:
    34.         print('无效输入！')
    35.         choose_city()
    36.     
    37.     print(dic)   # 根据输入信息得到的生成结果
    38.     '''输入城市名：珠海
    39.     {
    40.       "美食": "https:www.dianping.com{}ch10",
    41.       "休闲娱乐": "https:www.dianping.com{}ch30",
    42.       "结婚": "https:www.dianping.com{}wedding",
    43.       "电影演出赛事": "https:www.dianping.com{}movie",
    44.       "丽人": "https:www.dianping.com{}beauty",
    45.       "酒店": "https:www.dianping.com{}hotel",
    46.       "亲子": "https:www.dianping.com{}baby",
    47.       "周边游": "https:www.dianping.com{}ch35",
    48.       "运动健身": "https:www.dianping.com{}ch45",
    49.       "购物": "https:www.dianping.com{}ch20",
    50.       "家装": "https:www.dianping.com{}home",
    51.       "学习培训": "https:www.dianping.com{}education",
    52.       "生活服务": "https:www.dianping.com{}ch80",
    53.       "医疗健康": "https:www.dianping.com{}ch85",
    54.       "爱车": "https:www.dianping.com{}ch65",
    55.       "宠物": "https:www.dianping.com{}ch95"
    56.     }'''
    57.  
    58.     bag.Bag.save_json(dic, r'./menu.json')
    59.  
    60.  
    61. if __name__ == '__main__':
    62.     choose_city()

  3. 完整代码

    1. # choose.py
    2. # !/usr/bin/env python3
    3. # coding:utf-8
    4. import bag
    5.  
    6.  
    7. def choose_city():
    8.     session = bag.session.create_session()
    9.  
    10.     for cookie in bag.Bag.read_json(r'./cookies.json'):
    11.         session.cookies.set(cookie['name'], cookie['value'])
    12.  
    13.     session.headers['Connection'] = 'close'
    14.     js_data = bag.Bag.read_json('./city.json')
    15.     choose = input('输入城市名：')
    16.     judge = js_data.get(choose)
    17.  
    18.     if judge:
    19.         city = judge.split('/')[-1]
    20.         choose_1 = input('输入爬取类类型：')
    21.         js_data1 = bag.Bag.read_json('./menu.json')
    22.         judge1 = js_data1.get(choose_1)
    23.         if judge1:
    24.             return judge1.format(city), session
    25.         else:
    26.             print('开发中......')
    27.             return None
    28.     else:
    29.         print('无效输入！')
    30.         return None
    31.  
    1. # get_shop.py
    2. # !/usr/bin/env python3
    3. # coding:utf-8
    4. import bag
    5. import chooses
    6. import re
    7. from bs4 import BeautifulSoup
    8. from tqdm import tqdm
    9. import requests
    10.  
    11.  
    12. proxies = {
    13.      "http": "http://{}:{}",
    14. }
    15.  
    16.  
    17. def check():
    18.     url_ = r'https://www.dianping.com/zhuhai/ch10'
    19.     ip_ls = bag.Bag.read_json('../代理ip/IP地址.json')
    20.     index = 0
    21.     if len(ip_ls) == 0:
    22.         print('IP地址全部失效')
    23.         exit()
    24.     for ip_address in ip_ls:
    25.         proxies_ = {
    26.             "http": "{}:{}".format(ip_address[0], ip_address[1]),
    27.         }
    28.         resp = session.get(url_, proxies=proxies_)
    29.  
    30.         if resp.status_code == 200:
    31.             proxies['http'] = proxies['http'].format(ip_address[0], ip_address[1])  # 创建虚拟IP
    32.             bag.Bag.save_json(ip_ls[index:], r'../代理ip/IP地址.json')
    33.             print(f'[{index}] 更换ip成功')
    34.             return
    35.         index += 1
    36.  
    37.  
    38. url, session = chooses.choose_city()
    39.  
    40.  
    41. def get_types():    # 正常传参
    42.     check()
    43.     pattern = re.compile(r'<a.*?href="(.*?)".*?<span>(.*?)</span></a>', re.S)
    44.     if bool(url):
    45.         resp = session.get(url, proxies=proxies)
    46.         html = BeautifulSoup(resp.text, 'lxml')
    47.         soup = html.findAll('div', id='classfy')
    48.         links = re.findall(pattern, str(soup))
    49.         return links
    50.     else:
    51.         check()
    52.         get_types()
    53.  
    54.  
    55. def get_shop():
    56.     links = get_types()
    57.     pattern = re.compile(r'<div class="tit">.*?<a.*?data-shopid="(.*?)".*?href="(.*?)".*?title="(.*?)"'
    58.                          r'(?:.*?<div class="star_icon">.*?<span class="(.*?)"></span>.*?<b>(.*?)</b>)?'
    59.                          r'(?:.*?<b>(.*?)</b>)?'
    60.                          r'(?:.*?<div class="tag-addr">.*?<span class="tag">(.*?)</span>.*?<em class="sep">.*?<span class="tag">(.*?)</span>)?',
    61.                          re.S)
    62.     number = re.compile(r'data-ga-page="(.*?)"', re.S)
    63.  
    64.     result = []
    65.  
    66.     if not bool(links):
    67.         print('获取异常')
    68.         return
    69.  
    70.     for link in links:   # 获取第一页
    71.         try:
    72.             resp = session.get(link[0], proxies=proxies)
    73.             page = [int(i) for i in re.findall(number, resp.text)]
    74.             page_num = sorted(page, reverse=True)[0]
    75.             html = BeautifulSoup(resp.text, 'lxml')
    76.  
    77.             soup = html.findAll('li', class_='')
    78.             for i in soup:
    79.                 for j in re.findall(pattern, str(i)):
    80.                     result.append(j)
    81.             if page_num >= 2:   # 获取第一页往后
    82.                 for count in tqdm(range(page_num)[1:]):
    83.                     try:
    84.                         resp1 = session.get(link[0]+'p{}'.format(count+1), proxies=proxies)
    85.                         html1 = BeautifulSoup(resp1.text, 'lxml')
    86.                         soup1 = html1.findAll('li', class_='')
    87.                         for k in soup1:
    88.                             info = pattern.search(str(k))
    89.                             if info:
    90.                                 groups = list(info.groups())
    91.                                 for i in range(len(groups)):
    92.                                     if not groups[i]:
    93.                                         groups[i] = 'null'
    94.                                 result.append(tuple(groups))
    95.                     except requests.exceptions.RequestException as e:
    96.                         print(e)
    97.                         check()
    98.                     except Exception as e:
    99.                         print(e)
    100.                         continue
    101.             else:
    102.                 pass
    103.         except requests.exceptions.RequestException as e:
    104.             print(e)
    105.             check()
    106.         except Exception as e:
    107.             print(e)
    108.             check()
    109.     return result
    110.  
    111.  
    112. end = get_shop()
    113. bag.Bag.save_excel(end, './商店.xlsx')

五、成品展示

六、总结

  1. 使用selenium结合requests对网页数据进行采集可以避免很多弯弯绕绕的破解
  2. 大众点评反爬机制比较完善，爬取的时候为了防止ip被拉黑建议使用代理ip，代理ip使用方法可自行百度
声明：本文内容由网友自发贡献，转载请注明出处：【wpsshop博客】
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号