【Python爬虫+可视化案例】采集电商网站商品数据信息，并可视化分析_爬取购物数据信息进行数据分析

爬虫+可视化案例 ：苏宁易购

1. 案例所需要掌握的知识点：

• selenium的使用
• html标签数据解析方法

2. 需要准备的环境：

• python 3.8
• pycharm 2022专业版
• selenium python里面的第三方库 可以用来操作浏览器

爬虫代码展示

所需模块
【代码领取 请看文末名片】

import time
from selenium import webdriver  # 第三方库 操作浏览器驱动的 浏览器驱动用来操作浏览器的
from selenium.webdriver.common.by import By
import csv
1
2
3
4

新建文件

f = open('苏宁易购.csv', mode='a', encoding='utf-8', newline='')
csv_writer = csv.writer(f)
csv_writer.writerow(['title', 'price', 'comment', 'store_stock', 'href'])
1
2
3

1. 打开浏览器 是没问题的

driver = webdriver.Chrome()
1

2. 打开网站

driver.get("https://****/iPhone14/")
for i in range(15):
1
2

3. 将滚动条 拉到最下方

    # 通过js代码去操作 页面
    # deocument.documentElement.scrollHeight: 获取当前整个页面的高度
    # document.documentElement.scrollTop: 当前滚动条的位置
    # document.documentElement.scrollTop = document.documentElement.scrollHeight: 将当前滚动条的位置设置为 整个页面的高度
    for page in range(0, 14500, 2900):
        driver.execute_script('document.documentElement.scrollTop = ' + str(page))
        time.sleep(1)
1
2
3
4
5
6
7

4. 取数据

.product-box: 匹配到所有的商品标签

    goods = driver.find_elements(By.CSS_SELECTOR, ".product-box")
    ""代码获取：文末名片""
    for good in goods:
        price = good.find_element(By.CSS_SELECTOR, ".price-box").text
        title = good.find_element(By.CSS_SELECTOR, ".title-selling-point").text
        href = good.find_element(By.CSS_SELECTOR, ".title-selling-point a").get_attribute("href")
        comment = good.find_element(By.CSS_SELECTOR, ".evaluate-old.clearfix").text
        store_stock = good.find_element(By.CSS_SELECTOR, ".store-stock").text
        print(title, price, comment, store_stock)
        csv_writer.writerow([title, price, comment, store_stock, href])
    driver.find_element(By.CSS_SELECTOR, "#nextPage").click()
# 阻塞 不让程序结束 因为程序结束 浏览器就自动关闭了
# 退出浏览器
driver.quit()
1
2
3
4
5
6
7
8
9
10
11
12
13
14

可视化效果演示

一共就是以下三个表格，外加一个词云图

我觉得如果是大学生把这个交给老师，应该也许大概可能，老师会觉得你还不错吧，哈哈哈

开个玩笑，不过现在大学生基本都写完作业，早早的放假了

还是希望这篇文章可以帮助到大家吧，emm不过最近没怎么更新，已经没什么人看文章了哈哈

可视化代码

{
   源码领取，请看文末名片
 "cells": [
  {
   
   "cell_type": "code",
   "execution_count": 5,
   "id": "d19250a4",
   "metadata": {
   },
   "outputs": [],
   "source": [
    "import pandas as pd \n",
    "import jieba\n",
    "import time\n",
    "from pyecharts.charts import Bar,Line,Map,Page,Pie  \n",
    "from pyecharts import options as opts \n",
    "from pyecharts.globals import SymbolType"
   ]
  },
  {
   
   "cell_type": "code",
   "execution_count": 6,
   "id": "69c29f78",
   "metadata": {
   
    "scrolled": true
   },
   "outputs": [
    {
   
     "data": {
   
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>price</th>\n",
       "      <th>comment</th>\n",
       "      <th>store_stock</th>\n",
       "      <th>href</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Apple iPhone 14 128G 午夜色 移动联通电信5G手机</td>\n",
       "      <td>¥5999.00</td>\n",
       "      <td>1.3万+评价</td>\n",
       "      <td>苏宁自营</td>\n",
       "      <td>https://product.suning.com/0000000000/12391268...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Apple iPhone 14 Pro Max 256G 暗紫色 移动联通电信5G手机</td>\n",
       "      <td>¥9899.00</td>\n",
       "      <td>6300+评价</td>\n",
       "      <td>苏宁自营</td>\n",
       "      <td>https://product.suning.com/0000000000/12391268...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Apple iPhone 14 Pro Max 128G 暗紫色 移动联通电信5G手机</td>\n",
       "      <td>¥8999.00</td>\n",
       "      <td>6300+评价</td>\n",
       "      <td>苏宁自营</td>\n",
       "      <td>https://product.suning.com/0000000000/12391268...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Apple iPhone 14 Pro 256G 深空黑色 移动联通电信5G手机</td>\n",
       "      <td>¥8899.00</td>\n",
       "      <td>6400+评价</td>\n",
       "      <td>苏宁自营</td>\n",
       "      <td>https://product.suning.com/0000000000/12391268...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Apple iPhone 14 Pro 128G 暗紫色 移动联通电信5G手机</td>\n",
       "      <td>¥7999.00</td>\n",
       "      <td>6400+评价</td>\n",
       "      <td>苏宁自营</td>\n",
       "      <td>https://product.suning.com/0000000000/12391268...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1693</th>\n",
       "      <td>圣幻 iphone11手机壳苹果11pro硅胶套iphone11PROMAX全包防摔ipho...</td>\n",
       "      <td>¥46.00</td>\n",
       "      <td>200+评价</td>\n",
       "      <td>任意门数码专营店</td>\n",
       "      <td>https://product.suning.com/0070067325/11398343...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1694</th>\n",
       "      <td>圣幻 iphone11苹果11proMax手机壳薄透明苹果11全包边11Pro电镀软壳防摔1...</td>\n",
       "      <td>¥46.00</td>\n",
       "      <td>300+评价</td>\n",
       "      <td>任意门数码专营店</td>\n",
       "      <td>https://product.suning.com/0070067325/11398610...</td>\n"
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120