Selenium+IP爬虫刷新网页_爬虫刷新页面

一、环境配置及测试

1、Pycharm+python安装
2、安装selenium库，selenium是web自动化测试工具
3、下载浏览器驱动程序：
如果要编写自动化测试程序，需要下载与对应浏览器版本匹配的驱动，如Edge浏览器查看：
转到 edge://settings/help 查看浏览器版本：

Edge Driver下载

Chrome浏览器版本查看：浏览器设置->关于Chrome

Chrome驱动
将下载好的驱动程序放入python的安装目录：

4、驱动测试：
以Chrome浏览器为例：

# _*_ coding:utf-8 _*_
import random
from time import sleep
from selenium import webdriver
from selenium.webdriver.chrome.service import Service

url = "xxx"
driver_path = r'C:\Users\XXX\AppData\Local\Programs\Python\Python310\chromedriver.exe'

s = Service(driver_path)
#one_driver = webdriver.Edge(service=s)
one_driver = webdriver.Chrome(service=s)
one_driver.get(url)  # 获取网址

for i in range(10000):  # 刷新次数
    one_driver.refresh()  # 刷新网页
    print("刷新次数：" + str(i))
    sleep(10)  # 10秒一次
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

二、给予代理IP刷新网页

代码实现：

from selenium.webdriver.chrome.service import Service
from seleniumwire import webdriver
import time
import random
from fake_useragent import UserAgent

# 改成自己的URL
url = 'https://blog.csdn.net/xxx/article/details/120707894'
# Driver Path
driver_path = r'C:\\Users\\XXX\\AppData\\Local\\Programs\\Python\\Python310\\chromedriver.exe'
s = Service(driver_path)

# 使用自己可用的代理IP
proxy_arr = [
    '104.16.132.229:443',
    '39.175.75.5:30001',
    '101.43.9.150:80',
    '106.52.57.217:8080',
    '221.122.91.65:80',
    '182.61.201.201:80'
]

cnt = 0
for proxy in proxy_arr:
    cnt += 1
    proxies = '--proxy-server=' + proxy
    print("访问次数：%s" % cnt)
    print(proxies)
    options = webdriver.ChromeOptions()
    options.add_argument(proxies)  # 添加代理

    uar = UserAgent().random
    options.add_argument('--user-agent="'+uar+'"')  # 添加UserAgent
    # 创建Chrome浏览器实例的时候传入前面的options
    driver = webdriver.Chrome(service=s, chrome_options=options)
    driver.get(url)
    print(driver.title)
    time.sleep(10+random.randint(1, 10))
    driver.close()
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

运行即可。