Python爬虫案例Demo——小说的爬取_python爬小说网站demo

这个寒假闲来没事开始爬虫，学了几天了，期间自己爬了一些网站，现在大致整理一下：

import requests
import re
response = requests.get("http://www.shuquge.com/txt/96636/index.html")
# 万能设置编码
response.encoding = response.apparent_encoding
html = response.text

#正则表达式获取所有的章节目录和url网址
result = re.findall('<dd><a href="(.*?)">(.*?)</a></dd>',html)
print(result)

for url,name in result:
    new_url = "http://www.shuquge.com/txt/96636/" + url
    response1 = requests.get(new_url)
    response1.encoding = response1.apparent_encoding
    html = response1.text
    result1 = re.findall('<div id="content" class="showtxt">(.*?)</div>',html,re.S)
    # re.S是自动换行
    with open(str(name) + ".txt", mode="w", encoding="utf-8") as f:
        f.write(str(result1[0]).replace("&nbsp;&nbsp;&nbsp;&nbsp;","").replace("<br/>",""))

"""
print("http://www.shuquge.com/txt/96636/" + url)
print(name)
response1 = requests.get("http://www.shuquge.com/txt/96636/28767693.html")
response1.encoding = response1.apparent_encoding
html = response1.text
with open("1.html",mode="w",encoding="utf-8") as f:
    f.write(html)
"""

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

爬取的是小说《抢救大明朝》，Pycharm截止到我现在运行这些代码还是可以的，大家可以看截图：

爬取最后的小说界面是：

这是爬虫第一天敲的案例！