爬虫学习笔记01-爬去猫眼电影TOP100榜单_天空之城寺田农鹫尾真知子龟山助清

#! python3

import requests
import csv
from bs4 import BeautifulSoup

url = 'http://maoyan.com/board/4?offset='

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36'
}

class maoyan:
    def __init__(self, url):
        self.url = url
        self.csvfile = open('data.csv', 'w', newline='')
        self.fieldnames = ['排名', '电影名', '演员', '上映时间', '评分']
        self.writer = csv.DictWriter(self.csvfile, fieldnames=self.fieldnames)
        self.writer.writeheader()
        self.data = []

    def set_url(self, num):
        result = self.url + str(num)
        return result

    def parse_one_page(self, url):
        try:
            res = requests.get(url, headers=headers)
            res.encoding = 'utf-8'
            soup = BeautifulSoup(res.text, 'lxml')
            dds = soup.select('dd')
            if not dds:
                return
            for dd in dds:
                index = dd.select(".board-index")[0].text
                title = dd.select(".name")[0].text
                actors = dd.select('.star')[0].text.strip()[3:]
                time = dd.select('.releasetime')[0].text[5:]
                score = dd.select('.score')[0].text
                info = {
                    self.fieldnames[0]: index,
                    self.fieldnames[1]: title,
                    self.fieldnames[2]: actors,
                    self.fieldnames[3]: time,
                    self.fieldnames[4]: score
                }
                self.data.append(info)
        except:
        	print('无法打开网页')

    def write_to_csv(self):
        for i in range(10):
            url = self.set_url(i * 10)
            self.parse_one_page(url)
        if self.data:
            self.writer.writerows(self.data)
            self.csvfile.close()


if __name__ == '__main__':
    m = maoyan(url)
    m.write_to_csv()
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63

文本方式打开，部分结果如下：

排名,电影名,演员,上映时间,评分
1,霸王别姬,"张国荣,张丰毅,巩俐",1993-01-01,9.6
2,肖申克的救赎,"蒂姆·罗宾斯,摩根·弗里曼,鲍勃·冈顿",1994-10-14(美国),9.5
3,罗马假日,"格利高里·派克,奥黛丽·赫本,埃迪·艾伯特",1953-09-02(美国),9.1
4,这个杀手不太冷,"让·雷诺,加里·奥德曼,娜塔莉·波特曼",1994-09-14(法国),9.5
5,教父,"马龙·白兰度,阿尔·帕西诺,詹姆斯·肯恩",1972-03-24(美国),9.3
6,泰坦尼克号,"莱昂纳多·迪卡普里奥,凯特·温丝莱特,比利·赞恩",1998-04-03,9.5
7,唐伯虎点秋香,"周星驰,巩俐,郑佩佩",1993-07-01(中国香港),9.2
8,千与千寻,"柊瑠美,入野自由,夏木真理",2001-07-20(日本),9.3
9,魂断蓝桥,"费雯·丽,罗伯特·泰勒,露塞尔·沃特森",1940-05-17(美国),9.2
10,乱世佳人,"费雯·丽,克拉克·盖博,奥利维娅·德哈维兰",1939-12-15(美国),9.1
11,天空之城,"寺田农,鹫尾真知子,龟山助清",1992,9.1
12,喜剧之王,"周星驰,莫文蔚,张柏芝",1999-02-13(中国香港),9.2
13,大闹天宫,"邱岳峰,毕克,富润生",1965-12-31,9.0
14,辛德勒的名单,"连姆·尼森,拉尔夫·费因斯,本·金斯利",1993-12-15(美国),9.2
15,音乐之声,"朱莉·安德鲁斯,克里斯托弗·普卢默,埃琳诺·帕克",1965-03-02(美国),9.0
......

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

Excel打开，部分结果如下图所示：