blind网站爬虫

1 介绍

文本对https://www.teamblind.com/ 网站进行爬虫

对特殊的领域进行爬虫，用户可以先选择领域，然后进行爬虫，例如，文本是对https://www.teamblind.com/topics/General-Topics/Health-Wellness进行爬虫

2 主要代码

获取帖子内容

def get_comment(title_url, headers_list):
    headers = random.choice(headers_list)
    title_content = get_page(title_url, headers)
    title_soup = BeautifulSoup(title_content, 'html.parser')
    print(title_soup)
    title = title_soup.find_all(class_='word-break')[0].text.strip() # 获取标题
    contents = title_soup.find_all(id='contentArea')[0].text.strip() # 获取帖子的内容
    reviews_soup = title_soup.find_all(class_='comment_area') # 获取评论
    reviews = []

    for review_soup in reviews_soup:
        review = review_soup.find_all(class_='detail')[0].text.strip()
        if len(review) != 0:
            reviews.append(review)

    result = {}
    result["title"] = title
    result["contents"] = contents
    result["reviews"] = reviews
    return result
    定制化爬虫开发，联系Q 596520206
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

3 结果展示

保存为json格式，包含字段：title、 contents、reviews等