devbox
代码创新者
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

python 登录新浪微博爬取粉丝信息

作者：代码创新者 | 2024-02-03 14:23:22

赞

踩

python 登录新浪微博爬取粉丝信息

最近有个小需求，爬取新浪微博的粉丝信息，弄了好几天，终于搞定，送上代码：

环境：

系统：windows 7

版本：python 3.3

IDE: PyCharm 4.0.4

参考：http://blog.csdn.net/crystal_zero/article/details/51154632

  1. #!/usr/bin/env python3
  2. # -*- coding: utf-8 -*-
  3. import time
  4. import base64
  5. import rsa
  6. import binascii
  7. import requests
  8. import re
  9. import random
  10. try:
  11.     import cookielib
  12. except:
  13.     import http.cookiejar as cookielib
  14.  
  15. try:
  16.     from PIL import Image
  17. except:
  18.     pass
  19. try:
  20.     from urllib.parse import quote_plus
  21. except:
  22.     from urllib import quote_plus
  23.  
  24. '''
  25. 如果没有开启登录保护，不用输入验证码就可以登录
  26. 如果开启登录保护，需要输入验证码
  28. '''
  29.  
  30.  
  31. # 构造 Request headers
  32. agent = 'Mozilla/5.0 (Windows NT 6.3; WOW64; rv:41.0) Gecko/20100101 Firefox/41.0'
  33. headers = {
  34.     # "Host": "www.weibo.com",
  35.     'User-Agent': agent
  36. }
  37.  
  38. session = requests.session()
  39. session.cookies = cookielib.LWPCookieJar(filename='cookies')
  40. try:
  41.     session.cookies.load(ignore_discard=True)
  42. except:
  43.     print("Cookie 未能加载")
  44.  
  45.  
  46. # 访问 初始页面带上 cookie
  47. index_url = "http://weibo.com/login.php"
  48. try:
  49.     session.get(index_url, headers=headers, timeout=2)
  50. except:
  51.     session.get(index_url, headers=headers)
  52. try:
  53.     input = raw_input
  54. except:
  55.     pass
  56.  
  57.  
  58. def get_su(username):
  59.     """
  60.     对 email 地址和手机号码 先 javascript 中 encodeURIComponent
  61.     对应 Python 3 中的是 urllib.parse.quote_plus
  62.     然后在 base64 加密后decode
  63.     """
  64.     username_quote = quote_plus(username)
  65.     username_base64 = base64.b64encode(username_quote.encode("utf-8"))
  66.     return username_base64.decode("utf-8")
  67.  
  68.  
  69. # 预登陆获得 servertime, nonce, pubkey, rsakv
  70. def get_server_data(su):
  71.     pre_url = "http://login.sina.com.cn/sso/prelogin.php?entry=weibo&callback=sinaSSOController.preloginCallBack&su="
  72.     pre_url = pre_url + su + "&rsakt=mod&checkpin=1&client=ssologin.js(v1.4.18)&_="
  73.     pre_url = pre_url + str(int(time.time() * 1000))
  74.     pre_data_res = session.get(pre_url, headers=headers)
  75.  
  76.     sever_data = eval(pre_data_res.content.decode("utf-8").replace("sinaSSOController.preloginCallBack", ''))
  77.  
  78.     return sever_data
  79.  
  80.  
  81. # print(sever_data)
  82.  
  83.  
  84. def get_password(password, servertime, nonce, pubkey):
  85.     rsaPublickey = int(pubkey, 16)
  86.     key = rsa.PublicKey(rsaPublickey, 65537)  # 创建公钥
  87.     message = str(servertime) + '\t' + str(nonce) + '\n' + str(password)  # 拼接明文js加密文件中得到
  88.     message = message.encode("utf-8")
  89.     passwd = rsa.encrypt(message, key)  # 加密
  90.     passwd = binascii.b2a_hex(passwd)  # 将加密信息转换为16进制。
  91.     return passwd
  92.  
  93.  
  94. def get_cha(pcid):
  95.     cha_url = "http://login.sina.com.cn/cgi/pin.php?r="
  96.     cha_url = cha_url + str(int(random.random() * 100000000)) + "&s=0&p="
  97.     cha_url = cha_url + pcid
  98.     cha_page = session.get(cha_url, headers=headers)
  99.     with open("cha.jpg", 'wb') as f:
  100.         f.write(cha_page.content)
  101.         f.close()
  102.     try:
  103.         im = Image.open("cha.jpg")
  104.         im.show()
  105.         im.close()
  106.     except:
  107.         print(u"请到当前目录下，找到验证码后输入")
  108.  
  109.  
  110. def login(username, password):
  111.     # su 是加密后的用户名
  112.     su = get_su(username)
  113.     sever_data = get_server_data(su)
  114.     servertime = sever_data["servertime"]
  115.     nonce = sever_data['nonce']
  116.     rsakv = sever_data["rsakv"]
  117.     pubkey = sever_data["pubkey"]
  118.     showpin = sever_data["showpin"]
  119.     password_secret = get_password(password, servertime, nonce, pubkey)
  120.  
  121.     postdata = {
  122.         'entry': 'weibo',
  123.         'gateway': '1',
  124.         'from': '',
  125.         'savestate': '7',
  126.         'useticket': '1',
  127.         'pagerefer': "http://login.sina.com.cn/sso/logout.php?entry=miniblog&r=http%3A%2F%2Fweibo.com%2Flogout.php%3Fbackurl",
  128.         'vsnf': '1',
  129.         'su': su,
  130.         'service': 'miniblog',
  131.         'servertime': servertime,
  132.         'nonce': nonce,
  133.         'pwencode': 'rsa2',
  134.         'rsakv': rsakv,
  135.         'sp': password_secret,
  136.         'sr': '1366*768',
  137.         'encoding': 'UTF-8',
  138.         'prelt': '115',
  139.         'url': 'http://weibo.com/ajaxlogin.php?framelogin=1&callback=parent.sinaSSOController.feedBackUrlCallBack',
  140.         'returntype': 'META'
  141.         }
  142.     login_url = 'http://login.sina.com.cn/sso/login.php?client=ssologin.js(v1.4.18)'
  143.     if showpin == 0:
  144.         login_page = session.post(login_url, data=postdata, headers=headers)
  145.     else:
  146.         pcid = sever_data["pcid"]
  147.         get_cha(pcid)
  148.         postdata['door'] = input(u"请输入验证码")
  149.         login_page = session.post(login_url, data=postdata, headers=headers)
  150.     login_loop = (login_page.content.decode("GBK"))
  151.     # print(login_loop)
  152.     pa = r'location\.replace\([\'"](.*?)[\'"]\)'
  153.     loop_url = re.findall(pa, login_loop)[0]
  154.     # print(loop_url)
  155.     # 此出还可以加上一个是否登录成功的判断，下次改进的时候写上
  156.     login_index = session.get(loop_url, headers=headers)
  157.     uuid = login_index.text
  158.     uuid_pa = r'"uniqueid":"(.*?)"'
  159.     uuid_res = re.findall(uuid_pa, uuid, re.S)[0]
  160.     web_weibo_url = "http://weibo.com/%s/profile?topnav=1&wvr=6&is_all=1" % uuid_res
  161.     weibo_page = session.get(web_weibo_url, headers=headers)
  162.     weibo_pa = r'<title>(.*?)</title>'
  163.     # print(weibo_page.content.decode("utf-8"))
  164.     contents = re.findall(weibo_pa, weibo_page.content.decode("utf-8", 'ignore'), re.S)
  165.     if len(contents):
  166.         userID = contents[0]
  167.         print(u"欢迎你 %s, 你在正在使用 xchaoinfo 写的模拟登录微博" % userID)
  168.         session.cookies.save()
  169.         return True
  170.     else:
  171.         return False
  172.  
  173.  
  174. def getWeiboPageContent(url):
  175.     import requests
  176.     from selenium import webdriver
  177.     import time
  178.     # webdriver.DesiredCapabilities.PHANTOMJS['phantomjs.page.customHeaders.Cookie'] = 'SINAGLOBAL=3955422793326.2764.1451802953297; wb_publish_vip_1888964862=2; YF-Page-G0=9a31b867b34a0b4839fa27a4ab6ec79f; _s_tentry=123.sogou.com; Apache=3233757158741.355.1460597853009; ULV=1460597853022:23:7:4:3233757158741.355.1460597853009:1460533608651; YF-V5-G0=24e0459613d3bbdec61239bc81c89e13; YF-Ugrow-G0=3a02f95fa8b3c9dc73c74bc9f2ca4fc6; login_sid_t=9c9ff740ffffbdf23415b871aa319ec0; TC-Ugrow-G0=e66b2e50a7e7f417f6cc12eec600f517; TC-V5-G0=7e5b74ea4beaaa98b5f592db11c2eeb9; myuid=5898063885; TC-Page-G0=1e758cd0025b6b0d876f76c087f85f2c; wvr=6; SSOLoginState=1460604791; un=1654916845@qq.com; user_active=201604141220; user_unver=b37741d33507619aa07b6d512be0dd67; SUS=SID-5898063885-1460616073-XD-a0vlz-c7f218016d290c52b4297371d3942ce7; SUE=es%3D5ec8e6a39470b9d75e4cf930977a7449%26ev%3Dv1%26es2%3D2b500477fcc5dc1560b4aee4d5ca9be5%26rs0%3Daeb6LPcKR9mfwMB31yJquO6NSXgRd9BpSLAcKEdCQjtUOM7pIzDblRVIOOItI22Uc6pLjTxUc6k7s0zIgIxar9JlVdPjOiHEGijE2ucUP6GjFv%252BJWntWR7szF3qcWknPBJzemeHiThBGRF0bAcNFHcH%252BY9VfrTaIVjXCUsO93B8%253D%26rv%3D0; SUP=cv%3D1%26bt%3D1460616073%26et%3D1460702473%26d%3Dc909%26i%3D2ce7%26us%3D1%26vf%3D0%26vt%3D0%26ac%3D27%26st%3D0%26uid%3D5898063885%26name%3D1654916845%2540qq.com%26nick%3Dforfun2016%26fmp%3D%26lcp%3D; SUB=_2A256C0vZDeTxGeNG4loR9i3EwzmIHXVZYToRrDV8PUJbvNAPLXX2kW9LHet_3CJqcXTqa57ddP1X0MxCpd6bSA..; SUBP=0033WrSXqPxfM725Ws9jqgMF55529P9D9WhFdfTB9.lOPTcSpGQskYri5JpX5K-t; SUHB=0z1QUpglvHBm_i; ALF=1492143611; UOR=far.tooold.cn,widget.weibo.com,login.sina.com.cn; WBtopGlobal_register_version=ab9111fb56d70a2b'
  179.  
  180.     #这个地方可以设置 header, 代理等一些参数
  181.     cap = webdriver.DesiredCapabilities.PHANTOMJS
  182.     cap["phantomjs.page.settings.resourceTimeout"] = 1000
  183.     cap["phantomjs.page.settings.loadImages"] = False
  184.     cap["phantomjs.page.settings.disk-cache"] = True
  185.     cap["phantomjs.page.customHeaders.Cookie"] = 'SINAGLOBAL=1090764576516.7528.1472895523482;' \
  186.                                                  'ULV=1473054803412:3:3:2:8569250075753.363.1473054803408:1472968709618;' \
  187.                                                  'SCF=AsEbWSFBj4yI1a200LCVvAmkggqyRCHJ705_J2slnQgOZiIr6H6PjN2HTWiP8y_wK4rtnl9XPpwDIkzi3Do0iHg.;' \
  188.                                                  'SUHB=0wn1u-A905WJz_;' \
  189.                                                  'un=18704027874;' \
  190.                                                  'UOR=,,www.baidu.com;' \
  191.                                                  'wvr=6;' \
  192.                                                  'SUBP=0033WrSXqPxfM725Ws9jqgMF55529P9D9WFwo9sl7cB8PE.FyBvPnqYc5JpX5K2hUgL.Foq7eKe0S0.7SKz2dJLoIEBLxK-LBo2LBK5LxK.L1KzL1-qLxK-LB.eL1heLxK-LBo2LBK5t;' \
  193.                                                  'YF-Ugrow-G0=1eba44dbebf62c27ae66e16d40e02964;' \
  194.                                                  'YF-V5-G0=5f9bd778c31f9e6f413e97a1d464047a;' \
  195.                                                  'YF-Page-G0=8ec35b246bb5b68c13549804abd380dc;' \
  196.                                                  'WBtopGlobal_register_version=370d73e0c771001f;' \
  197.                                                  'login_sid_t=1a1f99aaff13878558a61d66c6b9c0a4;' \
  198.                                                  '_s_tentry=www.baidu.com;' \
  199.                                                  'Apache=8569250075753.363.1473054803408;' \
  200.                                                  'SSOLoginState=1473054827;' \
  201.                                                  'SUB=_2A256yvvYDeTxGeBO6lES9yfMzj6IHXVZvmoQrDV8PUJbmtBeLWyjkW8OibLNYiqudnUDwBp6GgtGcUnFuQ..' #我删掉了一大部分
  202.  
  203.     driver = webdriver.PhantomJS(executable_path="D:\\phantomjs-2.1.1-windows\\bin\\phantomjs.exe", desired_capabilities=cap)
  204.     driver.set_window_size(1120, 1100)
  205.     # print(url)
  206.     driver.get(url)
  207.  
  208.     page_source = driver.page_source
  209.     # print(t)
  210.     #
  211.     # contents = driver.find_elements_by_tag_name("<a>")
  212.     #
  213.     # for content in contents:
  214.     #     print(content.text)
  215.     return page_source
  216.  
  217.  
  218. from lib3 import spider
  219. from lib3 import fileWriter
  220. import os
  221. import sys
  222.  
  223. class weiboSpider:
  224.     def __init__(self,userAgent,userName,password):
  225.         self.sp = spider.Spider(userAgent)
  226.         self.loginStatus = login(userName, password)
  227.     def go(self,url):
  228.         status = True
  229.         pageIndex = 1
  230.         while status == True:
  231.             fix = str(pageIndex)+"#Pl_Official_HisRelation__63"
  232.             curUrl = url+fix
  233.             print(curUrl)
  234.             status = self.parse(curUrl)
  235.             pageIndex += 1
  236.     def cur_file_dir(self):
  237.          #获取脚本路径
  238.          path = sys.path[0]
  239.          #判断为脚本文件还是py2exe编译后的文件，如果是脚本文件，则返回的是脚本的目录，如果是py2exe编译后的文件，则返回的是编译后的文件路径
  240.          if os.path.isdir(path):
  241.              return path
  242.          elif os.path.isfile(path):
  243.              return os.path.dirname(path)
  244.  
  245.     def saveData(self,fansUrl,fansId):
  246.         fHandle = fileWriter.FileWriter(self.cur_file_dir(),'zhiHuData.txt','a')
  247.         fHandle.write(fansUrl+'\t'+fansId+'\r\n')
  248.         fHandle.close()
  249.  
  250.     def parse(self,url):
  251.         if self.loginStatus == True:
  252.             content = getWeiboPageContent(url)
  253.             strPattern = '''<a class="S_txt1" target="_blank" usercard=".*?" href="(.*?)">(.*?)</a>'''
  254.             resultsList = self.sp.parseReg(content,strPattern,2)
  255.             if len(resultsList):
  256.                 for i in range(0,len(resultsList)):
  257.                     fansUrl = "http://wei.com"+resultsList[i][0]
  258.                     fansId = resultsList[i][1]
  259.                     self.saveData(fansUrl,fansId)
  260.                     print(fansUrl,fansId)
  261.                 return True
  262.             else:
  263.                 print("no data")
  264.                 return False
  265.  
  266.         else:
  267.             print("sorry,can not login weibo")
  268.  
  269. if __name__ == "__main__":
  270.     demo = weiboSpider('Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko','在此填写用户名','密码')
  271.  
  272.     demo.go('http://weibo.com/p/1005053810677006/follow?relate=fans&page=')


上面引用的lib3是我自己写的一个http接口


fileWriter.py

  1. __author__ = 'zhengjinwei'
  2. #coding=utf-8
  3. import os
  4.  
  5.  
  6.  
  7. class FileWriter:
  8.     def __init__(self,fileDir,fileName,format):
  9.         self.mkDir(fileDir)
  10.         self.f = open(fileDir+u"/"+fileName,format)
  11.  
  12.     def mkDir(self,path):
  13.         isExists = os.path.exists(path)
  14.         if not isExists:
  15.             os.makedirs(path)
  16.     def write(self,contents):
  17.         return self.f.write(contents)
  18.  
  19.     def close(self):
  20.         self.f.close()


spider.py 

  1. #coding:utf-8
  2. __author__ = 'zhengjinwei'
  3.  
  4. import  re
  5. import requests
  6.  
  7. try:
  8.     import cookielib
  9. except:
  10.     import http.cookiejar as cookielib
  11.  
  12.  
  13. class Spider:
  14.     def __init__(self,userAgent):
  15.  
  16.         self.user_agent = userAgent
  17.         self.headers = {
  18.             'User-Agent' : self.user_agent
  19.         }
  20.     def getHttp(self,url,param=None):
  21.         return requests.get(url,param,headers=self.headers)
  22.  
  23.     def postHttp(self,url,postData=None):
  24.         return requests.post(url, data=postData,headers=self.headers)
  25.  
  26.     def sessionPostHttp(self,url,param=None):
  27.         session = requests.session()
  28.         session.cookies = cookielib.LWPCookieJar(filename='cookies')
  29.         try:
  30.             session.cookies.load(ignore_discard=True)
  31.             print(url,param)
  32.             return session.post(url,data=param,json=None,headers=self.headers)
  33.         except:
  34.             print("Cookie 未能加载")
  35.             return None
  36.  
  37.     def sessionGetHttp(self,url,param=None):
  38.         session = requests.session()
  39.         session.cookies = cookielib.LWPCookieJar(filename='cookies')
  40.         try:
  41.             session.cookies.load(ignore_discard=True)
  42.             return session.get(url, headers=self.headers, allow_redirects=False)
  43.         except:
  44.             print("Cookie 未能加载")
  45.             return None
  46.     def parseReg(self,content,strPattern,count):
  47.         pattern = re.compile(strPattern,re.S)
  48.         items = re.findall(pattern,content)
  49.  
  50.         contents=[]
  51.  
  52.         for item in items:
  53.             temArr = []
  54.             for i in range(0,count):
  55.                 temArr.append(item[i])
  56.             contents.append(temArr)
  57.  
  58.         return contents
  59.  
  60.     def getContents(self,url,strPattern,count,method="get",param=None):
  61.         page = ""
  62.         if method == "get":
  63.             page = self.getHttp(url,param).text
  64.         else:
  65.             page = self.postHttp(url,param).text
  66.  
  67.         pattern = re.compile(strPattern,re.S)
  68.         items = re.findall(pattern,page)
  69.         contents=[]
  70.  
  71.         for item in items:
  72.             temArr = []
  73.             for i in range(0,count):
  74.                 temArr.append(item[i])
  75.             contents.append(temArr)
  76.  
  77.         return contents
  78.  
  79.  
  80.  
  81. # demo = Spider("Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko")
  82. #
  83. # t = demo.getHttp("https://www.baidu.com/index.php?tn=02049043_23_pg")
  84.  
  85.  
  86.  
  87.

效果图：



注意事项：

1，新浪微博本身有查看限制，只能查看到前5页的数据

2，爬虫工具可以使用火狐浏览器，装上fireDebug 插件用于拦截消息，获取cookie和其他参数


声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/article/detail/56786?site
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号