当前位置:   article > 正文

Python爬虫练习:爬取高德地图地铁线路及站点数据_中国大城市地铁站点数据爬虫

中国大城市地铁站点数据爬虫

前言

本文的文字及图片来源于网络,仅供学习、交流使用,不具有任何商业用途,版权归原作者所有,如有问题请及时联系我们以作处理。

作者:Darcy频道

Python爬虫练习:爬取高德地图地铁线路及站点数据

 

  1. # coding=utf-8
  2. import requests
  3. import time
  4. import json
  5. import ast
  6. import os
  7. import utils
  8. from lxml import etree
  9. PAGE_URL = 'http://map.amap.com/subway/index.html?&1100'
  10. DATA_URL = 'http://map.amap.com/service/subway?srhdata='
  11. HEADER = {
  12. "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36"}
  13. def fetchAllCity(url, header):
  14. r = requests.get(url, header)
  15. html = r.content
  16. element = etree.HTML(html)
  17. options = element.xpath("//a[contains(@class, 'city')]")
  18. cities = []
  19. for option in options:
  20. city = {
  21. 'id': option.get('id'),
  22. 'name': option.get('cityname'),
  23. 'text': option.text
  24. }
  25. cities.append(city)
  26. return cities
  27. def parseCityData(citys):
  28. lw = open('./lwkt.txt', 'w')
  29. lw.write('wkt' + '\n')
  30. pw = open('./pwkt.txt', 'w')
  31. pw.write('wkt' + '\n')
  32. for city in citys:
  33. parseCityPointFromApi(city, lw, pw)
  34. def parseCityPointFromApi(city, lw, pw):
  35. url = DATA_URL + "{}_drw_{}.json".format(city['id'], city['name'])
  36. print(url)
  37. json_str = requests.get(url).text
  38. res = json.loads(json_str)
  39. res = res['l']
  40. for r in res:
  41. st = r['st']
  42. coords = []
  43. for s in st:
  44. _coords = s.get('sl', '').split(',')
  45. coords.append(_coords)
  46. pwkt = '"POINT ({} {})"'.format(_coords[0], _coords[1])
  47. pw.write(pwkt + '\n')
  48. new_coords = ','.join(['%s %s' % (coord[0], coord[1]) for coord in coords])
  49. lwkt = '"LINESTRING(%s)"' % new_coords
  50. lw.write(lwkt + '\n')
  51. def main():
  52. cities = fetchAllCity(PAGE_URL, HEADER)
  53. print(cities)
  54. parseCityData(cities)
  55. if __name__ == '__main__':
  56. main()

PS:如有需要Python学习资料的小伙伴可以加下方的群去找免费管理员领取

Python爬虫练习:爬取高德地图地铁线路及站点数据

 

可以免费领取源码项目实战视频PDF文件

Python爬虫练习:爬取高德地图地铁线路及站点数据

声明:本文内容由网友自发贡献,不代表【wpsshop博客】立场,版权归原作者所有,本站不承担相应法律责任。如您发现有侵权的内容,请联系我们。转载请注明出处:https://www.wpsshop.cn/w/Gausst松鼠会/article/detail/728835
推荐阅读
相关标签
  

闽ICP备14008679号