赞
踩
要提取google搜索的部分数据,发现google对于软件抓取它的数据屏蔽的厉害,以前伪造下 USER-AGENT 就可以抓数据,但是现在却不行了。利用抓包数据发现,Google 判断了 cookies,当你没有cookies的时候,直接返回 302 跳转,而且是连续几十个302跳转,根本抓不了数据。
因此,在发送搜索命令时,需要先提取 cookies 并保存,然后利用保存下来的这个cookies再次发送搜索命令即可正常抓数据了。这其实和论坛的模拟登录一个道理,先POST登录,获取cookies并保存,然后利用这个cookies访问就可以了。
1.
$cookie_jar
= dirname(
__FILE__
).
"/pic.cookie"
;
1.
$url
=
"http://1.2.3.4/"
;
2.
$ch
= curl_init();
3.
curl_setopt(
$ch
, CURLOPT_URL,
$url
);
4.
curl_setopt(
$ch
, CURLOPT_HEADER, 0);
5.
curl_setopt(
$ch
, CURLOPT_RETURNTRANSFER, true);
6.
curl_setopt(
$ch
, CURLOPT_COOKIEJAR,
$cookie_jar
);
7.
$content
= curl_exec(
$ch
);
8.
curl_close(
$ch
);
1.
$ch
= curl_init();
2.
curl_setopt(
$ch
, CURLOPT_URL,
'http://1.2.3.4/getCheckpic.action?rand=6836.185874812305'
);
3.
curl_setopt(
$ch
, CURLOPT_COOKIEFILE,
$cookie_jar
);
4.
curl_setopt(
$ch
, CURLOPT_HEADER, 0);
5.
curl_setopt(
$ch
, CURLOPT_RETURNTRANSFER, 1);
6.
$ret
= curl_exec(
$ch
);
7.
curl_close(
$ch
);
01.
$post
=
"name=2&userType=1&passwd=asdf&loginType=1&rand=6836&imageField.x=25&imageField.y=7"
;
02.
$ch
= curl_init();
03.
curl_setopt(
$ch
, CURLOPT_URL,
"http://1.2.3.4/loginstudent.action"
);
04.
curl_setopt(
$ch
, CURLOPT_HEADER, false);
05.
curl_setopt(
$ch
, CURLOPT_RETURNTRANSFER,1);
06.
curl_setopt(
$ch
, CURLOPT_POSTFIELDS,
$post
);
07.
curl_setopt(
$ch
, CURLOPT_COOKIEFILE,
$cookie_jar
);
08.
$result
=curl_exec(
$ch
);
09.
curl_close(
$ch
);
01.
$ch
= curl_init();
02.
curl_setopt(
$ch
, CURLOPT_URL,
"http://1.2.3.4/accountcardUser.action"
);
03.
curl_setopt(
$ch
, CURLOPT_HEADER, false);
04.
curl_setopt(
$ch
, CURLOPT_HEADER, 0);
05.
curl_setopt(
$ch
, CURLOPT_RETURNTRANSFER,0);
06.
curl_setopt(
$ch
, CURLOPT_COOKIEFILE,
$cookie_jar
);
07.
$html
=curl_exec(
$ch
);
08.
// var_dump($html);
09.
curl_close(
$ch
);
Copyright © 2003-2013 www.wpsshop.cn 版权所有,并保留所有权利。