爬出和反爬出是矛与盾的关系,
进化史
1,java原生自带url类:url.getContent();
2,httpclient,
3,jsoup,htmlclean.
4,htmlunit,
5,se.
1-2只是原生http链接,
3,做了解析层面的支持,比如页面html清理,xpath支持;
4添加了js支持,这是很大的改进,不过由于各种衍生js库,支持不是很好.
5,se,这就是走了捷径.
好的反爬出机制,能识别在电脑另一头的是虫子还是人
,eg.阿里系网站机制 anti spider.:供应商,搜索食品,
http://s.1688.com/company/company_search.htm?keywords=%CA%B3%C6%B7&sug=2_0&n=y&spm=a260k.635.1998096057.d1
短时间内连续翻页超过n页,n属于(8,20),自动让你登陆.
登陆后,继续翻超过n页,触发阿里反爬虫机制:http://sec.1688.com/query.htm?smApp=searchweb2&smPolicy=searchweb2-company-anti_Spider-html-checkcode&smCharset=GBK&smTag=MTIxLjIwNC4xOTUuMTkzLDE5MzIzNDI3NSxjNjEzZDJhOTQyMTM0ZjJhOTg4OTM5OWMyMTdlY2MyZg%3D%3D&smReturn=http%3A%2F%2Fs.1688.com%2Fcompany%2Fcompany_search.htm%3Fkeywords%3D%25CA%25B3%25C6%25B7%26button_click%3Dtop%26earseDirect%3Dfalse%26n%3Dy%26sortType%3Dpop%26pageSize%3D30%26beginPage%3D15&smSign=qxdnSrcI%2B5VTtjVJumzY7w%3D%3D
其实,阿里系也是有漏洞的.
采集器的前世今生
原文：http://my.oschina.net/u/856051/blog/466966
