猿问

scrapy抓取淘宝商品详情页,读取url随机强制302,跳转到h5.taobao。

使用scrapy+redis从一定量的淘宝详情页url获取商品详情
已设置user-agent,已传入cookie,已设置proxy-ip
获取url,response.status有时是200,有时是302,随机改变
1000个url,成功获取商品信息大概有400多
是否为cookie未传入成功,还是proxy-ip不稳定?或者其他原因。请帮忙分析,谢谢!
报错Traceback:
2017-07-1415:51:12[scrapy.core.engine]DEBUG:Crawled(200)(referer:None)
2017-07-1415:51:12[requests.packages.urllib3.connectionpool]INFO:StartingnewHTTPSconnection(1):rate.taobao.com
2017-07-1415:51:12[requests.packages.urllib3.connectionpool]DEBUG:"GET/detailCommon.htm?auctionNumId=10245430841HTTP/1.1"200None
2017-07-1415:51:12[scrapy.core.scraper]DEBUG:Scrapedfrom<200https://item.taobao.com/item.htm?id=10245430841&ns=1&abbucket=0>
None
2017-07-1415:51:12[taobao]DEBUG:Read1requestsfrom'taobao:start_urls'
2017-07-1415:51:12[scrapy.downloadermiddlewares.cookies]DEBUG:Sendingcookiesto:
2017-07-1415:51:12[scrapy.downloadermiddlewares.redirect]DEBUG:Redirecting(302)tofromem.htm?id=10245681616&ns=1&abbucket=0#detail>
2017-07-1415:51:12[scrapy.downloadermiddlewares.cookies]DEBUG:Sendingcookiesto:
2017-07-1415:51:12[scrapy.core.engine]DEBUG:Crawled(200)(referer:None)['partial']
2017-07-1415:51:12[scrapy.core.scraper]ERROR:Spidererrorprocessing(referer:None)
千万里不及你
浏览 1739回答 2
2回答
随时随地看视频慕课网APP

相关分类

JavaScript
我要回答