避免在刮擦过程中被发现

由于scrapy,我正在尝试抓取lacentrale.fr 的网站,但即使我轮换了我的用户代理和IP 地址(感谢TOR),该网站也会检测到我的机器人并向我发送错误值。请您检查我在中间件和设置中使用的代码,并告诉我是否出现问题。


中间件中的代码:


from tutorial.settings import * #USER_AGENT_LIST

import random


from stem.control import Controller

from toripchanger import TorIpChanger

from stem import Signal


class RandomUserAgentMiddleware(object):


def process_request(self, request, spider):

    ua = random.choice(USER_AGENT_LIST)

    if ua:

        request.headers.setdefault('User-Agent', ua)



def _set_new_ip():

    with Controller.from_port(port=9051) as controller:

        controller.authenticate(password='')

        controller.signal(Signal.NEWNYM)


ip_changer = TorIpChanger(reuse_threshold=10)

class ProxyMiddleware(object):

    _requests_count = 0


def process_request(self, request, spider):

    self._requests_count += 1

    if self._requests_count > 10:

        self._requests_count = 0

        ip_changer.get_new_ip()


        print("New Tor connection processed")


    request.meta['proxy'] = 'http://127.0.0.1:8118'

    spider.log('Proxy : %s' % request.meta['proxy'])

设置中使用的代码:


BOT_NAME = 'tutorial'


SPIDER_MODULES = ['tutorial.spiders']

NEWSPIDER_MODULE = 'tutorial.spiders'



ROBOTSTXT_OBEY = True



DOWNLOAD_DELAY = 1

RANDOMIZE_DOWNLOAD_DELAY = True

DOWNLOADER_MIDDLEWARES = {

    'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None,

    'tutorial.middlewares.RandomUserAgentMiddleware': 400,

    'scrapy.contrib.downloadermiddleware.httpcompression.HttpCompressionMiddleware':None, # to avoid the raise IOError, 'Not a gzipped file' exceptions.IOError: Not a gzipped file

    'scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware': 110,

    'tutorial.middlewares.ProxyMiddleware': 100

    }


繁星coding
浏览 145回答 1
1回答

摇曳的蔷薇

您可能会因多种因素被检测到,包括您的抓取工具是否下载/运行 javascript 文件。如果是这种情况,您可能需要将 selenium 之类的工具与 Python/Scrapy 结合使用,以进一步伪装成普通人类用户。我认为我无法为您的 TOR 设置可能出现的问题提供太多指导
打开App,查看更多内容
随时随地看视频慕课网APP

相关分类

Python