我正在尝试为黄页中的每个企业抓取 href。我对使用scrapy和第二天很陌生。我正在使用请求来获取实际 url 以使用蜘蛛进行搜索。我的代码做错了什么?我希望最终让scrapy 去每个企业并抓取它的地址和其他信息。
# -*- coding: utf-8 -*-
import scrapy
import requests
search = "Plumbers"
location = "Hammond, LA"
url = "https://www.yellowpages.com/search"
q = {'search_terms': search, 'geo_location_terms': location}
page = requests.get(url, params=q)
page = page.url
class YellowpagesSpider(scrapy.Spider):
name = 'quotes'
allowed_domains = ['yellowpages.com']
start_urls = [page]
def parse(self, response):
self.log("I just visited: " + response.url)
items = response.css('span.text::text')
for items in items:
print(items)
相关分类