使用 python 请求从网页中抓取数据

Question

我正在尝试抓取域搜索页面（您可以在其中输入关键字，并获得一些随机结果）并且我在网络选项卡中找到了这个 api url https://api.leandomainsearch.com/search?query=computer&count=all（对于关键字：计算机），但出现此错误

{'error': True, 'message': 'Invalid API Credentials'}

这是代码

import requests

r = requests.get("https://api.leandomainsearch.com/search?query=cmputer&count=all")

print(r.json())

Answer 1

该站点需要您设置Authorization和RefererHTTP 标头。例如：import reimport jsonimport requestskw = 'computer'url = 'https://leandomainsearch.com/search/'api_url = 'https://api.leandomainsearch.com/search'api_key = re.search(r'"apiKey":"(.*?)"', requests.get(url, params={'q': kw}).text)[1]headers = {'Authorization': 'Key ' + api_key, 'Referer': 'https://leandomainsearch.com/search/?q={}'.format(kw)}data = requests.get(api_url, params={'query': kw, 'count': 'all'}, headers=headers).json()# uncomment this to print all data:# print(json.dumps(data, indent=4))for d in data['domains']:    print(d['name'])print()print('Total:', data['_meta']['total_records'])印刷：...blackopscomputer.comallegiancecomputer.comnorthpolecomputer.commonumentalcomputer.comfissioncomputer.comhedgehogcomputer.comblackwellcomputer.comreflectionscomputer.comtowerscomputer.comoffgridcomputer.comredefinecomputer.comquantumleapcomputer.comTotal: 1727