使用 BeautifulSoup 抓取 shopee.co.id 时得到一个空列表

Question

我试图使用 BeautifulSoup 包在https://shopee.co.id/top_products上抓取顶级产品的名称和销售额。但无论我尝试什么，它总是返回一个空的产品列表。你能解释一下幕后发生的事情吗？这是我的代码：

import requests

from bs4 import BeautifulSoup as bs

headers = {

    'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36',

    'cookie': '_gcl_au=1.1.961206468.1594951946; _med=refer; _fbp=fb.2.1594951949275.1940955365; SPC_IA=-1; SPC_F=y1evilme0ImdfEmNWEc08bul3d8toc33; REC_T_ID=fab983c8-c7d2-11ea-a977-ccbbfe23657a; SPC_SI=uv1y64sfvhx3w6dir503ixw89ve2ixt4; _gid=GA1.3.413262278.1594951963; SPC_U=286107140; SPC_EC=GwoQmu7TiknULYXKODlEi5vEgjawyqNcpIWQjoxjQEW2yJ3H/jsB1Pw9iCgGRGYFfAkT/Ej00ruDcf7DHjg4eNGWbCG+0uXcKb7bqLDcn+A2hEl1XMtj1FCCIES7k17xoVdYW1tGg0qaXnSz0/Uf3iaEIIk7Q9rqsnT+COWVg8Y=; csrftoken=5MdKKnZH5boQXpaAza1kOVLRFBjx1eij; welcomePkgShown=true; _ga=GA1.1.1693450966.1594951955; _dc_gtm_UA-61904553-8=1; REC_MD_30_2002454304=1595153616; _ga_SW6D8G0HXK=GS1.1.1595152099.14.1.1595153019.0; REC_MD_41_1000044=1595153318_0_50_0_49; SPC_R_T_ID="Am9bCo3cc3Jno2mV5RDkLJIVsbIWEDTC6ezJknXdVVRfxlQRoGDcya57fIQsioFKZWhP8/9PAGhldR0L/efzcrKONe62GAzvsztkZHfAl0I="; SPC_T_IV="IETR5YkWloW3OcKf80c6RQ=="; SPC_R_T_IV="IETR5YkWloW3OcKf80c6RQ=="; SPC_T_ID="Am9bCo3cc3Jno2mV5RDkLJIVsbIWEDTC6ezJknXdVVRfxlQRoGDcya57fIQsioFKZWhP8/9PAGhldR0L/efzcrKONe62GAzvsztkZHfAl0I="'

}

shopee_url = 'https://shopee.co.id/top_products'

response = requests.get(shopee_url, headers=headers)

soup = bs(response.text, "html.parser")

products = soup.find_all("._3S8sOC _2QfAXF")

print(type(products))

print(products)

感谢您阅读到这里。我读过一些类似的答案，但它们似乎并不适用于此。

Answer 1

如果您愿意使用 selenium，您可以使用 selenium 提出请求：from selenium import webdriverfrom bs4 import BeautifulSoup as bsheaders = {    'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36',    'cookie': '_gcl_au=1.1.961206468.1594951946; _med=refer; _fbp=fb.2.1594951949275.1940955365; SPC_IA=-1; SPC_F=y1evilme0ImdfEmNWEc08bul3d8toc33; REC_T_ID=fab983c8-c7d2-11ea-a977-ccbbfe23657a; SPC_SI=uv1y64sfvhx3w6dir503ixw89ve2ixt4; _gid=GA1.3.413262278.1594951963; SPC_U=286107140; SPC_EC=GwoQmu7TiknULYXKODlEi5vEgjawyqNcpIWQjoxjQEW2yJ3H/jsB1Pw9iCgGRGYFfAkT/Ej00ruDcf7DHjg4eNGWbCG+0uXcKb7bqLDcn+A2hEl1XMtj1FCCIES7k17xoVdYW1tGg0qaXnSz0/Uf3iaEIIk7Q9rqsnT+COWVg8Y=; csrftoken=5MdKKnZH5boQXpaAza1kOVLRFBjx1eij; welcomePkgShown=true; _ga=GA1.1.1693450966.1594951955; _dc_gtm_UA-61904553-8=1; REC_MD_30_2002454304=1595153616; _ga_SW6D8G0HXK=GS1.1.1595152099.14.1.1595153019.0; REC_MD_41_1000044=1595153318_0_50_0_49; SPC_R_T_ID="Am9bCo3cc3Jno2mV5RDkLJIVsbIWEDTC6ezJknXdVVRfxlQRoGDcya57fIQsioFKZWhP8/9PAGhldR0L/efzcrKONe62GAzvsztkZHfAl0I="; SPC_T_IV="IETR5YkWloW3OcKf80c6RQ=="; SPC_R_T_IV="IETR5YkWloW3OcKf80c6RQ=="; SPC_T_ID="Am9bCo3cc3Jno2mV5RDkLJIVsbIWEDTC6ezJknXdVVRfxlQRoGDcya57fIQsioFKZWhP8/9PAGhldR0L/efzcrKONe62GAzvsztkZHfAl0I="'}driver = webdriver.Firefox(executable_path='c:/program/geckodriver.exe')shopee_url = 'https://shopee.co.id/top_products'driver.get(shopee_url)driver.implicitly_wait(15)response = driver.page_sourcedriver.close()soup = bs(response, "html.parser")products = soup.select(".shopee-item-card__lower-padding")#print(products)for product in products:    name = product.select_one('.shopee-item-card__text-name span')    print(name.get_text(strip=True))    price = product.select_one('.shopee-item-card__current-price')    print(price.text)请注意，您需要安装 Firefox、geckodriver 和 selenium。输出：Kuota Telkomsel Cek DadakanRp300Aigo 1GB 5HariRp7.375Paket Data Tri Mix Small 5GBRp4.500Paket Data Tri Mix Small 2.75GBRp9.500*PROMO MURAH* SURPRISE DEAL PAKET DATA INTERNET KUOTA TELKOMSEL SIMPATI AS LOOP OMG 20GB 30GB 50GBRp1.000Kuota Telkomsel Cek DadakanRp500Paket Data Telkomsel 12GBRp98.000Paket Data Tri AON Unlimited + 6GBRp58.000KUOTA INTERNET PAKET DATA INTERNET TELKOMSEL KUOTA DATA FLASH Kartu Simpati As Loop (KUOTA MURAH)Rp1.000[MURAH] PAKET DATA TELKOMSEL ∣ KUOTA INTERNET ∣ PAKET DATA ∣ SIMPATI  ∣ AS  ∣ LOOOPRp1.000Paket Data Tri AddOn 12GBRp48.000Paket Data  Indosat Freedom U 7GB+20GB Apps / 30 HariRp66.500PAKET DATA TELKOMSEL ISI ULANG SIMPATI 14GB/27GB/37GB - AS 6,5GB/10GB/17GB/32GB - LOOP 3GB/8GB/15GBRp38.000Paket Data Indosat Freedom Internet 4GB / 30 HariRp24.000Paket Data Indosat Freedom U 1GB+4.5GB Apps / 30 HariRp24.000KUOTA TELKOMSEL paket kuota data internet telkomsel termurah 52gb injek injeck kuota  promoRp1.000Kuota Telkomsel Simpati 52GB 50GB 24jam Langsung MasukRp1.000Paket Data Telkomsel 50GBRp195.000Paket Data Indosat Freedom Internet 10GB /30 HariRp48.000KUOTA TELKOMSEL paket data internet telkomsel kuota internet TELKOMSEL  termurah simpati as loopRp1.000Paket Data Telkomsel Data 25.000Rp23.500VOCHER VOUCHER KUOTA AXIS AIGO 1GB 5HARI NASIONALRp7.390PAKET DATA KUOTA INDOSAT [DISKON 15%] TERMURAHRp1.000VOUCHER Axis 1gb 5 hari 24 Jam KUOTA 1 GBRp7.500Paket Data Indosat Freedom Combo 14GB, 30 HariRp48.500[PROMO] KUOTA BESAR PAKET OMG TELKOMSEL MURAHRp1.000Paket Data Indosat Freedom Combo 8GB/ 30 HariRp33.500TELKOMSEL PEMBAYARAN RESELLER/ AGEN atau Penambahan Selisih Harga Produk ThalhahCellRp1.000Aigo 3GB 15HariRp18.500Paket Data Indosat Freedom U 3GB+15GB Apps / 30 HariRp49.900