使用Find_All函数返回意外的结果集

Question

我正在使用蟒蛇3.8.2和bs4美丽汤。我正在尝试查找标记的所有实例，并在结果集中列出每个实例，每行一个。但是，返回的结果集包含的行数多于网站的原始抓取。这是因为结果集的第一行包含标记的所有实例。下一行包含除第一个实例之外的所有实例，第三行包含除第一个实例和第二个实例之外的所有实例，依此类推，以及结果集的其余部分。

代码如下：

from bs4 import BeautifulSoup

import requests

url = "https://www.sainsburys.co.uk/shop/gb/groceries/drinks/seeall"

html_content = requests.get(url, timeout=5)

soup = BeautifulSoup(html_content.text)

test_1 = soup.find('ul',{"class": "productLister gridView"})

test = test_1.find_all("li", attrs={"class": "gridItem"})

我如何获得它，以便每个实例仅列出本身，每行一个。<li class: "gridItem">

Answer 1

网站加载了事件，该事件在页面加载后动态呈现其数据。JavaScriptrequests库将无法动态渲染。所以你可以使用 或 .事实上，有很多模块可以做到这一点。JavaScriptseleniumrequests_html现在，我们在表上确实有另一个选项，可以跟踪数据的呈现位置。我能够找到XHR请求，该请求用于从中检索数据并将其呈现给用户端。back-endAPI您可以通过打开开发人员工具来获取请求，并检查网络并根据呼叫类型检查发出的请求，例如XHRXHR/JSfetch您可以通过以下方式实现您的目标：请注意以下几点：website持有3068架item我增加了每页要使用的项目120parameter&nbsp;"pageSize": "120"所以 = 让我们说 ，这意味着每页 120 个项目，26 页。3068 / 12026因此，您将需要从哪个手段循环，依此类推，使用您将在循环下递增的参数。(0, 3120, 120)0 > 120 > 240"beginIndex": "0"for在下面，您可以实现您的目标，因为您没有向我们提供您的最终目标。但我相信你的目标是或（网址，img）或其他什么。你会发现它。namepriceimport requestsfrom bs4 import BeautifulSoupparams = {&nbsp; &nbsp; "langId": "44",&nbsp; &nbsp; "storeId": "10151",&nbsp; &nbsp; "catalogId": "10241",&nbsp; &nbsp; "categoryId": "12192",&nbsp; &nbsp; "parent_category_rn": "",&nbsp; &nbsp; "top_category": "12192",&nbsp; &nbsp; "pageSize": "120",&nbsp; &nbsp; "orderBy": "FAVOURITES_FIRST",&nbsp; &nbsp; "searchTerm": "",&nbsp; &nbsp; "catSeeAll": "true",&nbsp; &nbsp; "beginIndex": "0",&nbsp; &nbsp; "categoryFacetId1": "12192",&nbsp; &nbsp; "categoryFacetId2": "",&nbsp; &nbsp; "requesttype": "ajax"}def main(url):&nbsp; &nbsp; with requests.Session() as req:&nbsp; &nbsp; &nbsp; &nbsp; r = req.post(url, params=params).json()&nbsp; &nbsp; &nbsp; &nbsp; for item in r[5]['productLists']:&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; for nest in item['products']:&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; soup = BeautifulSoup(nest['result'], 'html.parser')&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; target = soup.find("div", class_="productNameAndPromotions")&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; name = target.h3.a.text.strip()&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; url = target.h3.a.get("href")&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; img = f"https"+target.h3.a.img.get("src")&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; price = soup.find(&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "p", class_="pricePerUnit").get_text(strip=True)&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; print(name, price, img, url)main("https://www.sainsburys.co.uk/webapp/wcs/stores/servlet/gb/groceries/drinks/AjaxApplyFilterSearchResultView")名称和价格的简要输出：Sainsbury's British Semi Skimmed Milk 2.27L (4 pint) £1.10/unitSainsbury's British Semi Skimmed Milk 1.13L (2 pint) 80p/unitSainsbury's British Whole Milk 2.27L (4 pint) £1.10/unitCravendale Purefilter Semi Skimmed Milk 2L £1.90/unitSainsbury's British Skimmed Milk 2.27L (4 pint) £1.10/unitSainsbury's British Semi Skimmed Milk, SO Organic 2.27L (4 pint) £1.80/unitSainsbury's Sparkling Water, Basics 2L 25p/unitSainsbury's British Skimmed Milk 1.13L (2 pint) 80p/unitSainsbury's 100% Pure Squeezed Smooth Orange Juice, Not From Concentrate 1L £1.30/unitSainsbury's Water, Basics 2L 25p/unitSainsbury's British Whole Milk 1.13L (2 pint) 80p/unitSainsbury's Smooth Pure Orange Juice 1L 95p/unitPepsi Max 2L £1.90/unitSainsbury's Caledonian Still Water 4x2L £1.50/unitHighland Spring Still Water 12x500ml £3.00/unitSainsbury's 100% Pressed Apple Juice, Not From Concentrate 1L £1.30/unitSainsbury's British Whole Milk, SO Organic 2.27L (4 Pint) £1.80/unitLactofree Semi Skimmed Lactose Free Fresh Dairy Drink 1L £1.50/unitDiet Coke 8x330ml £4.00/unitAlpro Roasted Almond Unsweetened UHT Drink 1L £1.80/unitRobinsons Orange Squash No Added Sugar 1L £1.65/unitSainsbury's Soda Water 1L 60p/unitSainsbury's Caledonian Sparkling Water 4x2L £1.60/unitTropicana Smooth Orange Juice 950ml £2.45/unitSainsbury's Diet Indian Tonic Water 1L 60p/unitSainsbury's Pure Apple Juice 1L 95p/unitRobinsons Apple & Blackcurrant Squash No Added Sugar 1L £1.65/unitSainsbury's Sparkling Flavoured Water, Lemon & Lime 1L 50p/unitSainsbury's Conegliano Prosecco, Taste the Difference 75cl £8.00/unitSainsbury's Unsweetened Soya Drink 1L 90p/unitSainsbury's British Semi Skimmed Milk, SO Organic 1.13L (2 pint) £1.15/unitSainsbury's Caledonian Sparkling Water 6x500ml £1.50/unitSainsbury's Apple & Blackcurrant Squash, No Added Sugar 1.5L £1.00/unitHighland Spring Still Water 6x1.5L £3.00/unitAlpro Roasted Almond Unsweetened Fresh Drink 1L £1.85/unitSainsbury's Semi Skimmed Long Life Milk 1L 90p/unitTropicana Smooth Orange Juice 1.6L £2.50/unitSainsbury's 100% Pure Squeezed Orange Juice with Bits, Not From Concentrate 1L £1.30/unitCravendale Purefilter Semi Skimmed Milk 1L £1.15/unitSainsbury's Caledonian Still Water Sports Cap 6x500ml £1.50/unitSainsbury's Double Strength Orange Squash, No Added Sugar 1.5L £1.00/unitDiet Coke 18x330ml £7.00/unitSainsbury's Indian Tonic Water 1L 60p/unitSainsbury's Pure Orange Juice 1L 85p/unitSainsbury's Pure Apple Juice 6x200ml £1.50/unitBuxton Still Natural Mineral Water 8x500ml £2.00/unitSainsbury's Whole Long Life Milk 1L £1.05/unitCravendale Purefilter Skimmed Milk 2L £1.90/unitSainsbury's Sparkling Flavoured Water, Blackcurrant & Cherry 1L 50p/unitInnocent Smooth Orange Juice 1.35L £3.00/unitAlpro Original Soya Fresh Drink 1L £1.55/unitSainsbury's Still Flavoured Water, Strawberry & Kiwi 1L 50p/unitSainsbury's British Filtered Semi Skimmed Milk 2L £1.35/unitSainsbury's Sparkling Flavoured Water, Mango & Passionfruit 1L 50p/unitSainsbury's Caledonian Still Water 5L £1.10/unitMcGuigan Estate Merlot 75cl £5.10/unitSchweppes Slimline Tonic Water 1L £1.50/unitPG tips Pyramid Tea Bags x240 696g £4.50/unitSainsbury's Sparkling Flavoured Water, Strawberry & Kiwi 1L 50p/unitSainsbury's Caledonian Sparkling Water 2L 55p/unitSainsbury's Sweetened Soya Drink 1L 90p/unitSainsbury's 100% Pure Squeezed Smooth Orange Juice, Not From Concentrate 1.75L £2.10/unitSainsbury's Diet Lemonade 2L 60p/unitSainsbury's Apple & Mango Juice, Not From Concentrate 1L £1.30/unitRobinsons Summer Fruits Squash No Added Sugar 1L £1.65/unitSainsbury's 100% Pure Squeezed Pineapple Juice, Not From Concentrate 1L £1.30/unitClearsprings Sauvignon Blanc 75cl £5.50/unitPhantom River Sauvignon Blanc 75cl £5.00/unitNestle Pure Life Still Spring Water 12x500ml £2.50/unitBuxton Sparkling Natural Mineral Water 8x500ml £2.10/unitBrancott Estate Sauvignon Blanc 75cl £6.75/unitSchweppes Slimline Lemonade 2L £1.30/unitMcGuigan Estate South Australian Shiraz 75cl £5.10/unitCoca-Cola Zero Sugar 8x330ml £4.00/unitVilla Maria Private Bin Sauvignon Blanc 75cl £9.25/unitDiet Coke Caffeine Free 8x330ml £4.00/unitSainsbury's British Skimmed Milk, SO Organic 1.13L (2 pint) £1.15/unitSainsbury's Kids Caledonian Still Water 6x300ml £1.10/unitCanti Prosecco 75cl £7.50/unitOatly Enriched with Calcium Oat UHT Drink 1L £1.50/unitSainsbury's Pure Orange Juice 6x200ml £1.50/unitSainsbury's Still Flavoured Water, Lemon & Lime 1L 50p/unitValdo Prosecco Marca Oro 75cl £8.50/unitOyster Bay Sauvignon Blanc 75cl £8.00/unitRibena Blackcurrant Squash 850ml £2.30/unitVolvic Mineral Water 6x1.5L £3.40/unitCampo Viejo Rioja Tempranillo 75cl £6.75/unitNescafé Azera Americano Instant Coffee 100g £4.60/unitTropicana Orange Juice Original 950ml £2.45/unitSainsbury's Double Strength Orange & Mango Squash, No Added Sugar 1.5L £1.00/unit&nbsp;&nbsp;Robinsons Lemon Squash No Added Sugar 1L £1.65/unitSchweppes Lemonade 2L £1.30/unitRobinsons Orange & Pineapple Squash No Added Sugar 1L £1.65/unitSainsbury's Diet Indian Tonic with Lime 1L 60p/unitSt Helen's Farm Semi Skimmed Goats Milk 1L £1.80/unitSainsbury's Double Strength Orange, Lemon & Pineapple Squash, No Added Sugar 1.5L £1.00/unitSainsbury's Double Strength Summerfruits Squash, No Added Sugar 1.5L £1.00/unit&nbsp; &nbsp;&nbsp;Alpro Oat UHT Drink 1L £1.80/unitInnocent Smooth Orange Juice 900ml £1.50/unitSainsbury's British Whole Milk, SO Organic 1.13L (2 pint) £1.15/unitSainsbury's Skimmed Long Life Milk 1L 80p/unitNescafé Gold Blend Instant Coffee 200g £7.00/unitHighland Spring Still Water Sports Cap 12x330ml £3.00/unitSainsbury's Cava Brut 75cl £6.00/unitAlpro Light Unsweetened Soya Fresh Drink 1L £1.55/unitSainsbury's Caledonian Still Water 2L 50p/unitKoko Coconut UHT Drink 1L £1.50/unitSainsbury's House Pinot Grigio 75cl £4.50/unitSainsbury's Cola Zero 2L 45p/unitSt Helen's Farm Whole Goats Milk 1L £1.80/unitSainsbury's Double Strength Cherries & Berries Squash, No Added Sugar 1.5L £1.00/unitSainsbury's Lemonade 2L 60p/unitSainsbury's Pure Orange Juice With Bits 1L 85p/unitSainsbury's Pinot Grigio, Taste the Difference 75cl £6.00/unitSchweppes Tonic Water 1L £1.50/unitSainsbury's Cranberry Juice Drink 1L 85p/unitNescafé Gold Blend Instant Coffee Refill 150g £3.50/unitSainsbury's Gold Roast Instant Coffee 200g £3.15/unitSainsbury's Pure Orange Juice with Juicy Bits 1L 95p/unitEdizione 789 Di Mondelli Prosecco 75cl £6.25/unit