使用 BeautifulSoup 查找名为 data-stats 的属性

Question

我目前正在开发一个网络爬虫，它可以让我从足球运动员那里提取统计数据。通常，如果我可以抓取 div，这将是一项简单的任务，但是，该网站使用名为 data-stats 的属性并将其用作类。这是一个例子。

<th scope="row" class="left " data-stat="year_id"><a href="/years/2000/">2000</a></th>

如果您想亲自查看该网站，请点击此处的链接。

https://www.pro-football-reference.com/players/B/BradTo00.htm

我尝试了几种不同的方法。要么它根本不起作用，要么我将能够启动一个 for 循环并开始将内容放入数组中，但是您会注意到并非表中的所有内容都是相同的 var 类型。

对不起格式和语法。

这是我到目前为止所拥有的，我确定它不是最好看的代码，它主要是我自己尝试过的代码以及在 Google 上搜索时混合的一些东西。忽略我正在尝试不同的东西的随机导入

# import libraries

import csv

from datetime import datetime

import requests

from bs4 import BeautifulSoup

import lxml.html as lh

import pandas as pd

# specify url

url = 'https://www.pro-football-reference.com/players/B/BradTo00.htm'

# request html

page = requests.get(url)

# Parse html using BeautifulSoup, you can use a different parser like lxml if present

soup = BeautifulSoup(page.content, 'lxml')

# find searches the given tag (div) with given class attribute and returns the first match it finds

headers = [c.get_text() for c in soup.find(class_ = 'table_container').find_all('td')[0:31]]

data = [[cell.get_text(strip=True) for cell in row.find_all('td')[0:32]]

        for row in soup.find_all("tr", class_=True)]

tags = soup.find(data ='pos')

#stats = tags.find_all('td')

print(tags)

Answer 1

您需要使用getBeautifulSoup 中的方法按名称获取属性参见：BeautifulSoup Get Attribute这是从表中获取您想要的所有数据的片段：from bs4 import BeautifulSoupimport requestsurl = "https://www.pro-football-reference.com/players/B/BradTo00.htm"r = requests.get(url)soup = BeautifulSoup(r.text, 'html.parser')# Get tabletable = soup.find(class_="table_outer_container")# Get headthead = table.find('thead')th_head = thead.find_all('th')for thh in th_head:&nbsp; &nbsp; # Get case value&nbsp; &nbsp; print(thh.get_text())&nbsp; &nbsp; # Get data-stat value&nbsp; &nbsp; print(thh.get('data-stat'))# Get bodytbody = table.find('tbody')tr_body = tbody.find_all('tr')for trb in tr_body:&nbsp; &nbsp; # Get id&nbsp; &nbsp; print(trb.get('id'))&nbsp; &nbsp; # Get th data&nbsp; &nbsp; th = trb.find('th')&nbsp; &nbsp; print(th.get_text())&nbsp; &nbsp; print(th.get('data-stat'))&nbsp; &nbsp; for td in trb.find_all('td'):&nbsp; &nbsp; &nbsp; &nbsp; # Get case value&nbsp; &nbsp; &nbsp; &nbsp; print(td.get_text())&nbsp; &nbsp; &nbsp; &nbsp; # Get data-stat value&nbsp; &nbsp; &nbsp; &nbsp; print(td.get('data-stat'))# Get footertfoot = table.find('tfoot')thf = tfoot.find('th')# Get case valueprint(thf.get_text())# Get data-stat valueprint(thf.get('data-stat'))for tdf in tfoot.find_all('td'):&nbsp; &nbsp; # Get case value&nbsp; &nbsp; print(tdf.get_text())&nbsp; &nbsp; # Get data-stat value&nbsp; &nbsp; print(tdf.get('data-stat'))您当然可以将数据保存在 csv 甚至 json 中，而不是打印它

Answer 2

目前还不清楚您究竟要提取什么，但这可能会对您有所帮助：import requestsfrom bs4 import BeautifulSoup as bsurl = 'https://www.pro-football-reference.com/players/B/BradTo00.htm'page = requests.get(url)soup = bs(page.text, "html.parser")# Extract tabletable = soup.find_all('table')# Let's extract data from each row in tablefor row in table:&nbsp; &nbsp; col = row.find_all('td')&nbsp; &nbsp; for c in col:&nbsp; &nbsp; &nbsp; &nbsp; print(c.text)希望这可以帮助！