如何抓取 HTML 表格格式的数据？
我正在尝试从https://www.msamb.com/ApmcDetail/ArrivalPriceInfo网站抓取数据。
这是我要抓取的数据。所以，在高亮的下拉选择框中有148个商品。
截至目前，我正在通过选择每个单独的商品来手动复制数据。这需要大量的手动工作来提取数据。
所以，为了让它自动化，我开始使用 Python。以下是我在 Python (3.7.8) 代码中使用的库。
硒
美汤
熊猫
这是我的 Python 代码。
from selenium import webdriver
from bs4 import BeautifulSoup
import pandas as pd
from selenium.webdriver.support.ui import Select
#from selenium.webdriver.common.by import By
driver = webdriver.Chrome(executable_path='G:/data/depend/chromedriver.exe')
driver.get('https://www.msamb.com/ApmcDetail/ArrivalPriceInfo/')
commodity = Select(driver.find_element_by_id("CommoditiesId"))
#able to select commodities by value
commodity.select_by_value('08005')
# Iterating over the all the commodity an fetching <td> element
for option in commodity.options:
#print(option.text)
soup = BeautifulSoup(option.text)
print(soup)
rows = soup.select('tr')
print(rows)
for row in rows[1:]:
td = row.find_all('td')
print(td)
APMC = td[0].text.strip()
print(APMC)
在这里，我可以从下拉选择框中通过等于CommoditiesId的id获取商品。
获取商品列表 (148) 后，我将尝试解析为该特定商品获取的 HTML 表格内容。在这里，我能够为每次迭代打印商品名称，但无法打印APMC 、Variety、Unit、Quantity、Lrate、Hrate、Modal列数据。
如果以上解决了，我想要分隔格式的输出~|~并想要添加两列，即Date, Commodity。因此，示例输出将如下所示（截至目前，手动准备以下数据文件）。
Date~|~Commodity~|~APMC~|~Variety~|~Unit~|~Quantity~|~Lrate~|~Hrate~|~Modal
2020-07-11~|~APPLE~|~KOLHAPUR~|~QUINTAL~|~17~|~8500~|~14500~|~11500
2020-07-11~|~APPLE~|~CHANDRAPUR-GANJWAD~|~QUINTAL~|~9~|~15000~|~17000~|~16000
2020-07-11~|~APPLE~|~NASHIK~|~DILICIOUS- No.1~|~QUINTAL~|~60~|~9500~|~16000~|~13000
2020-07-11~|~AMBAT CHUKA~|~PANDHARPUR~|~~|~NAG~|~7~|~10~|~10~|~10
2020-07-10~|~AMBAT CHUKA~|~PUNE-MANJRI~|~~|~NAG~|~400~|~3~|~6~|~4
2020-07-10~|~AMBAT CHUKA~|~PUNE~|~LOCAL~|~NAG~|~1300~|~4~|~5~|~4
肥皂起泡泡2回答
-
交互式爱情
此脚本将遍历所有页面并将它们保存到标准 csv 和~|~分隔文本文件中：import requestsimport numpy as npimport pandas as pdfrom bs4 import BeautifulSoupurl = 'https://www.msamb.com/ApmcDetail/ArrivalPriceInfo'detail_url = 'https://www.msamb.com/ApmcDetail/DataGridBind?commodityCode={code}&apmcCode=null'headers = {'Referer': 'https://www.msamb.com/ApmcDetail/ArrivalPriceInfo'}soup = BeautifulSoup(requests.get(url).content, 'html.parser')values = [(o['value'], o.text) for o in soup.select('#CommoditiesId option') if o['value']]all_data = []for code, code_name in values: print('Getting info for code {} {}'.format(code, code_name)) soup = BeautifulSoup(requests.get(detail_url.format(code=code), headers=headers).content, 'html.parser') current_date = '' for row in soup.select('tr'): if row.select_one('td[colspan]'): current_date = row.get_text(strip=True) else: row = [td.get_text(strip=True) for td in row.select('td')] all_data.append({ 'Date': current_date, 'Commodity': code_name, 'APMC': row[0], 'Variety': row[1], 'Unit': row[2], 'Quantity': row[3], 'Lrate': row[4], 'Hrate': row[5], 'Modal': row[6], })df = pd.DataFrame(all_data)print(df)df.to_csv('data.csv') # <-- saves standard csvnp.savetxt('data.txt', df, delimiter='~|~', fmt='%s') # <-- saves .txt file with '~|~' delimiter印刷：...Getting info for code 08071 TOMATOGetting info for code 10006 TURMERICGetting info for code 08075 WAL BHAJIGetting info for code 08076 WAL PAPDIGetting info for code 08077 WALVADGetting info for code 07011 WATER MELONGetting info for code 02009 WHEAT(HUSKED)Getting info for code 02012 WHEAT(UNHUSKED) Date Commodity APMC Variety Unit Quantity Lrate Hrate Modal0 18/07/2020 AMBAT CHUKA PANDHARPUR ---- NAG 50 5 5 51 16/07/2020 AMBAT CHUKA PANDHARPUR ---- NAG 50 5 5 52 15/07/2020 AMBAT CHUKA PANDHARPUR ---- NAG 100 9 9 93 13/07/2020 AMBAT CHUKA PANDHARPUR ---- NAG 16 7 7 74 13/07/2020 AMBAT CHUKA PUNE LOCAL NAG 2400 4 7 5... ... ... ... ... ... ... ... ... ...4893 12/07/2020 WHEAT(HUSKED) SHIRUR No. 2 QUINTAL 2 1400 1400 14004894 17/07/2020 WHEAT(UNHUSKED) SANGLI-MIRAJ ---- QUINTAL 863 4000 4600 43004895 16/07/2020 WHEAT(UNHUSKED) SANGLI-MIRAJ ---- QUINTAL 475 4000 4500 42504896 15/07/2020 WHEAT(UNHUSKED) SANGLI-MIRAJ ---- QUINTAL 680 3900 4400 41504897 13/07/2020 WHEAT(UNHUSKED) SANGLI-MIRAJ ---- QUINTAL 1589 3900 4450 4175[4898 rows x 9 columns]节省data.txt：0~|~18/07/2020~|~AMBAT CHUKA~|~PANDHARPUR~|~----~|~NAG~|~50~|~5~|~5~|~51~|~16/07/2020~|~AMBAT CHUKA~|~PANDHARPUR~|~----~|~NAG~|~50~|~5~|~5~|~52~|~15/07/2020~|~AMBAT CHUKA~|~PANDHARPUR~|~----~|~NAG~|~100~|~9~|~9~|~93~|~13/07/2020~|~AMBAT CHUKA~|~PANDHARPUR~|~----~|~NAG~|~16~|~7~|~7~|~74~|~13/07/2020~|~AMBAT CHUKA~|~PUNE~|~LOCAL~|~NAG~|~2400~|~4~|~7~|~55~|~12/07/2020~|~AMBAT CHUKA~|~PUNE~|~LOCAL~|~NAG~|~1700~|~3~|~8~|~56~|~19/07/2020~|~APPLE~|~KOLHAPUR~|~----~|~QUINTAL~|~3~|~9000~|~14000~|~115007~|~18/07/2020~|~APPLE~|~KOLHAPUR~|~----~|~QUINTAL~|~12~|~8500~|~15000~|~117508~|~18/07/2020~|~APPLE~|~NASHIK~|~DILICIOUS- No.1~|~QUINTAL~|~110~|~9000~|~16000~|~130009~|~18/07/2020~|~APPLE~|~SANGLI-PHALE BHAJIPALAM~|~LOCAL~|~QUINTAL~|~8~|~12000~|~16000~|~1400010~|~17/07/2020~|~APPLE~|~MUMBAI-FRUIT MARKET~|~----~|~QUINTAL~|~264~|~9000~|~12000~|~10500...来自 LibreOffice 的 csv 文件的屏幕截图： -
慕后森
您可以将它们保存到 txt 文件中，您可以这样做df = pd.read_csv("out.txt",delimiter='~|~')，或者date = df['Date'] commodity = df['Commodity']您可以将 apmc 附加到列表中，并在最后附加 read_as 数据框。
随时随地看视频慕课网APP
相关分类
Python