如何从具有不变 url 的网页中抓取 pdf 链接？

Question

我正在做一个关于网络抓取的项目，我被要求从一个网站抓取所有 pdf 链接：https: //www.sebi.gov.in/sebiweb/home/HomeAction.do ?doListing=yes&sid=3& s。该网站有 397 个页面，但每个页面都有相同的 URL。我尝试了 inspect element 工具，发现 javascript 代码有助于导航到不同的页面。但我仍然无法弄清楚如何为所有页面运行我的脚本。

下面是我的代码。

from bs4 import BeautifulSoup

import lxml

url = 'https://www.sebi.gov.in/sebiweb/home/HomeAction.do?doListing=yes&sid=3&s'

conn = urllib2.urlopen(url)

html = conn.read()

soup = BeautifulSoup(html)

links = soup.find_all('a')

urls=[]

for tag in links:

    link = tag.get('href',None)

    if link is not None and link.endswith('html'):

        #urls.append(link)

        purl=link

        new=urllib2.urlopen(purl)

        htm=new.read()

        sp=BeautifulSoup(htm)

        nl=sp.find_all('a')

        nm=sp.find_all('iframe')

        for i in nl:

            q=i.get('href',None)

            title=i.get('title',None)

            if q is not None and q.endswith('pdf'):

                print(q)

                urls.append(q)

        for j in nm:

            z=j.get('src',None)

            title=j.get('title',None)

            if z is not None and z.endswith('pdf')and title is not None:

                print(z)

                print(title)

                urls.append(z)

        

print(len(urls))

Answer 1

您可以使用他们位于 上的 APIhttps://www.sebi.gov.in/sebiweb/ajax/home/getnewslistinfo.jsp来加载数据。例如：from bs4 import BeautifulSoupfrom requests import getapi_url = 'https://www.sebi.gov.in/sebiweb/ajax/home/getnewslistinfo.jsp'payload = {    'nextValue': "1",    'next': "n",    'search': "",    'fromDate': "",    'toDate': "",    'fromYear': "",    'toYear': "",    'deptId': "",    'sid': "3",    'ssid': "-1",    'smid': "0",    'intmid': "-1",    'sText': "Filings",    'ssText': "-- All Sub Section --",    'smText': "",    'doDirect':  "1",}page = 0while True:    print('Page {}...'.format(page))    payload['doDirect'] = page    soup = BeautifulSoup(requests.post(api_url, data=payload).content, 'html.parser')    rows = soup.select('tr:has(td)')    if not rows:        break    for tr in rows:        row = [td.get_text(strip=True) for td in tr.select('td')] + [tr.a['href']]        print(*row, sep='\t')    page += 1印刷：...Page 1...Jun 25, 2020    Mindspace Business Parks REIT – Addendum to Draft Prospectus    https://www.sebi.gov.in/filings/reit-issues/jun-2020/mindspace-business-parks-reit-addendum-to-draft-prospectus_46928.htmlJun 25, 2020    Amrit Corp. Ltd. - Public Announcement  https://www.sebi.gov.in/filings/buybacks/jun-2020/amrit-corp-ltd-public-announcement_46927.htmlJun 24, 2020    NIIT Technologies Buyback - Post Buyback - Public Advertisement https://www.sebi.gov.in/filings/buybacks/jun-2020/niit-technologies-buyback-post-buyback-public-advertisement_46923.htmlJun 23, 2020    Addendum to Letter of Offer of Arvind Fashions Limited  https://www.sebi.gov.in/filings/rights-issues/jun-2020/addendum-to-letter-of-offer-of-arvind-fashions-limited_46941.htmlJun 23, 2020    Genesis Exports Limited - Draft letter of Offer https://www.sebi.gov.in/filings/buybacks/jun-2020/genesis-exports-limited-draft-letter-of-offer_46911.htmlJun 23, 2020    Genesis Exports Limited - Public Announcement   https://www.sebi.gov.in/filings/buybacks/jun-2020/genesis-exports-limited-public-announcement_46909.htmlJun 19, 2020    Coral India Finance and Housing Limited – Post Buy-back Public Announcement https://www.sebi.gov.in/filings/buybacks/jun-2020/coral-india-finance-and-housing-limited-post-buy-back-public-announcement_46900.htmlJun 19, 2020    Network Limited https://www.sebi.gov.in/filings/takeovers/jun-2020/network-limited_46890.htmlJun 17, 2020    KSOLVES INDIA LIMITED   https://www.sebi.gov.in/filings/public-issues/jun-2020/ksolves-india-limited_46996.htmlJun 10, 2020    Happiest Minds Technologies Limited https://www.sebi.gov.in/filings/public-issues/jun-2020/happiest-minds-technologies-limited_46843.htmlJun 08, 2020    IM+ Capitals Limited    https://www.sebi.gov.in/filings/takeovers/jun-2020/im-capitals-limited_46786.htmlJun 05, 2020    HealthCare Global Enterprises Limited   https://www.sebi.gov.in/filings/takeovers/jun-2020/healthcare-global-enterprises-limited_46773.htmlJun 02, 2020    Jaikumar Constructions Ltd. - DRHP  https://www.sebi.gov.in/filings/public-issues/jun-2020/jaikumar-constructions-ltd-drhp_46774.htmlJun 02, 2020    Mahindra Focused Equity Yojana  https://www.sebi.gov.in/filings/mutual-funds/jun-2020/mahindra-focused-equity-yojana_46767.htmlJun 02, 2020    GRANULES INDIA LIMITED - Dispatch advertisement https://www.sebi.gov.in/filings/buybacks/jun-2020/granules-india-limited-dispatch-advertisement_46765.htmlJun 02, 2020    GRANULES INDIA LIMITED  - Letter of Offer   https://www.sebi.gov.in/filings/buybacks/jun-2020/granules-india-limited-letter-of-offer_46764.htmlJun 02, 2020    Motilal Oswal Multi Asset Fund  https://www.sebi.gov.in/filings/mutual-funds/jun-2020/motilal-oswal-multi-asset-fund_46762.htmlJun 02, 2020    Principal Large Cap Fund    https://www.sebi.gov.in/filings/mutual-funds/jun-2020/principal-large-cap-fund_46761.htmlJun 02, 2020    Mahindra Arbitrage Yojana   https://www.sebi.gov.in/filings/mutual-funds/jun-2020/mahindra-arbitrage-yojana_46760.htmlJun 02, 2020    HSBC Mid Cap Equity Fund    https://www.sebi.gov.in/filings/mutual-funds/jun-2020/hsbc-mid-cap-equity-fund_46759.htmlJun 01, 2020    Tanla Solutions Limited - DLOF  https://www.sebi.gov.in/filings/buybacks/jun-2020/tanla-solutions-limited-dlof_46750.htmlJun 01, 2020    Axis Banking ETF    https://www.sebi.gov.in/filings/mutual-funds/jun-2020/axis-banking-etf_46748.htmlJun 01, 2020    Kalpataru Power Transmission Limited - Public Announcement  https://www.sebi.gov.in/filings/buybacks/jun-2020/kalpataru-power-transmission-limited-public-announcement_46746.htmlJun 01, 2020    Reliance Industries Limited - Addendum dated May 22, 2020   https://www.sebi.gov.in/filings/rights-issues/jun-2020/reliance-industries-limited-addendum-dated-may-22-2020_46745.htmlJun 01, 2020    Reliance Industries Limited - Addendum dated May 19, 2020   https://www.sebi.gov.in/filings/rights-issues/jun-2020/reliance-industries-limited-addendum-dated-may-19-2020_46744.htmlPage 2...Jun 01, 2020    Reliance Industries Limited - Addendum dated May 18, 2020   https://www.sebi.gov.in/filings/rights-issues/jun-2020/reliance-industries-limited-addendum-dated-may-18-2020_46743.htmlMay 29, 2020    Muthoottu Mini Financiers Limited- Prospectus   https://www.sebi.gov.in/filings/debt-offer-document/may-2020/muthoottu-mini-financiers-limited-prospectus_46769.htmlMay 29, 2020    Coral India Housing and Finance Limited - Advertisement https://www.sebi.gov.in/filings/buybacks/may-2020/coral-india-housing-and-finance-limited-advertisement_46732.htmlMay 29, 2020    TANLA SOLUTIONS LIMITED - Public Announcement   https://www.sebi.gov.in/filings/buybacks/may-2020/tanla-solutions-limited-public-announcement_46731.htmlMay 28, 2020    Tips Industries Limited - Dispatch Advertisement    https://www.sebi.gov.in/filings/buybacks/may-2020/tips-industries-limited-dispatch-advertisement_46723.htmlMay 27, 2020    KLM Axiva Finvest Limited - Prospectus  https://www.sebi.gov.in/filings/debt-offer-document/may-2020/klm-axiva-finvest-limited-prospectus_46755.htmlMay 26, 2020    Tips Industries Limited - Letter of Offer   https://www.sebi.gov.in/filings/buybacks/may-2020/tips-industries-limited-letter-of-offer_46708.htmlMay 26, 2020    Axis Capital Protection Oriented Fund - Series 7-10 https://www.sebi.gov.in/filings/mutual-funds/may-2020/axis-capital-protection-oriented-fund-series-7-10_46707.htmlMay 26, 2020    ICICI Prudential Alpha Low Vol 30 ETF   https://www.sebi.gov.in/filings/mutual-funds/may-2020/icici-prudential-alpha-low-vol-30-etf_46706.htmlMay 22, 2020    NIIT Technologies Ltd. - Letter of Offer    https://www.sebi.gov.in/filings/buybacks/may-2020/niit-technologies-ltd-letter-of-offer_46700.htmlMay 22, 2020    NIIT Technologies Ltd. - Dispatch Advertisement https://www.sebi.gov.in/filings/buybacks/may-2020/niit-technologies-ltd-dispatch-advertisement_46699.htmlMay 22, 2020    Coral India Finance and Housing Limited - Letter of Offer   https://www.sebi.gov.in/filings/buybacks/may-2020/coral-india-finance-and-housing-limited-letter-of-offer_46698.htmlMay 22, 2020    Jay Ushin Limited   https://www.sebi.gov.in/filings/takeovers/may-2020/jay-ushin-limited_46697.htmlMay 22, 2020    Pennar Industries - Post Buyback Public Announcement    https://www.sebi.gov.in/filings/buybacks/may-2020/pennar-industries-post-buyback-public-announcement_46696.htmlMay 22, 2020    Axis Global Equity Alpha Fund of Fund.  https://www.sebi.gov.in/filings/mutual-funds/may-2020/axis-global-equity-alpha-fund-of-fund-_46695.htmlMay 21, 2020    Axis Global Disruption Fund of Fund https://www.sebi.gov.in/filings/mutual-funds/may-2020/axis-global-disruption-fund-of-fund_46694.htmlMay 18, 2020    Reliance Industries Limited https://www.sebi.gov.in/filings/rights-issues/may-2020/reliance-industries-limited_46675.htmlMay 14, 2020    Public Advertisement of Spencer's Retail Limited    https://www.sebi.gov.in/filings/rights-issues/may-2020/public-advertisement-of-spencer-s-retail-limited_46693.htmlMay 12, 2020    Spencer's Retail Limited    https://www.sebi.gov.in/filings/rights-issues/may-2020/spencer-s-retail-limited_46692.htmlMay 12, 2020    Sequent Scientific Limited  https://www.sebi.gov.in/filings/takeovers/may-2020/sequent-scientific-limited_46662.htmlMay 11, 2020    Arvind Fashions Limited https://www.sebi.gov.in/filings/rights-issues/may-2020/arvind-fashions-limited_46659.htmlMay 05, 2020    JK Paper Limited - Public Announcement  https://www.sebi.gov.in/filings/buybacks/may-2020/jk-paper-limited-public-announcement_46647.htmlMay 05, 2020    Aurionpro Solutions Limited - Post BuyBack Advertisement    https://www.sebi.gov.in/filings/buybacks/may-2020/aurionpro-solutions-limited-post-buyback-advertisement_46646.htmlMay 04, 2020    KSOLVES INDIA LIMITED   https://www.sebi.gov.in/filings/public-issues/may-2020/ksolves-india-limited_46644.htmlMay 04, 2020    SBI ETF Consumption https://www.sebi.gov.in/filings/mutual-funds/may-2020/sbi-etf-consumption_46639.htmlPage 3...... and so on.

Answer 2

该网站似乎正在向 POST 请求getnewslistinfo.jsp并将新表格内容作为 html 取回。您可以打开您的网络（在 Firefox 上为 Ctrl+Shift+E），然后导航到下一页并查看发出的请求及其参数。您可以模仿该 POST 请求并使用urllib2（或者最好是requests ）更改下一页的适当参数（从我看到的应该是nextValueand ） 。获取内容后，您可以简单地使用 BeautifulSoup 解析它并按照您已经执行的方式提取 a 标签。doDirect另外给你一个提示：你应该把你的代码分成不同的函数来做不同的事情，比如getPage(pageNum)给定一个页码返回 html 内容，getLinks(html)给定一个 html 页面它从表中获取所有链接并将它们作为列表返回。这样您的代码将更具可读性并且更易于调试和使用。