使用 Python 的 requests.post 函数抓取动态 HTML 时如何指定“页码”？

Question

我正在尝试学习如何使用requestsPython 中的库使用动态 HTML 抓取网页。我已经能够获取 HTML，但我不知道如何实际获取不同页面的数据。如果您注意到图像底部有页码：

1. 如何指定页码requests.post

2. 我如何知道有多少页可用？它会像页面范围然后和一样简单try吗except？

3. 我如何知道哪些参数可用于我的data字典requests.post？

以下是我访问请求的方式：

import requests

r = requests.post('https://search.aap.org/nicu/', 

data = {'SearchCriteria.Level':'1', 'X-Requested-With':'XMLHttpRequest'}) #key:value

print(r.text)

这是输出：

https://pastebin.com/raw/HMBZWR36

这是显示页码的部分：

# <div id="myPager">

#     <div class="pagination-container"><ul class="pagination blue-pager"><li class="blue-pager active"><span>1</span></li></ul></div>

# </div>

Answer 1

该页面上的数据是使用 JavaScript 动态加载的，因此您不能仅通过该页面的请求来获取它。使用浏览器中的 Developer 标签，您可以拦截生成数据的 xhr 并将其用于请求。这就是本例中的情况。import requestsheaders = {&nbsp; &nbsp; 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:81.0) Gecko/20100101 Firefox/81.0',&nbsp; &nbsp; 'Accept': '*/*',&nbsp; &nbsp; 'Accept-Language': 'en-US,en;q=0.5',&nbsp; &nbsp; 'X-Requested-With': 'XMLHttpRequest',&nbsp; &nbsp; 'Connection': 'keep-alive',&nbsp; &nbsp; 'Referer': 'https://search.aap.org/nicu/',}params = (&nbsp; &nbsp; ('Model', 'X.PagedList.StaticPagedList`1[AAP.Website.Search.NICU.Web.ViewModels.NicuVm]'),&nbsp; &nbsp; ('jsonSearchVmString', '{"SearchCriteria":{"Name":null,"City":null,"State":null,"Zip":null,"Level":2,"LevelAssigner":null,"BedNumberRange":0},"States":null,"Levels":null,"LevelAssigners":null,"BedNumberRangeList":null,"NumberOfPages":0}'),&nbsp; &nbsp; ('page', '2'),&nbsp; &nbsp; ('class', 'blue-pager'),&nbsp; &nbsp; ('X-Requested-With', 'XMLHttpRequest'),&nbsp; &nbsp; ('_', '1603318486561'),)response = requests.get('https://search.aap.org/nicu/', headers=headers, params=params)参数之一是('page', '2')；如果您将请求包装在for适当范围内的循环中，您应该能够requests.get()在所有页面上执行此操作。

Answer 2

在这种情况下，您不必使用 POST 请求。如果单击下一页时查看 URL，您将看到可以更改的所有参数。此外，要获取所有页面，您可以解析 HTML 并查找<a rel="next">标签。如果标签不存在，则您位于最后一页（尝试将参数更改"LevelAssigner"为3其他值）：import requestsfrom bs4 import BeautifulSoupurl = 'https://search.aap.org/nicu/?Model=X.PagedList.StaticPagedList`1[AAP.Website.Search.NICU.Web.ViewModels.NicuVm]&jsonSearchVmString={"SearchCriteria":{"Name":null,"City":null,"State":null,"Zip":null,"Level":0,"LevelAssigner":"3","BedNumberRange":0},"States":null,"Levels":null,"LevelAssigners":null,"BedNumberRangeList":null,"NumberOfPages":0}&class=blue-pager&X-Requested-With=XMLHttpRequest&page='page = 1while True:&nbsp; &nbsp; print('Page {}..'.format(page))&nbsp; &nbsp; soup = BeautifulSoup(requests.get(url + str(page)).content, 'html.parser')&nbsp; &nbsp; for row in soup.select('.row.font-700'):&nbsp; &nbsp; &nbsp; &nbsp; print(row.get_text(strip=True, separator=' '))&nbsp; &nbsp; next_page = soup.select_one('a[rel="next"]')&nbsp; &nbsp; if not next_page:&nbsp; &nbsp; &nbsp; &nbsp; break&nbsp; &nbsp; page += 1印刷：Page 1..ANDERSON REGIONAL MEDICAL CENTER Meridian, MS 39301Ann & Robert H. Lurie Children's Hospital of Chicago Chicago, IL 60611BAYSTATE MEDICAL CENTER Springfield, MA 01199Beacon Children's Hospital South Bend, IN 46601-1033BEN TAUB GENERAL HOSPITAL Houston, TX 77030Page 2..BETH ISRAEL DEACONESS MEDICAL CENTER Boston, MA 02215BRIGHAM AND WOMEN'S HOSPITAL Boston, MA 02115CHILDREN'S HOSPITAL OF NEW ORLEANS New Orleans, LA 70118CLEVELAND CLINIC formally MERIDIA HILLCREST HOSPITAL Mayfield Heights, OH 44124EAST JEFFERSON GENERAL HOSPITAL Metairie, LA 70006Page 3..GEORGETOWN UNIVERSITY HOSPITAL Washington, DC 20007GREENWICH HOSPITAL Greenwich, CT 06830HELEN DEVOS CHILDREN'S HOSPITAL Grand Rapids, MI 49503INOVA LOUDOUN HOSPITAL Leesburg, VA 20176JACOBI MEDICAL CENTER Bronx, NY 10461Page 4..LAKEVIEW REGIONAL MEDICAL CENTER HOSPITAL Covington, LA 70433LOYOLA UNIVERSITY MEDICAL CENTER Maywood, IL 60153MELROSE WAKEFIELD HOSPITAL Melrose, MA 02176MERCYHEALTH HOSPITAL ROCKTON AVENUE CAMPUS formally ROCKFORD HEALTH SYSTEM Rockford, IL 61103RUSH FOUNDATION HOSPITAL Meridian, MS 39301Page 5..Shetal Shah Valhalla, NY 10595SLIDELL MEMORIAL HOSPITAL Slidell, LA 70458SOUTHERN NEW HAMPSHIRE MEDICAL CENTER Nashua, NH 03061St Mary Medical Center Apple Valley, CA 92307ST. AGNES HOSPITAL Baltimore, MD 21229Page 6..THE CHILDREN'S HOSPITAL AT NAVICENT HEALTH Macon, GA 31208-6000TOURO INFIRMARY New Orleans, LA 70115TULANE LAKESIDE HOSPITAL Metairie, LA 70001UNIVERSITY MEDICAL CENTER -&nbsp; SUNY STONY BROOK Stony Brook, NY 11794-8111UNIVERSITY OF IOWA CHILDREN'S HOSPITAL Iowa City, IA 52242Page 7..UNIVERSITY OF LOUISVILLE SCHOOL OF MEDICINE Louisville, KY 40232-5070UNIVERSITY OF MISSISSIPPI MEDICAL CENTER Jackson, MS 39216-4505UNIVERSITY OF ROCHESTER MEDICAL CENTER Rochester, NY 14642UNIVERSITY OF WASHINGTON MEDICAL CENTER Seattle, WA 98195WakeMed Cary Hospital Cary, NC 27518Page 8..WakeMed North Family Health & Women’s Hospital Raleigh, NC 27614WakeMed Raleigh Hospital Raleigh, NC 27610Wellstar Kennestone Hospital NICU Marietta, GA 30066WILLIS KNIGHTON CENTER FOR WOMEN'S HEALTH Shreveport, LA 71118WOMAN'S HOSPITAL Baton Rouge, LA 70817