Beautifulsoup 无法提取所有 html

Question

我尝试创建一个程序来提取 Spotify 中 Daily Mix 1 中的所有歌曲。我知道我必须使用的逻辑，但我无法获得整个源代码。

这是我写的代码：

import requests 

from bs4 import BeautifulSoup

headers = {"User-Agent": "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36"}

result = requests.get("https://open.spotify.com/playlist/37i9dQZF1E38L6D2gtQHWw", headers=headers)

src = result.content

soup = BeautifulSoup(src, 'lxml')

print(soup.prettify())

这是我得到的输出：

我使用的标题适用于亚马逊和维基百科等其他网站，所以我认为这不是问题。我也不认为问题与 javascript 有关，因为在其他用于抓取网站（例如亚马逊（也包含很多<script>标签））的程序中，代码显示得非常好。

请告诉问题是什么。

PS - 请不要在您的解决方案中推荐 selenium 或 scrapy。

Answer 1

您尝试抓取的日期是由 Javascript 填充的，因此您不会在页面的源代码中找到它，但您可以通过网站正在使用的 api 获取它：import  json , requestsfrom bs4 import BeautifulSoup as bsbase_url = 'https://open.spotify.com/playlist/37i9dQZF1E38L6D2gtQHWw'headers = {"User-Agent": "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36"}# Getting the access token first to send it with the header to the api endpointpage              = requests.get(base_url,headers=headers)soup              = bs(page.text,'html.parser')access_token_tag  = soup.find('script',{'id':'config'})json_obj          = json.loads(access_token_tag.text)access_token_text = json_obj['accessToken']endpoint = "https://api.spotify.com/v1/playlists/37i9dQZF1E38L6D2gtQHWw"headers.update({"authorization": f"Bearer {access_token_text}",                'referer': base_url,                'accept': 'application/json',                'app-platform': 'WebPlayer'})url_paramters = {'type': 'track,episode','market': 'EG'}data = requests.get(endpoint, params=url_paramters, headers=headers).json()tracks = data['tracks']['items']for index , track in enumerate(tracks,1):        print(f'{index } - ' , track['track']['name'] )输出：1 -  Tu Hi Haqeeqat2 -  Hasi - Female Version3 -  Kabhi Jo Baadal Barse4 -  Tere Bin Nahi Laage (Male Version)5 -  Dekhte Dekhte (Rahat Fateh Ali Khan Version) [From "Batti Gul MeterChalu"]6 -  Panchhi Bole7 -  Jame Raho8 -  Banjaara (From "Ek Villain")9 -  Mitwa10 -  Agar Tu Hota (From "Baaghi")11 -  Aasan Nahin Yahan12 -  Jiyo Re Bahubali13 -  Pyaar Manga Hai14 -  Kaun Hain Voh15 -  Mamta Se Bhari16 -  Zehnaseeb17 -  Dil Ibaadat18 -  Tu Hi Tu (Reprise)19 -  Haule Haule20 -  Manohari21 -  Ilahi (From "Yeh Jawaani Hai Deewani")22 -  Humsafar (From "Badrinath Ki Dulhania")23 -  Kiya Kiya24 -  Sunn Raha Hai (Female)25 -  Phir Le Aya Dil26 -  Tere Naal Nachna (From "Nawabzaade")27 -  Galliyan (From "Ek Villain")28 -  Valentine's Mashup 2019(Remix By Kedrock,Sd Style)29 -  Halka Halka30 -  Raabta (From "Agent Vinod")31 -  Mere Bina - Unplugged32 -  Agar Tum Saath Ho-Maahi Ve33 -  Swapn Sunehere34 -  Radha35 -  Behti Hawa Sa Tha Woh36 -  Mere Rashke Qamar37 -  Kehta Hai Pal Pal38 -  Maana Ke Hum Yaar Nahin39 -  Khoya Hain40 -  O Re Piya41 -  Jal Rahin Hain42 -  Zero Hour Mashup 2015(Remix By Dj Kiran Kamath)43 -  Aashiq Banaya Aapne44 -  Bikhri Bikhri45 -  Maula Mere Lele Meri Jaan46 -  Yadaan Teriyaan (Version 2)47 -  Tujh Mein Rab Dikhta Hai48 -  Veeron Ke Veer Aa49 -  Bolo Har Har Har (feat. Mohit Chauhan, Sukhwinder Singh, Badshah, Megha Sriram Dalton, Anugrah, Sandeep Shrivastava)50 -  Main Agar