我想使用 beautifulsoup 从 python 中的链接获取内容

Question

我正在尝试从https://www.indeed.ae/jobs?q=&l=dubai中删除数据

我想获取内容（点击上面链接上的每个职位即可获得）

我正在使用 python、requests 和 bs4

它的内容来自链接中的链接，我想获取每份工作的所有内容

import requests

from bs4 import BeautifulSoup

import lxml

import urllib.request

url = 'https://www.indeed.ae/jobs?q=&l=dubai'

response = requests.get(url)

soup = BeautifulSoup(response.text, 'lxml')

links = soup.find_all('a', {'class': 'jobtitle turnstileLink'})

for correct_url in (soup.find_all('a', {'class': 'jobtitle turnstileLink'})):

    correct_url = ("https://indeed.ae" + (links.get('href')))

    print(correct_url)

Answer 1

此脚本将打印所有作业的描述（页面通过 JavaScipt 加载此信息，但您可以使用requests模块来加载此信息）：import requestsfrom bs4 import BeautifulSoupurl = 'https://www.indeed.ae/jobs?q=&l=dubai'jobdesc_url = 'https://www.indeed.ae/rpc/jobdescs'soup = BeautifulSoup(requests.get(url).content, 'html.parser')jks = ','.join(jk['data-jk'] for jk in soup.select('[data-jk]'))descriptions = requests.get(jobdesc_url, params={'jks': jks}).json()for jk in soup.select('[data-jk]'):    print(jk.h2.get_text(strip=True))    print()    print(jk.find_next('span', class_='company').get_text(strip=True))    print('---')    print(BeautifulSoup(descriptions[jk['data-jk']], 'html.parser').get_text())    print('-' * 80)印刷：RETAIL SALES ASSOCIATEAl Ghazi---Hiring a Retail Sales Associate for Dubai.Key Responsibilities:Attend to walk in customers, phone calls and provide customer service.Maintain a conducive environment in the outlet through house keeping.Promote products and ensure high levels of customer satisfaction through customer journey mapping.Recommend and display items that match customers’ needs and preferences.Key Requirements:High school Diploma.One to two years experience as a Retail sales Associate.Presentable and well-groomedObserve strict punctuality.Able to work well in a team.Able to multi-task.--------------------------------------------------------------------------------Human Resource OfficerRaban Al-Safina---The RoleWe are looking for a HR Officer with 3 to 4 years UAE work experience.Candidate must be available currently in UAE and ready to join immediately. Candidates on Husband / Father Sponsor visas will be given more preference. Job Responsibilities: • Working closely with various departments, increasingly in a consultancy role, assisting line managers to understand and implement policies and procedures • Administering payroll and maintaining employee records • Interpreting and advising on employment law • Advising on pay and other remuneration issues, including promotion and benefits • Developing and implementing policies on issues like working conditions, performance management, equal opportunities, • Disciplinary procedures and absence management • Recruiting staff, which involves developing job descriptions and person specifications, preparing job adverts, checking application forms, shortlisting, interviewing and selecting candidatesRequirementsAs a Human Resources (HR) officer you'll develop, advise on and implement policies relating to the effective use of staff in an organisation. • In the role your aim is to ensure that the organisation you work for employs the right balance of staff in terms of skill and experience, and that training and development opportunities are available to colleagues to enhance their performance and achieve the company's business aims. • HR officers are involved in a range of activities whatever the size or type of business. These cover areas such as: • Conditions of employment • Equality and diversity • Negotiation with external work-related agencies • Pay roll management • Recruitment • Working practices.About the companyRaban Al-Safina Group of Companies stands as one of the most prominent industrial organizations in Iraq work in different fields in power industry, oil & gas, systems technology, and many others.--------------------------------------------------------------------------------...and so on.