要爬取的网站 http://quotes.toscrape.com/
**爬取名言，作者，标签。***
她们的Html为，通过beautiful库的html.parser解析，通过id,class选择器，提取我们需要的东西。
<span class="text">“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”</span>
<span class="text">“It is our choices, Harry, that show what we truly are, far more than our abilities.”</span>
<div class="tags">
Tags:
<a class="tag" href="/tag/change/page/1/">change</a>
<a class="tag" href="/tag/deep-thoughts/page/1/">deep-thoughts</a>
<a class="tag" href="/tag/thinking/page/1/">thinking</a>
<a class="tag" href="/tag/world/page/1/">world</a>
</div>
<div class="tags">
Tags:
<a class="tag" href="/tag/abilities/page/1/">abilities</a>
<a class="tag" href="/tag/choices/page/1/">choices</a>
</div>
<small class="author">J.K. Rowling</small>
相关的code如下
import requests
import os
from bs4 import BeautifulSoup
def get_html(url):
try:
header={'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.90 Safari/537.36 2345Explorer/9.3.2.17331', }
ht=requests.get(url,headers=header,verify=True)
ht.raise_for_status
ht.encoding=ht.apparent_encoding
return ht
except Exception as e:
print("error:",e)
def writeHtml(url):
text=get_html(url).content
path="C:\\Users\\Administrator\\Desktop\\python\\baidu.html"
with open(path,"wb") as f:
f.write(text)
print("success")
def getInfor(url):
finalSay=[]
html=get_html(url).text
soup=BeautifulSoup(html,"html.parser")
print(" the saying is:")
say_list=soup.select("span.text")
print("-------名言--------------")
print("the length=",len(say_list))
# print(say_list)
for elem in say_list:
te=elem.text
finalSay.append(te);
print(finalSay)
print("-------标签--------------")
finalTag=[]
tag_list=soup.select("div.tags")
print("the length of tag=",len(tag_list))
#print(tag_list)
for tag in tag_list:
a_tag=tag.select("a.tag")
#print(a_tag)
# for elem in a_tag:
# print(elem.text)
tag_list=[elem.text for elem in a_tag]
#print(tag_list)
finalTag.append(tag_list)
print(finalTag)
print("-------作者--------------")
finalAuthor=[]
author_list=soup.select("small.author")
print("the length of authoer=",len(author_list))
for elem in author_list:
finalAuthor.append(elem.text)
print(finalAuthor)
for i in range(len(author_list)):
print("the saying:",finalSay[i],"\t","the author:",finalAuthor[i],"\t","the tag:",finalTag[i])
结果如下
the saying: “The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.” the author: Albert Einstein the tag: ['change', 'deep-thoughts', 'thinking', 'world']
the saying: “It is our choices, Harry, that show what we truly are, far more than our abilities.” the author: J.K. Rowling the tag: ['abilities', 'choices']
the saying: “There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.” the author: Albert Einstein the tag: ['inspirational', 'life', 'live', 'miracle', 'miracles']
the saying: “The person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.” the author: Jane Austen the tag: ['aliteracy', 'books', 'classic', 'humor']
the saying: “Imperfection is beauty, madness is genius and it's better to be absolutely ridiculous than absolutely boring.” the author: Marilyn Monroe the tag: ['be-yourself', 'inspirational']
the saying: “Try not to become a man of success. Rather become a man of value.” the author: Albert Einstein the tag: ['adulthood', 'success', 'value']
the saying: “It is better to be hated for what you are than to be loved for what you are not.” the author: André Gide the tag: ['life', 'love']
the saying: “I have not failed. I've just found 10,000 ways that won't work.” the author: Thomas A. Edison the tag: ['edison', 'failure', 'inspirational', 'paraphrased']
the saying: “A woman is like a tea bag; you never know how strong it is until it's in hot water.” the author: Eleanor Roosevelt the tag: ['misattributed-eleanor-roosevelt']
the saying: “A day without sunshine is like, you know, night.” the author: Steve Martin the tag: ['humor', 'obvious', 'simile']完整 https://cn.bing.com/dict/search?q=snow
其翻译的信息主要在
<span class="pos">n.</span>
<span class="pos">v.</span>
<span class="pos">adj.</span>
<span class="pos web">网络</span>
<span class="def"><span>书；书籍；著作；部</span></span>
<span class="def"><span>书籍的；书本上的；账面上的</span></span>对应的正则表达式为
re0=r'<span class="(pos|pos web)">(.*?)</span>'
re1=r'<span class="def"><span>(.*?)</span></span>'相关code
主要是正则表达式的使用，已经写文件操作。
def getDictionary():
word=input("亲输入要翻译的词语:")
url="https://cn.bing.com/dict/search?q="
hurl=url+word
trant=[]
html=get_html(hurl).text
re0=r'<span class="(pos|pos web)">(.*?)</span>'
flag_list=[]
nav_list=re.findall(re0,html)
#print(hurl)
#print(html)
if len(nav_list)==0 and len(nav_list[0])<1:
return
for elem in nav_list:
flag_list.append(elem[1])
print(flag_list)
re1=r'<span class="def"><span>(.*?)</span></span>'
tran_list=re.findall(re1,html)
#print(tran_list)
for i in range(len(nav_list)):
tra="\t".join([flag_list[i],tran_list[i]])
print(tra)
trant.append(tra)
#trant.append("\n")
return trant
def writeTet(t):
path="E:\\infor.txt"
with open(path,"a+") as f:
for txt in t:
f.write(txt)
f.write("\n")
print("success")
结果为
亲输入要翻译的词语:Python
['n.', '网络']
n. 蟒；蚺蛇
网络 蟒蛇；巨蟒；派森
success
参考文章
beautifulsoup之CSS选择器
Beautiful Soup Documentation
Python 文件I/O
python对文件的操作读写追加等演示
随时随地看视频
