执行douban_spider出现问题

来源:2-1 scrapy的安装、和安装中遇到的问题

HamsterZ

2018-08-29 11:15

"UnicodeEncodeError",python3编码是utf-8,尝试用virtualenv解决:

(project_1) [root@hamster douban]# python3 -m scrapy crawl douban_spider

...

2018-08-28 22:59:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://movie.douban.com/top250> (referer: None)

2018-08-28 22:59:33 [scrapy.core.scraper] ERROR: Spider error processing <GET https://movie.douban.com/top250> (referer: None)

Traceback (most recent call last):

  File "/root/project_1/lib/python3.6/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks

    current.result = callback(current.result, *args, **kw)

  File "/root/douban/douban/spiders/douban_spider.py", line 14, in parse

    print(response.text)

UnicodeEncodeError: 'ascii' codec can't encode characters in position 349-352: ordinal not in range(128)

...

2018-08-28 22:59:33 [scrapy.core.engine] INFO: Spider closed (finished)

还是报错,检查了一下:

(project_1) [root@hamster douban]# python3

Python 3.6.5 (default, Aug 28 2018, 22:34:18) 

[GCC 4.4.7 20120313 (Red Hat 4.4.7-23)] on linux

Type "help", "copyright", "credits" or "license" for more information.

>>> import sys

>>> sys.getdefaultencoding()

'utf-8'

怀疑是"douban_spider.py"代码有误,但不知道错在哪,希望老师帮我解答一下,谢谢!?

写回答 关注

1回答

  • qq_MakiseKurisu_04324521
    2018-09-01 11:54:25
    已采纳

    你用virtualenv的时候,运行文件时,是否workon切换到正确的解释器了

    Hamste...

    我发现好像是print(response.text)的问题,我跟着课程继续往下做,可以成功爬取并保存数据。

    2018-09-04 11:22:51

    共 1 条回复 >

Python最火爬虫框架Scrapy入门与实践

做为爬虫工程师Python Scrapy主流爬虫框架你必须要会!

67418 学习 · 223 问题

查看课程

相似问题