比如我在settings.py中设置了
USER_AGENT = 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Firefox/31.0'
在某个spider中想覆盖掉这个值
我用了两种办法
方法一
from scrapy.conf import settings
settings.set('USER_AGENT','firefox')
方法二
from scrapy.settings import Settings
settings=Settings()
settings.set('USER_AGENT','firefox')
方法三
from scrapy.utils.project import get_project_settings
settings = get_project_settings()
settings.set('USER_AGENT','firefox')
方法二和方法三不起作用,方法一是生效的,但是from scrapy.conf import settings是被scrapy废弃掉了,可以兼容使用,但是官方不推荐
会提示
Module `scrapy.conf` is deprecated, use `crawler.settings` attribute instead
我折腾了半天,没有用crawler.settings搞出来,大家怎么解决的?
USER_AGENT = 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Firefox/31.0'
在某个spider中想覆盖掉这个值
我用了两种办法
方法一
from scrapy.conf import settings
settings.set('USER_AGENT','firefox')
方法二
from scrapy.settings import Settings
settings=Settings()
settings.set('USER_AGENT','firefox')
方法三
from scrapy.utils.project import get_project_settings
settings = get_project_settings()
settings.set('USER_AGENT','firefox')
方法二和方法三不起作用,方法一是生效的,但是from scrapy.conf import settings是被scrapy废弃掉了,可以兼容使用,但是官方不推荐
会提示
Module `scrapy.conf` is deprecated, use `crawler.settings` attribute instead
我折腾了半天,没有用crawler.settings搞出来,大家怎么解决的?