推荐学习书目
Learn Python the Hard Way
Python Sites
PyPI - Python Package Index
http://diveintopython.org/toc/index.html
Pocoo
值得关注的项目
PyPy
Celery
Jinja2
Read the Docs
gevent
pyenv
virtualenv
Stackless Python
Beautiful Soup
结巴中文分词
Green Unicorn
Sentry
Shovel
Pyflakes
pytest
Python 编程
pep8 Checker
Styles
PEP 8
Google Python Style Guide
Code Style from The Hitchhiker's Guide
yangjiaronga
V2EX  ›  Python

Scrapy+phantonjs 爬去速度过慢?

  •  1
     
  •   yangjiaronga · Jan 27, 2017 · 2994 views
    This topic created in 3394 days ago, the information mentioned may be changed or developed.
    scrapy 运行日志
    **************ProxyMiddleware not pass************171.38.66.23:9999
    2017-01-26 23:05:38 [selenium.webdriver.remote.remote_connection] DEBUG: POST http://127.0.0.1:57234/wd/hub/session {"desiredCapabilities": {"browserName": "phantomjs", "version": "", "platform": "ANY", "javascriptEnabled": true}, "requiredCapabilities": {}}
    2017-01-26 23:05:38 [selenium.webdriver.remote.remote_connection] DEBUG: Finished Request
    2017-01-26 23:05:38 [selenium.webdriver.remote.remote_connection] DEBUG: POST http://127.0.0.1:57234/wd/hub/session/e5c0aeb0-e3d8-11e6-a629-15e39efe8c5a/url {"url": "Matweb Engineering Materials List", "sessionId": "e5c0aeb0-e3d8-11e6-a629-15e39efe8c5a"}

    phantonjs 中间件代码:
    def process_request(self, request, spider):
    driver = webdriver.PhantomJS(executable_path=r"/Users/apple/phantomjs-2.1.1-macosx/bin/phantomjs")
    driver.get(request.url)
    body = driver.page_source
    print ("访问"+request.url)
    return HtmlResponse(driver.current_url, body=body, encoding='utf-8', request=request)


    每次执行到 selenium.webdriver.remote.remote_connection 。 都会卡上 20s , 请问这是为什么?
    No Comments Yet
    About   ·   Help   ·   Advertise   ·   Blog   ·   API   ·   FAQ   ·   Solana   ·   4784 Online   Highest 6679   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 31ms · UTC 09:52 · PVG 17:52 · LAX 02:52 · JFK 05:52
    ♥ Do have faith in what you're doing.