vgdev/stable/: scrapy-random-ua-0.3 metadata and description

Homepage Simple index

Scrapy Middleware to set a random User-Agent for every Request.

author cleocn
author_email cleocn@gmail.com
classifiers
  • Development Status :: 4 - Beta
  • Intended Audience :: Developers
  • Environment :: Console
  • License :: OSI Approved :: MIT License
  • Operating System :: OS Independent
  • Programming Language :: Python
  • Framework :: Scrapy
keywords scrapy random user-agent
license MIT
requires_dist
  • fake-useragent

Because this project isn't in the mirror_whitelist, no releases from root/pypi are included.

File Tox results History
scrapy_random_ua-0.3-py3-none-any.whl
Size
3 KB
Type
Python Wheel
Python
3

Does your scrapy spider get identified and blocked by servers because you use the default user-agent or a generic one?

Use this random_useragent module and set a random user-agent for every request.

Installing

Installing it is pretty simple.

pip install git+https://github.com/cleocn/scrapy-random-useragent.git

Usage

In your settings.py file, update the DOWNLOADER_MIDDLEWARES variable like this.

DOWNLOADER_MIDDLEWARES = {
    'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None,
    'random_useragent.RandomUserAgentMiddleware': 400
}

This disables the default UserAgentMiddleware and enables the RandomUserAgentMiddleware.

Now all the requests from your crawler will have a random user-agent.