Last updated: October 31, 2026.
How to Use and Rotate Proxies in Scrapy: A Python Guide
Scrapy includes a built-in HTTP proxy middleware, so you do not need to build a custom proxy system for every project. The important first step is choosing the right rotation model.
With a provider-managed rotating gateway, Scrapy connects to one proxy endpoint and the provider manages the exit IP. With client-managed rotation, your code selects an endpoint from a list of proxies.
This guide explains both approaches, shows how to configure proxy authentication, and walks through a real Scrapy 2.19.0 test that returned ten different exit IPs across ten successful requests.
The examples are intended for websites and data-collection workflows you own or are authorized to access. Proxy rotation does not override a website’s terms, access controls, or rate limits, and it does not guarantee that a request will avoid CAPTCHAs.
1. Choose the right Scrapy proxy rotation model
There are two common ways to rotate proxy exits.
Provider-managed rotation
Your spider sends requests to one gateway. The proxy provider selects the exit IP according to the account’s configuration and session behavior.
For example, your Scrapy project may use a single endpoint such as:
http://gateway.example.com:12000
The gateway remains the same while the provider manages the residential exit IPs behind it. Depending on the provider’s settings, a session can keep a consistent IP for a defined period, or requests can receive changing exits.
This approach is usually the simplest option when your proxy provider offers a rotating gateway. You do not need to maintain a local list of residential IP addresses or write a round-robin middleware just to use the gateway.
Client-managed rotation
Your project has multiple proxy endpoints, and a downloader middleware chooses one for each request. This gives your code control over endpoint selection, but it also means you must manage:
- Endpoint selection, such as random or round robin.
- Retry behavior and whether a retry should use another endpoint.
- Endpoint health and failure handling.
- Credential storage and safe logging.
- Concurrency and request-rate limits.
| Rotation model | What Scrapy configures | Who selects the exit |
|---|---|---|
| Provider-managed gateway | One gateway endpoint and authentication | Proxy provider |
| Client-managed pool | A selected endpoint from a proxy list | Your Scrapy middleware |
Choose the model your provider actually supports. If your account provides one gateway that rotates internally, start with provider-managed rotation. A client-side pool is only useful when you have multiple endpoints that your application is expected to select.
2. Create a Scrapy project
Create a virtual environment and install Scrapy. This example was tested with Scrapy 2.19.0 and Python 3.12.
python -m venv .venv
source .venv/bin/activate
python -m pip install "Scrapy==2.19.0" python-dotenv
On Windows PowerShell:
.venv\Scripts\Activate.ps1
Scrapy 2.13 and newer support the asynchronous start() spider method. The example in this guide uses that method, which is compatible with Scrapy 2.19.0. See the official Scrapy Spider documentation
.
A compact project structure for the tested provider-managed example is:
kindproxy-scrapy-proxy-rotation/
├── scrapy.cfg
├── requirements.txt
├── .env.example
├── .gitignore
├── README.md
└── scraper/
├── __init__.py
├── settings.py
├── proxy.py
├── middlewares.py
└── spiders/
├── __init__.py
└── proxy_test.py
The proxy.py file is part of the project: it builds an authenticated proxy URL from the endpoint and credentials. Do not omit it when copying the example.
Get the complete Python example on GitHub
You can clone or download the complete runnable project, including the Scrapy settings, proxy helper, downloader middleware, IP-check spider, and setup instructions:
KindProxy Scrapy Proxy Rotation on GitHub
Clone it with:
git clone https://github.com/kindproxy/kindproxy-scrapy-proxy-rotation.git
cd kindproxy-scrapy-proxy-rotation
Follow the repository README to install the dependencies and configure your own proxy gateway.
3. Configure a basic HTTP proxy
Scrapy enables HttpProxyMiddleware by default. For a one-off request, set the proxy URL in request.meta:
import scrapy
class ProxyCheckSpider(scrapy.Spider):
name = "proxy_check"
async def start(self):
yield scrapy.Request(
"https://api.ipify.org?format=json",
callback=self.parse_ip,
meta={
"proxy": "http://proxy.example.com:8080",
},
)
def parse_ip(self, response):
yield {
"status": response.status,
"body": response.text,
}
Replace the example endpoint with the host and port supplied by your proxy provider.
Scrapy’s built-in HttpProxyMiddleware also reads the http_proxy, https_proxy, and no_proxy environment variables. A request-level meta["proxy"] value takes precedence over those environment variables. See the Scrapy downloader middleware documentation
.
For a basic proxy connection, this built-in middleware is enough. Add custom downloader middleware only when your application needs behavior that the built-in middleware does not provide, such as selecting among multiple proxy endpoints.
4. Add proxy authentication safely
A provider may require a username and password. An authenticated HTTP proxy URL has this general format:
http://USERNAME:PASSWORD@proxy.example.com:8080
Some usernames or passwords contain reserved URL characters. URL-encode credentials before inserting them into a proxy URL. Never commit credentials to Git or print a complete authenticated proxy URL in application logs.
The helper below builds the URL and keeps credentials separate from the endpoint:
# scraper/proxy.py
from urllib.parse import quote, urlsplit, urlunsplit
def build_authenticated_proxy_url(
endpoint: str,
username: str,
password: str,
) -> str:
endpoint = endpoint.strip()
if not endpoint:
raise ValueError("Proxy endpoint is empty.")
if "://" not in endpoint:
endpoint = f"http://{endpoint}"
parts = urlsplit(endpoint)
if not parts.hostname or not parts.port:
raise ValueError(
"Proxy endpoint must include a hostname and port."
)
userinfo = (
f"{quote(username, safe='')}:"
f"{quote(password, safe='')}@"
)
netloc = f"{userinfo}{parts.hostname}:{parts.port}"
return urlunsplit(
(parts.scheme or "http", netloc, parts.path, parts.query, "")
)
This example assumes that your provider supports username/password authentication in the proxy URL. If your account uses IP allowlisting instead, follow the provider’s instructions to authorize the public IP of the machine running Scrapy.
Scrapy’s built-in HttpProxyMiddleware supports proxy URLs containing a username and password. Keep the resulting URL private because it contains authentication details.
5. Load proxy credentials from environment variables
Do not hard-code credentials in your spider or settings file. Use a local .env file for development and environment variables or a secret manager in production.
Example .env.example:
KP_PROXY_ENDPOINTS=http://gateway.example.com:12000
KP_PROXY_USERNAME=YOUR_USERNAME
KP_PROXY_PASSWORD=YOUR_PASSWORD
Copy it to .env and fill in your actual values:
cp .env.example .env
Install python-dotenv and load the file explicitly. Scrapy does not automatically load .env files.
The tested example uses the following settings:
# scraper/settings.py
import os
from pathlib import Path
from dotenv import load_dotenv
BASE_DIR = Path(__file__).resolve().parent.parent
load_dotenv(BASE_DIR / ".env")
BOT_NAME = "kindproxy_rotation_test"
SPIDER_MODULES = ["scraper.spiders"]
NEWSPIDER_MODULE = "scraper.spiders"
ROBOTSTXT_OBEY = False
LOG_LEVEL = "INFO"
CONCURRENT_REQUESTS = 1
CONCURRENT_REQUESTS_PER_DOMAIN = 1
DOWNLOAD_DELAY = 1.0
DOWNLOAD_TIMEOUT = 60
KP_PROXY_ENDPOINTS = os.getenv("KP_PROXY_ENDPOINTS", "")
KP_PROXY_USERNAME = os.getenv("KP_PROXY_USERNAME", "")
KP_PROXY_PASSWORD = os.getenv("KP_PROXY_PASSWORD", "")
DOWNLOADER_MIDDLEWARES = {
"scraper.middlewares.ProviderGatewayProxyMiddleware": 100,
}
RETRY_ENABLED = False
ROBOTSTXT_OBEY = False is used here only for the isolated IP-check endpoint in this diagnostic example. For a real crawl, configure and follow the target site’s robots.txt policy and other applicable access rules.
Keep .env out of version control:
.env
__pycache__/
*.py[cod]
output.jsonl
6. Route Scrapy requests through one rotating gateway
For a provider-managed gateway, a small middleware can attach the authenticated gateway URL to each request. The provider—not this middleware—selects the residential exit IP.
# scraper/middlewares.py
from scrapy import signals
from scraper.proxy import build_authenticated_proxy_url
class ProviderGatewayProxyMiddleware:
def __init__(self, proxy_url, username):
self.proxy_url = proxy_url
self.username = username
@classmethod
def from_crawler(cls, crawler):
endpoints = [
item.strip()
for item in crawler.settings.get(
"KP_PROXY_ENDPOINTS", ""
).split(",")
if item.strip()
]
username = crawler.settings.get("KP_PROXY_USERNAME", "")
password = crawler.settings.get("KP_PROXY_PASSWORD", "")
# This example demonstrates provider-managed rotation through
# one gateway; it is not a client-managed multi-endpoint pool.
if len(endpoints) != 1:
raise ValueError(
"Configure exactly one provider-managed gateway "
"in KP_PROXY_ENDPOINTS for this example."
)
if not username or not password:
raise ValueError(
"Set KP_PROXY_USERNAME and KP_PROXY_PASSWORD."
)
proxy_url = build_authenticated_proxy_url(
endpoints[0],
username,
password,
)
middleware = cls(proxy_url, username)
crawler.signals.connect(
middleware.spider_opened,
signal=signals.spider_opened,
)
return middleware
def spider_opened(self, spider):
spider.logger.info(
"Provider-managed proxy gateway enabled. Username: %s",
self.username,
)
def process_request(self, request):
if not request.meta.get("proxy"):
request.meta["proxy"] = self.proxy_url
# Diagnostic mode: ask the server to close the connection
# after the response, reducing the chance of connection reuse.
request.headers[b"Connection"] = b"close"
# Keep this diagnostic run simple: do not retry test requests.
request.meta["dont_retry"] = True
return None
The middleware hook signatures intentionally omit the spider argument, which is not part of the current downloader middleware method signatures in Scrapy 2.19.
The middleware sets request.meta["proxy"]. Scrapy’s built-in HttpProxyMiddleware then handles the proxy request. The custom middleware does not replace the built-in one.
Important: Connection: close and dont_retry are used here to make the rotation test easier to interpret. They are diagnostic settings, not universal production recommendations. After verifying the gateway, review whether your production spider should reuse connections and enable bounded retries.
7. Test proxy rotation with Scrapy
An IP-check endpoint helps you verify the public IP observed by the destination. The following spider sends ten sequential requests to api.ipify.org, adds a unique query parameter to each request, and records the returned IP.
# scraper/spiders/proxy_test.py
import json
import scrapy
class ProxyTestSpider(scrapy.Spider):
name = "proxy_test"
allowed_domains = ["api.ipify.org"]
custom_settings = {
"FEEDS": {
"output.jsonl": {
"format": "jsonlines",
"overwrite": True,
"encoding": "utf8",
}
}
}
async def start(self):
total = 10
for index in range(1, total + 1):
url = (
"https://api.ipify.org"
f"?format=json&scrapy_test={index}"
)
self.logger.info(
"Sending proxy test request %s/%s",
index,
total,
)
yield scrapy.Request(
url=url,
callback=self.parse_ip,
cb_kwargs={"request_number": index},
dont_filter=True,
headers={
"Cache-Control": "no-cache",
"Pragma": "no-cache",
"Connection": "close",
},
)
def parse_ip(self, response, request_number):
try:
payload = json.loads(response.text)
ip = payload.get("ip")
except (json.JSONDecodeError, AttributeError):
ip = None
item = {
"request": request_number,
"status": response.status,
"ip": ip,
"url": response.url,
}
self.logger.info(
"Request %s completed: HTTP %s, exit IP: %s",
request_number,
response.status,
ip or "NOT_PARSED",
)
yield item
Run the spider from the project directory:
scrapy crawl proxy_test
The results are written to output.jsonl:
cat output.jsonl
Each item contains the request number, HTTP status, and observed exit IP. Keep the output private if it contains full IP addresses or other information you do not want to publish.
Real test: 10 requests, 10 different exit IPs
We ran this diagnostic test with Python 3.12.3 and Scrapy 2.19.0 through a single KindProxy rotating gateway.

All ten requests returned HTTP 200, and the test observed ten different exit IPs. The screenshot masks the final octet of each address for privacy while preserving enough information to compare the observed exits. This confirms that the configured gateway rotated exits during this specific test run.
The result is an observation from one account configuration and one test session—not a guarantee that every request in every workload will always receive a unique IP. Rotation behavior can depend on the provider’s session settings, gateway policy, connection behavior, and request pattern.
8. Provider-managed rotation vs. client-side proxy pools
A single rotating gateway and a client-managed pool solve different problems.
Use a provider-managed gateway when your provider offers one endpoint that handles exit selection. Your Scrapy project can keep a stable gateway configuration while the provider manages the exits.
Use a client-managed pool when you have multiple proxy endpoints and need your application to decide which endpoint receives each request. In that case, implement and test endpoint selection separately. A round-robin list distributes requests across configured endpoints, but it does not guarantee that every endpoint maps to a unique residential IP.
Do not add a client-side rotation middleware on top of a provider-managed gateway unless you have a specific reason to select among multiple gateways. Extra rotation logic can make debugging harder without improving the provider’s own rotation behavior.
9. Does Scrapy automatically change proxies when retrying?
No. Scrapy’s RetryMiddleware retries requests for selected temporary failures, but it does not decide to use a different proxy endpoint.
The retry count is available in request.meta["retry_times"]. If you operate a client-managed pool and want retries to use another endpoint, add explicit selection logic and test the middleware order with your Scrapy version.
For a provider-managed gateway, the retry may continue through the same gateway URL. Whether the exit IP changes is determined by the provider’s session and rotation policy.
Do not rotate for every error indiscriminately:
- A timeout may justify a bounded retry after checking whether the proxy or destination was slow.
- A
407 Proxy Authentication Requiredresponse usually points to proxy authentication or endpoint configuration. - A
403 Forbiddenresponse may come from the destination site’s access policy. - A
429 Too Many Requestsresponse commonly indicates rate limiting. Reduce request frequency and follow the site’s rules rather than automatically retrying through more IPs.
See the official Scrapy RetryMiddleware documentation .
10. Troubleshoot common Scrapy proxy errors
| Symptom | Possible meaning | What to check |
|---|---|---|
407 Proxy Authentication Required | The proxy rejected or requires authentication | Username, password, authentication format, endpoint and allowlist settings |
| Connection refused | Scrapy cannot connect to the proxy endpoint | Host, port, protocol, DNS, firewall and network access |
| Timeout | The proxy route or destination did not respond in time | Which request timed out, provider status, target response time and bounded retries |
Target returns 403 | The destination refused the request | Response body, target access policy and request pattern |
Target returns 429 | The destination is rate-limiting requests | Reduce concurrency and request frequency; respect the site’s limits |
| The observed IP stays the same | A sticky session or provider rotation policy may be in effect | Session settings, gateway mode and the test method |
Scrapy’s default retry behavior is configurable. Its default retry HTTP status list includes 408, 429, several 5xx responses, and selected network exceptions; it does not treat every 403 or 407 as retryable by default. Check the official RetryMiddleware settings
before changing retry codes.
For diagnostics, log the request URL, response status or exception type, retry count, and a safe proxy label such as gateway-1. Never log the full authenticated proxy URL.
11. Tune concurrency and request rate
Proxy rotation does not replace responsible crawl settings. Start with a modest global concurrency, per-domain concurrency, and DOWNLOAD_DELAY. Increase them only when the target site permits it and your tests justify the change.
Scrapy’s DOWNLOAD_DELAY can reduce effective per-domain concurrency. AutoThrottle
can help adjust crawl speed based on observed responses.
Also review the target site’s terms, robots.txt policy, published access rules, and applicable restrictions. Prefer an official API or data feed when one is available. Stop or adjust the crawl when you receive an access restriction rather than treating proxy rotation as permission to continue.
12. Use a residential proxy with Scrapy
Residential proxies can be useful for authorized data-collection workflows that need network exits associated with particular markets. In Scrapy, the integration point remains the proxy URL assigned to each request. The provider’s dashboard determines the gateway, authentication mode, location options, and session behavior available to your account.
If you use KindProxy, you can generate or check your proxy endpoint and credentials in the KindProxy Proxy Traffic dashboard . Copy the exact endpoint and authentication details shown for your account. Start with a small number of requests, verify the observed exit IP, and check the destination response separately.
For projects that need residential proxy traffic, see KindProxy Residential Proxy plans . Choose the location and session behavior based on the requirements of your workflow, then validate the configuration with a small test before scaling up.
A regional exit does not guarantee that a website will display a particular language, price, or page. Those results can also depend on cookies, account settings, browser state, and the website itself.
FAQ
Does Scrapy support HTTP proxies?
Yes. Scrapy’s built-in HttpProxyMiddleware supports proxy URLs set in request.meta["proxy"] and standard proxy environment variables such as http_proxy and https_proxy.
Does Scrapy switch proxies automatically after a retry?
No. RetryMiddleware retries eligible requests, but it does not select a new proxy endpoint. Your project needs explicit endpoint-selection logic if a retry should use another endpoint.
Do I need custom middleware for a rotating residential gateway?
Not necessarily. A provider-managed rotating gateway may need only one configured endpoint. Custom middleware is mainly useful when your Scrapy client must select among multiple proxy endpoints.
Should I retry every 403 or 429 with a different proxy?
No. These responses may come from the destination site’s access policy or rate limiting. Check the response, adjust your request rate where appropriate, and follow the site’s rules.
Why does my IP-check test return the same IP repeatedly?
Possible reasons include a sticky session, provider rotation policy, connection reuse, or the way the test is configured. Compare multiple requests using the same gateway, review session settings, and test with a method that makes the connection behavior clear. Do not assume that one repeated IP automatically means the proxy is broken.
Conclusion
Scrapy’s built-in HttpProxyMiddleware handles basic proxy routing. For a provider-managed rotating gateway, one authenticated endpoint may be all your project needs. A custom downloader middleware is useful when your application must select from a pool of proxy endpoints or apply a deliberate retry-selection policy.
Start with a small test, verify the observed exit IP, distinguish proxy errors from destination-site responses, and keep credentials out of logs. Once the behavior is confirmed, tune connection reuse, concurrency, and retry settings for your actual workload.
Get the code and test it yourself
The complete example is available on GitHub, including the configuration files and the working IP-verification spider:
View the KindProxy Scrapy Proxy Rotation repository
If you need a residential proxy gateway for your own authorized Scrapy workflow, you can also explore KindProxy Residential Proxy plans .
